All the people accompanied him to his house.
Why, upon the completion of the services did he go specifically to his house?
The Talmud1 says, “Seven days before Yom Kippur they sequester the High Priest from his house. According to Rabbi Yochanan2 this practice is derived from the inauguration of the Tabernacle where Aaron and his sons were instructed, “To dwell seven days and nights at the entrance of the Tent of Meeting.”3 According to Reish Lakish it is derived from Mount Sinai that one who enters an area of intense sanctity needs seclusion away from his home in preparation for the event. After Moses received the commandments at Sinai, he went up to Heaven for forty days. At the start of the forty-day period Moses ascended the mountain. The Torah4 relates, “The glory of G-d rested upon Mount Sinai, and the cloud covered it for a six-day period. G-d then called Moses on the seventh day from the midst of the cloud.”5
Hence, the seclusion of the High Priest spent away from his home was not merely a preventive measure lest he become contaminated, but an integral detail of the Yom Kippur services. Consequently, just as the services commence with the High Priest being sequestered from his home, it concludes with his return home after Yom Kippur.
The lesson to be learned from this is the following: On Yom Kippur, the holiest day of the year, the High Priest, the holiest person of the community, after a period of seclusion, sequesters himself in the "Holy of Holies," the holiest place in the world -- and there he achieves the greatest revelation of G‑dliness. Nevertheless, he must bear in mind that this was not G-d’s ultimate interest in the creation. Rather, He desired to have an abode in the lower worlds,6 and this cannot be accomplished through seclusion from the world and sequestration within the Holy of Holies, but through involvement in the mundane world and observance of Torah and its precepts.
For this reason, immediately after completing the Yom Kippur service, the High Priest would return to his house to emphasize that the sublime elevation attained on this day must be transferred to his house and become a way of life for him and his household throughout the entire year. The same also applies to every individual on his particular level.