“And he would offer a short prayer there in the Sanctuary, after he left the Holy of Holies in order not to worry the people that perhaps he died in the Sanctuary.”

There is, however, another reason for the brevity of the High Priest’s prayer:

The service of the High Priest in the Holy of Holies represented the pinnacle of cleaving and elevation, in all of the three categories of “world,” “year,” and “soul”: the place was the most holy, the time was the most holy, and so was the person. On such a high level of attachment to the Divine there was no need for a prolonged prayer, and a short one sufficed.

(Likkutei Sichot, vol. 2, p. 646)