Although it is not mentioned in the Talmud, the custom to fast after accidentally dropping a Torah scroll or tefillin has indeed been widespread for many hundreds of years.
One of the earliest mentions of this practice is a responsa of Rabbi Israel of Brunna (present-day Brno, 15th century). He explains that dropping a Torah scroll or tefillin is a sign from Heaven that one has done something wrong for which one must repent. On the other hand, Rabbi Chaim Yosef David Azulai (18th century), commonly known as the ChIDA, cites another reason: The fast is to atone for the lack of care and respect which allowed the holy object to fall.
An important difference between Torah scrolls and tefillin: One must only fast after dropping his Tefillin if they are not in their protective case or bag. On the other hand, one must fast after a Torah falls—even if it is within its mantle.
Another difference is that although only the person who actually drops the Torah or tefillin is obligated to fast, under certain circumstances the rabbi of a community may declare a public penance for all of those present when a Torah falls. This may include studying the laws of respecting a Torah, a communal commitment not to speak during the Torah reading, charity, fasting, or a combination of some of the above.