Deuteronomy 23:11 "He may not enter this camp"

If a person is impure (because of any reason excepting that of contact with a dead body), he may NOT enter the enclosed area on the Temple Mount.

(There are also specific impurities that force the person to separate himself from the entire camp of Israel and not only the Temple Mount, for example, someone with Tzara'at. See, Positive Mitzvah 31.)