We look at our nails during havdalah because:
a. In order to make the blessing on the candle, it must be close and bright enough to be able to distinguish different currencies by its light. If we can distinguish between our nails and flesh we know that we're okay.
b. Nails are a siman brocho (sign of blessing) because they are constantly growing. So we start off our week by gazing at a siman brocho.
c. The midrash says that the original fire emanated from Adam's fingernails.
d. Another midrash says that on the first Motzoei Shabbos, when the world became dark for the first time, Adam said "woe is to me, for because of my sin the world has become dark." G‑d then gave Adam the wisdom to rub two flint-stones together, and when fire sprang forth Adam made the blessing (Borey Meoray Ha'aish), and realized that he was completely naked – aside from his nails.