It is indeed customary not to eat nuts on Rosh Hashanah, although this seems to be mentioned only in Ashkenazi halachic literature.
A practical reason for this is given by the Maharil (Rabbi Yaakov HaLevi Moelin), a talmudist of the 15th century who compiled and codified many of the customs of German Jewry. Nuts tend to increase saliva in one's mouth, making prayer difficult. Considering the amount of praying we do on Rosh Hashanah, and the importance of our words being enunciated clearly specifically on that day, nuts are avoided.
Another reason given is more mystical. The numerical equivalent of the word "nut" - egoz in Hebrew - is seventeen. Seventeen is also the numerical equivalent of the Hebrew word for sin, chet, not as it's properly spelled, but as it's pronounced. We stay far away from anything reminiscent of sin on Rosh Hashanah, nuts included.
The Eshel Avraham illuminates the custom with a deep insight. He notes that Rosh Hashanah is a time to be especially careful with food. On the first Rosh Hashanah in history, Adam and Eve sinned by eating the wrong food. We rectify this in part on Rosh Hashanah by eating foods with auspicious allusions, and avoiding those with negative connotations.
The custom of refraining from eating nuts, as well as both of these reasons, is cited by the Rema (Rabbi Moshe Isserles) in his notes that provide the Ashkenazic modifications to the Code of Jewish Law.
--Malkie Janowski for Chabad.org