Indeed it is customary to remove the tefillin on Rosh Chodesh before starting the Musaf prayer. Two reasons are given for this practice:

1) In many communities, the liturgy for the Musaf kedushah begins with the words "Keter Yitnu L'cha," ("A crown is given to You, L-rd our G‑d..."). This is a reference to the supernal crown which the angels on high, and the Jews below, bestow upon G‑d. The tefillin are also considered to be a crown, and therefore, the Zohar explains, it would be disrespectful for an individual to wear his own "crown" while crowning G‑d.

2) The Rosh Chodesh Musaf prayer speaks of the special sacrifices which were brought in the Holy Temple in honor of Rosh Chodesh and implores G‑d to grant us a prosperous and happy month. Now, on major Jewish holidays such as Passover and Rosh Hashanah, tefillin are not worn. The holiness of the day itself attests to the relationship G‑d shares with His nation, thus making tefillin, which is also a symbol of this relationship, unnecessary. Since Rosh Chodesh is also a holiday, albeit to a lesser degree, we don't wear the tefillin during the prayer which is devoted to the specialness of the day.1

Best wishes for a sweet new year,

Rabbi Baruch S. Davidson