We have been the day after Christmas and stayed until Jan 1st, and we also have arrived the first week in January. We loved being there for NYEve.
Like SoundGod said, the temps can vary. One year it was 29degrees on New Years Eve, and the high that day had been around 40. The topiaries were covered and the canals were freezing over. Another year, we were in short sleeves. Just be prepared for all kinds of weather in Dec and Jan.
If you can go earlier in Dec, I would choose that crowd wise, however, I think everyone should experience the fireworks on NY Eve at least once.