I'd love to go during 花見 （はなみ）, but only if I'd been there long enough to make a few friends first. It would be very sad to be all alone under the cherry blossoms. I'd also love to be there during 梅雨 （つゆ）, the rainy season, as I'm part Irish and a big fan of wet weather.
Probably the biggest consideration, I'd think, would be ゴールデンウィーク (Golden Week) in late April/early May, which is a string of holidays in Japan. Lots of people will be off of work and going around crowding things up, and prices for everything skyrocket from what I understand. So if you're planning a shorter visit, you're going to be paying quite a bit for hotel stays and such.
<insert something funny in Japanese here>