Can fellow redditors suggest a destination on the East coast...

I will be flying into NYC in a few weeks and although the Big Apple is a great place I also wanna do a short trip to somewhere. I am seeking suggestions for a city that

• is accessible from NYC by a not too long train ride (I don't drive...)
• is walking-friendly within the city
• arts museums and jazz bars would be great Night scene is secondary, as I'm not a big fan of clubbing. Any idea? Thanks!