Safety of Museum District?

We are looking for a rental house or townhome and seeing stuff available in the Museum District. Personal safety is important to me as I would sometimes be coming home after dark. My impression is that the neighborhood gets more sketchy as you go east, like around Almeda.

For people who are familiar with the area, which parts are safe? We would not be in an apartment building with security. Thanks.