Flat paint horror

Hello- the home I live in now has flat paint and every time I try to clean it the paint comes off on the towel, im renting so im trying to avoid an extra expense one our lease is up. Im using dawn dish soap and warm water btw. Any tips appreciated.