Starter relay problem? How does the main relay work?

Hi. I own a 1997 Toyota Estima. A few days ago, and today, I've had an issue starting it.

I turn the ignition on, all the dashboard lights and fans come on like normal, then I go to crank it and it just clicks once or a few times (like a dead battery) and the whole system seems to loose power, not even the open door indicator which doesn't even require a key in the ignition to work. Yet the battery is fine and seems properly connected.

So, the first time, I start running around, pulling out fuses to check them, etc. and I notice the door indicator is on again. I crank it and it's dead again. This happens a couple more times (by now, I'm fiddling with the engine bay fuse box). Then I try and it starts up just fine, perfectly normal.

Was fine all day yesterday for several tips.

Today, same problem, so I tried tapping all the relays, tapping the starter motor/solenoid unit with a hammer. Not much seemed to work (I did get the door indicator to come on once). The last thing I did was disconnect the battery, tap the connector onto the terminal a few times, before securing it back on, then I noticed the door indicator was on, and it started up fine.

Some Googling suggests it could be the starter solenoid contacts. But none of those people mention anything about losing all power after trying to start the engine.

It seems like a relay is turning the starter motor on (and it or another relay, turning everything else off), then it can't get the job done, cuts power to the starter and doesn't restore power to the rest of the system.

I've pulled the starter relay and the main power relay out, but there isn't much to see.

Also, I pulled the main power relay out and the car started fine, all system seemed to be fine. So, perhaps that is a normally-closed relay that cuts power when the starter relay is activated?

Any ideas on which I should try replacing first?