Spinalonga without crowds?

We will be staying in Agios Nikolaos in a couple weeks (late September), and I'm hoping to spend part of a day visiting Spinalonga. We are independent travelers and since we dislike crowds, we generally avoid group tours and packaged tourism whenever we can (I know that's not always possible, since access is restricted in popular places).

I'd like to go to Spinalonga and avoid being there when it gets crowded (to the extent that's possible) - basically, be there when the fewest possible other people are there (and yes, I know I contribute to tourist crowding wherever I go, too...and I'm aware of the irony of my question). We will have our own car, and can go early (or late) to avoid the busy times (I assume the busiest times are mid-day, say 11 am to 3 pm - same as with all day-trip destinations). It's on an island, so it's not like I can get there all by myself at the crack of dawn (though I'd probably like to).

What are my options to get from the mainland to Spinalonga and access the place when it's least crowded with other visitors? Thanks for any tips or insights.