FortiADC HA - Can't access the Active Peer's management interface

I'm having a weird issue when setting up a HA pair, I can't access the active peer through its management interface.

I've followed this guide https://docs.fortinet.com/document/fortiadc/8.0.2/administration-guide/60471/deploying-an-active-passive-cluster

If I reboot the active device through console, the problem switches so I can again access the secondary device. So the issue seems to be in the configuration, not in anything VMware/network related.