Wireguard issues with IPv4

I have wireguard server setup on flint 2. When I connect using IPv6 no issues at all. But when I use IPv4 it connects (handshake) but I can't open websites, not even the router admin page. The interesting part is, when I do

curl -4 ifconfig.me

I get my router static IP, so somehow it works but not fully.

Let me know if you need more details