Windows native VPN sending packets with wrong IP address

Trying to setup a Windows VPN client with Ikev2 on a Windows 10 PC. Went thru and set up the built in VPN. This PC has multiple ipv4 addresses configured. When I try to establish the VPN connection the source ip of the Ikev2 packets are using the wrong IP address. Does anyone know how to set the vpn client to send on a different IP address? The VPN server can only reply on a specific IP address