vscode won't use ssh key authentication

I'm connecting from Win11 to Linux server. When I open command line and do ssh account@myserver everything works and ssh key authentication is used. But when I connect with vscode to that same account, vscode prompts me for the password. Windows 11's own OpenSSH install is used. I tried to explicitly specify both the SSH binary and its configuration (it's in the default directory anyway), but nothing changed.

Funny thing is that on my home computer the same setup works fine in vscode. I am really running out of ideas what might be wrong here.