How can I remove the focus from my textarea when not tabbed?

I got a textarea and a few inputs. I obviously want the accessibility tab outline to work when you hit tab, just not when I click on it with the mouse to write. Can I somehow remove it only in those cases?

I figured that doing something like textarea:focus {outline: none;} and textarea:focus-visible {outline: solid 2px;} would do the trick but it just won't work.