Crazy scroll-restoration issue only on chrome (during native navigation)

Hey, I'm using Next.js 12 in a project. For client-side navigation (router/nextlink) scroll restoration works fine. However I have a problem with native browser navigation.

On Safari and Firefox: when I click on back button (browser) the scroll restoration works

On Chrome: when I click on back button (browser) I'm always at the top of the page. What's interesting if I disable javascript it works as it should.

Does anyone have an idea of what might be wrong here? I can't connect the dots - totally seems like some next.js internals but at the same time why does it differ between browsers?