Can per-pixel motion vectors provided by modern engines improve ASW which used chunkier motion vectors?
ASW 2.0 today uses the Nvidia Optical Flow SDK in order to get motion vectors that RTX cards can calculate (essentially for free) when given the color and the depth buffer. However, the motion vector resolution from that SDK is lower compared to the render resolution, which results in certain artifacts when ASW is active. And it was even worse before when ASW 1.0 could only use the flawed video encoder of GPUs to estimate motion vectors.
But that was at a time when a lot of engines and games weren't able to provide motion vectors themselves. Nowadays most are, at the native resolution. So my question is: does that mean we can do ASW better? It seems that motion vectors add some performance overhead, so it wouldn't be for free anymore (assuming the game wasn't already using TAA and thus calculating motion vectors), but could it be worth it for a better ASW? Perhaps in addition to DLSS 2.1? Could a new version of ASW using per-pixel motion vectors result in such little artifacting that we can confidently leave it on as a method for latency reduction and/or framerate upscaling?
Posted Sep 3, 2020, 9:43 PM UTC
Yes, it's an idea that's been toyed with over the years, but keep in mind depth-based Positional TW used in ASW 2.0 does not run per-pixel either. If we were to use motion vectors per-pixel like those used UE4 for TAA, the cost VR reprojection would go up significantly. So there's a balance between quality and cost in how good of a job we want to do with these techniques like ASW or ATW. Currently we try to keep the VR compositor cost to be < 1 ms per-frame. Also the kind of motion vectors needed for better ASW need to ignore head motion, and currently most game engines don't do that.
The idea has merit for sure, so I wouldn't say this is a no go, but it's not a "quick drop-in".
Thank you for the information Volga! Wasn't expecting to see a reply straight from the man himself haha.
While I have your attention, any thoughts/opinions on Nvidia DLSS 2.1 now being able to be used in VR?
With the growing resolutions and frame rates requirements for each new HMD, I'm hopeful that these kinds of techniques will help maintain sane GPU requirements.
I didn't like how the original DLSS required training per title, so it's good to see that they're starting to generalize things. I haven't seen DLSS in action specifically for VR yet. However it's showing promise. NVidia does need to make sure the DLSS reconstruction is stereo friendly in the sense that the algorithm applies plausible reconstruction that doesn't just look decent, but also plausible as far as stereo cues go especially along the edges of objects where depth discontinuities are the most obvious.