How does Oculus Link Work? The Architecture, Pipeline and AADT Explained
developer.oculus.com/blog/how-does-oculus-lin...
developer.oculus.comPosted Nov 22, 2019, 11:24 PM UTC
4630
developer.oculus.comH.264 instead of H.265. I guess that explains why there are still compression artifacts.
6
Ive only noticed artifacts while watching gameplay vids at half speed. Been playing with link for a few days and haven’t noticed it while playing either.
Regardless of how people feel, the engineering is incredible.
5
H.265 would be sharper, at the cost of latency. Which is probably why they went with H.264. An option to switch between the two would be nice though.
7
It could also be related to what the Snapdragon 835 can decode in hardware at 72 Hz. Carmack previously discussed such limitation on Go with decoding 5k@30 Hz
7
If I remember correctly from one of Carmack's speech/blog/tweets the two limitations are some total bandwidth on one hand and a hard limit for width of 4096 which is why he said it's better to use OU of you have higher res stereoscopic 180 content.
1
H.265 has compression artifacts. Also, do you understand H.265 would require a hardware encoder?
2
H.265 has compression artifacts.
H.265 looks twice as good at the same bitrate compared to H.264 and Link already has few artifacts.
Also, do you understand H.265 would require a hardware encoder?
Which as far as I know the Snapdragon 835 has. Virtual Desktop and ALVR both use it.
3
Encoder, not decoder bub. The device doing the rendering does the encoding. Yes, H.265 looks better than H.264...
5
Right, I misread. That's an even easier yes than. Almost every GPU since 2014 can do that and certainly every "VR Ready" GPU.
6
I'm assuming a hardware encoder is required anyway. Don't all VR-capable GPUs have them? On the Nvidia side, NVENC has supported h265 since the 900-series.
2
I'm assuming a hardware encoder is required anyway. Don't all VR-capable GPUs have them? On the Nvidia side, NVENC has supported h265 since the 900-series.
Most of the 900 series don't support HEVC properly, only one or two different revisions of a lower end 900 series card have the full support. 980 Ti got left out in the cold.
It's 10 series onwards, I use HEVC with my RTX card and it's a lot better than my 980 Ti using h264
2
I'm no expert on the Pascal encoder, even with hardware support it could have unacceptable encoding latency
1
Maybe, but I'd be more suspicious of the Quest's decoding performance.
1
No it wouldn't, works fine. You can check it with VD or ALVR. There may be other limitations on the Quest side (would have to go and look up the exact values again) for decoding though, but the 2016x1120 stream used now is far away from any such limitations.
1
USB3 pipeline explains artifacts.
There’s simply not enough data stream to stream full resolution, so the headset needs to “fill in” the gaps (which is foveated so that the center is full res)
Given the limitations, but all accounts they did a phenomenal job.
I’d still prefer my S for PCVR, but this is a fantastic 2 in 1 solution
0
No, the decoder speed is the limiting factor. USB 3 is capable of much higher speeds than 150 mbit/s. There is also no need to stream less than the full resolution, they could have just reduced the bitrate / increased compression.
That's all besides the point though, H.265 looks far better at lower bitrates. There likely wouldn't be any artifacts with H.265 at the cost of slightly higher latency.
5
With AADT, a pair of undistorted eye-buffers rendered at 3616x2000 can be compressed down to 2016x1120 saving close to 70% of the video codec budget. AADT not only saves codec bandwidth, but it also saves latency by making sure the encoder and decoder work on smaller images. While these numbers are tuned for the Oculus recommended-specification hardware, they might be updated for better visual fidelity in future releases.
This explains why in SteamVR it said 1808x200 per eye when set to 100% resolution. Good to see some additional confirmation that there may b exposed settings for the values in the future, it sure would be nice to be able to send something better than a 1008x1120 image to the 1440x1600 displays. I would also appreciate a switch to select h264 or h265 (especially having a 2080 with the improved nvenc) to be able to individually tweak image quality and latency as everyone perceives these differently.
4
Very proud of the team for all the work they put into this. It was great to see it all come together :)
2
I only just found about this blog post, here's my tl;dr I wrote up for r/OculusQuest:
- Frames are encoded and decoded in slices, so that the Quest can start decoding slices before it's received them all.
- The PC VR runtime is designed to predict the head and hand position at the time that the image will appear on the display, so that it can render the view correctly in advance. Oculus Link modifies this process to use the time the image will appear on the *Quest* display, taking into account the streaming delay.
- Adaptive Compositor Kickoff (ACK): at the very end of each frame, before displaying an image, the PC runtime does a last-minute visual correction (Asynchronous Timewarp or ATW) based on the latest input data. ACK makes sure this is done as late as possible, so that the input data is collected as close to display time as possible. Oculus Link modifies ACK to kick off just in time that it can do both ATW and streaming before the Quest displays its next frame.
- Axis-Aligned Distortion Transmission (AADT): before sending it over the connection, an image distortion is applied that effectively lowers the resolution of the outer edges of the frame while preserving full resolution in the center. This greatly reduces the resolution of the frame and the required bandwidth. After the Quest receives it, it reverses this distortion using its GPU, then sends it through the normal pipeline.
2
I really hope Carmack-less Oculus will still innovate like this.
-Oculus is a big company-
-Oculus is a big division of Facebook-
There are a lot of people working on VR at Facebook who are also really smart. They’ll probably be fine.
Facebook has the entire FB Reality Labs team, which spans multiple cities, dedicated to innovation. They've been working everything from hand-tracking to experimenting with BCI like the CTRL Labs bracelet. Carmack was important, but he wasn't the only employee pushing innovation at FB/Oculus. On top of all that, he'll still be a consultant anyway.
FB is a huge company with really smart people who are paid a crap load of money to innovate. They probably have many people on or above Carmack’s level.
FWIW Carmack is still a Facebook employee and is still involved in Oculus projects.