Event-based Near-eye Gaze Tracking at 10,000 Hz

https://www.computationalimaging.org/publications/event-based-eye-tracking/

ABSTRACT

The cameras in modern gaze-tracking systems suffer from fundamental bandwidth and power limitations, constraining data acquisition speed to 300 Hz realistically. This obstructs the use of mobile eye trackers to perform, e.g., low latency predictive rendering, or to study quick and subtle eye motions like microsaccades using head-mounted devices in the wild. Here, we propose a hybrid frame-event-based near-eye gaze tracking system offering update rates beyond 10,000 Hz with an accuracy that matches that of high-end desktop-mounted commercial trackers when evaluated in the same conditions. Our system builds on emerging event cameras that simultaneously acquire regularly sampled frames and adaptively sampled events. We develop an online 2D pupil fitting method that updates a parametric model every one or few events. Moreover, we propose a polynomial regressor for estimating the point of gaze from the parametric pupil model in real time. Using the first event-based gaze dataset we demonstrate that our system achieves accuracies of 0.45 degrees-1.75 degrees for fields of view from 45 degrees to 98 degrees. With this technology, we hope to enable a new generation of ultra-low-latency gaze-contingent rendering and display techniques for virtual and augmented reality.

Nice to see someone finally try out event cameras in a VR context. Hopefully the problem with cost gets solved one day though, since afaik they are insanely expensive to manufacture.

Also interesting to note that this research was actually done around a year ago and the first version of the paper was published then.

https://www.groundai.com/project/event-based-near-eye-gaze-tracking-beyond-10000hz/1

Also, looking at their results, it seems like the worst case precision (standard deviation vs average deviation?) means that if we wanted reliable foveated rendering using this implementation and on a 100 degree FOV headset, we'd need the safety margins of our foveation algorithm to be around 6 degrees (so if the center region of our real eye is 6 degrees, the rendering needs to output 12 degrees). Interestingly we could also vary the safety margin magnitude because apparently the precision gets worse the more the eyes look to the side. So if we have a headset that's 180 degrees, we can use a small margin when the user is looking ahead, and then a larger margin when the user is looking far to the side, up to around 6 degrees if the gaze is at 100 degrees.

That napkin calculation doesn't factor in slippage which the paper discusses a bit (like when the headset shifts a bit on your face when you turn your head), referring to it as "often the primary driver of error in head-mounted eye tracking products".