What's a suitable SoC/computer for a hardware input device (joystick, for example)?

I'm making a VR input device (something worn on hand and tracks hand movement) just for fun. I've read that using a Raspberry Pi for something like this introduces a level of latency. Is that true? Do I need to stick with something like a Teensy?

In all, there will be about 50 angle sensors and 30 stepper motors that need to be controlled based on input from those angle sensors. Obviously this far exceeds the number of pins in a Teensy or Pi or any other SoC. What do you recommend I do about this?

Thanks!