Need help combining keyframed animation to my RBD sim

Hello! I have an object that is keyframed to move every 40 frames or so. I have that object parented to my simulated object before RBD sim. When I try to simulate the keyframed object does not move at all and I want the keyframed object to influence my RBD sim, not replace it. How could I set this up to have this kind of motion? Gif from Bots by Media.work

https://i.redd.it/ejjzixh6v1ve1.gif