Midi Sync Delay (Pattern vs Sample Trigger)

Hello there, I've had the sp404 mk2 for about 6 months. Slowly wrapping my head around how it can integrate into my workflow. My question, has anyone noticed a delay in syncing via midi in pattern mode vs sample mode?

I currently have a bunch of breakbeats that I'm syncing with a DSI tempest. I've noticed when I trigger the breakbeats in pattern mode, there is a pretty significant delay and things sound messy.

My current workaround is triggering the samples directly using a voice on the tempest. This sounds a lot cleaner rhythmically but the downside is that it takes away 1 of 6 voices on the tempest during the sp404 sample trigger.

I can make this work by resampling patterns into samples and using pad link/pad mutes, but I'd love to have the flexibility to use pattern mode if possible and I'd love to not have to sacrifice a voice on the tempest. For midi clock i'm using the clockstep multi which sends out simultaneous signals on a 5pin din and trs midi jack - not currently looking to buy a multiclock or something of the likes. Thank you!