Massive Latency spike on xrun/delay

So I have a problem at work that has been escaping me for a few days now and I'm hoping someone here might be able to provide insight.

I'm running into a situation where it seems pipewire (or potentially lower in the linux audio stack) builds up a massive buffer when the graph is even remotely late. I'm working on a announcement system and about 1 in 15 announcements run into this problem. Normally there is next to no latency but when is xrun occurs latency can be as bad as a full second. Audio is never dropped... which is why I suspect a buffer is just growing massively in this situation. Does anyone know if there's a buffer in pipewire that's capable of growing that massive? If so, is there a way to make that stop? I'd rather lose the audio than let a full second build up in the system

Heres some information on the graph I'm creating:

Source node:

• audio.rate: 40/8000
• periods: 2
• pipewire stream client

Sink node:

• audio.rate: 256/48000
• alsa period-num: 2
• alsa sink node

Graph config:

• Quantum (both min and max as well): 256
• Link buffers: 2

My current theory is that.... When the graph misses its timing, whatever buffer is being used for for the resampling grows indefinitely until the timing of the between 40/80 and 256/48000 can be aligned. Unfortunately, my audio card doesn't seem to support a none power of 2 quantum so it will always negotiate 256. So there will always be a weird offset in timing between the source and sink in this particular case.

I had a whim to post this just now because its bugging me. I'll see if I can post my exact config tomorrow. Figured I'd ask around here before I go digging through the source code.