Take envelope freehand drawing and media item mouse modifiers

I use multiple mouse modifiers on media items and bottom half media items, so when I activate any take envelope on an item, I can only use my Cmd+Left-drag (Mouse modifiers > Envelope Segment) if I start drawing on the line of the envelope, if I don't Reaper logically think I'm want to use my Cmd+left-drag modifier for media item or bottom half.

Would you guys know about a work around for this ? Tried to write a script with AI to temporary remove mouse modifiers of the media item and bottom half but had no luck with it.