Weed on dextroamphetamine come down, will it cause anxiety or panic?

I've been taking dextroamphetamine 5-6 days a week for about a month now, usually 15-20mg. It works amazingly for my motivation and attention span and I love it, but it's almost impossible for me to relax or sleep until at least 5am if my last dose of dex is at midday. I've been thinking about smoking up a bit of bud to help ease this, but I've noticed that both drugs increase my anxiety to a moderate degree and high doses of either if I'm not in the right head space can cause a panic attack. If the dexamphetamine has totally worn off in terms of stimulation but I'm still feeling the side effects, will I be likely to have a panic attack if I get high?

Sorry this is a pretty specific question, was thinking that someone might have experienced something similar, or the same thing with Adderall (which is mostly dex anyway). Thank you!