Quit weed just before bed and now I wake up earlier and get tired more?

I've been feeling kinda groggy lately and confusion in my everyday life to a mild degree and I blamed the weed, because it limits rem sleep. I usually would have some edibles after dinner at 6pm. Be high for a while and go to sleep at about 10pm and wake up at 7 or 8.

For a few days I stopped taking edibles that late. If I did have weed it would be around noon-2pm if I didn't have work. But now fall asleep at the same time but waking up at 5 or 6, basically two less hours of sleep every night and I get 3pm 'crashes' from waking up that early. Even when I try and go back to sleep.

Should I just keep at it and see it changes, should I worry about it at all?