Add trim doors but drywall is making a gap.

We had a room that had wood paneling and replaced it with drywall. I want to trimming the doors but I'm not sure how to handle the gap that the drywall is creating with the door frames. Attached are pictures of the issue. Do I try to sand down the drywall so the trim will be flush with the door frame or add shims to the door frame so that it matches the drywall? Any advice?

See how the drywall is about a half an inch out from the door frame

pretty much like this around the whole door.

https://preview.redd.it/j2kclg51rzmf1.jpg?width=1848&format=pjpg&auto=webp&s=54040c340dbb9a3073e8b8e93e1b116570d9241b

https://preview.redd.it/m4ngrzn1rzmf1.jpg?width=1848&format=pjpg&auto=webp&s=46f771edd1f5ce461b496a6d14a327894bcaed6f