X11 eats up my memory

Hello. I know the title sounds a little newbie, but I'm having some issues with X11 on my computer. I'm running arch linux with i3 on X11 with all latest versions. After a couple of days my xorg running instance start claiming more and more memory and will not let go until I exit i3 and return to lightDM. I have tried investigating by disabling some software like compton and dunst without luck. I have also tried investigating with xrestop without anything that stands out of my mind.

uptime:

wall ~ ᐅ uptime
 13:11:41 up 1 day,  3:15,  1 user,  load average: 0,59, 1,04, 1,33

ps aux:

wall ~ ᐅ ps aux | egrep "i3|xorg|PID"
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       474 12.3 10.2 4140772 3365212 tty7  Rsl+ april04 202:58 /usr/lib/xorg-server/Xorg :0 -seat seat0 -auth /run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
wall      589  0.0  0.1 148836 39396 ?        Ss   april04   0:24 i3-with-shmlog

Free:

wall ~ ᐅ free -h
              total        used        free      shared  buff/cache   available
Mem:            31G         13G         12G        1,0G        6,1G         19G
Swap:           15G          0B         15G

I would like some help on how to investigate further, get tips and hopefully solve my issue.

EDIT, an update of the RAM usage:

wall ~ ᐅ free -h
              total        used        free      shared  buff/cache   available
Mem:            31G         17G        7,2G        1,2G        6,9G         13G
Swap:           15G          0B         15G

wall ~ ᐅ ps aux | egrep "i3|xorg|PID"
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root       474 14.2 24.2 8700896 7959036 tty7  Ssl+ april04 396:43 /usr/lib/xorg-server/Xorg :0 -seat seat0 -auth /run/lightdm/root/:0 -nolisten tcp vt7 -novtswitch
wall      589  0.0  0.1 148836 39396 ?        Ss   april04   0:32 i3-with-shmlog