MySQL - Random stalls during random times.

Production server (ECS)

Ubuntu 14.04.5 LTS
Kernel 3.13.0-30-generic
16 cores
64GB RAM
Apache2 and MySQL are sharing the same server.
InnoDB used on all tables
16GB innodb buffer pool
MySQL serves around 900 queries/second, most of them extremely small and fast.

I have this problem where MySQL becomes unresponsive for a couple of seconds. The stall isn't predictable and seems to happen at random times. Some days it doesn't happen, others it might happen 2-4 times a day. Each stall is around 1-8 seconds. At first I was thinking that I might be hitting max_connections but that wasn't the case. Then I checked my slow queries, there are a couple but I've never seen them run during the stalls. I've noticed that most disk activity seems to stop during the stalls. Also, if you check the Threads Connected there seems to be a gap in the log file, as if the script was not able to get these values during the stall. The web app that runs on this server runs into issues since mysql isn't responding. Most users get a corrupted cookie during the incident, which ends up with them logged out of the system.

I've run out of ideas and don't really know what else to look for.

Latest stall was 2017-08-25 10:33:53 - 10:34:00

I'm starting to feel hopeless about this, any help is appreciated. If you guys need some more data please don't hesitate to ask.

Update

CPU seems to spike to 1600% just before the stall and then drops to almost 0 a fraction of a second later.
Dirty pages increase during stall, what could be the cause of this?
Apache2 also seems to be unresponsive and connections seem to queue up during the stall. This might be directly related to the mysql stall though.
iostat consistently reports 0 disk usage during stall.
%nice seems to drop to 0 during stall while %system increases.

Posted Aug 25, 2017, 12:10 PM UTC · edited

226