DevOps team launch postmortem
Hey Guys,
I’m Joe, I’m a member of the Research Team for MAGLabs, as well as one of the many volunteer engineers involved with the DevOps team for MAGFest. The DevOps team handles ongoing development work on RAMS, our in-house application for managing registration and many other aspects of MAGFest, as well as handling all of the operational tasks associated with running and maintaining the servers.
I know that many of you are upset with the way registration launched this year, and to be honest we are too. I want to take this opportunity to share, from our perspective, what happened and what we’re going to work on to improve the system for next years’ launch. Since we’re a technical team, this post is going to get fairly deep into the technical details of what happened and what was done to resolve the issues.
Last year, the influx of traffic coupled with poorly optimized code caused CPU and Memory usage to grow far beyond our plans, which necessitated stopping badge sales for 3 days while profiling and optimization was done to survive the peak traffic demands. During that rush we sold roughly 3200 badges the first day, growing to ~5200 badges during the first 3 days.
In the aftermath of that ordeal, it became apparent that major changes would be required to cope with increasing traffic, and that part time development by contractors and volunteers wouldn’t suffice. The MAGFest Board of Directors created a full-time staff position for a developer to focus on improvement of our codebase, and Rob Ruana was hired in March.
During the first 30 minutes of our launch this year, it appeared that the server was up, however TCP errors were being logged to the system log:
kernel: [1731672.413723] TCP: drop open request from X.X.X.X/51395
Upon investigating and eliminating causes such as firewall rules and network issues, it was determined that the system’s TCP connection buffers were being exhausted. Increasing the tcp_max_syn_backlog kernel parameter resolved the connection issues.
Once the connection issue was resolved, the influx of traffic crashed the application running and required a restart. Looking into the logs, it appeared that the systems max file limits were hit. These limits were increased last year as part of our optimizations - however the increase in traffic hit these limits as well, and they were required to be increased as well. This change required an additional restart to the application server to apply.
This restart roughly corresponded with a decline in traffic, and no further issues were observed.
During the first six hours of registration, over 8000 badges were sold and peak traffic was over 10 times that observed during registration last year. The issues we faced were mainly related to server parameter tuning.
Going forward, we have reached the limit of performance increases that are possible within a single server environment. Migrating to a load balanced environment with multiple application servers will provide huge opportunities to scale appropriately. Currently there are two major changes required to support this architecture:
-
Currently, session data is stored on the filesystem, and there is no mechanism for avoiding session ID duplication between multiple servers.
-
Background threads to handle tasks such as email sending and other periodic background tasks need to be broken out into separate processes with a queueing mechanism to handle requests from multiple servers.
Implementing these changes will be one of the focuses of improvement for the code base throughout the next year.
Posted Aug 9, 2017, 2:25 AM UTC