Logbook of the World Queue
Does anybody know the story behind why logbook of the world has a queue that grows to hours sometimes when people upload their logbooks? It's currently at 30min delay but I've seen it well above 8 hours some times.
Forgive my ignorance as I'm not a DBA but I would have thought in this modern day and age a very modest computer running a database would have no problem processing millions of QSOs per hour without delay.
Posted Dec 30, 2022, 6:56 PM UTC
LoTW is a bit odd it's got more security than my bank but has always felt like it was cobbled together, nothing designed to scale. Well maybe better to say scaled like the 80's tech concepts it's built on que things up and batch process sequentially.
That is absolute the case. LoTW was developed starting in the late 90's and deployed circa 2003 and has remained mostly untouched since then aside from some hardware refreshes over the years. It all just lives in the basement at ARRL HQ in Newington and suffers high profile outages periodically due to hardware failure or insufficient redundances or availability of backup power. They have poor tools to monitor the environment and frequently don't seem to know there's an outage until significant time has elapsed.
They passed 1 billion QSL's in 2018. Long past the point where they should have partnered with AWS or Microsoft and put this all in a scalable geo-redundant environment.
To this day, LOTW's implementation standards don't support automatic syncing of individual QSO's from logging software, unlike every other major QSL service which is a zero touch affair after initial setup. They would choke on that ingestion rate.
But... At the end of the day. ARRL's budget and staff are limited and this is a service for hobbyists, it's not like lives depend on it, so they're probably fine with a 98-99% uptime rate. "LoTW 2.0" has been talked about for years, but there are no concrete milestones around it.
Now I wonder if it's on the tour.
I dont think I have any issue with the uptime or latency of LoTW at present, no matter how slow they are it's still faster than QSL cards (and nowhere as interesting as them).
The monitoring is fairly typical of small shops, would say rather see them move it to a commercial DC in the area I'm biased on that point as I've built several of them.
You are spot on. LoTW is 20 year old tech. It's kind of like the Windows XP of logging software as Ira the same age. That's the story of a lot of software that met the initial design specs, but did not age well.
For example, the way locations are handled in LoTW is beyond painful for today's POTA or SOTA use. It was just never designed for that.
This is irresponsible. With auto scaling and cloud computing, this should have been addressed.
Send them some money and your plan and maybe they'll let you volunteer to run it.
I’d love a peek behind the curtain as to how it sorts and confirms qso’s. The complexity figures are probably a horror show, but in this day and age it’s cheaper and more expedient to hurl computing power at the problem rather than make it more efficient.
It does not seem how they are doing it scales well with more power. Thinking it's a big serial task for some unknown reason that does not scale to modern multithreaded CPU's forget multiple systems.
Suspect it's a case of you dont want to see how the sausage is made.
Sometimes ya just get morbidly curious about the sausage.