Logbook of the World Queue

General

Does anybody know the story behind why logbook of the world has a queue that grows to hours sometimes when people upload their logbooks? It's currently at 30min delay but I've seen it well above 8 hours some times.

Forgive my ignorance as I'm not a DBA but I would have thought in this modern day and age a very modest computer running a database would have no problem processing millions of QSOs per hour without delay.

Posted Dec 30, 2022, 6:56 PM UTC

1223
silasmoeckel•

LoTW is a bit odd it's got more security than my bank but has always felt like it was cobbled together, nothing designed to scale. Well maybe better to say scaled like the 80's tech concepts it's built on que things up and batch process sequentially.

14
funbobGA [E]•

That is absolute the case. LoTW was developed starting in the late 90's and deployed circa 2003 and has remained mostly untouched since then aside from some hardware refreshes over the years. It all just lives in the basement at ARRL HQ in Newington and suffers high profile outages periodically due to hardware failure or insufficient redundances or availability of backup power. They have poor tools to monitor the environment and frequently don't seem to know there's an outage until significant time has elapsed.

They passed 1 billion QSL's in 2018. Long past the point where they should have partnered with AWS or Microsoft and put this all in a scalable geo-redundant environment.

To this day, LOTW's implementation standards don't support automatic syncing of individual QSO's from logging software, unlike every other major QSL service which is a zero touch affair after initial setup. They would choke on that ingestion rate.

But... At the end of the day. ARRL's budget and staff are limited and this is a service for hobbyists, it's not like lives depend on it, so they're probably fine with a 98-99% uptime rate. "LoTW 2.0" has been talked about for years, but there are no concrete milestones around it.

7
silasmoeckel•

Now I wonder if it's on the tour.

I dont think I have any issue with the uptime or latency of LoTW at present, no matter how slow they are it's still faster than QSL cards (and nowhere as interesting as them).

The monitoring is fairly typical of small shops, would say rather see them move it to a commercial DC in the area I'm biased on that point as I've built several of them.

2
kassett43•

You are spot on. LoTW is 20 year old tech. It's kind of like the Windows XP of logging software as Ira the same age. That's the story of a lot of software that met the initial design specs, but did not age well.

For example, the way locations are handled in LoTW is beyond painful for today's POTA or SOTA use. It was just never designed for that.

1
Northwest_Radio•

This is irresponsible. With auto scaling and cloud computing, this should have been addressed.

1
ElectroChuck•

Send them some money and your plan and maybe they'll let you volunteer to run it.

3
ND8DIndustrial RF Design Eng. •

I’d love a peek behind the curtain as to how it sorts and confirms qso’s. The complexity figures are probably a horror show, but in this day and age it’s cheaper and more expedient to hurl computing power at the problem rather than make it more efficient.

3
silasmoeckel•

It does not seem how they are doing it scales well with more power. Thinking it's a big serial task for some unknown reason that does not scale to modern multithreaded CPU's forget multiple systems.

Suspect it's a case of you dont want to see how the sausage is made.

2
ND8DIndustrial RF Design Eng. •

Sometimes ya just get morbidly curious about the sausage.

2
nickenziK1NZ - 87 1/2 fathom band afficionado•

Lol, that's nothing. LoTW used to be backed up for weeks after CQWW, ARRL DX, etc.

One example: http://web.archive.org/web/20121214080540/http://www.arrl.org:80/logbook-queue-status

7
1980techguyOPUSA [Extra]•

Wowza, I had heard mention of this on a forum when searching around the lotw queue. That's crazy.

1
ChucklzKC2SST [E]•

That's nothing. At the same time as this was happening lotw was numbering each submission, but they were just using sequential numbers, which would overflow! So if you submitted file say 42, and the counter overflowed before your submission was processed...poof, your file would be deleted and replaced by a new file 42. The arrl told people this was a small bug, no big deal, and not a reason to resubmit logs.

Bullshit it wasn't. I had over a dozen "new" qsls found this way.

3
MaxOverdrive6969•

Never gave it a thought. Logbook program uploads the Q's and then I download confirmations the next day.

5
Zanzibar_T_Cricketextra dm03•

I have a similar workflow and same reaction.

I'm not understanding the need for speed in the process.....

2
Miss_Page_TurnerExtra•

How nicely can I put this?

In a way, the overall technological vision of the ARRL, vis-a-vis the delivery of goods and services over the Internet, might seem to some people to be a bit behind the times.

4
GrognardDave•

This usually happens when there are many hundreds of uploads in a short period of time, such as in the 24 hours after a large international or domestic contest.

One would think that it could be quicker, but it sure beats the old pre-LoTW days.

3
funbobGA [E]•

The hamster that turns the wheel that powers their servers died. They had to go get a new one. Your ARRL member dues hard at work.

3
ElectroChuck•

How many of the LOTW users do you think are members? There are a shit ton of freeloaders that don't contribute a dime to the League for the use of LOTW. Maybe that's why they have to SELL awards.

2
ChucklzKC2SST [E]•

Yes, it's a result of how lotw processes each qso. I've forgotten the details but it involves the number of award accounts and call signs. The matching algorithm did something dumb... I remember thinking 'are you serious?' When I first read it. Wish I had the post saved, but yeah, bad design assumptions

2
threeion3ka [e]•

cough Shocker.

1
kawfeyN0SSC | StL MO | extra class millennial•

very modest computer running a database

If only it were that easy. Not saying that’s incorrect, just that LOTW isn’t that.

AA6YQ has a lot of responses explaining LOTW in detail on the ARRL groups.io, usually in response to someone’s complaints or feature requests.

2
[deleted]•

[removed]

1
N0RMAL_WITH_A_JOB•

They could hire a 16 year old to create a better program. I think the current software has a decrepit old man yelling, “get off my lawn!” For every QSO record.

2