It's back bay-be!

2021-10-25: GAME DOWNTIME AND RECOVERY

  • Hey all! So, you have probably noticed the game acting strange since about midnight last night and then it has been offline since about 7am game time. I’m going to try to breakdown what happened and what I’m doing about it.

FIRST, WHAT HAPPENED

  • The game stopped allowing new items to be given to players just before midnight game time last night. At first, I thought the database table for items had crashed, but that wasn’t the case. Here’s the technical answer: There’s an auto-incrementing unique ID per item per player. That type (MySQL INT) actually reached the max INT value of 2,147,483,647 so no new rows were being inserted into the table.
  • At first, I thought I would just change the type to BIGINT and keep going, but why did it hit 2.1 billion? We don’t have that many players or items, so it shouldn’t have happened. Turns out it was in ‘how’ I was adding items to the table. I have a unique constraint on the table for user + item, so inserts were happening even if the item was already in the table (which I thought was ok). With InnoDB the behavior by default is to keep incrementing the auto-incrementing value even if the insert didn’t actually happen. It was a bad combination of things and ultimately something I never considered could or would happen. I’ve changed how I’m adding to the table and redone the keys to be INT with correct values.

BACK ONLINE WITH A BACKUP OF THE DATABASE FROM JUST BEFORE MIDNIGHT

  • Everything went really haywire since the items table couldn’t accept new items. Hundreds, if not thousands of people, were getting items from playing and not actually getting the items in their Inventory. This affected every single part of the game from farming to the wishing well, pets, etc, everything. I had hundreds of support tickets from players that didn’t get their items from quests, crafting, etc, so I had to make the hard decision of reverting to the backup from last night.
  • This means if you played from about Midnight to 7am game time, your progress is lost. I hate that so much, I really do, but the game was a complete wreck due to the items not being added correctly. I had a lot of complete activity where no rewards were actually given, etc, and it is a giant train-wreck.

WHAT I’M DOING ABOUT ALL OF THIS

  • I’ve already fixed the table and the code and the ID column. Staff has tested things out and all feels ok again. I’m going to put up an “I’m really sorry” Help Request and give everybody some nice stuff today to make up for it, even if you weren’t affected by this you’ll be rewarded. I hate that it happened, and even though I try to mitigate this kind of thing from happening, it can happen still. So, I’m sorry once again.
  • If you bought Gold overnight I’ll make sure you have that Gold still.
  • So... that’s it. Thanks for reading and understanding and I hope that everybody can pick up from where we left off and continue enjoying the game and community.

SPECIAL HELP REQUEST AND XP BONUS EXTENDED

  • Check for the ‘Take this’ special help request +20% Skill XP extended through Tuesday evening now

ONE LAST THING

  • I realize that during the night some players found runestones or super rare items, and with rolling back to the previous backup, you no longer have that item. Send in a Support ticket and I’ll make it right for you. Allow me 24-48 hours to get through them all ;)

Posted Oct 25, 2021, 4:22 PM UTC

193
Loading comments…