r/help•
CorrectScale

Resolved: "Blocked" error when accessing reddit.com on Firefox

Admin Post

Hey all - we just reverted a change that resulted in reddit.com being blocked on Firefox for about 20 minutes.

All should be back to normal, but please let me know in this thread if you continue to see any errors.


Incident summary from u/PetGorignac:

Hi folks,

I was the incident commander for this one and came by to drop a bit of information about what happened here.

We were attempting to mitigate some problematic traffic that had been causing a low amount of site errors over the past few hours. In doing so, we identified some traffic characteristics that we believed correlated with the error rate and attempted to block it. It turns out this blocked Firefox traffic, which we noticed relatively quickly, leading us to revert the change.

Apologies for the disruption!

Posted Jan 14, 2022, 7:20 PM UTC · edited

129103
PetGorignac•

Hi folks,

I was the incident commander for this one and came by to drop a bit of information about what happened here.

We were attempting to mitigate some problematic traffic that had been causing a low amount of site errors over the past few hours. In doing so, we identified some traffic characteristics that we believed correlated with the error rate and attempted to block it. It turns out this blocked Firefox traffic, which we noticed relatively quickly, leading us to revert the change.

Apologies for the disruption!

(Also kudos to the commentor who had a great RCA, but sadly the comment got deleted before I could respond)

21
Merari01•

Please post this to r/shittychangelog

15
RagingCrane•

Seriously.

4
[deleted]•

[deleted]

1
Eldermuerto•

git commit -m “Site has error with browser. Blocking entire browser #Fixed”

git push origin master

4
PetGorignac•

Tsk tsk you should be doing your local dev work on a different branch than master! Wouldn't want to accidentally push to prod before code review...

2
nicolas-siplis•

OK honestly this just makes me even more curious. What traffic patterns are you noticing that would block Firefox requests exclusively, but not those made via cURL/Postman with the exact same headers? Can you go a bit more into detail here or is it too sensitive to discuss?

4
PetGorignac•

Unfortunately, I think I can't really expand on this much more as traffic management techniques are too abuse-adjacent

7
nicolas-siplis•

Completely understandable! Sucks that security through obscurity is sometimes the most viable option, but glad to see you guys being quick on a fix.

6
haykam821•

Too much road rage nowadays.

3
fluffycritter•

I'm very curious about this as well. My assumption was that it was something to do with the HTTP transport itself, like maybe there's one pervasive bot behavior that happens to exhibit the same timing or header ordering or something as Firefox.

Or maybe what they thought was an error rate due to bots was actually an error rate due to an HTTP spec violation/assumption on Reddit's side that was causing increased issues with Firefox.

3
nicolas-siplis•

My assumption was that it was something to do with the HTTP transport itself, like maybe there's one pervasive bot behavior that happens to exhibit the same timing or header ordering or something as Firefox.

But in that case wouldn't cURL requests copied from Firefox itself not work as well? I really hope the devs can chime in with some more info, otherwise I'm gonna spend the next few hours scratching my head trying to play digital Sherlock Holmes D:

3
fluffycritter•

Nah, "copy as cURL" would still be using cURL's HTTP transport stuff. There's more to HTTP packet analysis than just the headers.

(Edited for clarity and better phrasing)

3
6pointzen•

When I was checking on my end I could see that some of the fields in the request header were different between Chrome and Firefox I couldn't check them all as I refreshed the page and it was working again.

2
nicolas-siplis•

But that's actually expected. The simplest example would be the User-Agent header, which tells Reddit's backend which browser is making the request so barring spoofing it will obviously be different between Firefox and Chrome. The super weird thing is that the same exact request sent through Firefox and cURL failed in the former but not in the latter!

2
JBHUTT09•

I saw someone say that spoofing the user agent didn't result in the request being blocked. I can't confirm that, though.

1
6pointzen•

I remember specifically the cross origin one was different, for instance, but I don't remember which values it had.

I'm not that deep into this so l just looked at it for less than a minute or so as I was pretty sure l couldn't do anything about it and it was back online while I was on it

1
Mothringer•

Sounds like reddit needs to overhaul its change management system if something like this was allowed into production that should've been easily caught in a smoke test but wasn't.

2
burnalicious111•

Welcome to big tech companies.

Not testing this doesn't cause them to lose money, so they're not incentivized to. Hell, even at companies I've been at where it _did_ lose them money, they cared more about how much it looked like we were delivering more than anything else.

1
Eldermuerto•

Seriously, Just a couple weeks ago they broke ALL the top feeds and then just went on vacation for a week.

1

From the analysis in the Firefox bug tracker, it was related to cipersuite selection, which differs between the browsers (including Edge/Chrome) and curl/openSSL.

2
ipha•

Only think I can think of is something at the SSL layer.

1
VLXS•

Firefox users are the most likely to be viewing fewer ads. I noticed that on Chromium UBlock works differently; I can view ("promoted posts" on Chrome, while old.reddit on Firefox does not display those ads.

I wonder how many here use old reddit instead of www

1
Alx_xlA•

Are there people out there who aren't using old Reddit?

1
Security_Chief_Odo•

Probably some user-agent spoofing, or TLS cipher suite ordering that was blocked.

1
TaraBaraBoo•

You're awesome, and I'm so impressed that you are open about this, it just makes me love Reddit even more!

3
[deleted]•

[deleted]

3
Acclocit•

Thanks, still curious about what the characteristics you blocked were though.

2
RagingCrane•

Thanks for the update.

2
flash357•

LOL

WHOOPSIE!

cant pull the plug on us like that jefe!

we were in panic mode!

🤣🤣🤣

2

Hey

1
Eldermuerto•

So instead of investigating and fixing the errors you just blocked an entire major browser. Great stuff guys. Really showing how it's done.

1
jstosskopf•

The other part of the problem is that you guys don't test on Firefox particularly much.

The "Fancy Pants Editor" often blocks Copy and Paste, while Markdown editor is fine. Using a Chromium-based browser has no problems.

1
Nulono•

What does RCA stand for?

1
cmays90•

Root cause analysis: it's a process to find the core of issue and implement other processes to ensure this exact issue (and hopefully many similar ones) don't happen again.

Here: Root Cause was some security change (likely in the SSL validation stack) that cause Firefox to crash. So in the future, when SSL validation changes occur, ensure adequate testing occurs on Firefox to prevent this from happening again.

2
[deleted]•

[deleted]

1
redbull666•

Please do actual pre-production testing on Firefox and stop helping Chrome remain as the only browser.

10
Sachyriel•

Haha, why would they do that? Where else are we going to go? Hahahannnnh ;_;

2
bioemerl•

You literally can't paste into the stupid text box in firefox becaues it will RANDOMLY DELETE YOUR ENTIRE POST if you do.

I don't know what clusterfuck of a process let that code get out the door, but it needs fixed and it's been an issue for like a year and a half.

2
Lil_SpazJoekp•

Don't forget about safari!

1
ananos12-•

Use brave

-3
neighguard•

brave is chromium based

8
Pabszito•

firefox nightly + ublock origin + universal bypass works 10 times better than brave

3
tzaeru•

Brave blocks ads so it can show its own ads..

It has also secretly inserted its own referral codes to e.g. links to cryptocurrency exchanges.

It's also deep in the crypto bubble.

I don't trust it one bit. Privacy, security and so on are just a way for them to get people to their own ad and tracking platform.

2
thealterlion•

yeah I've been using it for a bit on android and I'm not convinced. I hate to have my homescreen be a crypto ad, even if the rest of the browser is quite good.

I may give Vivaldi a try and give Firefox for android another chance

2
Daverost•

You can blank the homescreen and I would definitely do that. That said, I have yet to find a better browser on mobile than Brave. Nothing else seems to block ads or clean up the ad space as well. I may have to try Firefox again, but mobile Firefox was a hot mess some years ago when I last tried it.

I have no real desire to use Brave on desktop, though.

1
neighguard•

What do you guys have against Firefox users :(

7
SpicyHotPlantFart•

Too much privacy settings :P

7
neighguard•

Do plants fart??

2
SpicyHotPlantFart•

You think that's air you're breathing now?

2
Sachyriel•

Reddit hates cute animals.

4
PetGorignac•

If nothing else, I can guarantee you that this is not true!

8
Sachyriel•

Okay I believe you. Please do not upset the firefox horde again, we protect our cute mascot.

2
PetGorignac•

Definitely the best browser mascot

3
[deleted]•

[deleted]

1
EmbarrassedHelp•

Will there be any information made available as to what the mistake was?

3
PetGorignac•

I made a top level comment with a brief explanation here

5
EmbarrassedHelp•

Thank you!

2
JBHUTT09•

Yeah, as a web dev I'm curious. Knowing what happened might help me not make that mistake in the future, lol.

3
xzinik•

y u hate us?

2
Sachyriel•

dey hate us cause they anus

1
nicolas-siplis•

Hey! Any chance you could give us a hint as to what went wrong? Since spoofing the User Agent didn't seem to do anything, my guess is that it was related to Firefox's Root Certificate Store which differs from the one present in the OS, which is the one Postman, cURL, etc. use. That would explain why the same request via those programs actually worked while browsing through Firefox did not.

2
JSTLF•

Since spoofing the User Agent didn't seem to do anything

You might have done it wrong, I was able to get around it by overriding my user agent

2
nicolas-siplis•

Nah trust me, a lot of people tried spoofing the User Agent and it did not work for them either. I even took a look at the request headers to make sure I was pretending to be Chrome. It's likely that the error just happened to get fixed after you changed your UA, but a lot of people now think that or disabling HTTP3 fixed it when it was most likely unrelated.

2
Sachyriel•

I changed my DNS settings for this! It came back after I changed them now IDK if I want to go back.

1
nicolas-siplis•

Doubt it's got anything to do with DNS, it's been fixed for pretty much everyone so now we all have different theories as to what solved it on our end when all indicates it was a problem on Reddit's side. I didn't change any DNS settings and it's working fine now.

1
JSTLF•

well ionno because I didn't fix my useragent on my laptop and it didn't fix it so it's kinda weird in that regard (although it is true that a patch was rolled out like 10 minutes later)

1
iammiroslavglavic•

I went on my android phone's firefox browser, incognito/private window thing and reddit.com, went onto a post I created on another sub, i clicked on my profile and it seemed fine. I just wasn't logged in on my phone browser.

1
nicolas-siplis•

Interesting! Just to verify, after checking that Reddit worked when accessed through Android, did you try accessing it through the desktop as well? In which case, did it keep failing or not? If it didn't it's possible you just happened to try that when the issue got fixed. If it did fail, maybe there's a discrepancy between Firefox's CA list in mobile vs desktop? Otherwise I guess it's back to square one, in which case I hope the devs can do a quick post mortem!

1
iammiroslavglavic•

my phone's firefox browser, not the reddit app. I just became back from eating lunch. Apparently the oldest comment on the million posts about this is 40 something minutes ago.

2
nicolas-siplis•

Yeah I understand that you managed to access Reddit through mobile Firefox, but I wanted to know if you tried accessing it via desktop afterwards. Depending on whether or not you did, and whether or not you succeeded, it contradicts my little theory.

2

Hi

1
Deleis•

Is this issue caused by the HTTP/3 bug in FF?

2

"Test environment? Never heard of it"

2
Disquo_303•

As an ff and proxy user, I thought my proxy was the issue, again. Anyway, looks like it's fixed on my end. Thx

1
iammiroslavglavic•

Yes, that happened to me. Google says I had to reset the password. I did. username+e-mail, waited for the e-mail, clicked on the link/button, picked a new password that I don't use anywhere else. Voila, it worked. I came on here to ask why that happened and voila I see the post I am replying to.

1
yayayatta•

voila

1
justcool393•

Thanks for fixing it

1
Bardfinn•

Glad to know it wasn't my DNS server flipping over into parental blocking mode

1
[deleted]•

[deleted]

1
[deleted]•

[removed]

1
iammiroslavglavic•

All should be back to normal, but please let me know in this thread if you continue to see any errors.

How does someone do that if they are "blocked"? Obviously I am not but if they see that blank-page-with-just-blocked-error. Then they can't reply here. Can they?

1
Malek_Deneith•

In this particular case it was only Firefox users that got blocked. So if it'd happen again they could just use any other browser to reply.

1
-JakHamma-•

I'm sure their handful of remaining users can breathe easy now, if that went on for any longer the might have actually had to get off the computer and do something with their lives!

1
[deleted]•

[removed]

1

I Have a Something went Wrong Because I have no connection

1

Oh Now I Have a Internet Connection

1

phew

1

Antiwork

1

ER I mean Antiwork now?? reddit.com.antiwork

1