We were all hands on deck tonight with our fire fighting gear on and not a damn thing happened. Smooth sailing. We called it at 19:15 CDT and checked out.
Anyone else have any leap second events or non events?
Edit: apparently the internet didn't handle it smoothly. https://twitter.com/DynResearch/status/616059353821487104/photo/1
Posted Jul 1, 2015, 12:14 AM UTC · edited
Everything went smoothly... except that Paraguay scored a goal during the leap second... hum... strange..
It's 9:30am now. I woke up at 2:00 am this morning to be available to remote in if problems occur.
Looks like our preparations paid off and only 3 servers out of several hundred are not right, but I have snoozed their alarms since they aren't critical.
Business day has started up and all the banking stuff going on that we do, and no one has lost money from algorithms being off.
We actually inserted the second a few hours early before Greenwich midnight because we are in investment banking, and it would have been added during trading hours.
Edit I am in Asia for those wondering about the times.
I'm curious how you added the extra second early...
Master ntp servers prolly
Done through our own master ntp servers..
If we didn't do it early it would have happened during trading hours. With millions or billions of dollars in high frequency trading going through our systems, it's way to risky to do it during active trading sessions.
We run a backend for major trading between large investment banks.
Its not billions if you are doing HFT.
No, it is billions. Not many billions but it is a few billion that goes through us everyday.
Hacked local physics.
I'm on vacation. Hahaha
... Checking next leap second and scheduling my vacation. ;)
you win
Dear god it all went to shit here. Juniper equipment did not handle the leap second well. High CPU errors on nearly all our SRX Firewall/Routers which caused all our tunnels (we have over 40) to intermittently drop which caused OSPF to reconverge continually. Kill me.
Oh man, my condolences. I hope there patches in place before the next shit storm.
I just pointed my stuff on ntp server that can do "Google trick" with smearing extra second instead of passing leap second bit, no problems
I should mention we're pretty much down/slow during the summer months so the outage went unnoticed by users thankfully. Working around the college sports schedule FTW. :-)
I should mention we're pretty much down/slow during the summer months so the outage went unnoticed by users thankfully. Working around the college sports schedule FTW. :-)
[deleted]
Is this a thing? Or is this a joke thread? Honest question.
It's a real thing. Time gets adjusted to account for the slowing of the earths rotation due to earthquakes, volcanoes, other natural phenomena. It's not a "regular" thing like Daylight Savings, so it's potentially more problematic when it comes up. One second may not sound like much, but it can throw off threaded and concurrent processes that expect things to run in a precise time.
I know this subreddit is viewed by more technical folks, but why the down votes? Show yer smart and let the dude know what's up?
The NYSE turned everything off because of the leap second and the problems it can cause. Other companies like Google do a smear, where they spread the leap second over the entire day.
http://www.theverge.com/2015/6/30/8866463/june-30th-leap-second-confuse-internet for a quick overview and links to specifics.
You know what a leap year is, where we adjust time by a day every 4 years since the earth doesn't go around the sun in exactly 365 days. It's actually closer to 365 days, 5 hours, 49 minutes, and 16 seconds. Adding a day every four years mostly fixes this, but it's still going to be off by 3 days every 400 years.
A leap second is just our way of making sure our time lines up with how long it actually takes the earth to orbit the sun in pretty much the same way a leap day does, just on a much smaller and much more accurate way.
None here. Disabled ntpd with config management a few days ago. Scotch o'clock time.
That's....bold. No databases or replication running on your systems?
Nothing that would be negatively impacted from slight time skew.
Had a small hiccup with our credit card processor.
Had to restart a service on the web facing front end then restarted the database. Minimal interruptions.
No one yelled. Life is good.
Yeah, I work for a trading firm and were fucked for about 2 hours, even after working all week trying to account for everything... Why the fuck they decide to do this at month's end in the middle of the work week boggles my mind..Either way throwing back a few when I get off in a hour
Bad luck man. Enjoy a few on the sysadmin boss, ya earned em.
Survivor checking in!
Welcome, drinks are over there. Snacks are on the table.
Management was freaking out, wanted to do rolling reboots and some other nonsense. We (sysadmins) said screw that noise, just waited and lo and behold, NTP handled it just fine. Psh.
We have an older version of TeamCity running an older version of Java on an older version of Ubuntu.
It spun out and I could not get Java to be happy. Luckily I remembered that the last time this happened, I just had to do a reboot to "fix the glitch". I am sure there is a more reasonable method, but the hammer works.
Hello, IT. Have you tried turning it off and on again?
Honestly this sounds like the best way to solve this problem.
In this case it was. It has something to do with the way Java and the Linux kernel interact. Rebooting resets whatever clock the kernel had that messes up Java.
I was on here to say just that
Our Ciena network had some undocumented issues with line cards resetting last night and in to this afternoon. Ciena isn't quite sure why but it is strangely coincidental. And as long as my Solarwinds server didn't die during the leap the Cisco side of things survived.
So far, no problems.
This may be due to shutting down ntp, for the transition. (we don't need that degree of accuracy. And the clocks aren't that inaccurate anyway.)
Tomorrow, it'll be switched back on in phases, after causing a time sync. probably to no effect :)
(we have some legacy systems which should be safe, but it's a better safe than sorry position.)
Why shut down ntp?
[deleted]
It sounds like a poor mans leap smear. NTPd have an option for that you know.
it also means I can control exactly when it starts happening, rather than at midnight UTC, when I'd prefer not to be working.
Ah, we're configured so our servers noticed the one second offset from the reference and slewed to adjust
All clear here, but I'd patched the most vulnerable systems myself. I'd say having nothing happen at all is pretty much the best result I could have hoped for.
Only thing here was that my core NTP relay node bitched that it was suddenly 998ms separate from its reference. It automatically chose a different upstream, and all was well.
Nothing broke out of a hundred or so nodes.
Nagios played a prank on me by sending an alert e-mail. Fortunately, it was just Chinese botters using a new IP block. All is well (CentOS 6)
Greece stole the extra second to pay down their porn bills.
I slept through it...
Windows drifts so badly anyway I wasn't worried
True. Sometimes having inaccurate servers lets you slide a bit.
Backups failed, but what else is new
So just another Tuesday, Eh?
Anyone else's network give 0 fucks about that level of accuracy?
One of the junior guys though he'd be the hero and stopped and disabled the time service on all our DC's. Boy was he proud of himself...
Oh wow :/
Give him a magnifying glass and make him go find the token that fell behind the racks and made the servers stop working.
My servers were like "Huh? Was that the same second? Oh it passed. Cool."
I sent him over to the building with our most problematic employee in it. The lady needs just about constant hand holding, and it's considered a win if she cries less than 10 times a day (I'm not kidding). Then I emailed her and told her that this guy was going to be in her building that day at her beck and call and that if she needed ANYTHING to go down and get him. Judging by the look on his face this morning I'd say he didn't have a good day yesterday.
Next, change his system font to comic sans. Might as well scar him for life.
Good for him. Just let him feel important.
Or, if you're a dick, send him this.
I wish, some moron promised to our customers..... and they saw it as a reason to grab all available magnifying glasses.
It's not the accuracy that matters per se, it's that some systems misbehave when they see the same second twice
Everything on my network evidently saw the same second twice and goes "huh... Didn't expect that to happen. Oh well... "
yeah it was specific gear that was affected. For example my SAN switches would have been (Nexus 5k) and my fabric interconnects could have been (luckily the code they are running on is old as shit).
There really wasn't a reason to touch windows machines. Some older Linux machines freaked out and their CPU usage spiked a bit.
https://access.redhat.com/articles/15145
I didn't reall read all that shit though. Just rebooted the servers affected since it occurred after hours in my timezone and everything is HA at the service.
Yeah take a look at some of the cisco nexus switches. Some versions can cause a total system lockup on a leap second.
We would have had this happen but i removed ntp yesterday and put it back today.
Is your NTP updating every second? The switches shouldn't see it twice unless it does update every second, which I didn't think they did.
The main reason for NTP on network devices is not accuracy but correlation.
If both your servers and network devices have same time it is much easier to correlate events from various sources and that helps a lot in debugging
I commented earlier about if this was a joke thread or not exactly for this reason. I suppose I can see that specific time-sensitive processes and logging can require a lot of attention for a leap second, but in a 75/25 Windows/Linux environment where the most important thing we really have is file shares? This wasn't even on my radar.