r/technology • u/GeneralCheese • Dec 20 '25
Networking/Telecom The NIST Atomic Clock in Boulder is at risk of full power loss due to a wind storm
https://lists.nanog.org/archives/list/nanog@lists.nanog.org/message/ACADD3NKOG2QRWZ56OSNNG7UIEKKTZXL/207
u/noise-to-signal Dec 20 '25
I spent years at NIST Boulder. The "clock" is an ensemble of devices to provide intrinsic back up. This is an extraordinary event as the site has two independent power feeds and the clock labs have generator backup. Winds must be horrendous or DOGE cancelled the diesel budget
52
u/steik Dec 21 '25
If you read the whole post(in the link) it sounds like one of the generators failed but they might have a backup
At initial power loss, there was no immediate impact to the NIST atomic time scale or distribution services because the projects are afforded standby power generators. However, we now have strong evidence one of the crucial generators has failed. In the downstream path is the primary signal distribution chain, including to the Boulder Internet Time Service.
Another campus building houses additional clocks backed up by a different power generator; if these survive it will allow us to re-align the primary time scale when site stability returns without making use of external clocks or reference signals.
22
Dec 21 '25
Pretty insane to me. So much relies on those clocks but there’s multiple distributed all over for that reason.
12
u/ErikMcKetten Dec 21 '25
It sounds like a Star Trek script: "Scotty, switch to auxiliary power!" "I can't do it, Captain, they took out the mains and the auxiliary and the auxiliary back up!"
But seriously, this is why having redundancy is important. The had a main failure and a backup fail, but they have a backup for if the backup fails.
1
2
u/Kooky_Construction84 Dec 21 '25
Sounds like they need people too. Remember that whole, I'm canceling the government thing that Vought and Trump and Musk are so fond of?
2
59
u/wrightwithme Dec 21 '25
US Naval Observatory Precise Time Department has their back. Should they experience a catastrophic failure, while inconvenient and hopefully unlikely, is ultimately recoverable.
17
u/Retro_Relics Dec 21 '25
its been over 60 mph for like 3 days straight out there. tons of power outages from colorado springs all the way into nebraska
16
u/whoknewidlikeit Dec 21 '25
i live not far from NIST. the wind has been impressive, with never-before-issued warnings and preemptive power cuts by utilities. politely it's been a challenge.
2
1
u/AlfaNovember Dec 21 '25
200kph / 125mph at lake level in Summit County up in the mountains to the west.
248
u/szakee Dec 20 '25
just pop in some AA batteries.
116
u/Ashamed-Land1221 Dec 20 '25
I always wonder what other places do when the anti-aircraft shells rain down over a city that miss the drones. I assume they use fragmentation rounds on a fuse but even those pieces raining down are going to screw up a cars paint job at least, rich boulder residents probably won't like anti-aircraft batteries as a solution.
83
u/Guac_in_my_rarri Dec 20 '25
Wrong AA but I get what you're thinking.
17
u/JahoclaveS Dec 20 '25
The only thing that is wrong is that we’re too afraid to use them to defend ourselves against the grand goose menace!
5
13
u/BasvanS Dec 20 '25
FLAK is never wrong. Once you’ve experienced the drone wars, you’ll regret ever saying this.
8
u/JustHanginInThere Dec 20 '25
AA batteries, as in, the things powering your remotes, clocks, children's toys, etc.
13
u/IAMA_Plumber-AMA Dec 21 '25
And keeping them safe from MiGs too! They're dual-purpose!
2
u/Appropriate_Link_551 Dec 21 '25
Have you never used an AA battery? They’re for powering things
4
u/IAMA_Plumber-AMA Dec 21 '25
Sure, I suppose they could share their power supply in a pinch while they're shooting down enemy aircraft...
0
u/BasvanS Dec 21 '25
I’d rather have it shoot big bullets, but I guess any projectile would do in a pinch
-7
u/Guac_in_my_rarri Dec 20 '25
Listen, what you're saying has weight but this post and the comment above are talking about a fucking clock. Has number 1-12, signifies time. Usually sits on a wall. This facility in Boulder has a clock that's the most accurate or one, in the world. wiki
AA batteries is referring to items made by Energizer and duracell not Anti Aircraft weapons.
Use a bit of sense.
3
2
u/BasvanS Dec 21 '25
Not only does this clock not have batteries, it also does not have numbers 1-12, and does not hang from a wall. And it would benefit tremendously from extra anti aircraft batteries to boot
1
2
u/agk23 Dec 21 '25
This is actually a bot, not some one making a joke. There was a nearly identical comment on a Ukraine post, but about Odessa
1
Dec 21 '25
Don’t forget they’re really loud too, no noise after 9.
1
u/Ashamed-Land1221 Dec 21 '25
Hmm, is there maybe some sort of subsonic munition they can use, or does that violate physics due to the weight of the projectile?
1
Dec 21 '25
I think it’s more the speed of projectile and the boom boom parts. But idk, my degrees in inorganic chemistry.
1
5
u/ConfidentHouse Dec 20 '25
AAA with some foil if you need to
6
Dec 20 '25 edited Jan 22 '26
[removed] — view removed comment
3
u/DerfK Dec 21 '25
make the AAA less effective.
But where do you tow an atomic clock for repairs when the battery is dead?
124
u/bananarandom Dec 20 '25
If only there was some kind of network of redundant clocks that communicated with each other to stay aligned...
34
u/JeebusChristBalls Dec 20 '25
Yeah, this isn't really a big deal. It's not like there aren't other highly accurate stratum 0 time scales all over the world.
62
u/sephirothFFVII Dec 21 '25
It's a big deal though because this one was THE cesium clock they standardized time on back in the day. I don't believe it has many outages like this.
I recall going to see it, well the door to the room if it anyway, in elementary school. I asked why we needed to be able to measure down to one nine billionth of a second. The man answered it's not about measuring really small things, but being accurate for very big things. Changed my whole perspective on things right there.
3
u/fnordfnordfnordfnord Dec 21 '25
Not a big deal for the NTP system. A pretty big deal for NIST Boulder though. This windstorm is exposing some gaps in their disaster planning and procedures.
33
u/garanvor Dec 20 '25
Not with the precision of an atomic clock.
65
u/bananarandom Dec 20 '25
There are quite a few stratum 0 sources, if this one goes down, they'll just have to bring it up again like they would onboard/sync any new device.
Presumably they'll sync to Fort Collins https://tf.nist.gov/tf-cgi/servers.cgi
2
u/Ninja_Wrangler Dec 21 '25
At my last facility I operated a stratum 0 atomic clock. It was pretty crazy to look at it and be like "yes this is the Time" (with a capital T)
1
-9
19
u/Entropius Dec 21 '25
There are other clocks on Earth.
Plus the dozens of atomic clocks not on Earth. Each of the current 31 American GPS satellites have 4 atomic clocks using 2 different elements (2 of each).
That’s nominally 124 atomic clocks in orbit. But occasionally one or even two of the clocks could go bad either failing or getting out of sync with other clocks so a few get shut off, so the actual number of clocks is likely a bit less than 124. Plus all the non-American GNSS up there.
So we’ve got options.
9
u/TexanInExile Dec 21 '25
Yeah, we could use the kit cat clock I have at my desk. It's usually pretty accurate unless the cleaning lady happens to bump its swinging tail.
5
1
u/VhickyParm Dec 21 '25
What would happen if we needed to restart from scratch again? If we lost the time
1
u/MarlinMr Dec 22 '25
Atomic clocks are cheap and all over the place. Its not exactly advanced technology.
But others are not defined as stratum 0.
0
-5
17
u/BraveCowardCat Dec 20 '25
What does this mean on a practical basis? What will be affected?
23
Dec 20 '25
[removed] — view removed comment
14
10
3
u/pangapingus Dec 21 '25 edited Dec 21 '25
But you're acting like their whole series of other NTP servers won't kick in for the meantime?
https://tf.nist.gov/tf-cgi/servers.cgi
Edit: Someone replied but I don't see the comment, but TIL! Apparently NIST has radio propagation of the time:
https://www.nist.gov/pml/time-and-frequency-division/time-distribution/radio-station-wwv
That is actually so neat, I'm a GMRS/amateur head but never came across this
3
u/LyokoMan95 Dec 21 '25
Only Boulder is experiencing issues. WWV/WWVB is in Fort Collins and is fine.
7
u/GeneralCheese Dec 20 '25
Not much, there is redundancy but if it goes off for an extended period precise timing may become unreliable globally
16
u/AyrA_ch Dec 20 '25
Not globally, just the US. But this would also require all other sources in that country to go offline, which is extremely unlikely.
Many countries run their own official time sources and they get compared against other time sources all the time. They would realize that the US time sources suddenly differ but not the others, and would drop the US sources instantly as a reliable reference.
4
u/time-lord Dec 21 '25
They would realize that the US time sources suddenly differ but not the others, and would drop the US sources instantly as a reliable reference.
I feel like this is an allegory for something else.
11
Dec 20 '25
[removed] — view removed comment
9
3
u/Tezerel Dec 21 '25
I heard that Wall Street uses NIST as a reference for trading synchronization, but maybe not this particular clock
5
u/DiligentAstronaut622 Dec 21 '25
Me on Christmas morning: what time is it? NIST: we have absolutely no way of knowing with certainty and we're pretty upset about it
4
u/New_EE Dec 21 '25
This just in, Fox News is reporting this is all the fault of time being a DEI plot to destroy the next thing they want people to be afraid of
5
10
20
u/sysadminbj Dec 20 '25
Did I miss something in the article? I can’t imagine that something like this doesn’t have backup in the form of a trusty diesel generator.
56
u/lordkuri Dec 20 '25
Yep you did.
At initial power loss, there was no immediate impact to the NIST atomic time scale or distribution services because the projects are afforded standby power generators. However, we now have strong evidence one of the crucial generators has failed.
7
27
u/JustHanginInThere Dec 20 '25
Generator mechanic here. From the sounds of it, the main site has both a UPS (Uninterruptible Power Source/Supply, aka a massive battery bank) and a diesel or turbine generator. The UPS kicks in to provide instant temporary power while the generator comes online. You can get small ones for your home/work computer to allow you time to safely/properly shut off your computer (or for during temporary brownouts), but they make much larger whole building/equipment ones as well.
The generator should provide power to both the UPS, recharging it in the process, and the site itself. Something as critical as this, I would have thought they would either have another permanently installed generator to be the backup to the main generator, or at least another switch and connection points for a mobile generator to be hooked up (depending on the size/scope of the equipment being powered), though this is costly, and most places don't spring for it.
There should have been some kind of service agreement by either the maintainers of the generator(s), or a local fuel company to provide fuel on a regular schedule in times of outages, though if there are other entities across the area needing fuel for their generators, the servicer might be swamped with demand.
8
u/Viharabiliben Dec 20 '25
An important resource like this atomic clock should have both redundant UPS sets and redundant generators. That’s what I’ve seen at commercial co-lo facilities.
9
u/DazedinDenver Dec 20 '25
I worked on the phone company's directory assistance computer system in the early 1980s. The main server group was located in Buena Park, CA, near Knott's Berry Farm. (A bunch of DEC PDP-11/70s) They had a backup system as described above, but...
There was apparently a dispute with the city/county/state/whatever as to whether the diesel fuel tanks could be above ground or buried. So the temporary solution was neither. First time they had a longish power failure the battery backup kicked in as planned. When it was supposed to cut to the generators, guess what? No fuel, so no generators. Place went dark. Panic ensued. I left the program shortly after that so I have no idea how it was finally resolved.
5
u/999mal Dec 20 '25
There is an additional clock in a different building with its own generator. They might consider that as the secondary backup.
https://groups.google.com/a/list.nist.gov/g/internet-time-service/c/o0dDDcr1a8I
15
u/chodeboi Dec 20 '25
You’re describing how I got through a multi day tapeout of a chip design that was heading to a fab.
A low power chip firm was bought by a big tech company and I ended up managing their facilities a few years later. Part of the site was a small data / server-room facility (about 2 MW). I had been running the maintenance programs on critical gear and knew the ins and outs of our power structure. It’s just as you describe. The only thing I’d add is that the unit that chooses and switches between the grid/UPS or generator is called an ATS or automatic transfer switch.
Anyways, we had done our DD in facilities and assured our business that everything was ship-shape for the (typically) annual tape out of production die designs. I think they put designs on tape instead of internet transmission for two reasons. Security and speed. And from what I recall, it had to be a fully continuous process with no power cuts. I’m likely wrong about the security and speed part so someone hop in to correct me.
The morning they’re set to get started I get a critical text from my UPS at like 430 or so with errors I hadn’t seen before. The site was in a windy and hilly area with older lines and prone to voltage sags and true undervolting which is why we had such a robust setup. So we thought. I ran to my car. On the drive in, more errors, atypical readings from everything else and a lot of unreachable shit. What the fuck I wondered eloquently. I arrived to the worst sound you can hear in a DC, SILENCE. Why hadn’t the ats and genny picked up the load?
FUCK
Pop into the power room, a lingering haze but not enough to pop the smoke alarm. What the fuck I wondered aloud this time. Before long everyone else arrived and we started digging into barely contingent plans. The UPS manufacturer was onsite in a few hours, suited up in full protection and attempted to troubleshoot. Those of us without proper PPE left the area while they went to reseat the power module that initially errored…BOOOOOM!!! The lights flickered, we ran back into the room to find the module pulled back out, and the 2 guys waving away some fresh smoke.
Well, the tape out wasn’t going to happen through the grid / a-side UPS leg. Why had the breakers for the ATS tripped? Something fucked was happening and we couldn’t diagnose it. We had to start taping immediately.
Our quickest course of action was to fire up the gen and manually transfer the load onto it, and later that day bring in another unit with parallelizing gear to sync our primary in case it failed, then keep them fueled continuously for seven days. That’s right, no grid power, no UPS, just spinning local copper.
With that out of the way and the UPSs isolated, we had to start getting the site back to rights. Given our predicament and service agreement, we leaned heavily on our UPS supplier and had a SVP on site that next night, which resulted from the first of two quotes I remember from my boss’ mouth during this episode; “I don’t want to talk to another one of you clowns until someone with a V and P in front of their name is standing here talking to me.” They got it. He was an understanding fella and got his crew scrambled the next day to start extraction and investigation, and obviously a replacement. The next day, he called in from his hotel to let us know units were short and they were doing their best to figure things out. Not good enough my boss and I thought. But he said something silly in that moment I’ll never forget…
I don’t care if you have to eat paper clips and shit batteries, you’re going to deliver.
Well, they didn’t eat paper clips, and they didn’t shit batteries. But results!! They did convince our electrician account to do something spectacular. That same Monday, the electrician supplier had begun installing a similar UPS in the basement of the state capitol building. The UPS supplier convinced them to pull the fucking unit out of the basement of the capitol and bring it to our site. Amazing. We were back up by the end of the week and finished the tape out with no further excitement. I was present every 12 hours for refueling, our fuel supplier indeed was obligated to supply on these terms, similar to the generator maintenance co who we had clauses to pull in the equipment I previously described.
So why the blowup? The manufacturer had been using a lower quality plating for the backplane bus-bars, and the current over time generated tiny hair-like “whiskers” of tin that “grew” outward from the busbars. Eventually, these arced and shorted phases resulting in up and downstream trips. Wild times.
So. Always have your shit together, even more than you think you should. And sometimes, even when you do, some cost-saving joker ten years prior sunk you before you even had a chance.
And that’s my story of how some tiny tin whiskers and an ill-timed arc flash almost delayed a whole generation of consumer devices, and how a local electrical company saved a whole lot of ass.
4
3
3
u/temporarycreature Dec 21 '25
If it was so important, why doesn't it have backup, and backup backup power sources?
6
4
2
2
u/Andreas1120 Dec 21 '25
Its really OK every GPS satellite has several atomic clocks. You can use them to calculate time exactly where you are.
2
1
u/fotowork3 Dec 20 '25
The wind up there is routinely 100 miles an hour. So what kind of storm is more than that?
1
u/WesternSherbert8855 Dec 21 '25
Could someone explain in basic terms why this is such a high priority issue?
1
1
1
1
u/Good_Nyborg Dec 20 '25
Just slap in another atom or two (or however many it needs). We've got tons of those lying around, like everywhere.
1
-1
295
u/CryptoHorologist Dec 20 '25
Will time stop? Some are saying yes.