I've read about how Elon Musk shut down one of Twitter's data centers in Sacramento. A display of narcissistic behavior, impulsiveness, and very little afterthought. Interesting read, especially when paying attention to his behavior.
The Batshit Crazy Story Of The Day Elon Musk Decided To Personally Rip Servers Out Of A Sacramento Data Center
Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of
shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees were quoted about how much work it would be to do that cleanly, noting that there’s a ton of stuff hardcoded in Twitter code referring to that data center (hold that thought).
That same day, Elon tweeted out that he had “disconnected one of the more sensitive server racks.”
CNBC is running an excerpt from the new Walter Isaacson book about Elon that details what happened with the closing of the data center, and it is way, way, way crazier than even I expected. When Musk talked about how he “disconnected one of the more sensitive server racks,” he meant that entirely literally, in that he literally unplugged it, involving a series of improbable (and ridiculously dangerous and stupid) decisions that resulted with him under the floorboards in the data center pulling the plug, after multiple people warned him not to.
But, let’s take a step back. As the book details, Musk wanted to shut down the data center because he was in drastic “stop paying bills” mania at the time, and the Sacramento data center was costing the company $100 million/year. Also, apparently, the data center (which appears to be run by NTT) had told a Twitter employee that it did not think Twitter would be financially viable for very long.
The article starts out with a vignette that basically says everything you need to know about Musk. He had asked an infrastructure manager about moving the servers in Sacramento to one of the other two US data centers Twitter had, in Portland, Oregon. Then this happened:
Another manager at the meeting said that couldn’t be done right away. “We can’t get out safely before six to nine months,” she said in a matter-of-fact tone. “Sacramento still needs to be around to serve traffic.”
Over the years, Musk had been faced many times with a choice between what he thought was necessary and what others told him was possible. The result was almost always the same. He paused in silence for a few moments, then announced, “You have 90 days to do it. If you can’t make that work, your resignation is accepted.”
The manager began to explain in detail some of the obstacles to relocating the servers to Portland. “It has different rack densities, different power densities,” she said. “So the rooms need to be upgraded.” She started to give a lot more details, but after a minute, Musk interrupted.
“This is making my brain hurt,” he said.
“I’m sorry, that was not my intention,” she replied in a measured monotone.
“Do you know the head-explosion emoji?” he asked her. “That’s what my head feels like right now. What a pile of f—ing bulls—. Jesus H f—ing Christ. Portland obviously has tons of room. It’s trivial to move servers one place to another.”
If it’s making your brain hurt to have someone explain to you some fairly basic issues about infrastructure, the problem may be with you, dude. And, yes, it may have been “trivial” to move servers around the last time Musk ran an internet company over two decades ago, but for a major service relied on by hundreds of millions of people, that also has a ton of sensitive data, it’s… not trivial at all.
But a cousin of Musk suggested to Musk that they just do it themselves, while they were flying from the Bay Area to Austin, and Musk literally had his plane diverted to go to Sacramento and try it out, leading to… whatever the -flick- this is:
They were somewhere over Las Vegas when James made his suggestion that they could move them now. It was the type of impulsive, impractical, surge-into-the-breach idea that Musk loved. It was already late evening, but he told his pilot to divert, and they made a loop back up to Sacramento.
The only rental car they could find when they landed was a Toyota Corolla. They were not sure how they would even get inside the data center at night, but one very surprised X staffer, a guy named Alex from Uzbekistan, was still there. He merrily let them in and showed them around.
The facility, which housed rooms of servers for many other companies as well, was very secure, with a retinal scan required for entry into each of the vaults. Alex the Uzbek was able to get them into the X vault, which contained about 5,200 refrigerator-size racks of 30 computers each.
“These things do not look that hard to move,” Elon announced. It was a reality-distorting assertion, since each rack weighed about 2,500 pounds and was eight feet tall.
“You’ll have to hire a contractor to lift the floor panels,” Alex said. “They need to be lifted with suction cups.” Another set of contractors, he said, would then have to go underneath the floor panels and disconnect the electric cables and seismic rods.
Musk turned to his security guard and asked to borrow his pocket knife. Using it, he was able to lift one of the air vents in the floor, which allowed him to pry open the floor panels. He then crawled under the server floor himself, used the knife to jimmy open an electrical cabinet, pulled the server plugs, and waited to see what happened. Nothing exploded. The server was ready to be moved.
The story gets dumber. Musk had a Tesla employee buy Apple AirTags to “track” the servers, and then the process of “moving them” should make everyone cringe:
Other workers at the facility watched with a mix of amazement and horror. Musk and his renegade team were rolling servers out without putting them in crates or swaddling them in protective material, then using store-bought straps to secure them in the truck. “I’ve never loaded a semi before,” James admitted. Ross called it “terrifying.” It was like cleaning out a closet, “but the stuff in it is totally critical.”
At 3 p.m., after they had gotten four servers onto the truck, word of the caper reached the top executives at NTT, the company that owned and managed the data center. They issued orders that Musk’s team halt. Musk had the mix of glee and anger that often accompanied one of his manic surges. He called the CEO of the storage division, who told him it was impossible to move server racks without a bevy of experts. “Bulls—,” Musk explained. “We have already loaded four onto the semi.”
The CEO then told him that some of the floors could not handle more than 500 pounds of pressure, so rolling a 2,000-pound server would cause damage. Musk replied that the servers had four wheels, so the pressure at any one point was only 500 pounds. “The dude is not very good at math,” Musk told the musketeers.
Note the pattern: a willingness to ignore the details of what could go wrong, YOLO it and just test it out, and the assumption that if nothing goes wrong when you do that, it means that everything is fine and nothing else could possibly go wrong.
I might never even ride in a Tesla after this.
And then this:
The moving contractors that NTT wanted them to use charged $200 an hour. So James went on Yelp and found a company named Extra Care Movers that would do the work at one-tenth the cost. The motley company pushed the ideal of scrappiness to its outer limits. The owner had lived on the streets for a while, then had a kid, and he was trying to turn his life around. He didn’t have a bank account, so James ended up using PayPal to pay him.
The second day, the crew wanted cash, so James went to a bank and withdrew $13,000 from his personal account. Two of the crew members had no identification, which made it hard for them to sign into the facility. But they made up for it in hustle. “You get a dollar tip for every additional server we move,” James announced at one point. From then on, when they got a new one on a truck, the workers would ask how many they were up to.
Remember, these are servers full of information, some of it sensitive, and Musk is basically hiring literally undocumented workers off the street and tipping them a dollar for each rack they can move.
And, if you think anyone at Twitter cares about the privacy of your data, think again:
The servers had user data on them, and James did not initially realize that, for privacy reasons, they were supposed to be wiped clean before being moved. “By the time we learned this, the servers had already been unplugged and rolled out, so there was no way we would roll them back, plug them in, and then wipe them,” he says. Plus, the wiping software wasn’t working. “F—, what do we do?” he asked. Elon recommended that they lock the trucks and track them.
So James sent someone to Home Depot to buy big padlocks, and they sent the combination codes on a spreadsheet to Portland so the trucks could be opened there. “I can’t believe it worked,” James says. “They all made it to Portland safely.”
This is… bad. Really bad. Like this could have been a massive disaster for an awful lot of people. This is the kind of thing that the FTC
should go after Musk for. He was playing fast and loose with data that could have created very serious problems. And we have no way of knowing that the data was actually safe, despite the “big padlocks” from Home Depot.
Again, because “it worked,” some people will argue that Elon was right to do it this way. But just because your crazily stupid move didn’t create immediate chaos doesn’t mean that it’s the right move.
And, of course, it didn’t really “work.” As we detailed,
Twitter toppled over a few days later, and this excerpt admits it was because of the “server move.” The article does note that Musk himself eventually said he shouldn’t have done this and it did cause a fair bit of problems for the site, including the disastrous “Twitter Spaces” where Ron DeSantis tried to launch his Presidential campaign.
“In retrospect, the whole Sacramento shutdown was a mistake,” Musk would admit in March 2023. “I was told we had redundancy across our data centers. What I wasn’t told was that we had 70,000 hard-coded references to Sacramento. And there’s still shit that’s broken because of it.”
Yeah, I mean, maybe if you didn’t tell the person trying to explain stuff to you that it hurt your brain to hear the details, you would have heard them telling you things like this.
Even dumber is that the “lesson” that Walter Isaacson seems to take from this little episode is not that Musk’s impulsivity is a dangerous, out of control wrecking ball that is going to destroy some serious stuff, but that (ex)Twitter’s employees had to learn how to better “manage” the man-baby in charge:
His most valuable lieutenants at Tesla and SpaceX had learned ways to deflect his bad ideas and drip-feed him unwelcome information, but the legacy employees at X didn’t know how to handle him. That said, X survived. And the Sacramento caper showed X employees that he was serious when he spoke about the need for a maniacal sense of urgency.
There’s something to be said for pushing back on needless rules and bureaucracy, but it helps if you actually understand stuff before doing so, rather than doing something like this that had half a dozen ways it could have ended in serious disaster and possible tragedy. The fact that it “only” resulted in Twitter falling over every few weeks for months likely means that Musk and his supporters got the very wrong lesson out of this.
But the lesson I would take from it: have nothing to do with Elon Musk or any product he sells. It’s bound to be stupidly, unnecessarily risky.
Back on Christmas Eve of last year there were some reports that Elon Musk was in the process of shutting down Twitter’s Sacramento data center. In that article, a number of ex-Twitter employees wer…
www.techdirt.com
www.indiatoday.in
www.middleeasteye.net
thenationalpulse.com
