Total blackout in Argentina and neighboring countries... a warning?

[fabric]

I was also wondering about what could cause such a large area to be affected. They're saying that the failure started in the distribution line between two major hydroelectric plants, but it seems rather strange that one failure would make the whole system crash.

When I heard the news I was immediately reminded of the 2003 blackout in northeastern US/Canada. In that case it was one failure that led to a cascade which collapsed the power grid. Although these systems are designed with that in mind (ie, to prevent cascade failures), aging infrastructure, buggy software etc can lead to these kinds of outcomes as it did in Canada. I’m curious as to what the investigation will reveal in the case of Argentina but in terms of scale it was almost the same number of people affected.

Findings

In February 2004, the U.S.-Canada Power System Outage Task Force released their final report, placing the causes of the blackout into four groups:[8]

1. FirstEnergy (FE) and its reliability council "failed to assess and understand the inadequacies of FE's system, particularly with respect to voltage instability and the vulnerability of the Cleveland-Akron area, and FE did not operate its system with appropriate voltage criteria."
2. FirstEnergy "did not recognize or understand the deteriorating condition of its system."
3. FirstEnergy "failed to manage adequately tree growth in its transmission rights-of-way."
4. Finally, the "failure of the interconnected grid's reliability organizations to provide effective real-time diagnostic support."

The report states that a generating plant in Eastlake, Ohio, a suburb northeast of Cleveland, went offline amid high electrical demand, putting a strain on high-voltage power lines (located in Walton Hills, Ohio, a southeast suburb of Cleveland) which later went out of service when they came in contact with "overgrown trees". This trip caused load to transfer to other transmission lines, which were not able to bear the load, tripping their breakers. Once these multiple trips occurred, multiple generators suddenly lost parts of their loads, so they accelerated out of phase with the grid at different rates, and tripped out to prevent damage. The cascading effect that resulted ultimately forced the shutdown of more than 100 power plants.

Computer failure

A software bug known as a race condition[vague]existed in General Electric Energy's Unix-based XA/21 energy management system. Once triggered, the bug stalled FirstEnergy's control room alarm system for over an hour. System operators were unaware of the malfunction. The failure deprived them of both audio and visual alerts for important changes in system state.[9][10]

Unprocessed events queued up after the alarm system failure and the primary server failed within 30 minutes. Then all applications (including the stalled alarm system) were automatically transferred to the backup server, which itself failed at 14:54. The server failures slowed the screen refresh rate of the operators' computer consoles from 1–3 seconds to 59 seconds per screen. The lack of alarms led operators to dismiss a call from American Electric Power about the tripping and reclosure of a 345 kV shared line in northeast Ohio. But by 15:42, after the control room itself lost power, control room operators informed technical support (who were already troubleshooting the issue) of the alarm system problem.[11]

I was in Toronto at the time and it was quite something. We did have cases of ‘good Samaritans’ stopping what they were doing to manage traffic intersections. No payment systems worked, although I always carried cash with me so I was able to buy things but many could not (I was in the middle of buying water and cigarettes when it happened). And cash registers are often designed with an override to manually open if there is no power). If you were out of gas, you were out of luck. Tons of people stuck in the subway, streetcars frozen, no access to bank machines, dark streets, etc - the list is almost endless. Absolutely everything relies on electricity, including your water and natural gas (pumps and supply stations need to get it into the city). In a way normalcy bias does have its use I guess – since people managed to carry on and not have things descend into complete chaos. Although had it been a week without power? Who knows. I doubt even Canadians would still be polite.

In any case it was a wakeup call. It’s one of those things that we take for granted but once it’s gone, we realize how integral it is to society. And with society’s addiction to technology, having that suddenly taken away is essentially like quitting heroin cold turkey. The first day you can tough it through but it’s unlikely you’ll make a whole week without losing your sanity.

Since I lived alone it was easy to keep cash on hand and a week’s worth of water and food (although they say try to have 2 weeks or more if possible). That’s a bare minimum. There’s also lots more one can do but it takes effort and work. Preparedness is not just about having stuff ‘just in case’ it also means maintenance, and attitude. Food needs to rotated, supplies checked etc, although something is better than nothing if/when one does find themselves in a disaster situation. The article Gary referenced above is a good one to keep in mind.

fwiw

 

[msante]

Well, I'm surprised the shop was still open. Bank/cash machines won't work without power (so you can't get cash), the payment systems in shops won't work without power, even cash registers/tills won't work in most shops without power. Did you try buying anything at the time?

The supermarket had an emergency power generator; it is common here for banks, supermarkets, hospitals, etc. to have such equipment to operate during a blackout for several hours. However, when I left for the supermarket I took enough cash because my first thought was "no communications -> no online payments"... but to my surprise the supermarket had its card payment system working. Anyway it is necessary to clarify that the online payment systems here (in Argentina) still work partially connected through the wired telephone lines (the old copper pair). During the event, even though the cell phone networks almost completely collapsed, traditional telephony continued to work, in fact I was able to communicate with my mom through that medium.

 

[Chu]

Yes Chu, I agree, I was thinking in the same line, I realize that so far I have not thought seriously about this whole thing; I have kept myself informed but I have been procrastinating about doing something concrete to prepare myself. I will take this particular event as a personal warning,... although I also think it might be taken like a warning for others in my situation.

Based on something like what Gary shared, you can make a list, and work every day or every week towards preparing. Only little thing at a time is better than nothing. I think it's important to act, but without panicking. Just taking precautions. It's part of applying knowledge.

 

[msante]

From Sputnik in Spanish (Qué hubo detrás del apagón histórico que dejó a oscuras a Argentina y Uruguay):

Translation:

"Usually the lightning strikes the lines and that causes something called 'recierre', that is, the protections take those lines out of service for milliseconds and reconnect them so they don't have a fault," he explained.

According to the expert, it is likely that on Sunday one of the lines had a fault and could not be reconnected after reclosing.

The fact that the fault became generalized instantaneously had to do with a series of processes that were unleashed afterwards, based on a mechanism designed to continuously ensure the balance between the power generated and the 'load' consumed by the users, always with a frequency of 50 Hertz.

According to Di Lavello, the initial failure caused a 'load shot' that decreased the amount of energy consumed. As the load decreased, the generation became greater and, due to the automated mechanisms to keep the system stable, the frequency began to increase.

"The frequency must have gone up to such an extent that it activated the protection of the generators that takes them out of service if the frequency is too high so that they do not run the risk of breaking", he explained, in relation to what is known as 'Automatic Generation Trigger' (DAG).

The disconnection of some generators abruptly decreased the generation. The load was kept at the same level, so another of the automatic mechanisms, the 'Frequency Automatic Trip' (DAF), was activated, which decreases the load to stabilize again at 50 Hz.

The two consecutive 'shots' in the system, noted the engineer, caused oscillations that, in turn, activated the protections of the high voltage lines. Thus, generators that had not been disconnected at first were disconnected at this stage.

Finally, low-frequency operation, with very little generation and high load "caused the two systems to collapse".

It seems that there is quite consensus about how to explain what happened after "the event" and why collapsed all the system; what remain a mistery yet is what was the initial cause of the failure. I say that natural causes should not be discarded...

Although these systems are designed with that in mind (ie, to prevent cascade failures), aging infrastructure, buggy software etc

In fact buggy system is a possibility, even more so when these software systems have to face conditions that they have never faced before and for which they were not prepared,... in other words, very unusual conditions.

 

[Wandering Star]

There is a cover of the magazine "The Economist" of the year 2017, with the title "The World If".

In the upper left corner is drawn a magnet in a space nebula, releasing energy to a tower of high electrical tension rendering it useless.

Everything points to that there is "technology" behind these latest and spectacular electric blackouts, but ... What if something else is happening?

 

[whitecoast]

There is a cover of the magazine "The Economist" of the year 2017, with the title "The World If".

In the upper left corner is drawn a magnet in a space nebula, releasing energy to a tower of high electrical tension rendering it useless.

Everything points to that there is "technology" behind these latest and spectacular electric blackouts, but ... What if something else is happening?

There was a recent article on SOTT (Stephen Cohen's 2017 Prophecy About The Nuclear Threat of Russiagate is Coming True -- Sott.net) that had this also:

The New York Times has published an anonymously sourced report titled "U.S. Escalates Online Attacks on Russia's Power Grid" about the "placement of potentially crippling malware inside the Russian system at a depth and with an aggressiveness that had never been tried before" which could potentially "plunge Russia into darkness or cripple its military," with one anonymous official reporting that "We are doing things at a scale that we never contemplated a few years ago."

I’m not saying it’s related, but it makes me wonder.