A major Rogers outage has cut off 25 percent of Canada’s internet traffic

loreta

The Living Force
FOTCM Member
Very strange, don't you think?



"The outage has also cut off a crucial Canadian banking network used for debit cards and ATMs

Canadian telecom Rogers is suffering a major outage affecting landline phones, cellular connections, and internet connectivity throughout Canada that started early this morning. Downdetector listed thousands of reports for the issues that flooded in as people started to get up around 5AM ET and couldn’t get online.

Rogers first addressed the outage in a tweet from its official support account just before 9AM ET and then went silent for a couple of hours. On Friday afternoon, the company tweeted that its technical teams are working to restore services “alongside our global technology partners, and are making progress.” There is still no ETA for restoration, even after services have been available for about 12 hours nationwide. In a follow-up tweet, the company promised that it would be “proactively crediting all customers,” and said that it had “every technical resource and partner” working on getting its network back up.

Cloudflare Radar shows communication between its load-sharing network and Rogers in Canada exhibiting its normal nightly pattern before dropping to essentially zero at around 3–4AM ET and registering a flat line ever since. At that same time, there was a massive spike in BGP (Border Gateway Protocol) announcements for the network that indicate changes in routing.

In a blog post published Friday afternoon, Cloudflare personnel speculated that the outage is the result of an internal error at Rogers, as opposed to a cyberattack or other cause. They cited that pattern of BGP announcements, noting that with the Rogers network now failing to announce its presence, the rest of the internet can’t find it. BGP is a fundamental part of the technology that helps move information from one place on the internet to another, and a problem with BGP routing information was the cause of a massive Facebook network outage last fall.

The outage has also disconnected Interac, the network Canadian banks use, disabling debit cards, ATMs, and the e-Transfer services that recently notched over 1 billion transactions in one year.

Passport offices and Canada’s tax-collecting Revenue Agency are among the government services that are unavailable today due to the outage. Both agencies also warned users the outage is cutting off multifactor authentication codes sent by voice or text message, so people who are logged out may not be able to log in at this time.

Even the Canadian Radio-television and Telecommunications Commission (CRTC), which regulates broadcasting and telecoms, says its phones aren’t working due to the outage.

Internet network disruption tracker NetBlocks shared real-time data showing Canada’s national connectivity has dropped to just 75 percent of its normal levels.

What is BGP, and what role did it play in Facebook’s massive outage




A report by CBC News noted Rogers subsidiary Fido is having problems, too. The article says problems have affected card payment processing and ATMs across Canada. A CBC radio station in Kitchener, Ontario, is also offline, and some telephone services for Ottawa’s transit agency have been knocked out. Pictures from Toronto show people crammed into coffee shops or heading to the library to access Wi-Fi now that their other options are offline.

TekSavvy customer service has told customers the outage extends to “Ontario, Quebec and the Eastern Provinces,” with no ETA for resolution, while the CBC reports that Thunder Bay operator Tbaytel says the Rogers outage is nationwide.

Toronto police tweeted early in the morning after some people in the city experienced connection issues calling 911. “The Rogers Network is experiencing some technical difficulties,” said the Toronto Police Operations Twitter account. “We are working to resolve these issues.” In a follow-up tweet, the police confirmed their 911 center is operational and advised anyone who has to call to stay on the line if they connect and to retry calls if they don’t.



Source
 
The "practice runs," whether deliberate or cosmic, are getting more frequent, aren't they?
On the bright side, it's great to see people not using their phones for once...
But it is a very stressful situation for many people. I can imagine myself not be able to receive money from the bank and not have cash at home. No phone it is good, but no bank it is horrible.
 
But it is a very stressful situation for many people. I can imagine myself not be able to receive money from the bank and not have cash at home. No phone it is good, but no bank it is horrible.
Yes, indeed, very stressful!

I experienced this Rogers outage and it really was a complete mess affecting so many services. Although Rogers claimed service was restored to nearly all customers in 24 hours, this simply wasn't true and many people, including myself, had to wait days to get service restored. Customers were beyond angry at Rogers lack of communication and wait times to speak to any tech for help were minimum 3 hours, if you were lucky. A lot of people also aren't satisfied with the explanation for the outage, and why it took so long for the company to communicate anything at all about it to their customers. All in all, it was total incompetence at the very least by a company that has its tentacles in so many services necessary to keep society functioning.

The silver lining to the whole thing was that many people who don't ever use cash and rely on internet for everything suddenly saw how very vulnerable this system is. I know a few people who prided themselves on never carrying cash that have completely changed their tune!
 
There is a wikipedia page about this that gives as good a summery as we're likely to get. It says about 25% of Canada lost internet connectivity and that all Interac services were taken offline. Rogers network routes all Interac transactions (in Canada), so when it goes down, so does your debit. Although:

On July 11, 2022 Interac stated they were adding another provider in addition to Rogers to strengthen their network redundancy. [9]

Back to the outage, this is what happened, officially I guess:

Causes

A report by Cloudflare suggested that the outage was due to internal, rather than external, causes. It identified spikes in BGP [Border Gateway Protocol] updates, as well as withdrawals of IP prefixes, noting that Rogers was not advertising its presence, causing other networks to not find the Rogers network.[1] As of the day after the outage, the cause remained unknown. Public Safety Canada stated that it was not a cyberattack.[20] [Because if it was, why not blame Russia? In fact, why didn't they blame Russia?] The outage was later said to be caused by a maintenance upgrade that caused routers to malfunction.[21] A similar outage occurred in April 2021 which was attributed to a software update.[4]

Postmortem

In a letter to the CRTC, Rogers stated that the deletion of a routing filter on its distribution routers caused all possible routes to the internet to pass through the routers, exceeding the capacity of the routers on its core network. The deletion occurred during the sixth phase in a seven phase update to its core IP network. The failure occurred at 4:45 a.m. and resulted in a nationwide loss of access to the company's phone and internet services, as well as internal access to systems such as the company's VPN to its core network nodes, hampering the ability of the company's employees to mobilize a team and identify the issue. Some employees were able to connect through “emergency SIMs” on alternate networks, which they had due to a practice established through reciprocal agreements made in 2015. Others traveled to centralized locations to establish network access.

At 6 a.m., the company's CTO Jorge Fernandes, who has since been replaced [by Ron McKenzie- former president of Rogers for Business], contacted leadership at rival companies Bell and Telus, advising them of the issues they were facing, as well as warning "to watch-out for possible cyber-attacks." Both competitors offered assistance to Rogers, though the company determined that it was not possible to switch its users over to alternate networks, citing the inability to access its user database, as well as the excess load that the sudden addition of more than 10 million users would put on the other networks. A Memorandum of Understanding on how the carriers can work together in the future will be submitted in September to the Minister of Innovation, Science and Economic Development. Rogers plans to physically separate its wireless and wireline core networks to help mitigate the fallout of future outages.[22]
 
Back
Top Bottom