Rackspace hosted Exchange suffered a catastrophic failure starting December 2, 2022 and is still continuous since 12:37 AM December fourth. At first described as connectivity and login problems, the guidance was eventually updated to announce that they were dealing with a security incident.
Rackspace Hosted Exchange Issues
The Rackspace system went down in the early morning hours of December 2, 2022. At first there was no word from Rackspace about what the issue was, much less an ETA of when it would be solved.
Clients on Buy Twitter Verification reported that Rackspace was not responding to support e-mails.
This has been quite the day with #Rackspace. Every hosted exchange client has been down for 14 hours or two. Support isn’t reading/responding to tickets. Updates are unhelpful.
I am concerned now that they succumbed to something bad like the ProxyNotShell PoC hack. https://t.co/jchKsAO3Z7
— Joe Sinkwitz (@CygnusSEO) December 2, 2022
A Rackspace client independently messaged me over social media on Friday to relate their experience:
“All hosted Exchange clients down over the previous 16 hours.
Uncertain how many business that is, however it’s substantial.
They’re serving a 554 long delay bounce so individuals emailing in aren’t knowledgeable about the bounce for numerous hours.”
The main Rackspace status page offered a running upgrade of the failure however the preliminary posts had no details besides there was an interruption and it was being examined.
The first authorities upgrade was on December 2nd at 2:49 AM:
“We are examining a concern that is affecting our Hosted Exchange environments. More information will be published as they become available.”
Thirteen minutes later Rackspace started calling it a “connectivity concern.”
“We are examining reports of connection concerns to our Exchange environments.
Users may experience an error upon accessing the Outlook Web App (Webmail) and syncing their email customer(s).”
By 6:36 AM the Rackspace updates described the continuous issue as “connectivity and login issues” then later on that afternoon at 1:54 PM Rackspace revealed they were still in the “investigation phase” of the blackout, still attempting to determine what failed.
And they were still calling it “connectivity and login concerns” in their Cloud Office environments at 4:51 PM that afternoon.
Rackspace Recommends Migrating to Microsoft 365
Four hours later Rackspace referred to the situation as a “significant failure”and began providing their consumers complimentary Microsoft Exchange Plan 1 licenses on Microsoft 365 as a workaround up until they comprehended the issue and might bring the system back online.
The official guidance specified:
“We experienced a significant failure in our Hosted Exchange environment. We proactively shut down the environment to avoid any further issues while we continue work to bring back service. As we continue to work through the root cause of the concern, we have an alternate option that will re-activate your ability to send and get e-mails.
At no cost to you, we will be supplying you access to Microsoft Exchange Plan 1 licenses on Microsoft 365 till additional notice.”
Rackspace Hosted Exchange Security Incident
It was not until nearly 24 hr later on at 1:57 AM on December 3rd that Rackspace formally announced that their hosted Exchange service was struggling with a security incident.
The statement further revealed that the Rackspace professionals had powered down and disconnected the Exchange environment.
“After more analysis, we have actually identified that this is a security occurrence.
The known impact is separated to a portion of our Hosted Exchange platform. We are taking essential actions to assess and protect our environments.”
Twelve hours later that afternoon they updated the status page with more details that their security group and outside specialists were still dealing with fixing the interruption.
Was Rackspace Service Impacted by a Vulnerability?
Rackspace has not launched information of the security event.
A security event normally includes a vulnerability and there are two extreme vulnerabilities currently in the wile that were covered in November 2022.
These are the 2 most current vulnerabilities:
Microsoft Exchange Server Server-Side Demand Forgery (SSRF) Vulnerability
A Server Side Demand Forgery (SSRF) attack enables a hacker to check out and alter data on the server.
Microsoft Exchange Server Remote Code Execution Vulnerability
A Remote Code Execution Vulnerability is one in which an assailant is able to run destructive code on a server.
An advisory released in October 2022 explained the impact of the vulnerabilities:
“A verified remote assaulter can perform SSRF attacks to escalate opportunities and carry out arbtirary PowerShell code on susceptible Microsoft Exchange servers.
As the attack is targeted against Microsoft Exchange Mailbox server, the aggressor can possibly get to other resources via lateral movement into Exchange and Active Directory environments.”
The Rackspace failure updates have actually not shown what the specific problem was, just that it was a security occurrence.
The most current status update as of December fourth stated that the service is still down and customers are encouraged to move to the Microsoft 365 service.
Rackspace published the following on December 4, 2022 at 12:37 AM:
“We continue to make development in resolving the event. The availability of your service and security of your information is of high significance.
We have committed extensive internal resources and engaged first-rate external knowledge in our efforts to minimize unfavorable effects to customers.”
It’s possible that the above noted vulnerabilities belong to the security occurrence impacting the Rackspace Hosted Exchange service.
There has been no announcement of whether client information has been compromised. This event is still continuous.
Included image by Best SMM Panel/Orn Rin