Рукописи не горят!

"Many thanks for all the empathy messages you sent us today!" tweeted OVHcloud founder Octave Klaba. "I want to thank the teams who have been working all night/day. It's been the worst day for the last 22y and there is no word strong enough to say how sorry I feel today. We keep working hard to restart SBG1 / 3/4 asap!"

OVHcloud customers with servers in the Strasbourg facility will be offered replacement servers elsewhere in France: "We have a stock of new servers at the Roubaix and Gravelines sites, ready to be delivered to the majority of affected customers," says the most recent OVHcloud statement. "We will further enhance availability in these datacentres, with the production of nearly 10,000 new servers in the coming weeks. Affected customers will be notified about this process as soon as possible."

The site's fibers have been checked and were not affected, so it will be possible to get the facilities back online once the network equipment is patched and any repairs made.

Overnight, new network equipment was sent to Strasbourg to set up a temporary network room. The worklist includes rebuilding a network room referred to as "network room B (SBG5).".

To restart the Strasbourg site, OVHcloud's plans for the next two weeks include rebuilding the 240V circuits for SBG1/SBG4, a process expected to be done by Monday March 15, and rebuilding the 20kV power supply at SBG3 which should be done by March 19.

Although SBG1 lost four rooms, the network room survived, which means it can come back online with SBG4.

During the initial disturbance 3.6 million websites, on 464,000 distinct domains were taken offline by the fire, according to network monitor site Netcraft.

Websites that went offline included government websites such as the Polish Financial Ombudsman; the Ivorian DGE; the French government procurement site Plateforme des achats de l’Etat; the Welsh Government’s Export Hub; and the UK Government’s Vehicle Certification Agency website. The latter site moved swiftly, getting a new SSL certificate by 10am and shifting to a UK hosting company.

The causes are still unknown, but The Register points out that SBG1, SBG2, and SBG4 had a three-and-a-half hour power outage in 2017, prompting a major upgrade to the power system on the site, and quotes a Gartner analyst speculating that an UPS failure may be the root cause.

Other Internet sites have scoured the OBHcloud update pages and noticed that OVHcloud has been replacing power supply cables at other data centers. "We are going to intervene in the bay in order to replace a set of power cable that may have an insulation fault," says an update for the Beauharmois data center (BHS-1). A similar set of cables were replaced at Limburg (LIM-1) yesterday, and similar updates have been spotted at other data centers back to December.

With an incident like this, an official investigation has to take place, and any connection with these maintenance issues can't be assumed. If nothing else, one might expect OVHcloud to prioritize any and all maintenance work with any connection to fire prevention before the full cause is known.

OVHcloud is planning to float on the stock market with an IPO; no change has been announced to this, given that it won't be taking place until the end of this year at the soonest.

Update: OVHcloud has published a FAQ page for customers needing to understand the effects of the fire on their service

For more breaking data center news, features, and opinions, be sure to subscribe to DCD's newsletter