Cloud hosting infrastructure outage

Incident Report for W3C

Resolved

Our tests show that performances of W3C services are back to normal.

Posted Jun 28, 2021 - 10:49 UTC

Monitoring

Our cloud provider reported that the issue affecting their storage backend was resolved and that disk performance should be back to normal.

Posted Jun 26, 2021 - 13:35 UTC

Update

Additional update from CSAIL's Infrastructure Group:
"Update on storage issues" - https://lists.csail.mit.edu/pipermail/openstack-users/2021-June/000968.html

Posted Jun 23, 2021 - 00:09 UTC

Update

Unfortunately this cloud issue is still ongoing and has even aggravated over the weekend, which made the W3C website unavailable on several occasions.
Please also note that this is the cause of the delay in serving IRC logs and minutes.

More details in CSAIL's Infrastructure Group message:
"Continued storage issues and degraded OpenStack availability" - https://lists.csail.mit.edu/pipermail/openstack-users/2021-June/000967.html

Posted Jun 21, 2021 - 09:00 UTC

Update

We put some mitigation in place to make the Blogs and Community/Business Groups pages more responsive.

Posted Jun 03, 2021 - 14:50 UTC

Identified

A number of services continue to time out or be slow due to an ongoing outage on the storage infrastructure of our cloud provider. Blogs, Community/Business Groups, mail, mailing list archives, CVS, Calendar, IRC are impacted.
Our provider is working on the issue and plan for the situation to continue at least through to next week. Our apologies for the inconvenience.

Posted Jun 01, 2021 - 16:55 UTC

This incident affected: Website, User & Group Management, TR Publication, Email & Mailing Lists, IRC, API, Validation Services, and Chapters.