Connectivity issues across Hasura Cloud

Incident Report for Hasura

Postmortem

On June 21st, between the hours of 06:27 UTC and 07:42 UTC, Cloudflare experienced a major service disruption across 19 of its datacenters. During this time, depending on your location, you may have been unable to access Hasura services (Dashboard, APIs, Console, etc.) including your project’s GraphQL API.

In summary, a configuration change being deployed by Cloudflare engineers to their Multi-Colo PoP (MCP) locations, resulted in a re-ordering of terms which caused Cloudflare to withdraw a critical subset of prefixes (Where prefixes are a collection of adjacent IP addresses which are advertised to peers on the global network). Due to this withdrawal, our service which is routed via Cloudflare’s network, unfortunately experienced a period of inaccessibility for a large portion of our user base.

Even though the 19 datacenters which experienced the issue account for 4% of Cloudflare’s total network, these datacenters handle a total of about 50% of the total traffic that flows via CF’s backbone.

A more detailed blog has been published by Cloudflare, which details the issue and the sequence of events. Please see the blog here: https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/

Posted Jun 22, 2022 - 13:00 UTC

Resolved

Traffic is back to normal on our side. If you're still facing issues, reach out to support@hasura.io

See more details about the Cloudflare incident at https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj

Posted Jun 21, 2022 - 07:45 UTC

Update

Most services and regions have recovered following Cloudflare's fix. We're continuing to monitor the situation.

Posted Jun 21, 2022 - 07:34 UTC

Monitoring

Cloudflare indicates that a fix has been implemented. We're monitoring the situation.

More details at https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj

Posted Jun 21, 2022 - 07:24 UTC

Update

We are still monitoring the situation an assessing impact. As of now the Cloudflare incident is impacting most of our services.

Posted Jun 21, 2022 - 07:09 UTC

Investigating

We are investigating connectivity issues with Hasura Cloud Dashboard, APIs and certain projects due to an incident with Cloudflare, who is our DNS and Network Security provider.

More details about the Cloudflare incident at https://www.cloudflarestatus.com/

Posted Jun 21, 2022 - 06:51 UTC

This incident affected: Hasura Cloud Dashboard and APIs, Hasura Console, Hasura EE (Hybrid Setup) (Self-hosted EE(Hybrid Setup) - Monitoring), GCP: Metrics & Reporting (GCP: Iowa (us-central1), GCP: Frankfurt (europe-west3), GCP: Los Angeles (us-west2)), AWS: Metrics & Reporting (AWS: London (eu-west-2), AWS: Sydney (ap-southeast-2), AWS: Tokyo (ap-northeast-1), AWS: Frankfurt (eu-central-1), AWS: N. Virginia (us-east-1), AWS: Mumbai (ap-south-1), AWS: Oregon (us-west-2), AWS: Singapore (ap-southeast-1), AWS: Ireland (eu-west-1), AWS: Central (ca-central-1), AWS: N. California (us-west-1), AWS: Ohio (us-east-2)), Hasura Cloud AWS Regions (AWS: Tokyo (ap-northeast-1), AWS: N. Virginia (us-east-1), AWS: London (eu-west-2), AWS: Frankfurt (eu-central-1), AWS: Oregon (us-west-2), AWS: Sydney (ap-southeast-2), AWS: Singapore (ap-southeast-1), AWS: N. California (us-west-1), AWS: Mumbai (ap-south-1), AWS: Ohio (us-east-2), AWS: Ireland (eu-west-1), AWS: Central (ca-central-1)), and Hasura Cloud GCP Regions (GCP: Los Angeles (us-west2), GCP: Iowa (us-central1), GCP: Frankfurt (europe-west3)).