Connectivity issues across Hasura Cloud
Incident Report for Hasura
Postmortem

On June 21st, between the hours of 06:27 UTC and 07:42 UTC, Cloudflare experienced a major service disruption across 19 of its datacenters. During this time, depending on your location, you may have been unable to access Hasura services (Dashboard, APIs, Console, etc.) including your project’s GraphQL API.

In summary, a configuration change being deployed by Cloudflare engineers to their Multi-Colo PoP (MCP) locations, resulted in a re-ordering of terms which caused Cloudflare to withdraw a critical subset of prefixes (Where prefixes are a collection of adjacent IP addresses which are advertised to peers on the global network). Due to this withdrawal, our service which is routed via Cloudflare’s network, unfortunately experienced a period of inaccessibility for a large portion of our user base.

Even though the 19 datacenters which experienced the issue account for 4% of Cloudflare’s total network, these datacenters handle a total of about 50% of the total traffic that flows via CF’s backbone.

A more detailed blog has been published by Cloudflare, which details the issue and the sequence of events. Please see the blog here: https://blog.cloudflare.com/cloudflare-outage-on-june-21-2022/

Posted Jun 22, 2022 - 13:00 UTC

Resolved
Traffic is back to normal on our side. If you're still facing issues, reach out to support@hasura.io

See more details about the Cloudflare incident at https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj
Posted Jun 21, 2022 - 07:45 UTC
Update
Most services and regions have recovered following Cloudflare's fix. We're continuing to monitor the situation.
Posted Jun 21, 2022 - 07:34 UTC
Monitoring
Cloudflare indicates that a fix has been implemented. We're monitoring the situation.

More details at https://www.cloudflarestatus.com/incidents/xvs51y9qs9dj
Posted Jun 21, 2022 - 07:24 UTC
Update
We are still monitoring the situation an assessing impact. As of now the Cloudflare incident is impacting most of our services.
Posted Jun 21, 2022 - 07:09 UTC
Investigating
We are investigating connectivity issues with Hasura Cloud Dashboard, APIs and certain projects due to an incident with Cloudflare, who is our DNS and Network Security provider.

More details about the Cloudflare incident at https://www.cloudflarestatus.com/
Posted Jun 21, 2022 - 06:51 UTC
This incident affected: Hasura Cloud Dashboard and APIs, Hasura Console, Hasura Cloud GCP Regions (GCP Cloud: Free tier, GCP: Los Angeles (us-west2), GCP: Iowa (us-central1), GCP: Frankfurt (europe-west3)), GCP: Metrics & Reporting (GCP: Iowa (us-central1), GCP: Frankfurt (europe-west3), GCP: Los Angeles (us-west2)), Hasura Cloud AWS Regions (AWS Cloud: Free tier, AWS: Tokyo (ap-northeast-1), AWS: N. Virginia (us-east-1), AWS: London (eu-west-2), AWS: Frankfurt (eu-central-1), AWS: Oregon (us-west-2), AWS: Sydney (ap-southeast-2), AWS: Singapore (ap-southeast-1), AWS: N. California (us-west-1), AWS: Mumbai (ap-south-1), AWS: Ohio (us-east-2), AWS: Ireland (eu-west-1), AWS: Central (ca-central-1)), AWS: Metrics & Reporting (AWS: London (eu-west-2), AWS: Sydney (ap-southeast-2), AWS: Tokyo (ap-northeast-1), AWS: Frankfurt (eu-central-1), AWS: N. Virginia (us-east-1), AWS: Mumbai (ap-south-1), AWS: Oregon (us-west-2), AWS: Singapore (ap-southeast-1), AWS: Ireland (eu-west-1), AWS: Central (ca-central-1), AWS: N. California (us-west-1), AWS: Ohio (us-east-2)), and Hasura EE (Self hosted EE - Monitoring).