Dedicated VPC customers connecting to metadata database in AWS Oregon us-west-2
Incident Report for Hasura
Postmortem

On May 24th, between 15:29 UTC and 16:45 UTC, Dedicated VPC users running in the AWS us-west-2 region, experienced issues connecting to the regional metadata DB. During this time, these projects experienced a period of downtime.

Root cause of the problem was the result of a triggered regional migration job which overloaded metadata DB and subsequently caused issues with our lux proxy service. The database was stabilized during the incident and service restarts were performed on the lux proxy service.

The Hasura SRE team will be reviewing this incident in full detail and will implement additional safeguards and fixes as follow up items in response to our internal RCA.

Posted May 25, 2022 - 18:56 UTC

Resolved
This incident has been resolved. A postmortem will be created by EOD 05/25
Posted May 24, 2022 - 17:26 UTC
Monitoring
A fix has been implemented and the SRE team is continuing to monitor
Posted May 24, 2022 - 16:57 UTC
Investigating
Hasura SRE Team is currently investigating an issue where our dedicated VPC customers are having issues connecting to the metadata RDS instance in the AWS us-west-2 region.
Posted May 24, 2022 - 16:32 UTC