On May 5th, a Kubernetes service which handles GitHub deployments & region migrations, experienced a period of downtime. The underlying issue was related to our Vault service which experienced downtime during this window. A temporary fix was implemented by the Infrastructure team (Which resolved the immediate issue) and a more permanent one was rolled out on May 9th to update our Vault configuration to be compliant with new specifications introduced in newer versions of Kubernetes. In addition, the SRE team is implementing additional monitors to have the team be alerted more quickly in case there are any issues with the previously mentioned Kubernetes service.
Posted May 09, 2022 - 20:22 UTC
Resolved
This incident have been resolved.
Posted May 05, 2022 - 23:53 UTC
Monitoring
A fix has been implemented and we are continuing to monitor our services.
Posted May 05, 2022 - 23:22 UTC
Investigating
GitHub deployments (via GitHub integrations) as well as region migrations are currently impacted. Hasura SRE team is currently investigating.