Dashboard and Apps Service Disruption
Incident Report for Subsplash
Postmortem

Early in the morning on September 21st, our monitoring systems alerted on-call engineers to an issue which impacted the Subsplash dashboard, mobile apps, and web embeds. From 4:08am-4:40am PDT, these systems were inaccessible to clients and end users.

We determined that the issue was caused by a database backup process which inadvertently prevented queries from receiving a response. We restarted the database to quickly restore service, then disabled the offending process to immediately prevent the issue from happening again. We are currently refining aspects of the backup process to prevent queries from being blocked or delayed under any circumstances.

Fortunately, due to the early hour of the disruption, we did not receive any reports from clients or end users about the issue.

Posted 3 months ago. Sep 26, 2018 - 00:48 UTC

Resolved
We've isolated the problem and have restored service. We've also disabled a process which inadvertently led to the issue. We'll follow up later with more details.
Posted 3 months ago. Sep 21, 2018 - 11:53 UTC
Identified
Our monitoring system has alerted us to an issue with the Subsplash Dashboard as well as apps that is preventing access to both. We're currently investigating and attempting to resolve.
Posted 3 months ago. Sep 21, 2018 - 11:42 UTC
This incident affected: Mobile apps and Dashboards.