Dashboard and Apps Service Disruption
Incident Report for Subsplash
Postmortem

Early in the morning on September 21st, our monitoring systems alerted on-call engineers to an issue which impacted the Subsplash dashboard, mobile apps, and web embeds. From 4:08am-4:40am PDT, these systems were inaccessible to clients and end users.

We determined that the issue was caused by a database backup process which inadvertently prevented queries from receiving a response. We restarted the database to quickly restore service, then disabled the offending process to immediately prevent the issue from happening again. We are currently refining aspects of the backup process to prevent queries from being blocked or delayed under any circumstances.

Fortunately, due to the early hour of the disruption, we did not receive any reports from clients or end users about the issue.

Posted about 1 year ago. Sep 25, 2018 - 17:48 PDT

Resolved
We've isolated the problem and have restored service. We've also disabled a process which inadvertently led to the issue. We'll follow up later with more details.
Posted about 1 year ago. Sep 21, 2018 - 04:53 PDT
Identified
Our monitoring system has alerted us to an issue with the Subsplash Dashboard as well as apps that is preventing access to both. We're currently investigating and attempting to resolve.
Posted about 1 year ago. Sep 21, 2018 - 04:42 PDT
This incident affected: Mobile apps and Dashboards.