Early in the morning on September 21st, our monitoring systems alerted on-call engineers to an issue which impacted the Subsplash dashboard, mobile apps, and web embeds. From 4:08am-4:40am PDT, these systems were inaccessible to clients and end users.

We determined that the issue was caused by a database backup process which inadvertently prevented queries from receiving a response. We restarted the database to quickly restore service, then disabled the offending process to immediately prevent the issue from happening again. We are currently refining aspects of the backup process to prevent queries from being blocked or delayed under any circumstances.

Fortunately, due to the early hour of the disruption, we did not receive any reports from clients or end users about the issue.