Service Interruption
Incident Report for Subsplash
Postmortem

On October 22, 2023, the Subsplash Platform experienced a service interruption that impacted the functionality of some of our products. During a routine technical update to one of our services, there was a misconfiguration that impacted the autoscaling technology of Subsplash Giving, Websites, and our Dashboard for some of our clients—we saw roughly 1% of traffic requests fail during this time and understand the impact this had on your end users. During the 2 hours of interruption these products operated with an uptime of 99% and 99.99% for the day. Other services were entirely unaffected and maintained an uptime of 99.99%-100%. Our engineers identified the root cause and corrected the issue.

The Subsplash team is actively working to prevent this issue from occurring in the future, including expanding our load testing capabilities and implementing better observability into our use of the impacted services in the event of future updates.

Posted Oct 24, 2023 - 15:41 PDT

Resolved
This incident has been resolved.
Posted Oct 22, 2023 - 09:15 PDT
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 22, 2023 - 08:45 PDT
Identified
The issue has been identified and a fix is being implemented.
Posted Oct 22, 2023 - 08:38 PDT
Update
We are currently investigating the issues affecting multiple areas of our platform.
Posted Oct 22, 2023 - 08:26 PDT
Update
We are continuing to investigate this issue.
Posted Oct 22, 2023 - 08:01 PDT
Update
We are continuing to investigate this issue.
Posted Oct 22, 2023 - 07:38 PDT
Update
We are continuing to investigate this issue.
Posted Oct 22, 2023 - 07:18 PDT
Investigating
We are currently investigating an issue affecting multiple areas of the platform, including dashboard access, Giving, websites, and Check-in.
Posted Oct 22, 2023 - 06:49 PDT
This incident affected: Giving, SnapPages, Dashboards, and Church Management.