Subsplash Dashboard Service Disruption

Incident Report for Subsplash

Postmortem

On July 23, 2024, Subsplash experienced an unexpected infrastructure change to a core networking system. While this networking system was interrupted, many of our Subsplash backend services were unavailable. The Dashboard, Giving, Live, SnapPages, Mobile Apps, and Messaging were among these services. This interruption affected both Clients managing the services and End-Users interacting with them.

Recovering the services required manual intervention to properly reconfigure the networking system. We are prioritizing the improvement of our misconfiguration alerts and network resilience.

Posted Jul 31, 2024 - 05:50 PDT

Resolved

This incident has been resolved.

Posted Jul 23, 2024 - 06:48 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jul 23, 2024 - 06:10 PDT

Identified

The issue has been identified and a fix is being implemented.

Posted Jul 23, 2024 - 06:06 PDT

Update

We are continuing to investigate this issue.

Posted Jul 23, 2024 - 05:30 PDT

Investigating

We are currently investigating an issue preventing Clients from logging into the Dashboard. Clients may receive an error when attempting to log into the Dashboard. This issue is also affecting other Subsplash services from loading such as Mobile Apps, SnapPages, Giving, and Media & Live.

Posted Jul 23, 2024 - 04:37 PDT

This incident affected: Mobile apps, Giving, Subsplash Live, SnapPages, Subsplash Messaging, and Dashboards.