Dashboards slow or failing to load
Incident Report for Subsplash
Postmortem

We made a change to our file service in an attempt to speed up retrieving images from the database. The high load we experience on Sundays then resulted in degraded service where some customer experiences included not receiving some image, video, or audio files. Podcasts and other media lookups were also affected.

The root issue was fixed on Sunday by rolling back the change to the database. We also performed a database failover to bring the database back into a healthy state. Lastly, we made improvements to our caching systems to improve performance and reliability.

We will calibrate and adjust our application and database timeouts to ensure they are working correctly to protect the platform in the event of increased system load.

Posted Jun 30, 2022 - 12:56 PDT

Resolved
This incident has been resolved.
Posted Jun 12, 2022 - 10:58 PDT
Monitoring
We have seen an improvement in the services impacted, we are continuing to monitor for further issues.
Posted Jun 12, 2022 - 10:47 PDT
Update
Live stream viewing pages are encountering loading issues in some cases. Some syndicated streams may also be disrupted. We are continuing to investigate.
Posted Jun 12, 2022 - 09:32 PDT
Update
We are continuing to investigate this issue.
Posted Jun 12, 2022 - 08:38 PDT
Update
Mobile apps are also experiencing slow loading times in some cases. We are continuing to investigate this issue.
Posted Jun 12, 2022 - 07:39 PDT
Update
SnapPages websites and Subsplash TV apps are also experiencing slow loading times or failing to load in some cases.
Posted Jun 12, 2022 - 07:19 PDT
Update
We are continuing to investigate this issue.
Posted Jun 12, 2022 - 07:05 PDT
Investigating
Subsplash media and live stream dashboard pages are loading slow or timing out for some users.
Posted Jun 12, 2022 - 06:54 PDT
This incident affected: Mobile apps, Subsplash Live, TV apps, SnapPages, and Dashboards.