Live Streaming Disruption

Summary:

A service disruption on March 8th impacted Subsplash Live performance and access to the Live section of the dashboard. During a period of high traffic, a rate limiting setting on our recently updated web gateway incorrectly flagged internal system traffic as an overload. This prevented some users from accessing their Subsplash Live accounts and caused a drop in livestream connection success rates.

Timeline, Resolution & Recovery:

• Incident start: 6:00 AM PST
• Fix Applied: Our engineering team identified the bottleneck and disabled the restrictive setting at 7:30 AM PST, restoring full service by 7:38 AM PST.
• Video Backfill: All "Live-to-On-Demand" videos that were delayed during the outage have been successfully processed and are now visible in your library.

Next Steps: 

To ensure long-term stability and as part of our ongoing infrastructure enhancements, we have refined our internal routing protocols. This change allows essential system traffic to bypass traffic filtering while maintaining the integrity of the security measures protecting Subsplash Live.