We made a change to our file service in an attempt to speed up retrieving images from the database. The high load we experience on Sundays then resulted in degraded service where some customer experiences included not receiving some image, video, or audio files. Podcasts and other media lookups were also affected.
The root issue was fixed on Sunday by rolling back the change to the database. We also performed a database failover to bring the database back into a healthy state. Lastly, we made improvements to our caching systems to improve performance and reliability.
We will calibrate and adjust our application and database timeouts to ensure they are working correctly to protect the platform in the event of increased system load.