Analytics post mortem
From 10/11/18 10:00 PM- 10/12/18 10:25 AM we experienced a disruption to our analytics servers that left about 1/3 of clients unable to view analytics in their dashboard. The disruption was caused when we deployed an analytics improvements. One of our servers was not updated with the correct config settings causing it to have trouble responding to analytics requests. Safeguards are currently in the works to put automated processes in place to check for inconsistent configuration settings prior to deployment of a new service.
Posted Oct 17, 2018 - 07:21 PDT
Resolved
All issues with analytics visualization have been resolved. Less than 1/3 of traffic to the analytics service was affected. We discovered a discrepancy in one out of three servers that was not successfully updated with a config change during a recent deployment to improve analytics. The impact of this issue was less than initially assumed.
Posted Oct 12, 2018 - 10:38 PDT
Investigating
We are currently investigating an issue that is preventing some clients from accessing their analytics in the Subsplash Dashboard. All other systems are unaffected, and the underlying analytics service is fine. But dashboard analytics visualizations are unavailable for some clients. We'll keep you posted as we work to rectify the issue.