On 6/26/19 we deployed a code change that resulted in the analytics page failing to display data. All dashboard users would have seen an error message when viewing app analytics during this time. Once discovered, a rollback was deployed to a previous code version and this fixed the issue. The total time of service disruption was 15 hours 42 minutes. This did not affect the actual analytic micro service and all analytic data integrity was maintained. Learning and remediation from this disruption include a specific focus to ensure that all new deployments take into account dependencies and that our rollback procedures are better defined for faster response.
Posted Jun 27, 2019 - 13:16 PDT
Resolved
We're all clear on this one! During the service disruption, the analytics page of the app dashboard did not display graphics but did continue to collect data so all of the stats are accounted for.
Posted Jun 26, 2019 - 08:14 PDT
Monitoring
A fix has been implemented and we are monitoring the results. Analytics are now displaying in the dashboard as expected.
Posted Jun 26, 2019 - 07:55 PDT
Update
We are continuing to work on a fix for this issue.
Posted Jun 26, 2019 - 07:46 PDT
Identified
The underlying issue was discovered and we're working to implement a resolution.
Posted Jun 26, 2019 - 07:06 PDT
Investigating
We are currently investigating an issue with our analytics microservice. When accessing the analytics section of the app dashboard users will receive an error message. All other systems are unaffected and we'll keep you posted as we identify and resolve this issue.