Final Update: Tuesday, 18 October 2016 06:34 UTC
We’ve confirmed that all systems are back to normal with no customer impact as of 10/18, 06:00 AM UTC. Our logs show the incident started on 10/17, 08:00 PM UTC and that during the 10 hours that it took to resolve the issue ~63% of customers in would have not received alerts from proactive detection feature.We have processed all the backlog data and now the feature is working as expected.
We’ve confirmed that all systems are back to normal with no customer impact as of 10/18, 06:00 AM UTC. Our logs show the incident started on 10/17, 08:00 PM UTC and that during the 10 hours that it took to resolve the issue ~63% of customers in would have not received alerts from proactive detection feature.We have processed all the backlog data and now the feature is working as expected.
- Root Cause: The failure was due to one of our back end systems which process the requests from proactive detection feature was not working as expected. We have fixed our back end system which helped in mitigating the issue.
- Lessons Learned: We have collected telemetry logs and identified steps to be taken to avoid these kind of scenarios in future.
- Incident Timeline: 10 Hours – 10/17, 08:00 PM UTC through 10/18, 06:00 AM UTC
We understand that customers rely on Application Insights as a critical service and apologize for any impact this incident caused.
-Madhuri
Initial Update: Tuesday, 18 October 2016 05:08 UTC
We are aware of issues within Application Insights and are actively investigating. Some customers would have not receive alerts from proactive detection feature. Initial findings indicate that the problem began at 10/17 10:00 PM UTC
We are aware of issues within Application Insights and are actively investigating. Some customers would have not receive alerts from proactive detection feature. Initial findings indicate that the problem began at 10/17 10:00 PM UTC
- Work Around: None
- Next Update: Before 10/18 09:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Madhuri