This article is contributed. See the original author and article here.
Final Update: Sunday, 13 December 2020 20:02 UTC
We’ve confirmed that all systems are back to normal with no customer impact as of 12/13, 19:40 UTC. Our logs show the incident started on 11/20, 02:40 UTC and that during the duration of ~23 days that it took to resolve the issue some customers experienced alerting failures. For new alerts that fired post 12/13 19:40 UTC, these are accurately reflecting the state of the resource health. For alerts in Fired/Resolved status between 11/20 02:40 UTC and 12/13 19:40 UTC, please depend on notifications that are setup through Action groups for accurate status of resource health.
-Anupama
We’ve confirmed that all systems are back to normal with no customer impact as of 12/13, 19:40 UTC. Our logs show the incident started on 11/20, 02:40 UTC and that during the duration of ~23 days that it took to resolve the issue some customers experienced alerting failures. For new alerts that fired post 12/13 19:40 UTC, these are accurately reflecting the state of the resource health. For alerts in Fired/Resolved status between 11/20 02:40 UTC and 12/13 19:40 UTC, please depend on notifications that are setup through Action groups for accurate status of resource health.
Root Cause: Root cause has been isolated to data from older version of code which was impacting Metric Alerts in Azure Portal.
Incident Timeline: ~23 days – 11/20, 02:40 UTC through 12/13, 19:40 UTC
-Anupama
Update: Sunday, 13 December 2020 18:35 UTC
Root cause has been isolated to data from old version of code which was impacting Metric Alerts in Azure Portal. Mitigation is complete on most of the instances and is in progress for last couple of instances.
Root cause has been isolated to data from old version of code which was impacting Metric Alerts in Azure Portal. Mitigation is complete on most of the instances and is in progress for last couple of instances.
- Work Around: None
- Next Update: Before 12/13 21:00 UTC
Update: Sunday, 13 December 2020 15:06 UTC
Root cause has been isolated to data from old version of code which was impacting Metric Alerts in Azure Portal. To address this issue we have rolled out a hotfix deployment. Some customers may continue to experience alerting failures.
Root cause has been isolated to data from old version of code which was impacting Metric Alerts in Azure Portal. To address this issue we have rolled out a hotfix deployment. Some customers may continue to experience alerting failures.
- Work Around: None
- Next Update: Before 12/13 18:30 UTC
We are working hard to resolve this issue and apologize for any inconvenience.
-Sandeep
Initial Update: Sunday, 13 December 2020 11:18 UTC
We are aware of issues within Metric Alerts and are actively investigating. Some customers may see alerts still active in the Azure Portal even though they receive resolved notification. Our logs show the incident started on 12/07, ~15:00 UTC.
-Sandeep
We are aware of issues within Metric Alerts and are actively investigating. Some customers may see alerts still active in the Azure Portal even though they receive resolved notification. Our logs show the incident started on 12/07, ~15:00 UTC.
- Work Around: None
- Next Update: Before 12/13 15:30 UTC
-Sandeep
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.
Recent Comments