This article is contributed. See the original author and article here.
Final Update: Thursday, 14 January 2021 02:20 UTC
We’ve confirmed that all systems are back to normal with no customer impact as of 01/14, 01:12 AM UTC. Our logs show the incident started on 01/13, 11:45 PM UTC and that during the 1 hour and 27 minutes that it took to resolve the issue while 8034 subscription’s of customers in WUS experienced intermittent data latency and incorrect alert activation.
We’ve confirmed that all systems are back to normal with no customer impact as of 01/14, 01:12 AM UTC. Our logs show the incident started on 01/13, 11:45 PM UTC and that during the 1 hour and 27 minutes that it took to resolve the issue while 8034 subscription’s of customers in WUS experienced intermittent data latency and incorrect alert activation.
- Root Cause: The failure was due to An increase in incoming traffic for certain data types led to congestion on a backend service responsible for processing the incoming data.
- Mitigation steps: The issue self-healed after the affected backend service auto-scaled to handle the increased volume of requests. Additionally we have worked on manually scaling up the service to avoid reoccurrence of the issue.
- Incident Timeline: 1 Hours & 27 minutes – 01/13, 11:45 PM UTC through 01/14, 01:12 AM UTC
- Latency increase: 60 minutes
-Arish B
Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.
Recent Comments