Final Update: Monday, 16 November 2015 09:00 UTC
Earlier today between 01:10 and 02:20 UTC, some customers may have experienced intermittent "Internal Server Error" or DNS-related errors while accessing Visual Studio Online services, especially during login.
• Root Cause: The failure was due to a planned DNS record change that was not implemented correctly. There was a brief period where no DNS record existed and therefore the "no record available" (NXDOMAIN) response from the authoritative DNS servers was cached by downstream DNS servers for up to an hour. This change was necessary to resolve an ongoing intermittent performance problem that can result in some requests taking up to an additional 9 seconds.
• Chance of Reoccurrence: Low. We do not plan to make any further changes to this DNS record.
• Lessons Learned: We have updated our DNS change procedures with additional validation steps to minimize the chance of this issue reoccurring.
• Incident Timeline: 1 hour & 10 minutes 16 November 2015 01:10 UTC through 02:20 UTC
We understand that customers rely on VS Online as a critical service and apologize for any impact this incident caused.
Sincerely,
VS Online Service Delivery Team