Incident Summary:

On the morning of May 3rd, our third-party 2FA (two-factor authentication) vendor experienced an outage that resulted in users with 2FA enabled in their ATS accounts being temporarily unable to log in. 

Customer Impact:

Users employing 2FA in their ATS accounts could not access their accounts from 6:00 AM to 9:30 AM PT.

Technical Analysis:

The issue was caused by interruptions between our third-party authentication provider and their messaging service vendor, leading to denied authentication requests with 401 errors. This blocked all 2FA users from logging into our services. 

Response:

• 6:22 AM PT: Our team was alerted about the inability to receive 2FA codes necessary for logging into our services. Our team responded quickly by updating our status page and establishing communication with the third-party service provider.
• 9:58 AM PT: Our third-party vendor confirmed they successfully implemented a fix and our team confirmed that 2FA functionality was restored, and access to accounts was reestablished.

Ongoing and Future Remediation Plans:

• Enhanced Monitoring: We are enhancing our monitoring solutions to more swiftly detect and manage potential future disruptions.
• Incident Response Protocol Review: We are refining our incident response strategies to include automated alerting and on-call triggering.

Conclusion:

2FA functionality has been fully restored, and all systems are now operational. We apologize for any inconvenience caused and are committed to enhancing our systems to prevent such disruptions. Our focus remains on maintaining a secure and reliable service for our users.

Contact Information:

For further details or inquiries, please do not hesitate to contact our Support Team by submitting a ticket or chatting with us on the ATS platform.

We appreciate the patience and continued support of our users during this incident.