Some accounts may experience an issue with Yahoo email deliverability

Incident Report for Applicant Tracking Software

Postmortem

Incident Summary:

Beginning May 6, 2024 at approximately 5:00 AM PT through May 21, 2024, some of our users experienced delayed, and sometimes bounced, email deliveries specifically to Yahoo and AOL email addresses as a result of unexpected changes to those providers’ email processing protocols. This issue was identified as a significant increase in Yahoo/AOL email delivery times, which impacted timely communication for our users relying on prompt email notifications.

‌

Customer Impact:

This issue was limited to emails sent to Yahoo/AOL addresses and, of those emails, only some encountered delays of up to 3 days or bounces during this period. Users were notified of any bounced emails via a standard notification on the dashboard. Users were advised to utilize alternative communication methods (SMS or a different email provider) for impacted recipients.

‌

Root Cause:

Upon thorough investigation, we determined that the delays were due to changes in the email processing protocols used by Yahoo/AOL, which were not communicated to us. The adjustments Yahoo/AOL made to their email filtering criteria led to an unexpected throttling of emails originating from our servers. This disruption was unintended

‌

Resolution Steps:

Our engineering team responded swiftly upon identifying the issue and took the following actions:

‌

Immediate Communication: We contacted Yahoo/AOL to confirm the cause and scope of the issue.
Technical Adjustments: Adjustments were made to our email sending practices to align with Yahoo’s new protocols, and to broaden our distribution of email traffic and increase categorization of email types to further enhance deliverability.
Monitoring: We closely monitored the email flow until we confirmed that delivery times had returned to normal.

‌

Preventive Measures:

To prevent such occurrences in the future, we have implemented several new measures:

‌

Enhanced Monitoring: We have implemented additional measures to monitor changes in processing protocols from major email providers.
Infrastructure Optimization: We have identified optimizations to our email infrastructure that will further improve deliverability.

‌

We apologize for any inconvenience caused and appreciate your understanding as we navigated this external challenge. Ensuring the reliability and speed of our email services remains a top priority, and we are committed to continuous improvements in our service delivery. Thank you for your patience and continued trust in our service.

‌

If you have any questions or require further details, please do not hesitate to contact our support team.

Posted May 31, 2024 - 13:43 PDT

Resolved

We are pleased to inform you that the recent issue with our Applicant Tracking System's messaging feature have been resolved. Our team of tech wizards has been diligently administering digital cough syrup to the system, and we are happy to report that messaging is now functioning as expected.

If you are still experiencing any issues with messaging, we recommend clearing your cache and cookies before reaching out to support. This will help to ensure that you are viewing the most up-to-date version of the Applicant Tracking System.

We sincerely apologize for any inconvenience this issue may have caused, and we appreciate your patience and understanding as we worked to resolve it.

Posted May 24, 2024 - 13:40 PDT

Update

We've identified the gremlin responsible for the issue in messaging and have implemented a fix to keep your messages flowing smoothly, but we're still keeping an eye on things just in case.

In the meantime, you should be able to resume sending and receiving messages without continued shenanigans. If you encounter any further issues, please don't hesitate to reach out to our Support Team.

Posted May 23, 2024 - 11:37 PDT

Update

This issue is nearing resolution, and the majority of emails are being delivered as expected. Our team will continue to monitor this issue for stability.

Posted May 22, 2024 - 09:00 PDT

Update

We are closely monitoring the situation to ensure that email deliverability continues without interruption. We sincerely apologize for any inconvenience this issue may have caused, and we appreciate your patience and understanding as we work to resolve it.

Please do not hesitate to contact us if you have any further questions or concerns.

Posted May 20, 2024 - 12:31 PDT

Monitoring

Rest assured that our team is actively monitoring this issue, as it remains our top priority.

We'll continue to keep you updated as we progress in our endeavors to restore the system to its full functionality.

Thank you for your patience and understanding as we work to clear up this technical issue. We're committed to providing you with a seamless communication experience throughout the hiring process.

Posted May 15, 2024 - 14:55 PDT

Update

We are continuing to work on a fix for this issue.

Posted May 13, 2024 - 16:04 PDT

Update

Our Applicant Tracking System is experiencing a temporary disruption in messaging. But worry not! Our tech team is already hard at work administering a dose of digital cough syrup to get those messages flowing smoothly again.

In the meantime, we apologize for any inconvenience this may cause. We know how vital communication is during the hiring process, so rest assured, we're working as fast as we can to clear up the issue.

While we handle the technical side, we recommend using alternative communication channels, like email or phone calls, to keep your conversation with candidates flowing. We'll keep you updated with any progress, and soon those messages will be back on track in no time.

Thank you for your patience and understanding as we clear up this technical issue.

Posted May 13, 2024 - 09:16 PDT

Identified

Our Engineering Team is actively working with Yahoo to identify the root cause so they can implement a solution.

Here's what we know so far:
-New emails to Yahoo inboxes are being throttled.
-The issue is affecting some users.

In the meantime, please:
-Reach out to Yahoo recipients externally or request an alternative email address to use within the ATS

We will send an additional update as soon as possible.

Posted May 08, 2024 - 13:29 PDT

Update

We've identified a temporary technical issue causing slight delays with emails sent to Yahoo users. We understand the importance of email deliverability, and we sincerely apologize for any inconvenience this may cause.

Here's what we know: Messages sent to Yahoo inboxes are being delayed.
What we're doing: Our team of tech wizards are diligently investigating the issue and working tirelessly to resolve it promptly.
What you can do: Reach out to applicants externally for an alternative email address so that you can continue correspondence within the ATS, if necessary. Sit back, relax, and enjoy a refreshing cup of applicant tracking system-themed tea. We'll handle the rest! You can rest assured that email deliverability remains our top priority.

We'll keep you updated as we progress in our endeavors to restore the Applicant Tracking System to its full functionality. Thank you for your continued patience and trust!

Posted May 07, 2024 - 09:02 PDT

Update

We are continuing to monitor for any further issues.

Posted May 06, 2024 - 12:09 PDT

Monitoring

A fix has been implemented and we are monitoring the results.

Posted May 06, 2024 - 10:46 PDT

Investigating

We are currently investigating this issue.

Posted May 06, 2024 - 08:18 PDT

This incident affected: Messages.