Some users may experience difficulties with searching for applicants and viewing candidate resumes.

Incident Report for Applicant Tracking Software

Postmortem

Incident Summary:
On July 1, 2024, users reported difficulties searching for applicants and viewing candidate resumes. The issue impacted applications received on June 29, 2024 and involved problems with search indexing, resume parsing, and other background updates.

Customer Impact:
Users encountered issues with searching for applicants who applied on June 29, 2024, and with updates to applicant profiles processed on that date. These issues affected resume parsing and other updates that are generally processed in the background. A workaround was provided, allowing users to access applicant profiles and download resumes directly from the file section in the application viewer.

Root Cause:
The incident was caused by scheduled system maintenance that overwhelmed our background processing system. This led to the failure of background jobs, including search indexing and resume processing. The system's capacity degraded over the course of the day, resulting in a total outage from 8:55 pm to 10:12 pm PT on June 29, 2024. The failed background jobs were initially missed during that outage.

Resolution Steps:
Background jobs that failed on June 29, 2024 were identified and reprocessed on July 1, 2024.

Preventative Measures:
We implemented several improvements to enhance our system's performance and reliability. New alerts have been established to monitor background job completions and detect errors more efficiently. We've added an additional monitor to catch issues during low traffic periods and increased system capacity to manage higher loads. An alert now triggers when background jobs exceed a reasonable threshold, ensuring timely intervention. Enhancements have been made to the background processing system, and we've reduced dependency on critical components to prevent total outages during failures.

If you have any further questions or concerns, please do not hesitate to reach out to our support team.

Posted Jul 22, 2024 - 20:47 PDT

Resolved

A fix has been implemented. Users should no longer experience issues searching for applicants who applied on Saturday, June 29, or encounter issues with changes to applicant profiles that didn't get processed on that date. Thank you for your patience as we worked to resolve this matter. If you are still experiencing issues, please reach out to our support team for further assistance.

Posted Jul 01, 2024 - 20:22 PDT

Identified

After further investigation, our team has identified additional issues impacting applicant profiles and their updates. This affects not only applicant search and resume parsing but also other updates that are generally processed in the background. We are actively working on implementing a solution and prioritizing this as a high-impact concern, dedicating our resources to restore these changes as quickly as possible.

Posted Jul 01, 2024 - 13:30 PDT

Investigating

Our team is aware of an issue impacting applicant search and resume parsing, specifically affecting applications received on Saturday, June 29. We are actively working on implementing a solution and prioritizing this as a high-impact concern, dedicating our resources to resolve the problem as quickly as possible. As a workaround, the applicant's profile is available under the job listings, and the resume can be downloaded from the file section in the application viewer. We apologize for any inconvenience this may cause.

Posted Jul 01, 2024 - 11:48 PDT

This incident affected: Application Viewer.