Finally got my hiring algorithm to stop flagging veterans for no reason

I spent 4 hours last Saturday digging through our candidate scoring model after I noticed it kept ranking military experience as a negative. Turns out the training data had like 80% civilian resumes and the model learned to penalize anything that looked different. I had to go back and rebalance the dataset with more veteran examples before retraining. Has anyone else found weird hidden biases in their recruitment tools that took forever to track down?

3 comments

3 Comments

max_brown1mo ago

Dude, that is wild but totally believable. My buddy works at a mid sized tech company and he told me they found their chatbot for customer support was literally saying "I don't understand" to anyone who typed in Spanish. Like, it wasn't even a translation thing, the model just completely ignored non English sentences and would give canned responses about checking the FAQ. They spent weeks digging through the logs before they realized the training data was 99% English transcripts. It took them forever to fix because they had to go find thousands of Spanish support calls and manually label them all. The worst part was management kept asking why the "AI was racist" and my buddy had to explain it was just bad data, not a conspiracy.

jamie_webb671mo ago

Wait, did you have to dig through the training data yourself or did you have a data team helping you out? I had a friend who worked at a call center software company and they found their voice-to-text model flat out ignored anyone with a Southern accent. It was like the AI just went "nope" and typed gibberish for half the calls. They ended up having to bring in ten people just to record themselves saying common phrases in different accents so the model could finally learn. The worst part was they kept a spreadsheet of all the weird stuff the model did before they fixed it, and one entry literally said "sounds like customer is speaking English but no words detected.

jamie_adams1mo ago

That spreadsheet detail hits close to home. Had a similar nightmare with a voice model at my old gig. It would just drop entire sentences from people with certain speech patterns. We found it in the logs later. The AI literally labeled chunks of audio as "silence" when people with a lisp were talking. Took us months of begging for more diverse training data. Management never got why simple fixes took so long.