2
I thought the hiring algorithm at my old job was fair until I saw the data
For two years, I helped run a system that filtered resumes for a big company in Chicago. I was sure it was neutral. Then, during a review, we pulled the numbers and saw it was passing over candidates from certain zip codes at a rate three times higher, even with names and schools hidden. The bias was in the job history patterns it learned from old, flawed hiring data. It felt like a gut punch because I had defended the system. Has anyone else had a tool they trusted turn out to have a hidden bias like that?
4 comments
Log in to join the discussion
Log In4 Comments
emeryj664mo ago
Three times higher for certain zip codes? That's crazy. It makes you wonder what else those systems are picking up on. I always figured hiding names would fix it but job history patterns... that's sneaky. How do you even start to clean data that's already that messed up?
8
amy_craig284mo ago
Hiding names is a good first step, but it's not enough. The system can still connect zip codes to mostly minority neighborhoods. It might also use school names or even the types of jobs listed. Cleaning it means you have to find and remove all those hidden links, which is a huge job. They might need to rebuild the whole system from scratch.
6
jana_black4mo ago
Our old system flagged every applicant from the 02119 zip code.
2
kaigibson1mo ago
Actually I gotta push back a little on the idea that hiding names and schools is a "good first step." In my experience, that kind of masking can actually make things worse if you're not careful. Here's why: if you strip out names and schools but the system is still learning from the same old biased hiring decisions, it's just going to find new proxies for the same biases like zip codes or job history patterns. You end up with a false sense of fairness while the underlying discrimination stays intact. Your mileage may vary but I've seen teams spend months on masking only to discover the algorithm just shifted its bias to a different set of features. It's a bandaid on a broken bone, not a real fix.
4