My resume screening bot rejected my own cousin

Last month I was testing a new hiring algorithm for our small HR firm in Austin. I fed it resumes from 50 applicants including my cousin who has 8 years of experience in data entry. The bot flagged her as a high risk hire because her address was in a low income zip code... I caught it right before the rejections went out. Has anyone else seen geographic bias sneak into their tools like that?

2 comments

Log in to join the discussion

Log In

2 Comments

kai_webb911mo ago

Your cousin's zip code is a proxy for a lot of real, measurable stuff though. Schools, commute times, even how likely someone is to have stable internet at home. That's not the bot being biased, that's it picking up on patterns we all kinda know exist. I've seen it where a person from a rough area is actually more reliable because they've had to fight for every shift. But the data doesn't lie, it just doesn't tell the whole story. Maybe the real issue is you didn't tune the weights right, not that geography itself is poison.

9

sean_green441mo ago

Haven't you ever seen the data punish somebody for something that wasn't their fault though? I get that zip code lines up with certain things, but that doesn't mean it's fair to use it as a shortcut for judging a person. That's just lazy model building, and it's gonna bite you in the butt eventually.

6