Credit: Mauro Pimentel/AFP/Getty Images

In news we never thought we’d be reporting, it looks as though Amazon accidentally created sexist AI. Let’s back up: As Reuters reports, in 2014, Amazon began designing a computer program to help hiring managers sift through resumes. Ideally, the program would select the strongest candidates and pass them along to an actual human for review.

The problem: the AI program overtly favored male applicants, especially in tech-related roles.

It turns out that when building the program, the engineers designed the algorithm using resumes and hiring decisions submitted over the previous 10 years, and the overwhelming majority of those resumes were from male candidates. The algorithm observed that more men than women were hired in the past, and translated that into a preference for male candidates.

The system not only favored male candidates, but it also reacted negatively to resumes featuring feminine words or phrases. Reuters reported that resumes that included words like “women’s” (as in, say, “Women’s Studies major” or “women’s college,” lowered an applicants’ rank.

Amazon told Mashable that the AI program was only ever used in trials, and the sexist algorithm was fixed in 2015—but the program was shut down in 2017 because it wasn’t “returning relevant candidates.” Additionally, the company told Reuters that the program never replaced traditional hiring practices, instead serving as a supplementary system for recruiters.

However, the algorithm’s design and execution speaks to a larger issue of sexism within the workforce, especially within tech-related fields. The algorithm was inherently biased in selecting candidates because it was built upon an already biased, severely flawed system in a culture that continues to discourage women from pursuing careers in STEM.

Let’s hope this was an eye-opening experience for Amazon—and for major companies everywhere.