Automated decision-making systems are increasingly being used to make important decisions in key areas of people’s lives. The vast majority of large employers in the U.S. — including 99 percent of Fortune 500 companies — use software to evaluate job applicants, and once hired, similar software is increasingly used to continuously surveil employees. Ninety percent of landlords use screening services, which are often powered by automated systems, to evaluate potential tenants.
These automated tools are a prevalent and entrenched part of our modern economic and social systems, and there is clear evidence that they can create and enable pervasive harm. For example, the automated tools frequently used by landlords to screen potential tenants are notoriously riddled with errors, locking consumers — disproportionately Black and Latinx renters — out of accessing housing. In the context of employment, the ACLU and other civil rights and technology organizations have repeatedly raised concerns about hiring discrimination facilitated or exacerbated by these technologies. There are many examples of algorithmically driven or amplified racial discrimination, gender discrimination, and disability discrimination in this area. In financial services, discriminatory automated systems are regularly used in high-stakes areas, impacting people’s ability to access credit and mortgages.
Automated decision-making systems are increasingly being used to make important decisions in key areas of people’s lives.
It is clear that relying on voluntary efforts of companies building and deploying automated decision-making systems has not been and will not be sufficient to address these harms. That’s why last month, the ACLU responded to the Federal Trade Commission (FTC)’s request for comment on harms stemming from commercial surveillance and lax data security practices. We are calling on the commission to adopt binding rules to identify and prevent algorithmic discrimination.
These problems are multi-faceted. Automated decision-making systems are often built and deployed in ecosystems and institutions already marked by entrenched discrimination — including in health care, the criminal legal system, and the family regulation system. Built and evaluated by humans, automated decision-making tools are often developed using data that reflects systemic discrimination and abusive data collection practices. Attempting to predict outcomes based on this data can create feedback loops that further systemic discrimination. These compounding issues can rear their heads throughout an automated decision-making system’s design and deployment. Moreover, these systems are often operated and deployed in such a way that impacted individuals and communities might not even know that they are interacting with these systems, let alone how they work — yet could still be materially affected by the system’s decision-making process and errors.
These automated tools are a prevalent and entrenched part of our modern economic and social systems, and there is clear evidence that they can create and enable pervasive harm.
The commission should establish tailored requirements for companies to undergo independent external audits of their automated decision-making systems. As highlighted in our comment, the commission can adopt these rules to prevent discrimination without contravening the First Amendment or Section 230 of the Communications Decency Act. The commission should consider requiring companies to adopt a comprehensive auditing framework to govern the use of automated decision-making systems and set clear standards for that framework. That’s because the harms of automated decision-making systems are best assessed as part of holistic efforts to understand the selection, design, and deployment of such tools.
These efforts should include meaningful engagement with people directly impacted by the deployment of automated decision-making systems. Companies should be required to undergo evaluations of the potential risks and harms of their algorithmic systems both before they are built and continuously if they are deployed. When these evaluations demonstrate the potential for algorithmic bias or other harms, companies can and should decommission or terminate the tools. To promote objectivity in evaluating algorithmic systems, these assessments should be carried out by independent external auditors who are provided with meaningful access to internal company systems under appropriate privacy controls.
That’s why last month, the ACLU responded to the Federal Trade Commission (FTC)’s request for comment on harms stemming from commercial surveillance and lax data security practices.
To enact these new rules, the commission should also collaborate with external researchers, advocacy organizations, and other government agencies. For example, the ACLU has previously called for the commission to collaborate with other civil rights agencies to address technology-driven housing discrimination and employment discrimination. New requirements established by the commission can and should co-exist with standards and guidance currently being developed by other agencies, such as the National Institute of Standards and Technology’s (NIST) Artificial Intelligence (AI) Risk Management Framework. The ACLU recently also provided recommendations to NIST to ensure that the AI Risk Management Framework centers impacted communities in efforts to address the harms of AI systems.
In a digital age, protecting our civil rights and civil liberties demands that we address the harms of algorithmic systems. Strong protections that address algorithmic discrimination have the potential to benefit all people and can make AI systems work better for everyone. The commission should act now to provide much-needed protection from the harms of automated decision-making systems.