Algorithmic Profiling and the Future of Individual Autonomy

-- By EricaPedersen - 11 Oct 2019

Privacy and Autonomy in the Internet Age: Beyond the Individual

Policymakers in the United States are finally beginning to acknowledge the tech industry’s exploitation of users through extensive monitoring and data collection. Unfortunately, the current policy debate is narrowly focused on enshrining limited individual rights to notice and consent with respect to the sale of personal data and the right to request deletion of personal data (see e.g., California Consumer Privacy Act which also includes broad carveouts allowing data collectors and third party “service providers” to ignore consumers’ deletion requests altogether).

These trivial ‘protections’ belie legislators’ erroneous belief that privacy will be sufficiently maintained through deletion rights and businesses’ feigned attempts to dissociate personally identifiable information from non-PII data. However, merely preventing one’s own data from entering a particular data pool will not protect individuals from privacy intrusions derived from algorithmic profiling or discriminatory limitations on freedom of choice. Laws which so narrowly frame the harms of data privacy exploitation will ultimately undermine our ability to conceptualize and address the significant environmental threats that big data pose to individual and societal freedom, autonomy, and self-determination.

The fundamental importance of a right to privacy lies in the necessity of protecting the fundamental human rights of individual autonomy and self-determination. Preservation of these rights is integral to both freedom and functional democracy. By focusing on a superficial individual right to anonymity, modern American lawmakers obscure and insulate the broader societal harms inherent to current methods of monetizing personal data.

Potential Harms of Data Analytics and Algorithmic Profiling

Algorithms provide an economically efficient means of analyzing huge data sets to gain new insights based on complex statistical correlations. As such, algorithmic methodologies are enormously powerful research tools. The Cambridge Analytica leaks have shown the power of data analytics to conduct mass psychological warfare and manipulate human behavior on a global scale. Moreover, algorithmic profiling has increasingly emerged as a gatekeeper controlling individuals’ abilities to access to a wide array of choices and opportunities in the real world.

Businesses seeking new ways to minimize costs have been quick to capitalize on algorithms’ predictive capabilities. Consequently, algorithms trained to maximize organizational welfare are increasingly used to infer, predict, and shape individuals’ personal preferences, interests, behavior, attitudes, movements, or health. Companies use algorithmic profiling to target and identify “ideal” job applicants, as well as to make training, compensation, promotion, and termination decisions about current employees (over whom management exercises increasingly broad and intrusive surveillance rights). Judges’ sentencing decisions are guided by insights from privately-developed and largely unaccountable predictive algorithms, despite the likelihood that these tools are perpetuating the same systemically discriminatory outcomes that plague our criminal justice system. Other applications include housing and policing.

Algorithms and statistical profiling now guide (or supplant) human decision-making in a wide variety of fora, often with guiding or determinative effects on individual freedoms. Despite the significant impact that these algorithms will have on our lives and the potential for their proprietors to manipulate en masse, very little attention is paid to the absence of accountability for these social impacts. For the most part, these processes are invisible to those whose freedoms they have restricted. Affected individuals typically have no right to notice, consent, explanation, nor any ability to effectively dispute the applicability, efficacy, or disparate impact of the methodology. Developers, empowered by trade secrets law, staunchly refuse to reveal the source code of the tools that are shaping our future.

The Value of Transparency

We must effectively regulate the abuse of profiling in data analytics without unnecessarily impeding technological development and the innumerable benefits to be derived from algorithmic insights. Individual autonomy, societal freedoms, and corporate accountability would be preserved far more effectively and sustainably through algorithmic transparency than proprietary control.

When data privacy is conceptualized within individual rights to ‘anonymity’ and deletion, personal data is framed in quasi-property terms and the regulatory solution to preserving autonomy appears to lie in enhancing individuals’ abilities to control access to their personal information. Even if such control could be exercised effectively, individuals remain vulnerable to privacy-infringing inferences of predictive algorithms and limitations on their own autonomy. Profiles will continue to be developed and used to make statistical inferences and predictions about any individual, regardless of whether that particular individual was able to prevent her own data from being collected and used to train the algorithm. These algorithms will be trained on data sets which are increasingly skewed and those attempting to enforce their own data privacy may find themselves increasingly marginalized and starved of algorithmically controlled opportunities.

Individual autonomy is strengthened by improving individuals’ access to information so that they can make informed decisions and contest erroneous assumptions. Thus, regulations designed to protect individual autonomy should emphasize transparency and attempt to reduce informational asymmetries. Individuals should be notified when the choices and opportunities available to them may be artificially limited by algorithmic insights. Individuals should have a right to an explanation of how the algorithm functions, including the data on which it was trained, the “target variables” it is designed to identify, and the inferences drawn from the statistical correlations that the algorithm has found. Algorithms used to guide choices and opportunities related to fundamental human rights should be continuously tested and held accountable for their social impacts. They should be open to the public and freely available for study and critique by scholars and government agencies.

Transparency would facilitate refinement of algorithms to more accurately achieve their intended goal, address biases overlooked by developers, and ensure ethical data processing and applications. Transparency would also provide a means of ensuring that algorithmic insights are not used in an arbitrarily determinative manner with respect to limiting individuals’ freedoms simply because they fall on the wrong side of a statistical inference. If enhanced market efficiency is truly the goal of algorithmic decision-making, then the exacerbation of informational disparities will only lead us in the wrong direction.

---

You are entitled to restrict access to your paper if you want to. But we all derive immense benefit from reading one another's work, and I hope you won't feel the need unless the subject matter is personal and its disclosure would be harmful or undesirable. To restrict access to your paper simply delete the "#" character on the next two lines:

• #Set ALLOWTOPICVIEW = TWikiAdminGroup, EricaPedersen
• #Set DENYTOPICVIEW = TWikiGuest

Note: TWiki has strict formatting rules for preference declarations. Make sure you preserve the three spaces, asterisk, and extra space at the beginning of these lines. If you wish to give access to any other users simply add them to the comma separated ALLOWTOPICVIEW list.