Power, Human Control, and Distributed Decisions
By Javier Surasky
.png)
When an institution uses an artificial intelligence system for decision-making, who is really deciding? The answer usually moves between two extremes: those who claim that the machine decides, and those who argue that the decision remains human as long as a person retains the power to approve, reject, or modify the output produced by the system.
I believe
neither answer adequately describes what actually happens. AI does not transfer
human decision-making to a machine or a model, but it does change the way
decisions are produced. It does so through functions configured by an authority
capable of defining the problem, producing knowledge, setting criteria,
selecting alternatives, intervening, and answering for the consequences. In
reality, however, that entire process is distributed among people,
organizations, data, models, providers, and infrastructures. This requires us
to replace the question of who makes the decision with another one: how to
reconstruct the chain of interventions, analyze the capacities held by each
party involved, and determine the real possibility each one has of influencing
the final decision.
In this
blog, although I have addressed it in previous entries, I do not analyze how
responsibility has evolved in relation to this decision-making structure, which
we may call “distributed.”
Decision-Making as a Distributed Process
Institutions
tend to represent decision-making as a clearly bounded act: a judge issues a ruling, a bank grants a loan, a company selects a candidate, and so on.
This
representation has legal and organizational reasons behind it, since it makes
it possible to assign competence, authority, and responsibility. But it tells
us little about the material process by which the decision was produced, even
though, in decision-making systems, the visible outcome is only one stage in a
sequence.
In AI
systems, before a model issues a score, a classification, or a recommendation,
someone has already defined the problem it is meant to solve and selected the
data, categories, variables, thresholds, and response criteria. These choices
delimit the field of decision-making and, for that reason, cannot be treated as
neutral or merely preparatory to the decision.
Cobbe, Lee,
and Singh (2021) propose understanding automated decision-making as a
sociotechnical process composed of human, technical, and organizational
elements. This process begins with the conception and procurement of the
system, continues through its design, development, deployment, and use, and
extends to its consequences and any later investigation of outcomes. Meaningful
accountability therefore requires considering the entire process, from “the
commissioning of the system” to “the effects of those decisions and any
subsequent investigations” (p. 4).
This
perspective prevents us from equating the decision with the model. It makes
multiple entry points for the generation of problems clearly visible, showing
that the model is only one component within a broader decision-making
architecture (Cobbe et al., 2021).
AI
contributes a specific form of knowledge production based on classification,
correlation, probability, and prediction. This means that deciding is not
simply a matter of choosing among previously given alternatives; it also
involves intervening in the production of the categories, evidence, and
possibilities within which those alternatives become imaginable.
Even before
AI became a mass-use tool, Feenberg (1991) rejected both the idea of technology
as an empty instrument and its portrayal as an autonomous and inevitable force.
Because technical design incorporates values, priorities, and forms of social
organization, apparently technical decisions can stabilize power relations and
limit alternatives before, or even without, any intervention by a formal
authority.
I therefore
argue that the decision is located neither entirely in the technical system nor
in the person who uses its output. Rather, it emerges from a sociotechnical
configuration involving designers, providers, public officials, experts,
databases, models, norms, procedures, and infrastructures.
The Fictions of Distributed Decision-Making
Recognizing
that decision-making is a distributed process can lead to a new simplification:
assuming that all the actors involved effectively participate in the decision.
That
equivalence is fictional, and it rests on a plurality of “metafictions.” I use
this term because they do not directly describe the decision-making process
itself, but rather the representations that allow its distribution to be
presented as horizontal and controllable. Among them, because of their
explanatory and demonstrative value, I am especially interested in two: the
metafiction of distributed decisional capacity and the metafiction of human
control.
The Metafiction of Distributed Decisional Capacity
Many actors
intervene in a decision mediated by AI, but not all of them have an effective
capacity to alter its direction.
An
operator, for example, may enter data without being able to modify the
categories through which that data will be processed. A public official may
receive a score without knowing how it was constructed. And the user
institution may retain formal authority while depending on external providers
to audit or modify the model.
At the
other end of the process, the affected person may provide, even involuntarily,
data that feeds the system, while lacking the means to challenge their classification.
The
sociotechnical distribution of the process does not take the form of an
equivalent distribution of capacities to influence the decision. Identifying
those capacities requires observing segmented elements of the process: who can
define objectives, select criteria of relevance, modify the system, provide
infrastructure, impose interpretations, challenge its use, or review its
consequences?
The
inequality between actors with limited capacities and others who exercise
strategic functions over the process as a whole marks a first difference: while
a decision is operationally distributed across multiple components, the
capacity to configure its conditions remains concentrated.
As O’Neil
(2017, p. 21) notes, “Models are opinions embedded in mathematics.” This means
that every model entails a series of selections and exclusions. That does not,
in itself, make every model arbitrary, but it does warn us that model-building
incorporates human and organizational choices. Whoever defines variables,
indicators, and optimization functions therefore participates more intensely in
the decision than whoever merely receives or executes the result.
The Metafiction of Human Control
This
metafiction begins with the claim that the decision remains under control
because a person has the final say, while the system is assigned the role of
assistant. This, in turn, offers an apparently simple answer to the problem of
responsibility: if a person validates the result, that person is answerable for
its consequences.
However,
human presence is not the same as effective human control. For meaningful
control to exist, the person must have sufficient information, understand the
result, have time to evaluate it, and possess real authority to depart from the
algorithmic recommendation, as well as alternatives for doing so.
Moreover,
the abstract possibility of rejecting an output has little value if the
organization systematically rewards acceptance, if there are no alternative
sources of information, or if the person lacks the technical capacities needed
to question the model.
Along these
lines, Santoni de Sio and van den Hoven (2018, p. 1) argue that human control,
to be meaningful, has two necessary conditions: tracking, which requires the
system to respond to relevant human reasons and to facts in the environment;
and tracing, which requires that its outputs be traceable to at least one
person who understands their role and the capacities of the system.
Green
(2022, p. 7) is more forceful in his critique. After reviewing 41 public
policies and the evidence on human-algorithm interaction, he concludes that
“people are unable to provide reliable oversight of algorithms,” creating a
“false sense of security” and legitimizing the incorporation of flawed or
controversial algorithms into public processes (p. 9).
This is not
a problem of people making mistakes. It is a problem rooted in the difficulty
of overseeing an algorithmic recommendation. That difficulty leads
institutional policy into a possible contradiction: it expects a person to
trust the system because it supposedly predicts better, while also expecting
that same person to detect the cases in which the prediction is wrong.
Elish
(2019) uses the expression “moral crumple zone” to describe the situation in
which the person closest to the point of execution absorbs the moral and legal
consequences of a system failure, even when their capacity to control the
system is limited:
Just as the
crumple zone in a car is designed to absorb the force of impact in a crash, the
human in a highly complex and automated system may become simply a
component—accidentally or intentionally—that bears the brunt of the moral and
legal responsibilities when the overall system malfunctions. (p. 41)
This
creates a critical separation between control and responsibility. The operator
appears as the author of a decision whose process they do not control simply
because they are located at its final stage. At the same time, this makes
invisible—and protects—those who made the most critical decisions during the
sociotechnical decision-making process of a model they configured without the
operator’s participation.
Human
presence, indispensable in certain contexts, is not a guarantee of control.
Turning that presumption into a ritual formula may even prevent the original
question from being asked: should the system have been used to make that
decision at all?
Power, Infrastructure, and Knowledge in AI-Mediated Decisions
What has
been said leads me to view distributed decision-making as a scenario of power
distribution, where multiple actors coexist alongside a high concentration of
strategic capacities.
Yeung
(2018) shows that algorithmic regulation can be broken down into three moments:
standard-setting, information collection and production, and intervention in
behavior. Each moment raises specific questions about decisional capacity, and
the answers to those questions are always shaped by actual power relations.
To begin
with, power appears in the capacity to configure the system, where it is
determined who defines the problem. This means establishing which aspects of a
situation will be translated into variables and which will be left out, thereby
determining, in the same act, which outcome will be optimized. Then, whoever
sets thresholds decides what level of risk will be considered tolerable.
Power can
also be seen in infrastructure. Crawford (2021) shows that AI is a material and
political infrastructure sustained by natural resources, labor, supply chains,
classificatory systems, and state and corporate capacities. Its operation
extends far beyond the algorithm and even conditions the possibility of the
algorithm’s execution.
That
infrastructure is highly concentrated in a small number of companies that
control cloud services, computational capacity, storage, models, and
development tools. An institution may therefore formally retain the competence
to decide while depending on external actors to execute, understand, modify, or
replace the AI systems it uses.
That
technical dependence becomes political capacity: the capacity to place a second
delimitation on the range of possible options, access to information, and the
conditions under which an AI system can operate in practice.
No less
important is the epistemic power to decide what will be recognized as evidence,
which correlations will become relevant, and which results will appear
trustworthy. All of these elements are inscribed in each particular form of
knowledge production that makes the decision possible. Amoore (2020) argues
that algorithms do not merely describe the world, but contribute to determining
what can be recognized and which futures appear probable or possible. Sadin
(2020), from a stronger position, characterizes this transformation as the
emergence of a power to state the truth.
For all
these reasons, asking who decides also means asking who establishes the field
of what is decidable.
Towards a New Way of Thinking About AI Decision-Making
AI does not
create the distributed character of decisions in an abstract space. It forms
part of a “world-environment” (Costa et al., 2023, p. 7) that permeates society
through multiple transformations, including changes in forms of
decision-making.
Although
institutions have always depended on rules, experts, documents, bureaucracies,
and technical instruments, what I see today is, fundamentally, a change in the
scale of that dependence, within a context of technical opacity.
To say that
something is an “AI decision” is to ignore the distribution inherent to that
decision-making process and to help conceal the differences in power and the
human and organizational choices inscribed in the system. But to maintain that
it was “the person” who validated the final output who decided is to ignore the
material and cognitive conditions that frame that validation, as well as the
power structure that gives those conditions their specific form.
Saying that
“multiple actors participated” in the decisional process is correct, but it
must be accompanied by attention to the distribution of functions and to the
actual distribution of decisional capacity.
To think
about decision-making in the age of AI, it is essential to move from treating
the decision as an isolated act to examining the broader decision-making
architecture in which it is embedded. This means reconstructing how the problem
was defined, how the underlying knowledge was selected and produced, what
infrastructure enabled and conditioned the decision, and the framework within
which its results were interpreted. It is also necessary to ask who could
understand and change the decision, and who would be reduced to executing it or
bearing its effects.
The answer
will inevitably be a map of unequal power relations that AI governance must
make visible so that the distributed decisions characteristic of this field can
be politically interpretable, reviewable, and controllable.
References
Amoore, L.
(2020). Cloud ethics: Algorithms and the attributes of ourselves and others.
Duke University Press.
Cobbe, J.,
Lee, M. S. A., & Singh, J. (2021). Reviewable automated decision-making: A
framework for accountable algorithmic systems. In Proceedings of the 2021
ACM Conference on Fairness, Accountability, and Transparency (pp. 598–609).
Association for Computing Machinery. https://doi.org/10.1145/3442188.3445921
Costa, F.,
Mónaco, J. A., Covello, A., Novidelsky, I., Zabala, X., & Rodríguez, P.
(2023). Desafíos de la inteligencia artificial generativa: Tres escalas
y dos enfoques transversales. Question/Cuestión, 3(76), 1–24. https://doi.org/10.24215/16696581e844
Crawford,
K. (2021). Atlas of AI: Power, politics, and the planetary costs of
artificial intelligence. Yale University Press.
Elish, M.
C. (2019). Moral crumple zones: Cautionary tales in human-robot interaction. Engaging
Science, Technology, and Society, 5, 40–60. https://doi.org/10.17351/ests2019.260
Feenberg,
A. (1991). Critical theory of technology. Oxford University Press.
Green, B.
(2022). The flaws of policies requiring human oversight of government
algorithms. Computer Law & Security Review, 45, Article 105681. https://doi.org/10.1016/j.clsr.2022.105681
O’Neil, C.
(2017). Weapons of math destruction: How big data increases inequality and
threatens democracy. Penguin Books.
Sadin, É. (2020). La inteligencia artificial o el desafío
del siglo: Anatomía de un antihumanismo radical (M. Martínez, Trans.). Caja Negra. (Original work published
2018)
Santoni de
Sio, F., & van den Hoven, J. (2018). Meaningful human control over
autonomous systems: A philosophical account. Frontiers in Robotics and AI, 5,
Article 15. https://doi.org/10.3389/frobt.2018.00015
Yeung, K. (2018). Algorithmic regulation: A critical interrogation. Regulation & Governance, 12(4), 505–523. https://doi.org/10.1111/rego.12158