The right to opacity as a conceptual response to law enforcement’s emotional surveillance under EU law – European Law Blog
In dystopian fiction, like Orwell’s 1984, thought criminals get punished for thinking against the state. In such unwanted scenarios, a thought is consciously manifested in people’s behaviour (e.g., ideas written in a diary); and that behaviour is observed by humans (e.g., the thought-police-officers, who would see that diary and catch the thought-criminal).
Today, AI-emotion-inferring is taking Orwellian surveillance from the humanly observable to the AI-legible; and from thinking to pre-thinking, the pre- or even unconscious.
Emotional surveillance via AI is aimed at interpreting AI-readable data and attributing emotions to them (Bouchagiar). As such, it may entail the black-box-processing of humanly invisible data – like facial micro-patterns that can solely be read by AI – and information consciously or unconsciously disclosed by a person (e.g., fear arising prior to conscious thinking).
This post argues that law enforcement’s emotional surveillance via AI has the potential to be outrageously excessive where it targets humanly invisible and pre- or unconscious emotional data. The post proposes the recognition of the right to opacity of such data, with the objective of making EU law catch up to a world, where the line between inner and outer life may be technologically erased.
Before proceeding to the discussion, it is worth distinguishing between manifested/conscious and unmanifested/unconscious emotions.
On the one hand, manifested emotions are and should be regulated. For instance, judges, deciding on the length of a sentence, may take into account frustration, as disclosed during the commission of a crime. Moreover, the law is, in principle, concerned with conscious, voluntary emotional states. For example, criminal liability requires a voluntary action and a guilty mind (e.g., Weigend; Chiao).
On the other hand, unmanifested emotions have traditionally remained and arguably should remain untouched by regulation. No person is punished for an emotion that s/he felt but never expressed. Similar can be the case with unconscious emotions: for example, one cannot get mistreated for an act committed, whilst in a state of unconsciousness (here, there may be exceptions, where, e.g., one knowingly causes her/his own state of unconsciousness, such as voluntarily taking drugs before driving).
Having drawn a clear line between the untouchable inner and the regulatable outer, let us look at the EU’s permissive approach to emotional surveillance.
The audacity to permit the possibly unreliable inference of an undefined mental state for law enforcement goals
In the past, the access of the police to invisible pre- or unconscious emotional data was impossible. There was, therefore, no need to ban or even regulate this impossibility. Now that such access is technologically possible, we would expect regulators to prohibit it outright. This absolute prohibition would be desirable, because AI may be misused for the establishment of dystopian thought-control-like regimes; and because we, humans, have not yet clearly defined what emotions are (Posner); and because AI-emotion-inferring may be unreliable (e.g., in view of cultural/individual variability, lab-vs-field gaps or bias).
In fact, in its amendments to the European Commission’s initial proposal for what would become the AI Act, the European Parliament (EP) had recommended an unconditional ban on emotion-inferring for law enforcement goals. The logic behind the ban was straightforward: there are major abuse-risks, given ‘serious concerns about the scientific basis of AI systems aiming to detect emotions’, including (1) cultural, individual and contextual variability, (2) ‘limited reliability (emotion categories are neither reliably expressed through, nor unequivocally associated with, a common set of physical or physiological movements)’, (3) ‘lack of specificity (physical or physiological expressions do not perfectly match emotion categories)’ and (4) ‘limited generalisability (the effects of context and culture are not sufficiently considered)’ (see Amendments, recital 26c (new); Article 5(1)(dc new)).
Notably, this ban could be supported by the EP’s proposed general principles that would apply to all AI practices (Amendments, Article 4a), e.g.: ‘human agency and oversight’, requiring that AI be solely used to serve the people and respect dignity (Article 4a(a)); ‘privacy and data governance’, demanding that any AI be compliant with the right to privacy (Article 4a(c)); ‘transparency’, meaning that all AI-tools must be explainable and inform humans on their interaction with AI and its possible impact (Article 4a(d)); and ‘diversity, non-discrimination and fairness’, under which every AI-use should avoid undue discrimination/bias (Article 4a(e)).
Problematically, the final text of the AI Act completely ignored these major abuse-risks that may arise especially in the area of law enforcement, as pointed out by the EP (Amendments, recital 26c (new)). Moreover, the AI Act’s final version accepted only few of the EP’s unreliability concerns (e.g., context-dependence, cultural/individual-variability or error-rates); but this, in areas other than law enforcement (i.e., the workplace and education; see recital 44 of the AI Act). The EP’s general principles, that would govern all AI-practices, were eventually reduced to certain obligations and requirements solely applicable to ‘high-risk’ AI-uses (Chapter III of the AI Act).
What the above means for law enforcement is that, without defining ‘emotions’ and after accepting unreliability concerns, the AI Act sees AI-emotion-recognition as ‘high-risk’ (Article 6(2) and Annex III, para 1(c)). It follows that designers or deployers of AI systems must meet certain obligations/requirements (Articles 8-27 AI Act), which are, however, limited, accompanied by exceptions favouring uses by the police (e.g., transparency obligations to inform on exposure to emotion recognition do not, in principle, apply to criminal investigations; see , Article 50(3) AI Act).
Overall, it is questionable whether the AI Act’s conformity or other assessments could sufficiently protect us from police-abuses that the EP had intended to avoid: misusing the potentially unreliable inference of what we have not yet clearly defined; and this, for law enforcement goals. This was an audacious choice from the part of the EU legislature because it potentially leaves our humanly invisible and pre- or unconscious emotional information in the hands of any ‘petty officer’, who may, then, play with our rights and freedoms (see Olmstead, para 474).
Given that what dystopian imagination required a totalitarian state to achieve, the AI Act gave us through polite exemptions and gentle risk-classifications, it is worth exploring how existing laws could set out a categorical ban, whose unconditionality would be guaranteed by the recognition of a human right to opacity.
Bans are desirable, but not enough
If we accepted that, as Warren and Brandeis (p. 213) argued within their discussion on the right to be let alone, the ‘principle which protects (…) emotions (…) is the right to privacy’, then the regulation of emotional surveillance would be primarily a matter of privacy law.
In the EU, we have the General Data Protection Regulation (‘GDPR’) and the Law Enforcement Directive (‘LED’). But they say nothing about emotions. Still, the European approach to the concept of ‘personal data’ is likely broad enough (Purtova) to cover emotions as inferred via AI. In fact, inferred emotions (e.g., happiness) can, with the aid of additional information (e.g., biometrics), meet the identifiability-criterion, under which it is the capacity to identify someone that makes data personal (FRA and CoE, pp 88-93). Inferred emotions can, indeed, be relied upon to single out a person (on identifiability and the use of identifiers, see Opinion 4/2007, pp 12-15).
Furthermore, emotions as inferred via AI can qualify as highly sensitive personal data. As can be the case with the GDPRand the LED’s special categories of personal data, the context of their processing may ‘create significant risks to the fundamental rights and freedoms’ (recital 51 of the GDPR). Compared to known sensitive data, inferred emotions may be equally intimate (Lench and Carpenter); e.g., shame could be as private as sexual orientation. They may be equally exploitable (Bakir et al), e.g., resulting in manipulation. They may be equally revealing, e.g., showing health-related data (Morie et al). And, importantly, they may be (equally or) more pervasive; e.g., emotional information could be gathered continuously in any context without a concrete collection operation (contrary to other data that demand, e.g., sample-taking) – our mere existence in a watched place can suffice.
In view of the unreliability concerns of AI-inferred emotions, the fact that such inferences may be untrue (for instance, frustration erroneously attributed to a facial expression) does not mean that such falsely inferred data do not constitute sensitive data (see, by analogy, C-492/23 Russmedia Digital, paras 52-53).
In view of these considerations, EU regulators could introduce a new category of highly sensitive personal data to expressly safeguard ‘inferred emotions’, including our humanly invisible and pre- or unconscious aspects. This would be a realistic and immediate response to law enforcement’s emotional surveillance, in the sense that it could come as an amendment to the GDPR (Article 9(1)) and the LED (Article 10) – an amendment that could be deemed necessary by the Commission, in view of latest AI-developments (Article 97(5) GDPR; Article 62(5) LED).
The proposed prohibition-by-default could be accompanied by sector-specific bans in other areas of law. Of particular relevance is criminal law and procedure. If the police inferred our emotions via AI, we would risk being unduly mistreated on the basis of our looks (as read by AI), our involuntary (pre- or unconscious) mental states, our future actions that we may (but also may not) take, as well as erroneously inferred emotions.
In such unfair mistreatment-scenarios, we could bring our case before courts. And judges would make questions on reliability (e.g., can the way in which evidence was gathered create doubts on reliability? did the defence have the opportunity to contest reliability?) or questions on the role (e.g., determinative/non-decisive) that evidence played in the outcome of the case (before the ECtHR, see e.g.: Dragos, para 49; El Haski, para 83; Jalloh, paras 95-96; Schenk, para 48; before the CJEU, see La Quadrature du Net, paras 222-226).
On the question of the weight attached to evidence, inferred emotions could problematically be accepted, if they played an (allegedly) non-decisive role in the outcome of the case. For instance, judges might say they used emotion-inferring for corroboration, in conjunction with other evidential materials, which (other evidential materials) can independently support their decision (see, by analogy, Schenk, para 48). In practice, this would mean that judges might be permitted to consider inferred emotions, insofar as they were informed on AI’s limitations (see, by analogy, experience from the US Loomis-case).
In light of the foregoing considerations, a comprehensive prohibitive framework would, in the context of fair trials, require a ban on all uses of emotional surveillance via AI in the area of law enforcement; so that no inferred emotion gets into the courtroom (or any phase of the criminal proceedings).
An important remark must be made on the above-proposed prohibitive scheme via data protection and other laws. The suggested ban could only work, if it were unconditional and categorical.
More precisely, the recommended sensitive-data-categorisation should not treat emotion-inferring as a processing issue to be handled carefully. The problem is not how law enforcement can process humanly invisible and pre/unconscious emotional data. Rather, the problem is that law enforcement can process such data in coercive contexts. If inferred emotions were merely seen as a sensitive personal data category, they might be subject to balancing exercises and, e.g., be overcome by an (allegedly) sufficiently compelling public interest (see exceptions to the ban on sensitive personal data processing: Article 9(2) GDPR; Article 10(a)-(c) LED). That would be a loophole that the ‘police-must-be-fit-for-the-AI-age’-advocates might use. Hence, inferred emotions should be put ‘higher’ than other sensitive data, because, as this post argued, emotional surveillance can make what dystopian fiction had not imagined reality.
In the same vein, the fair-trials-ban should be unconditional. Any exception to the recommended rule of inadmissibility-by-default might result in unfairness. For instance, consent of the individual concerned would not work as a valid exception; it would be coerced, not freely given, in light of power asymmetry.
In other words, there should be no procedural workaround (e.g., consent or compelling public interest), legitimising the state’s access to the humanly invisible and the pre/unconscious self of a person. The guarantee of the unconditionality of this categorical ban could come with a human-rights-reform.
Concluding with the right to opacity: Opacity belongs to our inner life, not the government’s decision-making processes
In most, if not all, democracies, states must give reasons for their actions. Contrary, we, citizens, owe no justification to no entity for our inner, unmanifested self. Emotional surveillance via AI turns this upside down: the state becomes unintelligible; and citizens get legible. The recognition of a human right to opacity could fix this.
It could be introduced as a new human right, linked to privacy (and other rights/freedoms, like the right to a fair trial); e.g., same as the right to the protection of personal data came as a distinct, enumerated right in the Charter of Fundamental Rights (‘CFR’, Article 8). As such, it would have its own weight and irreducible core that no (allegedly compelling) public interest could surpass. Drawing inspiration from absolute bans (e.g., on torture), as well as inviolable human rights (e.g., human dignity; Article 1 CFR), regulators could define it as the right not to have one’s humanly invisible and pre/unconscious emotions and other mental states inferred or otherwise accessed by the state.
The doctrinal grounding could be linked to the inviolable core of human dignity, an idea long recognised by German constitutional law, for example (Schwartz and Peifer). EU regulators could and should accept that there is a sphere of humanhood so intimate that absolutely no state invasion is permitted, regardless of proportionality and regardless of what technology makes technically possible. Our humanly invisible and pre/unconscious emotional data fall within that sphere and should, in the law enforcement context, remain illegible to anyone and anything.
To conclude, if AI gives us the possibility of accessing what no regulation has ever touched and no practice has ever reached, let us take advantage of this possibility in areas, where abuse-risks would be totally absent and the benefit for the people would be obvious (e.g., use in health to diagnose a disorder, upon explicit consent). But let us be very careful in other areas, where emotional surveillance may give governments access to our humanly illegible and pre/unconscious selves; that is, to the one space that has always naturally belonged to the person alone
