Prohibited AI Practices

Article 5 of the EU AI Act [2] lists all the AI systems that are not allowed in Europe:

• subliminal techniques or purposefully manipulative or deceptive techniques, with the objective or the effect of materially distorting the behavior of a person or a group of persons by appreciably impairing their ability to make an informed decision;

• systems that exploit any of the vulnerabilities of a natural person or a specific group of persons due to their age, disability, or a specific social or economic situation, with the objective or the effect of materially distorting the behavior of that person(s) in a manner that (likely) causes significant harm;

• systems for the evaluation or classification of natural persons based on their social behavior or known, inferred or predicted personal or personality characteristics, with the social score leading to either or both of the following:

  • detrimental or unfavorable treatment in social contexts that are unrelated to the contexts in which the data was originally generated or collected;

  • detrimental or unfavourable treatment that is unjustified or disproportionate to their social behaviour or its gravity;

• systems for making risk assessments of natural persons in order to assess or predict the risk of a natural person committing a criminal offence, based solely on the profiling of a natural person or on assessing their personality traits and characteristics, unless these systems are used only to support the human assessment;

• systems that create or expand facial recognition databases through the untargeted scraping of facial images from the internet or CCTV footage;

• systems to infer emotions of a natural person in the areas of workplace and education institutions, unless these systems are used for medical or safety reasons;

• biometric categorisation systems that categorize individually natural persons based on their biometric data to deduce or infer their race, political opinions, trade union membership, religious or philosophical beliefs, sex life or sexual orientation, except lawfully acquired biometric datasets in the area of law enforcement;

• “real-time” remote biometric identification systems in publicly accessible spaces for the purposes of law enforcement, unless and in so far as such use is strictly necessary for one of the following objectives:

  • the targeted search for specific victims of abduction, trafficking in human beings or sexual exploitation of human beings, as well as the search for missing persons;

  • the prevention of a specific, substantial, and imminent threat to the life or physical safety of natural persons or a genuine and present or genuine and foreseeable threat of a terrorist attack;

  • the localization or identification of a person suspected of having committed a criminal offence, for the purpose of conducting a criminal investigation or prosecution or executing a criminal penalty for offences referred to in Annex II of the EU AI Act and punishable in the Member State concerned by a custodial sentence or a detention order for a maximum period of at least four years.

In particular, the use of real-time biometric identification systems is investigated in Article 5 of the EU AI Act, where it is also specified that a real-time biometric identification system:

• shall be deployed for the purposes set out in that point only to confirm the identity of the specifically targeted individual, and it shall take into account the following elements:

  • the nature of the situation giving rise to the possible use, in particular the seriousness, probability, and scale of the harm that would be caused if the system were not used;

  • the consequences of the use of the system for the rights and freedoms of all persons concerned, in particular the seriousness, the probability, and the scale of those consequences.

• shall comply with necessary and proportionate safeguards and conditions in relation to the use in accordance with the national law authorizing the use thereof, in particular as regards the temporal, geographic, and personal limitations.

As a general norm, the use of such systems shall be authorized only if the law enforcement authority has completed a fundamental rights impact assessment. The EU AI Act envisages some derogation in justified cases of urgency, provided that such authorization is requested within 24 hours. The authorization shall be requested to a competent judicial authority or an independent administrative authority whose decision is binding.

In any case, the use of the “real-time” remote biometric identification systems must be necessary for, and proportionate to, achieving one of the objectives specified above (see last point of the first list in this entry) and remains limited to what is strictly necessary concerning the period of time as well as the geographic and personal scope.

This entry was re-adapted from the Artificial Intelligence Act by Francesca Pratesi and Umberto Straccia.