Security
Contents
Security#
In brief#
The goal of security encompasses the protection of several operational dimensions of an AI system when confronted with possible attacks, trying to take control of the system or having access to design, operational or personal information. A secure system is capable of maintaining the integrity of the information that constitutes it. This includes protecting its architecture from the unauthorised modification or damage of any of its component parts. A secure system also keeps confidential and private information protected even under hostile or adversarial conditions. 1
More in detail#
Security must be an integral part of the AI process. Protecting AI systems, their data and their communications is critical to the security and privacy of users, as well as protecting business investments. The AI systems themselves are incredibly expensive and possess valuable intellectual property to protect against disclosure and misuse. The confidentiality of the program code associated with AI systems may be considered less critical, but access to it, as well as the ability to manipulate this code, can result in the disclosure of important and confidential assets.
Several kinds of attacks against AI systems have been reported. Currently, the most prominent attack vector categories are [2]: adversarial inputs [3]; data poisoning attacks [4]; model stealing techniques [5, 6]; model poisoning [7], data leakage [8] and neural network Trojans [9], among others. Attack vectors directed against the AI systems’ deployment or training environment are equally applicable. These may be attack vectors directed against servers, databases, protocols or libraries utilised within the AI system [10].
Currently, AI systems often lack sufficient security assessments [11]. This may be the result of the mutually independent development of AI methods and their implementation in applications: while the application should have a security assessment, embedded AI (via APIs or frameworks) is rarely considered in terms of its security vulnerabilities by application developers and/or practitioners. While AI developers may follow coding standards and guidelines for secure software development, they will not assess the potential attack surface of an AI system (i.e., the means by which an attacker may enter, extract data or manipulate the system in question) using the system.
Bibliography#
- 1
Leslie David. Understanding artificial intelligence ethics and safety. The Alan Turing Institute, 2019. URL: https://doi.org/10.5281/zenodo.3240529.
- 2
Xiaofeng Liao, Liping Ding, and Yongji Wang. Secure machine learning, a brief overview. In 2011 Fifth International Conference on Secure Software Integration and Reliability Improvement-Companion, 26–29. IEEE, 2011.
- 3
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
- 4
Avi Schwarzschild, Micah Goldblum, Arjun Gupta, John P Dickerson, and Tom Goldstein. Just how toxic is data poisoning? a unified benchmark for backdoor and data poisoning attacks. In International Conference on Machine Learning, 9389–9398. PMLR, 2021.
- 5
Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction APIs. In 25th USENIX security symposium (USENIX Security 16), 601–618. 2016.
- 6
Raül Fabra-Boluda, Cèsar Ferri, José Hernández-Orallo, Fernando Martínez-Plumed, and María José Ramírez-Quintana. Identifying the machine learning family from black-box models. In Conference of the Spanish Association for Artificial Intelligence, 55–65. Springer, 2018.
- 7
Minghong Fang, Xiaoyu Cao, Jinyuan Jia, and Neil Gong. Local model poisoning attacks to Byzantine-Robust federated learning. In 29th USENIX Security Symposium (USENIX Security 20), 1605–1622. 2020.
- 8
Panagiotis Papadimitriou and Hector Garcia-Molina. Data leakage detection. IEEE Transactions on knowledge and data engineering, 23(1):51–63, 2010.
- 9
Yu Ji, Zixin Liu, Xing Hu, Peiqi Wang, and Youhui Zhang. Programmable neural network trojan for pre-trained feature extractor. arXiv preprint arXiv:1901.07766, 2019.
- 10
Kim Hartmann and Christoph Steup. Hacking the ai-the next generation of hijacked systems. In 2020 12th International Conference on Cyber Conflict (CyCon), volume 1300, 327–349. IEEE, 2020.
- 11
Why ai needs security. https://www.synopsys.com/designware-ip/technical-bulletin/why-ai-needs-security-dwtb-q318.html, 2020.
This entry was written by Jose Hernandez-Orallo, Fernando Martinez-Plumed, Santiago Escobar, and Pablo A. M. Casares.