Artificial Intelligence (AI) is one of the most disruptive technologies of this century. AI is now delivering significant economic and social value across many industries. In addition, a growing number of companies and organizations are deploying AI in critical infrastructures. These include healthcare, finance, transportation, and energy systems. It is therefore important that AI systems meet a set of criteria that make them sufficiently trustworthy and minimize the risk of material or immaterial harm. This is where Trustworthy Artificial Intelligence comes in, addressing the development of such AI systems.
In this article, you’ll learn the basics of Trustworthy AI and why it’s important. You’ll also explore the different subfields of Trustworthy AI.
Overview
This blog post is divided into the following sections:
- What is Trustworthy Artificial Intelligence?
- Why is Trustworthy Artificial Intelligence important?
- What are the subdomains of Trustworthy Artificial Intelligence?
What is Trustworthy Artificial Intelligence?
The field of Trustworthy AI is concerned with the development of AI systems that comply with legal regulations and ethical principles. In addition, trustworthy AI systems are characterized by four important properties:
- The AI system achieves good performance.
- The AI system is highly reliable.
- The AI system can interact with humans.
- The purpose of the AI system is aligned with the needs of society.
The field of Trustworthy AI is therefore not only concerned with the development of AI algorithms. It also encompasses the entire lifecycle of AI systems: from problem specification to data aggregation and processing to evaluation and monitoring of AI systems. Therefore, project managers should not only involve the development team in the development process of a trustworthy AI system. Rather, the requirements and needs of various stakeholders who will need to trust the AI system in the future should also be considered in the development of the AI system. These stakeholders may be:
- Public or private organizations that will use the AI system as part of their business processes.
- End users who will work directly or indirectly with the AI system.
- Regulators and policy makers who will review the AI system and decide on its compliance.
- The general public who will be directly or indirectly affected by the decisions of the AI system. (e.g., credit applicants whose creditworthiness will be evaluated by the AI system)
Why is Trustworthy Artificial Intelligence important?
In the past, when developing AI systems, companies and organizations typically only considered the performance of the AI algorithms when evaluating the systems. However, it is important to include indicators other than performance to evaluate the trustworthiness of AI systems. This can minimize the risk of AI systems causing economic or societal harm, such as discrimination, social inequality, or even loss of life.
In addition, legal or regulatory requirements may demand that AI systems be trustworthy. For example, on April 21, 2021, the European Union published the Artificial Intelligence Act, the first legislative proposal to strengthen the regulation of AI systems. The trustworthiness of AI systems is also necessary for them to be accepted by society and used by individuals and businesses.
“A decision aid, no matter how sophisticated or ‘intelligent’ it may be, may be rejected by a decision maker who does not trust it, and so its potential benefits to system performance will be lost.”
– Bonnie M. Muir, psychologist at University of Toronto
What are the subdomains of Trustworthy Artificial Intelligence?
The field of Trustworthy AI can be divided into 7 sub-areas: generalization, security, fairness, explainability, transparency, privacy, and accountability (see Figure 1). These 7 sub-areas are discussed in more detail below.
Generalization
An AI system has good generalizability if it performs correctly on a variety of different input data and in a variety of different situations, even if they were not included in the training data. In practice, however, AI systems often suffer significant performance degradation over time due to changes in the underlying data distribution. The distribution of input data in the productive operation of the AI system then no longer matches the data distribution of the training data to which the AI algorithm was originally adapted.
Especially for critical infrastructure applications, the performance degradation of an AI system due to lack of generalization can lead to various security risks. For example, AI systems in autonomous vehicles must be able to safely control the vehicle under different weather conditions (such as sun, snow, and rain). In the worst case, if an AI system fails to perform reliably in a given situation, it can put human lives at risk.
Increasing the robustness of AI systems to changes in data distribution can be achieved by regularly retraining the AI algorithm.
Security
Similar to software systems, AI systems must be tested for vulnerabilities that attackers can exploit. Otherwise, there is a risk that hackers could gain access to an AI system and steal or modify important data. This could cause the AI system to make incorrect decisions or even fail. Therefore, AI systems, especially in safety-critical areas, need to be highly resilient and robust against attacks.
However, traditional cybersecurity approaches are not sufficient. For example, AI algorithms (especially deep learning algorithms) are typically highly vulnerable to adversarial attacks. Adversarial attacks are attacks on AI systems that manipulate either the training data or the input data during production operation. The goal of the attacks is to degrade the performance of an AI system. In addition, attacks on AI systems can also be aimed at stealing confidential information in the training data or information about the underlying AI algorithm.
Therefore, it is important to take various measures to detect attacks on AI systems early and increase the resilience of AI systems.
Fairness
Another important aspect of Trustworthy AI is fairness. AI systems must be prevented from disadvantaging certain groups or individuals by making systematically unfair decisions, thereby causing negative social effects. For example, an AI system that makes hiring decisions should not systematically favor male candidates and discriminate against female candidates.
In practice, however, the data sets used to train AI systems often contain biases. Such biases can negatively affect the fairness of AI systems. They can lead to unintentional discrimination against certain groups of people, increasing the exclusion of minorities and disadvantaged groups.
For this reason, datasets should be screened for discriminatory biases at the collection stage and cleaned if necessary. Hiring people from different social, cultural, and disciplinary backgrounds can also help. This increases the diversity of opinions within an AI development team and improves the detection of discriminatory bias.
Explainability
Complex AI algorithms such as deep neural networks are often black-box models. This means that it is not possible for humans to understand the decision-making process of these AI algorithms. Moreover, it is very difficult for humans to trust an AI system if they do not understand the reasons behind its decisions. A solution is offered by Explainable AI methods. They make AI systems more understandable to humans by generating appropriate explanations for the behavior of AI systems.
It is particularly important that the explanations are adapted to the knowledge and needs of the addressed stakeholders. For example, a layperson needs different explanations than an employee of a regulatory agency. In the example of an AI-based lending system, a loan applicant needs simple explanations of why his loan application was rejected. A regulator, on the other hand, is less interested in explanations for individual decisions made by the AI system. Rather, the regulator needs global and detailed explanations of the entire decision-making process of the AI system in order to verify its compliance with regulations.
Transparency
Making an AI system transparent requires the disclosure of a wide range of information about the entire lifecycle of the AI system. In particular, information about the purpose of the model, the origin of the data, and the individual development steps must be recorded and published. This includes documentation of the training data sets used, including data collection and labeling, as well as documentation of the AI algorithms deployed. In addition, various metrics that quantify the trustworthiness of the AI system (including its security, fairness, and explainability) should be published. This should also include information about potential uncertainties in these metrics.
This information should be openly communicated in the form of reports and fact sheets. The presentation and level of detail of the information will depend on the target audience. For example, companies are primarily interested in information that demonstrates that their AI systems are sufficiently accurate and robust to attack. This helps companies manage the risks of AI systems. Regulators, on the other hand, need information to verify that an AI system complies with all policies and regulations.
Data and privacy protection
Another important aspect of the trustworthiness of artificial intelligence is maintaining data protection and privacy throughout the lifecycle of an AI system. AI systems must not make unauthorized use of data that could directly or indirectly identify an individual or household. In addition, it must be ensured that neither the training data nor the information generated by an AI system can be used to draw conclusions about specific individuals or households. This particularly applies to personal information such as name, age, gender, sexual orientation, ethnic origin, religious or philosophical beliefs.
Government regulations, such as the European Union’s General Data Protection Regulation (GDPR), require companies to take steps to protect data and privacy. To prevent data breaches or privacy intrusions, various protection techniques can be used in an AI system’s data collection, management, and processing. These include methods such as data anonymization and federated learning.
Accountability
To enable regulation and audit of AI systems, mechanisms must be in place to ensure accountability for AI systems and the decisions they make. Internal and external auditors must be able to assess whether the algorithms, data, and design processes underlying an AI system comply with all applicable regulations and laws. In particular, AI systems in critical infrastructure and AI systems whose decisions may violate the fundamental rights of certain groups of people should be able to be independently audited at any time.
Accountability is also closely related to the transparency of an AI system, as a transparent AI system greatly facilitates auditing.
Further reading
In this section you will find further reading to help you learn more about Trustworthy Artificial Intelligence.
Books
Scientific publications
Articles
Summary
In this post, you learned about the different subfields of Trustworthy AI and why they are important.
Specifically, you learned:
- Trustworthy AI is about developing AI systems that meet the needs of society while complying with laws and ethical principles.
- Trustworthy AI is important to minimize the risk of AI systems causing economic or social harm.
- Trustworthy AI encompasses 7 important sub-areas: generalization, security, fairness, explainability, transparency, privacy, and accountability.
Do you have any questions?
Feel free to post your questions in the comments below and I’ll do my best to answer them.
P.S.: Of course, I also welcome any constructive feedback on this blog post 😊
Hi, my name is René Heinrich. I am a data scientist doing my PhD in the area of trustworthy artificial intelligence. On this blog I share my experiences and everything I have learned on my own knowledge journey.
