top of page

A Framework for Measuring the Benefits of AI


Introduction 


Significant research has been invested in studying AI risks, a response to the rapid pace of deployment of highly capable AI models across a wide variety of use cases. Over the last year, governments around the world have established AI Safety institutes tasked with developing methodologies to assess the impact and probability of various AI risks. These range from existential threats, such as the potential 'loss of control' over AI systems, to more immediate concerns, including the perpetuation of historical systemic biases.


While the investment and research into AI risks are crucial (Trustible is proud to contribute to the US AI Safety Institute Consortium) focusing only on the potential harms is omitting the other side of the equation when it comes to AI ethics: benefits. The basis for many ethical decisions is that the benefits of using AI for a certain task outweigh the risks. However, you can only make that assessment if the potential benefits of an AI system are explicitly identified and quantified.


While several organizations, including NIST and MITRE, have started to create taxonomies and frameworks to categorize AI risks, we believe it is equally important to apply a similar structured approach to AI benefits. Developing a framework for AI benefits will help organizations conduct AI impact assessments, make more informed ethical decisions, and ensure that they are investing in AI capabilities that can yield measurable benefits for their organizations, users, and society at large. 


In this whitepaper, we outline our framework and taxonomy for measuring the benefits of AI. We include the stakeholder groups considered, the categories of benefits, and an initial list taxonomy labels along with proposed metrics for measuring each one. The goal of this blog post is to enable organizational AI leaders with practical guidance they can use to ethically justify the use of AI and measure their intended benefits. 


Stakeholders


Following our ethical AI framework, our benefits taxonomy focuses on 3 key stakeholder groups: People, Organizations, and Society.


People refers to individuals who benefit directly from an AI system. They may be the ‘user’ of the AI system, such as a student using AI to help them learn a new concept, or the ‘target’ of an AI system, such as when AI is used to help veterans fill out confusing insurance claims paperwork.


Organizations refers to businesses, non-profit groups, government agencies, or any other institution that develops or deploys AI. While individuals at these organizations may personally benefit, we typically think of benefits at this level as more abstract concepts such as increased productivity, improved reputation, or direct financial gains.


Society refers to communities, groups of people, and cultural elements. Societal AI benefits could include things like improved public health, reduced community violence, or improved safety of vulnerable populations. 


Categories of AI Benefits


In addition to identifying each type of beneficiary, we also identify different categories of benefits for each stakeholder group. We typically think of and measure benefits for each group and category differently.


People

Organizations

Society

Physical Health

Impacts to overall physical health of an individual

Financial

Impacts to key financial aspects of an organization including business costs and revenue

Justice

Impacts to crime, violence, and access to justice

Psychological

Emotional or mental impacts to an individual

Customer Satisfaction

Direct impacts to customer satisfaction outcomes

Public Health & Safety

Impacts to public health

Financial

Direct impacts to income or economic standing of an individual

Operational

Impacts to organizational efficiency or ability to scale

Environmental

Impacts to the environment

Privacy

Impacts to protection of personal information

Reputational

Direct impacts to the reputation of an organization

Opportunity

Impacts on access to economic, education, or other opportunities 

Freedom

Impacts to an individual’s freedom and autonomy

Legal

Direct impacts to an organization’s regulatory and civil liability or exposure

Community

Impacts to social interactions and shared experiences

Reputational

Impacts to an individual’s social standing, reputation, or dignity

ESG

Impacts to environmental, social, and governance factors of an organization

Culture

Impacts to key cultural traditions, norms, or values

Opportunity

Impacts to access to economic, educational, or other opportunities




Taxonomy


In our effort to help individuals and organizations develop a pragmatic approach to artificial intelligence, we have developed a comprehensive taxonomy of AI benefits. This taxonomy serves as a structured framework to systematically identify and measure the positive impacts that AI can have on various organizational processes and outcomes. By delineating clear categories and metrics, this taxonomy helps stakeholders understand the full spectrum of advantages AI can offer, facilitating informed decision-making and strategic investments in AI technologies.


Name

Description

Stakeholder

Benefit Type

Reduced Task Time

Reducing the time it takes an individual to perform a discrete task.

Personal

Financial

Reduced Task Cost

Reducing the financial costs to an individual to perform a discrete task.

Personal

Financial

Increased Scalability

Enabling a process to efficiently scale to additional users, sectors, or business units.

Organizational

Operational

Improved Task Accuracy

Increased accuracy or performance for a discrete task.

Organizational

Customer Satisfaction

Product Personalization

Customization or personalization of a product/service that increases usability or user value.

Personal

Psychological

Increased Creativity

Increased access to new ideas, or innovative ways of creating products/services.

Personal

Psychological

Improved Knowledge Discovery

Easier and more accurate access to information necessary to accomplish a task. 

Organizational

Operational

Improved Customer Retention

Increased probability that a customer continues to use a product/service.

Organizational

Customer Satisfaction

Improved Physical Health

Improved physical health outcomes or information about bodily health.

Personal

Physical Health

Increased Diversity

Improved diversity of communities or groups of people.

Societal

Community

Increased Inclusion

Increased acceptance of people with diverse backgrounds.

Societal

Community

Improved Environmental Sustainability

Improved environment including lowered carbon emissions, reduced waste, or using recycled materials.

Societal

Environmental

Repetitive Task Automation

Automating highly inefficient repetitive tasks, particularly those that contribute to harmful bottlenecks or bloated bureaucracy.

Organizational

Operational

Reduced Risk

Reduction in potential for harm, especially potential legal harm.

Organizational

Legal

Improved Learning

Improved educational outcomes or experiences including faster learning rates or academic achievements.

Personal

Opportunity

Improved Accessibility

Increased accessibility in society for people with disabilities.

Societal

Freedom

Improved Organizational Efficiency

Improvements to overall operational efficiency, including helping facilitate cross team processes, reducing organizational friction, or reducing redundancies.

Organizational

Operational

Increased Product Revenue

Direct increases in product/service revenues or contract growth rates over time.

Organizational

Financial

Improved Public Reputation

Improved social standing and reputation with core stakeholders.

Organizational

Reputational

Increased Product Safety

Improved safety features, detection mechanisms, or failsafes.

Personal

Physical Health

Reduced Carbon Emissions

Reduction in energy usage, water consumption, or direct carbon emission outputs.

Societal

Environmental

Increased Access to Education

Improved access to educational services, institutions, or programs.

Societal

Environmental

Reduced Violence

Reduction in criminal activities or physically harmful incidents.

Societal

Justice


תגובות


bottom of page