This week, we welcomed Anand Rao to the Responsible AI podcast. With a PhD in AI, an MBA, and over two decades of experience working in AI technology and consulting, Anand brings science and business expertise to his role as the global AI lead for PwC. His responsibilities include consulting over 25 countries on their national AI strategy. We talked with Anand about the conversations he has with his clients and the approaches he uses to help organizations implement AI responsibly.
You can watch the video or stream/download the audio version below. Read on for a summary of our conversation.
What is Responsible AI?
When we asked Anand what responsible AI meant to him, he immediately pointed out that ideally, we won’t need this term for much longer. “Anyone who does AI should be doing it responsibly, right? And I joke with my colleagues: You can’t go and tell a client, we’re doing AI but we’re doing it irresponsibly.”
Terminology aside, responsible AI in Anand’s mind means looking beyond the immediate task that AI is solving. Data scientists have traditionally focused on model accuracy. But teams also need to be thinking about broader questions like how the model works, whether it’s taking in institutional bias or historical bias, whether it is explainable to a regulator, what the process for governance is, and how the model functions in society.
Organizations should think about AI as what people in the academic world call a sociotechnical system. This puts the focus on the interface between humans and technology and asks if we are using technology in the right way. Ethical questions of right and wrong are sometimes translated into regulations, but not always. Being responsible is about how you behave even when there are no laws guiding your actions.
What does “doing the right thing” mean?
In the absence of AI regulations, what standards should companies hold themselves to? It comes down to how you respond to your users. Anand gave the example of a company building a model that decides whether or not to grant a home loan. “If the customer comes in and asks, why was my loan denied, and you’re using a model to come up with that, you need to have an adequate explanation that meets the customers’ criteria,” Anand said. “You can’t just say ‘the algorithm came up with the answer, I don’t know if it was fair or not, I was just using the model.’ That’s not a good defense.”
This might mean telling the customer that you examined five different factors, and they fell short by a certain percentage, and here are the actions they might take to get a better outcome in the future. And different stakeholders might need different explanations. For example, a clinical expert using a model to examine X-rays might want complex charts and metrics on the model’s behavior. The average person applying for a home loan, on the other hand, is probably looking for an explanation that’s easy to read and understand.
Harm mitigation and the five dimensions of AI risk
Implementing responsible AI isn’t black or white, Anand explained. There are gray areas where tradeoffs will need to be made. To help organizations make these strategic decisions, Anand has written an article identifying five dimensions of AI risk:
- Time: Is the risk near-term or long-term?
- Stakeholders: Who benefits from AI, and who is affected?
- Sectors: The financial services and health care industries take on a type of risk that’s different from the average company.
- Use cases: You can’t simply classify risk as “reputational.” It’s important to go deeper and classify what can go wrong in each scenario.
- Sociotechnical: In this dimension, teams should think about how the AI is being used, what the interface looks like, and what the relationship with the user is.
In addition to a risk assessment, a harm assessment can help teams make tradeoffs. The team should ask themselves how many people will be using the model, and how will these people will be affected if something goes wrong. Compare the effect of seeing an ad that’s poorly targeted with being denied a job or a loan. Or even worse, having the wrong assessment made about your health care. Understanding the potential for harm can go a long way toward knowing whether you need to impose a rigorous process or whether you can give your team slightly more room to experiment.
Why too many frameworks can be a bad thing
Right now, according to a global survey that Anand and his colleagues conducted, most organizations are talking about responsible AI, but their actions remain in the “experimentation” stage. “That’s because there is no right tooling, no regulations, too many frameworks, too many documents…[it’s] a lot of work,” Anand said. But regulations are likely coming soon that will force standardization and action, and companies need to get ready.
Why has there been hesitancy? “Companies are loath to get onto yet another new bandwagon,” Anand said, and adopt a new and unproven process. That’s why Anand borrows from well-established systems for handling risk with AI, such as the kind of model risk management used in the financial services industry for around 25 years.
Anand has found that thinking about three lines of defense can be a simple yet effective way for organizations to get started.
- The first line of defense is the people building the system. The data scientists and engineers who implement the model need to have a clear understanding of what they’re building and be on the lookout for risk
- The second line of defense is a compliance group that is close to the engineering team but independent. They check up on what the first line is doing and measure model accuracy, examine training data, perform tests with holdout samples, etc.
- The third line of defense is an internal audit that looks at the process from end-to-end, beginning with model development and continuing into ongoing monitoring.
Why building models is different from building traditional software
With software, once you know that you’ve passed all tests, you can rest assured that the program will perform as advertised until the specification changes. You can’t say the same thing about machine learning models. They depend on data, and if the data changes for some reason—if the training data wasn’t representative of the production data, if user behavior or preferences change, or if the global environment shifts—the model accuracy will change as well. For this reason, Anand explained, models require a much higher level of monitoring.
Tools can help models become easier to deploy, monitor, and scale. Anand said that “so far data science has been more like an artisan shop,” with each data scientist making a custom work of craftsmanship. In the future, Anand predicts that creativity will happen at a larger scale and enable so-called model “factories” to churn out models that are consistent and easier for the average team to use and maintain.
If you found this conversation interesting, we recommend reading Anand’s article Six stage gates to a successful AI governance.
For previous episodes, please visit our Resources section. You can also find us on Clubhouse, where we host a chat and have a club on Responsible AI and ML.
If you have any questions or would like to nominate a guest, please contact us at podcast@fiddler.ai.
