Explainable AI goes mainstream. But who should be explaining?

Bias in AI is an issue that has really come to the forefront in recent months — our recent blog post discussed the Apple Card/Goldman Sachs alleged bias issue. And this isn’t an isolated instance: Racial bias in healthcare algorithms and bias in AI for judicial decisions are just a few more examples of rampant and hidden bias in AI algorithms.

While AI has had dramatic successes in recent years, Fiddler Labs was started to address an issue that is critical — that of explainability in AI. Complex AI algorithms today are black-boxes; while they can work well, their inner workings are unknown and unexplainable, which is why we have situations like the Apple Card/Goldman Sachs controversy. While gender or race might not be explicitly encoded in these algorithms, there are subtle and deep biases that can creep into data that is fed into these complex algorithms. It doesn’t matter if the input factors are not directly biased themselves — bias can, and is, being inferred by AI algorithms.

Companies have no proof to show that the model is, in fact, not biased. On the other hand, there’s substantial proof in favor of bias based on some of the examples we’ve seen from customers.  Complex AI algorithms are invariably black-boxes and if AI solutions are not designed in such a way that there is a foundational fix, then we’ll continue to see more such cases. Consider the biased healthcare algorithm example above. Even with the intentional exclusion of race, the algorithm was still behaving in a biased way, possibly because of inferred characteristics.

Prevention is better than cure

One of the main problems with AI today is that issues are detected after-the-fact, usually when people have already been impacted by them. This needs to be changed: explainability needs to be a fundamental part of any AI solution, right from design all the way to production – not just part of a post-mortem analysis. We need to have visibility into the inner workings of AI algorithms, as well as data, throughout the lifecycle of AI. And we need humans-in-the-loop monitoring these explainability results and overriding algorithm decisions where necessary.

What is explainable AI and how can it help?

Explainable AI is the best way to understand the why behind your AI. It tells you why a certain prediction was made and provides correlations between inputs and outputs. Right from the training data used to model validation in testing and production, explainability plays a critical role. 

Explainability is critical to AI success

With the recent launch of Google Cloud’s Explainable AI, the conversation around Explainable AI has accelerated. Google’s launch of Explainable AI completely debunks what we’ve heard a lot recently – that Explainable AI is two years out. It demonstrates how companies need to move fast and adopt Explainability as part of their machine learning workflows, immediately.

But it begs the question, who should be doing the explaining? 

What do businesses need in order to trust the predictions? First, we need explanations so we understand what’s going on behind the scenes. Then we need to know for a fact that these explanations are accurate and trustworthy, and come from a reliable source. 

At Fiddler, we believe there needs to be a separation between church and state. If Google is building AI algorithms and also explains it for customers -without third party involvement – it doesn’t align with the incentives for customers to completely trust their AI models. This is why impartiality and independent third parties are crucial, as they provide that all important independent opinion to algorithm-generated outcomes. It is a catch-22 for any company in the business of building AI models. This is why third party AI Governance and Explainability services are not just nice-to-haves, but crucial for AI’s evolution and use moving forward.

Google’s change in stance on Explainability

Google has changed their stance significantly on Explainability. They also went back and forth on AI ethics by starting an ethics board only to be dissolved in less than a fortnight. A few top executives at Google over the last couple of years went on record saying that they don’t necessarily believe in Explainable AI. For example, here they mention that ‘..we might end up being in the same place with machine learning where we train one system to get an answer and then we train another system to say – given the input of this first system, now it’s your job to generate an explanation.’  And here they say that ‘Explainable AI won’t deliver. It can’t deliver the protection you’re hoping for. Instead, it provides a good source of incomplete inspiration’.

One might wonder about Google’s sudden launch of an Explainable AI service. Perhaps they got feedback from customers or they see this as an emerging market. Whatever the reason, it’s great that they did in fact change their minds and believe in the power of Explainable AI. We are all for it.

Our belief at Fiddler Labs

We started Fiddler with the belief of building Explainability into the core of the AI workflow. We believe in a future where Explanations not only provide much needed answers for businesses on how their AI algorithms work, but also help ensure they are launching ethical and fair algorithms for their users. Our work with customers is bearing fruit as we go along this journey.

Finally, we believe that Ethics plays a big role in explainability because ultimately the goal of explainability is to ensure that companies are building ethical and responsible AI.  For explainability to succeed in its ultimate goal of ethical AI, we need an agnostic and independent approach, and this is what we’re working on at Fiddler.

Founded in October 2018, Fiddler's mission is to enable businesses of all sizes to unlock the AI black box and deliver trustworthy AI experiences for their customers. Fiddler’s next-generation Explainable AI Engine enables data science, product and business users to understand, analyze, validate, and manage their AI solutions, providing transparent and reliable experiences to their end users. Our customers include pioneering Fortune 500 companies as well as emerging tech companies. For more information please visit www.fiddler.ai or follow us on Twitter @fiddlerlabs.                  

The never-ending issues around AI and bias. Who’s to blame when AI goes wrong?

We’ve seen it before, we’re seeing it again now with the recent Apple and Goldman Sachs alleged credit card bias issue, and we’ll very likely continue seeing it well into 2020 and beyond. Bias in AI is there, it’s usually hidden, (until it comes out), and it needs a foundational fix.

This past weekend we saw just how quickly the Apple Card, managed by Goldman Sachs, issue spiralled out of control. What started as a tweet thread with multiple reports of alleged bias (including from Apple’s very own co-founder, Steve Wozniak and his spouse), eventually led to a regulator opening an investigation into Goldman Sachs and their algorithm-prediction practices.

Apple Card from Apple and Goldman Sachs

The problem

If we dig into the allegations to find the source of the problem, two issues stand out:

  1. The algorithm making credit decisions for the Apple card is biased
  2. The customer support teams from Goldman Sachs and Apple had zero insight into how the algorithm worked when they were asked to explain certain decisions

For (1) above we saw multiple responses to the original tweet thread that corroborated the allegation. Multiple people had faced a similar outcome where all other input factors being the same (or in some cases higher: like a higher annual income or credit score), and gender being the only difference, they were given significantly lower credit limits than their male spouse. This definitely comes across as problematic. Following the allegations, a separate group of non-related women and men ran a test experiment to check for bias and noticed significant differences in credit limits. Men with bad credit scores and irregular income got better offers than women with high incomes and good credit scores. 

Was the algorithm biased against women? We can’t say for sure because we don’t know. This is the key – we don’t know what’s going on inside the algorithm to analyze the root-cause. But based on external outcomes, we’re guesstimating that this is likely what happened.

Impact of this problem

The primary issue here is with the black-box algorithm that generated Apple’s credit lending decisions. As laid out in the tweet thread, Apple Card’s customer service reps were rendered powerless to the algorithm’s decision. Not only did they have no insight into why certain decisions were made, they were unable to override it. 

Humans don’t want a future ruled by algorithms, especially biased ones. Algorithms permeate all aspects of our lives today from lending and housing decisions to decisions in criminal justice. If we continue to let algorithms operate the way they do today, in the black-box and without human oversight, it results in a dystopian view of the world where unfair decisions are made by unseen algorithms operating in the unknown.

The solution

How could this issue have been avoided or at least handled better? Let’s come back to the initial statement above on how bias in AI needs a foundational fix. It’s very likely that in this particular credit lending decision, the algorithm was trained on biased data to begin with. To better understand this, let’s look at the high-level lifecycle of an AI solution: 

  1. Identify a use case for AI (credit lending, criminal justice, cancer prediction, etc.)
  2. Access historical data to build models
  3. Import this data to train and test models
  4. Test and validate models 
  5. Deploy models into production 
  6. Monitor models in production to ensure optimal performance 

If the data is flawed to begin with, this flaw permeates into everything that an algorithm does going forward. What we need is a way to check for bias and other issues in both data and models through all stages of the AI lifecycle. What we also need is human oversight – AI is just simply not ready to function on its own. We need humans-in-the-loop who will ensure that AI is functioning as it should. 

In the Apple credit card example above, it’s likely this issue could have been avoided if humans had visibility into every stage of the AI lifecycle. They could have seen examples in the test and validate stage of how the model was behaving when a certain input factor was isolated and compared with the global dataset. They could have also had the ability to override an algorithm’s prediction in the test/validate stage if they felt it was unfair or incorrect. This would have resulted in an algorithm that was getting trained in the right way to produce accurate results when in production. 

How Fiddler solves for this

This is exactly what we’re addressing at Fiddler. We’re working to unlock the AI black-box and empower all relevant stakeholders with more visibility into their AI than exists today. We’re working on infusing visibility and insight with explainable AI into every stage of an AI solution’s lifecycle: right from the data and training of it to when a model is deployed and in production.

  • We dive into the details to explain each prediction a model makes – whether a training, test, or production model – so users can understand the why behind individual decisions.
  • Our goal is to empower users with easy to grasp explanations of AI decisions. This empowers different stakeholders in an organization: data scientists are empowered to build the best and most accurate models, risk officers are empowered to publicize models with minimal-risk, and customer support representatives are empowered to answer customer questions around the why behind decisions.
  • With Fiddler’s explainable AI built into the AI lifecycle, it helps teams ensure they are compliant with regulations and are protecting their algorithms from inherent and hidden bias.

We’re continuing to build capabilities into Fiddler to ensure explainability is infused throughout the AI lifecycle and are working with a variety of customers to  build this functionality into their existing and new models. If you’re interested in working with us, please reach out.