The 33rd annual NeurIPS conference has now wrapped up. By the numbers, NeurIPS has become a behemoth, with over 1,400 papers accepted and around 13,000 people registered. The quickly growing field of Explainable AI (XAI) made a noticeable appearance in this multitude of explainable AI papers and people. Additionally, many papers not geared specifically toward explainability turned out to be quite relevant to the field as well. I attended on behalf of Fiddler, and here are some of my top takeaways.
An increased understanding of dataset shift
Informally, dataset shift is when the process that generates a machine learning model’s data changes. A commonly-used example is the difference that comes from running an image classifier trained on photos taken by professional photographers on smartphone pictures taken by everyday people.
Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift was not a new paper to us, so it was a treat to dive deep at the poster. The tl;dr of this extensive set of experimental results was that when models take in hundreds to thousands of inputs, it can be hard to understand when the world in which your model lives changes. The authors presented compelling experimental results on high-dimensional multiclass classification image datasets (MNIST and CIFAR10) that suggested simple univariate testing of individual model outputs was a tough technique to beat for checking if something has changed in the data distribution.
Some might interpret these results as suggesting that more complex theory is useless, but we take a more optimistic view. Some of the more complex techniques like MMD were not far behind the univariate tests presented, and they may be better suited to a wider variety of settings.
Luckily, I actually went by the poster at the same time as Dougal Southerland, who had just a few days earlier presented some fascinating work on these exact kinds of complex statistical tests. Dougal's talk in the tutorial on Interpretable Comparison of Distributions and Models is a must-watch for anyone interested in the space (Dougal starts around the 40min mark). It was great to see the complementary interaction of theoretical and empirical research in this research challenge, and we're expecting exciting progress in this space in the near future!
More reasons why dataset shift is dangerous
Why is distribution shift such an important concept to pay attention to? Well, just look to Can You Trust Your Model’s Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift. In this work, researchers demonstrated that a number of techniques designed to quantify the confidence of a deep neural network in its predictions are only effective on data that closely resembles the kind of data the model has seen before.
When researchers showed models heavily distorted pictures, most of the tested confidence scoring techniques suggested strong overconfidence in incorrect predictions. This serves as a chilling and important lesson in the dangers of evaluating model only on historical data in cases when the use-case can evolve over time or when the training data may not be representative of the real world. Luckily, not all is lost. The results showed that an ensemble-based approach was actually a pretty robust uncertainty estimation technique able to cope with most of the distribution shifts the authors tested. We are glad that more and more researchers are exploring the very real dangers that come when techniques built around and tested no-distribution-shift datasets are used in a more realistic setting with some distribution shift.
Concept activation vectors get a big usability upgrade
Concept Activation Vectors were introduced when Testing with Concept Activation Vectors (TCAV) debuted at ICML 2018. This technique established a simple, human-friendly approach for probing how deep neural networks reason with respect to concepts that have semantic meaning to humans. For example, TCAV let's us test how a model classifying horses and zebras is sensitive to the concept of "stripes."
However, in the original presentation of the technique, concepts had to be hand-designed by human practitioners. Thanks to Towards Automatic Concept-based Explanations, a follow-up work published at NeurIPS 2019, this has changed. Though the technique is presented for image models, the success of Automated Concept-based Explanation (ACE) in image models suggests that simply searching over patches of inputs may serve as a powerful technique for models consuming other types of unstructured input as well (for example, sequences of text input).
Conclusion
Even in the Explainable AI space, these works represent only a small fraction of the work presented at NeurIPS. Overall, the conference was an incredible meeting of minds, and we at Fiddler are excited to see growing research interest in tackling the kinds of trust and reliability issues that necessitate XAI.
Bonus: video recap of tutorial day
For those interested, here's a quick video recap of the NeurIPS tutorial day. (Around 1:00, I make a point comparing Emti Khan’s VOGN optimizer to ADAM which is phrased a bit confusingly. To clarify, VOGN was benchmarked against ADAM for speed and accuracy on the test set, and it beat ADAM in accuracy, not speed.)