Tools  |    |  August 25, 2020

Addressing bias in large-scale AI applications: The LinkedIn Fairness Toolkit

Toolkit library created by Sriram Vasudevan and Krishnaram Kenthapadi (work done while at LinkedIn).

The LinkedIn Fairness Toolkit (LiFT) is a Scala/Spark library that enables the measurement of fairness in large scale machine learning workflows. The library can be deployed in training and scoring workflows to measure biases in training data, evaluate fairness metrics for ML models, and detect statistically significant differences in their performance across different subgroups. It can also be used for ad-hoc fairness analysis.

The LiFT library has broad utility for organizations who wish to conduct regular analyses of the fairness of their own models and data. 

  • It can be deployed in training and scoring workflows to measure biases in training data, evaluate different fairness notions for ML models, and detect statistically significant differences in their performance across different subgroups. It can also be used for ad hoc fairness analysis or as part of a large-scale A/B testing system. 
  • Current metrics supported measure: different kinds of distances between observed and expected probability distributions, traditional fairness metrics (e.g., demographic parity, equalized odds), and fairness measures that capture a notion of skew like Generalized Entropy Index, Theil’s Indices, and Atkinson’s Index.
  • LiFT also introduces a novel metric-agnostic permutation testing framework that detects statistically significant differences in model performance (as measured according to any given assessment metric) across different subgroups. This testing methodology will appear at KDD 2020.

LiFT has been open sourced and is available on GitHub. For more details on the library and the various metrics supported, we invite you to take a look at our GitHub documentation for the most up-to-date information.