Evaluating whether machines improve on human performance is one of the central questions of machine learning. However, there are many domains where the data is selectively labeled in the sense that the observed outcomes are themselves a consequence of the existing choices of the human decision-makers. For instance, in the context of judicial bail decisions, the outcome of whether a defendant fails to return for their court appearance is observed only if the judge decides to release the defendant on bail. Comparing the performance of humans and machines on data with this type of bias can lead to erroneous estimates and wrong conclusions. In this talk we present a novel framework for evaluating the performance of predictive models on selectively labeled data. We develop a methodology that is robust to the presence of unmeasured confounders (unobservables) and allows us to evaluate the effectiveness of any given black-box predictive model and benchmark it against the performance of human decision-makers. We demonstrate the framework on criminal court judges deciding whether to jail a defendant. We show that machine learning can reduce crime by up to 24.8% with no change in jailing, or reduce jail populations by 42.0% with no increase in crime. Such gains can be achieved while simultaneously reducing racial disparities as well as reducing all categories of crime, including the most violent. We also develop methods to identify reasons for judicial error—judges overfit the unobserved “noise”. These findings suggest that machine learning and prediction tools can be used to understand and improve human decisions.

Bio:
Jure Leskovec is associate professor of Computer Science at Stanford University and chief scientist at Pinterest. Computation over massive data is at the heart of his research and has applications in computer science, social sciences, economics, marketing, and healthcare. This research has won several awards including a Lagrange Prize, Microsoft Research Faculty Fellowship, the Alfred P. Sloan Fellowship, and numerous best paper awards. Leskovec received his bachelor's degree in computer science from University of Ljubljana, Slovenia, and his PhD in in machine learning from the Carnegie Mellon University and postdoctoral training at Cornell University.

More Info: http://www.cs.cmu.edu/calendar/tue-2017-03-21-1630/machine-learning-special-seminar