Workshop on the Interface of Machine Learning and Statistical Inference (18w5054)

Arriving in Banff, Alberta Sunday, January 14 and departing Friday January 19, 2018

Organizers

(Cornell University)

(University Pierre and Marie Curie)

(Stanford University)

(University of Pittsburgh)

Objectives

The intention of this workshop is to bring together Computer Scientists, Statisticians and Mathematicians working at the interface of Machine Learning and formalized statistical theory. In the past few decades, the field of Machine Learning (ML) has produced highly successful methods to obtain prediction functions from large data bases. Methods such as neural networks, random forests and support vector machines have enjoyed a great deal of empirical success at tasks ranging from handwriting recognition to predicting movie preferences to classifying astronomical signals. In contrast to parametric modeling approaches, ML makes few assumptions about the nature of the relationship between covariates and outputs. A consequence of this is that the resulting prediction functions tend to be algebraically complex, acting as "black-boxes" with little scope for human understanding or intuition. A further result is that there has been little attention given to applying statistical approaches to uncertainty quantification to these models. While there is considerable theoretical development for some ML approaches, these focus on minimizing global (ie expected) prediction error, rather than on individual predictions or understanding the relationships between inputs and outputs in these systems.



In spite of this, the last decade has seen the start of a statistical theory of ML. In particular, statistical theory has been developed to produce theoretical results for random forests and related ensemble methods -- some of the most widely used ML procedures. This theory includes results on consistency and asymptotic normality for predictions, enabling confidence intervals to be given for specific predictions for the first time. While these developments represent real advances, they remain incomplete, often only applying to simplified versions of methods used in practice and with poor understanding of extrapolation, convergence rates and other quantities. In addition, other ML methods, including boosting, support vector machines and neural networks do not have the same level of theoretical development.



At the same time, the literature in ML has started exploring model interpretation and interpretability; as evidenced by the Workshop on Interpretable Models at the latest International Conference on Machine Learning. These lie either in the creation of, or search for, algebraically simple models, or in developing interpretable summaries of more complex models. Many of these summaries point toward a mechanistic understanding of the underlying processes which, in turn, suggest the need for statistical uncertainty quantification in the form of confidence intervals and hypothesis tests. Some of these needs can be addressed by the results above, but much work remains to be done.



A third important development has been in the use of ML methods within causal inference. Here, we wish to model a causal effect -- for example the effect of a medical prescription (there are also applications in economics and other areas) -- from observational data, while accounting for biases associated with differences in who gets treated. Machine learning methods now have a dual role: they need to simultaneously provide high-quality non-parametric estimates of both the treatment propensities and the treatment effects.



The workshop will be structured around these three areas, as well as other related advances. Our intention is to mix both theoretical development with practical applications in order to ensure that new inferential tools match practical scientific needs. A final day will be devoted to round-table discussions about future directions and open challenges. We intend to leave plenty of time for informal interaction to stimulate collaboration. Recent theoretical developments in both statistics and computer science make 2018 a particularly suitable time to bring researchers from these fields together, both to share progress and build collaborations across disciplinary boundaries and to map out future directions.



Program:



DAY 1: Interpretability and interpretable models



One primary interface between ML and Statistics has been in model interpretation and interpretability. As noted above, many ML methods result in prediction functions that have complex algebraic expressions and are not easily understood or interpreted. Nonetheless, there has been a small literature on ways to produce interpretable summaries of these models: measures of the importance of individual covariates, of interactions, one dimensional summaries of variable dependence and the like have all be introduced. At the same time, the ML community has started to develop their own literature on interpretable models, as witnessed by a workshop on the topic in this year's International Conference on Machine Learning.



This day will feature new developments in ML interpretation and interpretable models, and will provide an excellent opening into more formalized problems later in the week. Specific problems to be addressed include



- Local versus global summaries of prediction functions



- Searches over model space and model structure



- Interpretation appropriate to specific ML methods and ways to compare between methods.



Both ML and Statistics researchers have been involved in these endeavours and the interpretation of ML models leads to natural statistical questions: variable importance suggests hypothesis tests, graphs of dependence demand confidence intervals etc.



Day 2: Theoretical developments for Random Forests and Other Ensemble Methods



The area in which the development of a statistical understanding of ML methods has been most developed has been for Random Forests and other ensemble methods. These operate by building many ML models on subsets of the data and then combining them. Random Forests are the most well-known example of this approach and this structure has been key to developing statistical theory for these methods.



While these structures have allowed some statistical development, there remains real theoretical holes that need to be filled. Topics for this day include:



- Development of convergence rates for ensemble methods, - Estimation of variance parameters, - Extensions from subsampling techniques to sequentially-estimated models such as boosting.



Day 3: Other developments



While the three themes alluded to above will consume a lot of our time, we have reserved some time for other topics. These include other learning methods, the use of more classical probably approximately correct frameworks for statistical inference and applications. In keeping with BIRS tradition, Wednesday afternoon will be reserved for an outing where we hope this broader set of concerns will stimulate discussion.



Day 4: Causal Inference



The application of machine learning to causal inference presents both considerable challenges and opportunities. As evidenced by a wealth of recent results, naive out-of-the-box applications of machine learning methods to causal problems can lead to biased and misleading results; but these problems can often be fixed by drawing from classical ideas in the econometrics literature.



The focus of our discussion will be on both theoretical and pragmatic questions, including:



- How can classical semi-paramteric efficiency theory inform current developments in machine learning?



- What are the best ways of building adaptive models that highlight causal effects?



- How can we use modern optimization tools to improve classical techniques such as matching?



DAY 5: Roundtable discussions



We will devote the final day to a series of round-table discussions. These will focus on future challenges for theoretical and methodological development. The aim here is to develop a set of directions which are both feasible and practically relevant. We hope this will foster new ideas and collaborations among the participants.