by Abhishek Gupta

Ever got excited about a piece of new machine learning research that you saw come out on arXiv or your favorite research lab’s blog hoping it will finally solve that last bit of optimization you need in your own work that will make you the ML superstar of your team? But after spending days trying to get the same results, you end up failing despite having tried everything in the paper including looking through their Github page, contacting the authors, etc. If this sounds familiar, you’re not alone! Everyday researchers and practitioners alike spend countless hours trying to replicate results from new ML research coming out but inevitably lose precious time and compute resources failing to achieve the required results. We’re facing a massive reproducibility crisis in the field of machine learning. There has been a rise in the ease of use of tools to develop machine learning (ML) based solutions, e.g. AutoML (1) for those with limited ML experience, Keras (2) as a high level API to do more in depth ML work, etc. At the same time, there are a lot more public datasets (3) available, increasingly so to do socially oriented research, e.g. in bias detection, loan approvals, criminal risk score predictions, etc. With more people entering the field coming from diverse trainings (4), it is not necessary that all adhere to rigorous standards of scientific research. This is evidenced by recent calls by the technical research community at conferences like NeurIPS (5). We see that a lack of reproducibility in ML research will be a key hindrance in meaningful use of R&D resources. There is currently a lack of a comprehensive framework for doing reproducible machine learning. We as Pythonistas who love to write well-maintained, up-to-date, Pythonic code can do something to help this! Through my own work in this domain and the work of the intern cohort that worked on the Reproducibility in Machine Learning project this summer at the Montreal AI Ethics Institute, let’s talk through some of the social and technical aspects of this problem and how you can take these principles from the talk today and become the superhero of your ML team elevating the quality of the work coming from your team and helping others build on top of your work - which is something that we pride ourselves in the Python community! We’ll walk through the following principles and apply them to a case study to understand how this simple yet effective mechanism can help address a ton of the issues that we face in the field. Our framework combines existing tooling with policy applied to the following areas: 1) solution design, 2) data collection, 3) model development, 4) data and model legacy, 5) deployment performance tracking Each section provides necessary and sufficient information to reproduce results and guide policy decisions and social changes that rely on such research. Each of these 5 parts comes with a checklist, suggested technical tools and metrics that aid in rigorous scientific development. References: 1) https://cloud.google.com/automl/ 2) https://keras.io 3) https://www.kaggle.com/datasets, https://github.com/awesomedata/awesome-public-datasets 4) https://jfgagne.ai/talent/ 5) “Call for Papers” by Neural Information Processing Systems Conference https://link.medium.com/ZXHwZFkQQV