We’re Jillian and Zeke, members of the team at Replicate. In this post, we’ll introduce some improvements we’re making to the way models are documented on Replicate.
What are model cards?
In the world of open-source software, README files have become an indispensable part of every well-documented project. For most users, the README is the first thing you see. It’s the starting point for learning about what a project is, how it works, and whether it’s right for your needs.
In the sub-world of software that is machine learning, the plot thickens: Unlike traditional software, machine learning models are not just source code, but rather the result of a training process that includes both code and training data. With ML software, it’s no longer possible to understand a program by simply “reading the source”.
In 2019, researchers at Google published an academic paper proposing a framework called Model cards. Model cards are “short documents accompanying trained machine learning models that provide benchmarked evaluation in a variety of conditions, such as across different cultural, demographic, [phenotypic, or intersectional groups] that are relevant to the intended application domains. Model cards also disclose the context in which models are intended to be used, details of the performance evaluation procedures, and other relevant information.”
