You’ve decided that you need to train a model. You have your company’s data and are gearing up to do some data engineering on it to clean and normalize the data: remove nulls, fill in missing data, dedupe, make sure that enum fields are correct, define a data dictionary, etc. You’ve taken a snapshot of the data and exported it to a non-production database instance.
Once you’ve done that, you’re ready to train a price model based on gradient-boosted trees (or maybe you’re going to fine-tune or train an LLM!). But you need to get the data in a format that’s ready to be used for training. You estimate you’ve got 500GB of training data. What do you do next? You’re going to train this in the cloud because you don’t have an H200 laying around. You’ve decided that a modern data storage format like Parquet is the way to go.
Throw your data into Git?
Like all other developers, you use Git to version control your code. This dataset is going to go through some iterations as you massage the data and clean it up, and other developers on your team will also be working on the data cleanup and preparation. Makes sense to also upload this dataset to Git, right? You’ve uploaded large files to Git and know about Git LFS (Large File Storage).
