You know the moment. You've got a Parquet file or a fat CSV, you write the obvious...
You know the moment. You've got a Parquet file or a fat CSV, you write the obvious pd.read_parquet(...), and Python sits there. Then either it finishes in its own sweet time, or you get the message everyone who's done data work in Python has seen at least once:
MemoryError: Unable to allocate ... for an array
Enter fullscreen mode
Exit fullscreen mode
The file is bigger than your RAM, pandas wants the whole thing in memory at once, and your laptop says no.
Python Generators and Iterators: Process Large Data Without Blowing Up Memory When Python...
Why the obvious approach crashes, and how a few generator functions keep memory flat no matter how...
I rebuilt the core ideas behind Parquet — encodings, compression, row groups, projection & predicate pushdown — in ~3,000 lines…
The code that processes a CSV in your tests looks reasonable. Read the file, parse the rows,...
Four commands to look at a new file, then the arguments that stop read_csv guessing: dtype, parse_dates with a format, na_values…
Messy CSV exports are a tax on every data task: stray whitespace, duplicate rows, inconsistent...
