Python Generators and Iterators: Process Large Data Without Blowing Up Memory
When Python scripts start consuming hundreds of megabytes of RAM, the instinct is often to reach for a faster language or a fancier database. More often than not, the real problem is much simpler: the code loaded an entire dataset into memory at once. Generators—Python's lazy evaluation workhorses—let you process data one item at a time, and they are one of the highest-leverage concepts you can add to your Python toolbox.
The Memory Problem in Plain Sight
Consider a common task: reading a large log file and counting how many lines contain the word "error". The straightforward approach looks harmless:
def count_errors(path):
