I could have written about another tool I’ve picked up on my data engineering journey, but I found something a bit more fundamental.
Recently, while exploring PySpark and building out a modular ETL pipeline, I caught myself looking at the data and asking:
"Why am I cleaning this? How is this different from the normal cleaning I do in data science?🤔"
I'd already spent plenty of time cleaning datasets for analysis and machine learning. But as I started building production-oriented pipelines, I realized something:
The transformations themselves might look similar, but the problems we're solving are often very different.
