This post was written by the data engineering team at Thrive Learning. The views and experiences expressed are their own.
Before Snowflake, Thrive's legacy analytics ran on a mix of scheduled batch jobs and MongoDB change streams triggering refreshes on a schedule. When the team moved to Snowflake, they built on Dynamic Tables from the start, driven by a core requirement to sync and transform platform data continuously as changes arrived, rather than in delayed batches.
That freshness isn't a back-office nicety; it's part of the product. Thrive's Analyse suite (the ability for customers to explore all their learning data) is a core part of what the Thrive platform offers. We've also noticed that better performance and data freshness translate directly into customer satisfaction and competitive strength in sales bids. Customers don't buy analytics as a separate line item, but its quality directly influences whether Thrive wins and retains business. More recently, that same data and Dynamic Table foundation underpins Thrive's first paid add-on product, Analyse with AI, which layers AI on top of the data pipelines at hand.
In this post, we'll share how we improved price-performance for a Dynamic Table using new custom incrementalization capabilities. Rob Howe, a senior data engineer at Thrive, led this work. He identified that custom incrementalization was a great fit for the challenges we were facing and came up with this solution, tested it and got it deployed to production within a couple of weeks.
