Good problems

We experienced hockey stick growth after we launched the Shop app. We were glued to our dashboards and saw millions of users onboard onto the app. This was gratifying, but we were becoming more nervous as our backend was pushed closer to its limit.

We wrote the backend in Ruby on Rails, and used a MySQL database (Shopify managed system called KateSQL). The first order of business was to identify the bottlenecks. We iterated, horizontally scaling our background job system, caching system, and used a horizontally scaled Message bus where appropriate. We then invested into detecting the usual suspects: slow queries, limited connections etc. We also dropped the “The” from “The Shop app” because it was cleaner.

As MySQL is the primary datastore, this would become the main bottleneck as well. To deal with this, we started off by splitting the primary database into separate parts – Rails makes it easy to interact with multiple databases. We identified groups of large tables that could exist on separate databases, and used GhostFerry to move a few tables onto a new database. We also created new databases for tables in separate domains – this had the added benefit of limiting the blast radius of issues with one domain impacting others.