Reading a Paginated API Without Holding the Whole Thing in Memory

Your API hands out 50 records at a time across 400 pages. You need all of them. You do not need them all at once.

Here's a very familiar situation that shows up constantly on the backend. Some API returns data in pages, 50 or 100 records at a time, and you need to walk every page: sync them to your database, export them to a file, run a report. The endpoint gives you a cursor or a page number and you keep asking until there's nothing left.

The way most of us write it the first time looks like this:

async function getAllRecords() {

const all = [];