Short answer: emit one custom counter for each operational failure, put those counters on a small dashboard, and have a Node.js job poll short windows for threshold breaches before handing notifications to an email provider.
For an edtech SaaS, this is a better starting point than turning every application log into an alert. It creates one source of truth for failures in the AI agent loop while leaving latency and cost attribution explicit. The constraint is important: a metric can tell the team that lesson-generation failures spiked, but it cannot explain every failed execution by itself.
Keep less, on purpose.
How do I report a custom failure metric from Node.js?
Report a counter only after the application knows the operation failed. A retry attempt isn't automatically a terminal failure, and counting both would overstate the rate. For an agent loop, the boundary might be the point after its permitted attempts have ended; for a webhook, it might be the point where the delivery policy returns control to the application.
