Introduction
Our team had been using CloudWatch Logs as the log storage layer for our identity management system, but as the service grew, the associated cost became a concern. This article describes how we built and migrated to a log storage architecture using Amazon S3 Tables, referred to here as S3 Tables, and the cost optimization results we achieved.
Rising CloudWatch Logs Costs as the Service Grew
Amazon CloudWatch Logs, referred to here as CloudWatch Logs, is an AWS service for storing and analyzing logs. In many common architectures, application logs are stored directly in CloudWatch Logs. When an issue occurs, teams query logs using CloudWatch Logs Insights and configure alerts for specific error logs in combination with CloudWatch Alarms. Our identity management platform followed a similar pattern. Logs from ECS were delivered to CloudWatch Logs through a log router, and we used Amazon Managed Service for Grafana, referred to here as Grafana, for day-to-day development and operations.
As our services grew, the identity management platform, which handles authentication and authorization requests across all services, saw a corresponding increase in request volume and feature scope. As a result, logging costs grew to a level that could no longer be ignored. We had made steady improvements through weekly reviews by removing unnecessary logs and consolidating others, but these efforts alone were not enough to achieve substantial cost optimization. We therefore began considering a more fundamental solution.
