Our real-time data pipeline was failing due to mysterious throughput limits. The culprit? A little-known Kinesis Data Firehose limitation that has nothing to do with shards or provisioned throughput. In this scenario, we expose the undocumented partition key limit that nobody talks about.

Introduction to Kinesis Data Firehose

Kinesis Data Firehose is a fully managed service that captures, transforms, and loads data into Amazon S3, Amazon Redshift, Amazon Elasticsearch, and Splunk. It's commonly used for real-time data processing and analytics.

import { PutRecordCommand } from '@aws-sdk/client-kinesis';

import { KinesisClient } from '@aws-sdk/client-kinesis';