Financial markets move faster than human cognition. A geopolitical headline can trigger automated oil liquidations within milliseconds. A single earnings report can wipe out a company’s valuation before a retail trader finishes reading the first paragraph.
I set out to build a production-grade system that could automatically ingest unstructured global financial news feeds, parse the entities affected, determine the sentiment polarity, and expose the results as machine-readable market signals.
This post details the technical architecture of the Market Sentiment API, the data engineering pipeline, and how I solved critical edge cases like LLM cost optimisation and ticker hallucinations.
1. Program Overview
The program processes incoming data through a pipeline designed to minimise LLM token overhead and optimise latency.
