A lightweight two-stage approach for parsing, preprocessing and storing semi-structured home automation data
The full configuration example is available on GitHub.
Home automation systems tend to generate a surprisingly heterogeneous collection of logs and measurements.
In this project, most data originates from Domoticz, but additional information is produced by ESP8266/D1 devices, alarm scripts, monitoring tools and other IoT components. Some records are plain text, some contain timestamps and metadata, while others contain JSON embedded inside a log line.
Rather than forcing every producer into a single format, I use a two-stage processing architecture based on Fluent Bit and Elasticsearch.
