Hello everyone! This is the second article about the development of RUSEON-core, a Zero-Copy video streaming server for AI platforms and Edge video infrastructure. In the first article, I talked about the fundamental reason why we decided to create our own server in the first place. I also covered the main problem with most similar solutions — the "thundering herd" — and how we managed to squeeze out 8 Gbps on a single CPU core. By the way, I forgot to mention in that article that besides simple streaming, we also record the streams in fMP4 format. It’s stored locally for N amount of time, and it can fly off to an S3 bucket (depending on how long the clients want to keep the recordings).
This article is precisely about a non-obvious (well, at least to me, maybe for someone else it's an everyday thing) problem related to data storage and its specifics across all Operating Systems. So, let's dive in.
We rolled out our first release to production (100 cameras), made the clients happy, and started working. About an hour passed, and the alerts started flying. I SSH into the server, open htop, and see there's only 100 MB of free RAM. Uh-oh. I should clarify that the production server had 32 gigs of RAM. The expected behavior was that the CPU is chilling, the network card is chewing through the traffic, RAM usage is around 250-300 MB, and the disks are not heavily loaded. So, when you see numbers like that in htop, you start blaming yourself and your crooked hands that wrote this piece of "garbage". But still, we decided to go to Google, ChatGPT, and the like. Fortunately, the answer was found quickly, and we stopped beating ourselves up.
