Cloudera and Vast Data partner to accelerate data delivery and prevent GPUs from starving
Big-data management company Cloudera Inc. said today it’s partnering with storage infrastructure firm Vast Data Inc. on what the companies have termed a “unified AI factory.”
They claim to have designed and built a highly scalable production environment that can be used to continuously ingest, refine, govern and deliver vast volumes of enterprise data that can be used to train and run artificial intelligence systems. It’s a jointly developed infrastructure platform that can run both on-premises and inside cloud environments, giving enterprises the flexibility to deploy AI in the most suitable place based on their performance, cost and compliance needs.
The companies say it’s a timely innovation, because enterprises are scrambling to try and automate more business processes with autonomous AI agents. But as they do so, they’re quickly finding out that their existing data architectures simply aren’t up to the job, because they were never designed to handle the continuous data pipelines needed to funnel massive amounts of information to those AI systems.
As a result, many agentic initiatives have been held back by a problem known as “GPU starvation,” which is where the expensive graphics processing units that power those AI workloads are constantly sitting idle, waiting for the data they need to process. As a result, GPU utilization is much lower than it should be, which means enterprises struggle to pay for the cost of those expensive chips.
