A peer-reviewed paper accepted to the 6th International Artificial Intelligence and Blockchain Conference (AIBC 2025) argues that idle consumer GPUs, exemplified by Nvidia’s RTX 4090, can meaningfully reduce the cost of running large language model inference when used alongside traditional datacenter hardware.
Titled Idle Consumer GPUs as a Complement to Enterprise Hardware for LLM Inference, the study from io.net is the first to publish open benchmarks of heterogeneous GPU clusters on the project’s decentralized cloud. The analysis compares clusters of consumer cards against datacenter-grade H100 accelerators and finds a clear cost-performance tradeoff that could reshape how organizations design their inference fleets.
According to the paper, clusters built from RTX 4090 GPUs can deliver between 62 and 78 percent of the throughput of H100s while operating at roughly half the cost. For batch workloads or latency-tolerant applications, token costs fall by as much as 75 percent. The researchers underscore that these savings are most compelling when developers can tolerate higher tail latencies or use consumer hardware for overflow and background tasks such as development, batch processing, embeddings generation and large-scale evaluation sweeps.
Aline Almeida, Head of Research at IOG Foundation and Lead Author of the study, said, “Our findings demonstrate that hybrid routing across enterprise and consumer GPUs offers a pragmatic balance between performance, cost and sustainability. Rather than a binary choice, heterogeneous infrastructure allows organizations to optimize for their specific latency and budget requirements while reducing carbon impact.”
Hybrid GPU Fleets
The paper does not shy away from H100s’ strengths: Nvidia’s datacenter cards sustain sub-55 millisecond P99 time-to-first-token performance even at high load, a boundary that keeps them indispensable for real-time, latency-sensitive applications such as production chatbots and interactive agents. Consumer GPU clusters, by contrast, are better suited to traffic that can tolerate extended tail latencies; the authors point to a 200–500 ms P99 window as realistic for many research and dev/test workloads.
Energy and sustainability are also part of the calculus. While H100s remain roughly 3.1 times more energy-efficient per token, the study suggests that harnessing idle consumer GPUs can lower the embodied carbon footprint of compute by prolonging hardware lifetimes and leveraging grids that are rich in renewable generation. In short, a mixed fleet can be both cheaper and greener when deployed strategically.
Gaurav Sharma, CEO of io.net, said, “This peer-reviewed analysis validates the core thesis behind io.net: that the future of compute will be distributed, heterogeneous, and accessible. By harnessing both datacenter-grade and consumer hardware, we can democratize access to advanced AI infrastructure while making it more sustainable.”
Practical guidance from the paper is aimed squarely at MLOps teams and AI developers. The authors recommend using enterprise GPUs for real-time, low-latency routing while routing development, experimentation and bulk workloads to consumer clusters. They report an operational sweet spot in which four-card RTX 4090 configurations hit the best cost per million tokens, between $0.111 and $0.149, while delivering a substantial portion of H100 performance.
Beyond the benchmarks, the research reinforces io.net’s mission to expand compute by stitching together distributed GPUs into a programmable, on-demand pool. The company positions its stack, combining io.cloud’s programmable infrastructure with io.intelligence’s API toolkit, as a full solution for startups that need training, agent execution and large-scale inference without the capital intensity of buying solely datacenter hardware.
The full benchmarks and methodology are available on io.net’s GitHub repository for those who want to dig into the numbers and reproduce the experiments. The study adds an important, empirically grounded voice to the debate about how to scale LLM deployments affordably and sustainably in the years ahead.
