Preprint March 2026
Benchmarking Multi-Agent LLM Architectures for Home Energy Management: Real-World Tariff Validation and Cross-Model Cost-Efficiency Analysis
- Author
- Besnik Sulmataj
- Institution
- Independent
- besniksulmataj@gmail.com
- Date
- March 17, 2026
- Published
- Zenodo (Preprint)
DOI 10.5281/zenodo.19074522 PDF / Zenodo LinkedIn
- 108 Simulations
- 4 LLMs Benchmarked
- 20%+ Cost Reduction (3 of 4 models)
- $0.005 DeepSeek-V3 API cost per day
Abstract
Summary
This paper benchmarks multi-agent LLM architectures for residential home energy management using a full factorial experiment: four frontier models (Llama 4 Maverick, DeepSeek-V3, GPT-4.1, and Claude Sonnet 4.6), three US utility tariff profiles, three household archetypes, and three random seeds across 108 seven-day simulations at hourly resolution. The central finding is that LLM-driven agents reliably achieve statistically equivalent cost reductions above 20% versus an unmanaged baseline, validating their practical viability for real-world deployment. Tariff complexity proves a stronger model differentiator than household size, with performance gaps widening significantly under real-time pricing structures. When savings performance is statistically equivalent across three models, the deployment decision reduces to cost: a 17x spread in API cost across models makes this a choice about efficiency, not capability.
Methodology
The full factorial.
108 runs = 4 models × 3 tariffs × 3 household types × 3 random seeds
-
4
LLM Models
Llama 4 Maverick, DeepSeek-V3, GPT-4.1, Claude Sonnet 4.6. Open-weight and proprietary frontier models.
-
3
US Utility Tariffs
ComEd Hourly (real-time pricing), PG&E E-TOU-C (time-of-use), and SCE TOU-D-4 (tiered time-of-use with demand charge).
-
3
Household Archetypes
Small suburban (5 kW solar, 10 kWh battery, 60 kWh EV), large suburban (10 kW solar, 20 kWh battery, 100 kWh EV), and apartment (no solar, 5 kWh battery, 40 kWh EV).
-
3
Random Seeds
Each configuration is run three times with different stochastic seeds to assess variance and ensure statistical robustness of reported savings.
Results / Savings vs. Oracle Gap
Three of four clear the threshold.
Figure 1
Axis 0 to 65%
Energy Cost Savings vs. Unmanaged Baseline (%)
20% threshold
Figure 2
Axis 0 to 95%
Oracle Gap, % Above Optimal (lower is better)
Table I / Household Archetypes
The three homes under test.
| Parameter | Small Suburban | Large Suburban | Apartment |
|---|---|---|---|
| Base load (avg kW) | 0.5 | 1.2 | 0.3 |
| EV capacity (kWh) | 60 | 100 | 40 |
| EV daily use (kWh) | 12 | 15 | 8 |
| Solar peak (kW) | 5 | 10 | 0 |
| Home battery (kWh) | 10 | 20 | 5 |
Scroll the table sideways for Large Suburban and Apartment.
Figure 3 / Tariff Generalization
Tariff complexity is the differentiator.
Figure 3
Axis 0 to 100%
Energy Savings by Tariff Profile, Grouped by Model (%)
ComEd Hourly
PG&E E-TOU-C
SCE TOU-D-4
Cost Efficiency & Monthly Savings
Equivalent savings, a 17x spread in cost.
Figure 4
Log x, linear y
Cost-Efficiency Scatter, API Cost vs. Energy Savings
- Llama 4 Maverick $0.001/day · 17.4% savings
- DeepSeek-V3 $0.005/day · 37.8% savings
- GPT-4.1 $0.039/day · 44.8% savings
- Claude Sonnet 4.6 $0.088/day · 49.3% savings
Figure 5
Axis $0 to $100
Monthly Net Savings by Household Type (USD)
Apartment
Small Suburban
Large Suburban
Deployment Recommendations
Which model, for which home.
-
Apartment / No Solar
39.8%
GPT-4.1
Best balance of savings, API cost, and latency in the top tier. Optimal for constrained setups without solar generation or large battery assets.
-
Small Suburban / Moderate Solar
56.0%
Claude Sonnet 4.6
Highest savings in the 5 kW solar configuration. Reasoning depth translates to superior solar arbitrage and EV scheduling under time-of-use tariffs.
-
Large Suburban / High Solar + EV
54.3%
Claude Sonnet 4.6
6.2% oracle gap, closest to theoretical optimum. Handles complex multi-asset coordination (10 kW solar, 20 kWh battery, 100 kWh EV) with the highest fidelity.
Cite This Work
The record is public.
The preprint, the method, and every figure above are published under a DOI, so the numbers can be checked against the source rather than taken on trust.
BibTeX
@misc{sulmataj2026benchmarking,
author = {Sulmataj, Besnik},
title = {Benchmarking Multi-Agent LLM Architectures for Home Energy Management: Real-World Tariff Validation and Cross-Model Cost-Efficiency Analysis},
year = {2026},
publisher = {Zenodo},
doi = {10.5281/zenodo.19074522},
url = {https://doi.org/10.5281/zenodo.19074522}
}