Replacing Large Language Models with Jev Decision Models for Low-Latency Edge Service Orchestration

arXiv · Delong Li, Xu Wang, Haochen Gong, Rui Lang, Guangsheng Yu · 2026-10-04

This item is no longer on the current shelf. Its page remains available, and editorial corrections may be made. See what is featured now.

Jev slashes edge-orchestration latency by up to two-thirds, offering a cheap, low-latency alternative to LLM-based admission decisions.

What it does

The paper integrates Jev's decision-oriented API into an edge service orchestrator, extracting four to eight bounded intent fields per request. It then runs a shared validator, admission policy, and scheduler while accounting for decision waiting time. Experiments on 8,280 verified requests across 33 test conditions compare Jev against two self-hosted decision models and three hosted LLMs. Across all conditions Jev reduces median decision latency by 22.7-64.5% relative to the fastest LLM and cuts API fees per correct decision by 59.7-80.9% on four-field contracts.

Why it matters

Latency-bound admission is a bottleneck for real-time edge services; shaving off even tens of milliseconds can keep more requests on time. The cost reduction also makes large-scale deployment financially viable where LLM fees would otherwise dominate. By showing that decision models can handle unseen services as accurately as known ones, the work opens a path to hybrid pipelines that keep the best of LLM reasoning while avoiding its latency tail.

How it applies

Practitioners can replace the LLM call in their admission path with a Jev-style validator when contracts are bounded to a few intent fields. The approach works with existing schedulers because the decision latency stays flat regardless of input size or catalog breadth. Caching repeated descriptions yields similar latency, so the gains come from fresh decisions on novel requests.

The short version

When milliseconds matter, a decision model can beat an LLM at its own game.

My take

I think the key takeaway is that we should treat LLMs as one tool in a toolbox, not the default for every decision point. When the contract space is bounded, a lightweight decision model can give you both speed and cost benefits. It also forces us to think harder about how we structure service requests so they’re amenable to such compact validation. In practice, swapping the LLM out for a Jev-style validator could let you scale edge services without over-provisioning compute.

How it connects

• Trend toward hybrid pipelines that combine LLM reasoning with specialized, low-latency components.
• Growing focus on cost-effective AI inference at the edge.
• Shift from monolithic LLM APIs to modular decision services for specific contract types.

Bottom line

Replace LLM admission calls with a bounded decision model wherever you can, and you’ll gain latency and cost headroom instantly.

Takeaways

• Four-to-eight bounded intent fields are enough for Jev to make accurate admission decisions.
• Median latency drops 22.7-64.5% compared to the fastest LLM across 33 scenarios.
• API fees per correct decision fall 59.7-80.9% on four-field contracts.
• Jev maintains 0.91-0.95 exact-on-time request rate where LLMs collapse below 0.1.
• The substitution works even when the service catalog is sent with each request.

Topics: edge computing, latency reduction, decision models, LLM substitution, service orchestration