科研速览 · Science Skim继续刷下去 · Keep skimming →
◆ Journal of Science Innovation and Technology Research2026-07-31· Computer science

Contamination-Free LLM Routing on LiveBench Reasoning Tasks: Accuracy-Cost-Latency Tradeoff Learning

Grace Xu

原始摘要(英文原文)· Original abstract
Dynamic benchmarks make model selection harder and more useful at the same time: a router must identify when a low-cost policy is reliable without learning benchmark-specific artifacts. This study evaluates prompt-level routing on 200 LiveBench reasoning questions (100 zebra puzzles, 50 spatial questions, and 50 Web-of-Lies questions). Two reproducible, deterministic reasoning back ends are compared. FastLite uses exact reasoning only for recognized low-complexity cases and bounded heuristics otherwise; SolvePro uses full constraint, Boolean, and analytic spatial procedures. A router predicts whether FastLite is safe from task, length, release, constraint, and structural features available before an answer is generated. SolvePro reaches 100.0% exact accuracy, whereas FastLite reaches 67.0% exact and 69.9% answer-component accuracy. Under five-fold out-of-fold evaluation, a logistic router reaches 97.0% exact accuracy with a 32.5% strong-call rate and a 55.0% normalized cost saving relative to always using SolvePro. A transparent complexity rule preserves 100.0% exact accuracy with a 36.5% strong-call rate and 52.7% cost saving. Errors are concentrated in spatial and zebra items, confirming that prompt length alone is insufficient. The findings show that fresh, objectively scored benchmark items can support auditable accuracy-cost-latency routing when features encode verifiable computational structure. Because both back ends are deterministic policies rather than hosted LLM APIs, the reported latency and normalized cost values are controlled experimental quantities, not vendor performance claims.
读原文 · Read the paper ↗

AI 追问PRO

登录后使用 AI 追问

讨论区

登录后参与讨论

相关论文 · Related

Contamination-Free LLM Routing on LiveBench Reasoning Tasks: Accuracy-Cost-Latency Tradeoff Learning — 科研速览 Science Skim