Decentralized AI networks face a hidden payments problem: how do you fairly pay nodes for model outputs when there’s often no “correct” answer to check against? DGrid AI’s latest paper — the fourth in its Proof of Quality (PoQ) research series — confronts that gap head-on, publishing a reference-free scoring approach and the numbers behind it.
Why this matters for crypto-native AI
- In decentralized inference systems, independent nodes respond to user queries and are paid based on output quality. Accurate, scalable scoring is therefore central to incentive design and token flows.
- Full cryptographic verification of every computation would be secure but cost-prohibitive at scale. The practical solution so far has been automated evaluation with small “scorer” models — but those usually rely on having a ground-truth answer to compare to, which rarely exists in live, open-ended settings.
What DGrid changed
DGrid found that off-the-shelf options failed as reference-free scorers. For example, an NLI cross-encoder meant to judge entailment produced a negative Pearson correlation (−0.363) when asked to rank answers without a reference — essentially preferring worse outputs over better ones.
Instead, DGrid trained three specialized “judges” to score answers on a 0–10 scale using only the question and response — no correct answer provided. The judges vary in size and latency trade-offs (a fast lightweight model to heavier, slower ones), and were trained in two stages:
1. Pretraining on UltraFeedback, a public dataset of GPT-4-graded responses, to build a broad baseline.
2. Fine-tuning on the network’s own task distribution so the judges align with real query types.
Performance and trade-offs
- On a held-out test set of 300 examples, the DeBERTa-based judge achieved a Pearson correlation of 0.747 against the paper’s ground-truth proxy — notably higher than the prior reference-based evaluators (max 0.647), which did have access to correct answers.
- The authors emphasize the reason: prior systems measured semantic similarity to a reference embedding, while the new judges were optimized end-to-end for the scoring task itself — not an architectural miracle, but a better-aligned optimization objective.
- Important caveat: the ground-truth in these experiments is a proxy metric (token-level word overlap), not human judgment. The judges correlate well with that proxy, but whether that reflects human notions of quality remains unsettled.
Deployment features and costs
- A cascading pipeline routes queries to the lightweight judge first and escalates to heavier judges only when scores are ambiguous. At the most aggressive threshold, this reduces evaluation costs by up to 72.7%, although correlation drops to roughly 0.51 in that configuration.
- An automated online calibration mechanism adjusts signal weights without manual tuning. Over time it identified semantic quality as the dominant signal, raising its weight to about 4.7× the initial setting.
Where it works — and where it doesn’t
- Task breakdowns are revealing: the judges correlate strongly on question-answering (0.830) but perform poorly on summarization (0.199). DGrid attributes the latter to the weakness of the token-overlap metric used during training — raw word overlap is a poor proxy for summarization quality, so models trained against it only learn to track that weak signal.
- The paper calls this metric mismatch the main open problem: improving the ground-truth signal is critical, particularly for tasks like summarization.
Context and tone
DGrid’s approach builds on earlier PoQ layers: latency-aware payouts, adversarial-robustness mechanisms to counter manipulative scorers, and a decomposition of “quality” into inspectable components. Across four papers, the team has methodically closed gaps and reported failures as transparently as successes — treating PoQ more as engineering research than a marketing pitch.
Bottom line
DGrid’s trained, reference-free judges show that you can meaningfully score decentralized model outputs without a ground-truth answer — and do so cost-effectively. But limitations remain: evaluation proxies drive what models learn, and better ground-truth signals (especially for summarization-style tasks) are still needed before PoQ can fully replace reference-based approaches in production decentralized networks.
Disclosure: This content is based on a third-party paper. Neither this platform nor the author endorses any product mentioned. Do your own research before taking action.
Read more AI-generated news on: undefined/news
