Running LLM Evals On-Device With MLX: Zero-Cost Batch Judging on a Mac

Short answer: Yes. A quantized 3-8B judge model running on MLX on Apple Silicon can run LLM-as-judge evaluations locally at effectively zero marginal cost. It holds up well on objective, rubric-clear tasks; keep a cloud judge in reserve for subjective quality calls.

I moved most of my offline eval batches off the OpenAI API and onto my Mac this year. The API bill for judging was quietly becoming the most expensive part of my eval loop — every regression run re-judged hundreds of golden cases with a GPT-4-class model. On-device inference made that number go to zero, and for the tasks I actually care about the agreement was close enough to keep.

Why on-device fits eval work

Eval batches have a specific shape that maps cleanly onto local hardware:

• Batch, not interactive — you run them nightly or per-PR, so per-token latency barely matters.
• Latency-tolerant — nobody is waiting on a spinner; a batch can take minutes.
• Cost-sensitive — you re-run the same golden set constantly, so per-call cost compounds fast.
• Privacy-sensitive — traces often contain user data you would rather not ship to a third-party API.

Small evaluators already cost 10-100x less than GPT-4 for objective correctness checks. Running them on-device removes even that residual cost: the marginal cost of one more judge call is electricity.

Where local judges actually hold up

The accuracy claim is not hand-waving. Small fine-tuned evaluators in the Llama 3B/8B class reach 0.88-0.95 accuracy on agentic tasks (Galileo's Luna-2 line). That range is competitive with much larger judges — but it lands there specifically on objective tasks: groundedness, faithfulness to context, and tool-call correctness. Those are where small and local judges are strongest, because the rubric is concrete and the model is checking a fact rather than exercising taste.

Where I still call the cloud:

• Subjective quality — "is this the better of two good answers," tone, helpfulness nuance.
• Long-context reasoning the small model can't hold.
• Anything where I have not yet measured agreement against a trusted judge.

Setup: quantized judge, batched cases, calibration

The workflow is three steps: load a 4-bit quantized judge in MLX, stream your golden set through it, and calibrate against a cloud judge on a sample before you trust it.

from mlx_lm import load, generate
 
# 4-bit quantized 8B judge, downloaded once
model, tokenizer = load("mlx-community/Meta-Llama-3.1-8B-Instruct-4bit")
 
RUBRIC = """You are a strict grader. Given CONTEXT and ANSWER,
return JSON {"grounded": true|false, "reason": "..."}.
'grounded' is true only if every claim in ANSWER is supported by CONTEXT."""
 
def judge(case):
    prompt = tokenizer.apply_chat_template(
        [{"role": "system", "content": RUBRIC},
         {"role": "user", "content": f"CONTEXT:\n{case['context']}\n\nANSWER:\n{case['answer']}"}],
        add_generation_prompt=True, tokenize=False)
    return generate(model, tokenizer, prompt, max_tokens=200)
 
# calibrate first: run cloud judge + local judge on a sample,
# measure agreement (Cohen's kappa). Ship local only if kappa is high enough.
verdicts = [judge(c) for c in golden_set]

The calibration step is the part people skip. Run both judges on a stratified sample of your golden set, compute agreement (Cohen's kappa or simple percent agreement per rubric dimension), and only promote the local judge for dimensions where it agrees. Groundedness usually passes; "overall quality" usually does not. Re-calibrate whenever you change models or rubrics.

Throughput and quantization tradeoffs

The practical tradeoffs on Apple Silicon:

• 4-bit vs 8-bit — 4-bit roughly halves memory and speeds up generation, with a small quality hit. For a binary/JSON verdict task, 4-bit is usually fine; verify it in calibration rather than assuming.
• Model size — an 8B judge is the sweet spot for objective rubrics; 3B is faster and viable if calibration holds, but tends to slip on multi-step tool-call grading.
• Memory — a 4-bit 8B model fits comfortably in unified memory on a 16GB machine, leaving room for the batch runner. 8-bit or larger models want 32GB+.
• Throughput — you are memory-bandwidth bound, not compute bound; expect tens of tokens/sec on an M-series chip. Since verdicts are short (a JSON blob), a few hundred cases finish in minutes.

Run objective eval batches on a local quantized judge for zero marginal cost; keep the cloud judge for the subjective calls and validate agreement before you trust the swap.

Related: more writing.