Benchmarks

pllm.benchmark(...) measures supported native compiled regions and returns an immutable EvidenceReport. pllm.deployment_benchmark(request) validates observations supplied by the caller, but cannot prove where they came from. The CLI also exposes the existing real-role loopback runtime as a non-interactive diagnostic through pllm benchmark run:

pllm benchmark run \
  --model Qwen/Qwen2.5-0.5B-Instruct \
  --prompt-file prompt.txt \
  --max-output-tokens 24 \
  --repetitions 3 \
  --output benchmark.json

The command chooses an ephemeral loopback port, starts the client, preparation, and inference roles, waits for readiness, runs each request, reads the authoritative archive record, and stops the process group. --format json returns a machine-readable CLI envelope and --dry-run starts no services.

Every record identifies whether it covers a compiled region, protocol, deployment, complete model, privacy test, or quality test. Compare runs only when the model, body fingerprint, plan, workload, numeric settings, environment, threads, SIMD settings, warmup, repetitions, and cold or warm mode match.

The CLI report and local dashboard help diagnose a loopback deployment, but they do not create a canonical research evidence record. See metrics and research experiments.

This SDK example requires a PLLM source checkout because it reads the checked-in schemas/fixtures/compile-request.valid.json compile request.

Python SDK example

from pathlib import Path
import pllm

fixture = Path("schemas/fixtures/compile-request.valid.json")
plan = pllm.compile(fixture.read_text(encoding="utf-8"))
report = pllm.benchmark(
    plan,
    weights=bytes([1, 2, 3, 4, 5, 6]),
    input=b"".join(value.to_bytes(4, "little") for value in (7, 8, 9, 10, 11, 12)),
    id="docs-region",
    privacy_cohort="masked-linear",
    numeric_cohort="wrap32",
    environment={"fixture": fixture.name},
    warmups=1,
    repetitions=3,
)
document = report.to_dict()
assert document["scope"] == "region"
assert document["plan_lock_digest"] == plan.plan_lock_digest
assert len(document["samples"]) == 3

API: pllm.compile, pllm.benchmark

This checked EvidenceReport is distinct from the loopback diagnostic JSON.