pllm benchmark run

Examples

Run a fast transport smoke test

Generated tiny weights exercise the real local role topology without downloading a checkpoint.

pllm benchmark run --tiny --max-output-tokens 1 --repetitions 1

Measure a Python experiment

Resolve a typed Experiment, then run its real client, preparation, and inference roles.

pllm benchmark run --experiment examples/benchmarks/qwen_prepared.py:cpu_4 --trust-python --max-output-tokens 8 --repetitions 1

Build a benchmark experiment with a factory

Factory targets construct the typed experiment before the benchmark driver resolves its profile and role topology.

pllm benchmark run --experiment examples/composition.py:build_experiment --factory --trust-python --max-output-tokens 8 --repetitions 1

Compare two matched experiments

Repeated --experiment options run candidates sequentially; --save-best exports a winner only when their measured cohorts are comparable.

pllm benchmark run --experiment examples/benchmarks/qwen_prepared.py:cpu_1 --experiment examples/benchmarks/qwen_prepared.py:cpu_4 --trust-python --max-output-tokens 8 --repetitions 3 --save-best best-experiment.json

Measure declarative configuration

A YAML experiment follows the same resolution and execution path as its Python form.

pllm benchmark run --experiment examples/benchmarks/qwen-prepared-cpu-4.yaml --max-output-tokens 8 --repetitions 1 --output benchmark.json

Options

usage: pllm benchmark run [-h] [--format {human,json,jsonl}] [--quiet]
                          [--no-color] [--no-input] [--dry-run]
                          [--experiment TARGET] [--factory] [--trust-python]
                          [--model MODEL] [--model-id MODEL_ID] [--tiny]
                          [--prompt PROMPT | --prompt-file PROMPT_FILE]
                          [--max-output-tokens MAX_OUTPUT_TOKENS]
                          [--warmups WARMUPS] [--repetitions REPETITIONS]
                          [--timeout TIMEOUT] [--show-dashboard]
                          [--output OUTPUT] [--save-best PATH] [--force]

options:
  -h, --help            show this help message and exit
  --format {human,json,jsonl}
                        result format (default: human)
  --quiet               suppress non-error diagnostics
  --no-color            disable colored output
  --no-input            fail instead of prompting
  --dry-run             report work without persistent writes or service
                        startup
  --experiment TARGET   Experiment .json/.yaml or explicit Python
                        path.py:object/module:object; repeat to compare
                        pipelines
  --factory             call each Python target as a zero-argument factory
  --trust-python        allow import and execution of explicit Python
                        experiment targets
  --model MODEL         Hugging Face model ID or local checkpoint path
  --model-id MODEL_ID   stable model identity stored in the report
  --tiny                use generated random weights for a transport smoke
                        test
  --prompt PROMPT       prompt text (visible in shell process listings)
  --prompt-file PROMPT_FILE
                        read prompt text from this file
  --max-output-tokens MAX_OUTPUT_TOKENS
                        maximum generated tokens per run (default: 24)
  --warmups WARMUPS     warmup runs retained in the report (default: 0)
  --repetitions REPETITIONS
                        measured runs (default: 1)
  --timeout TIMEOUT     startup and per-run timeout in seconds (default: 900)
  --show-dashboard      open the local dashboard in a browser (default:
                        hidden)
  --output OUTPUT       write the sanitized JSON report
  --save-best PATH      write the Experiment with lowest measured median full
                        latency as canonical JSON
  --force               replace --output if it exists