pllm serve inference

Examples

Serve the inference role from Python configuration

Set PLLM_API_KEY and PLLM_PROVIDER_PUSH_API_KEY before starting this public-model role.

pllm serve inference --experiment examples/composition.py:experiment --trust-python

Build the inference role configuration with a factory

Add --factory when the trusted Python target constructs the Experiment on demand.

pllm serve inference --experiment examples/composition.py:build_experiment --factory --trust-python

Serve the inference role from YAML

The same declarative experiment used by gateway and benchmark can configure a standalone provider role.

pllm serve inference --experiment examples/pllm.yaml

Options

usage: pllm serve inference [-h] [--format {human,json,jsonl}] [--quiet]
                            [--no-color] [--no-input] [--dry-run]
                            [--config CONFIG] [--experiment TARGET]
                            [--factory] [--trust-python]
                            [--privacy-mode {public,proprietary}]
                            [--protocol {guarded,blinded,secure,direct}]
                            [--host HOST] [--port PORT]
                            [--inference-url INFERENCE_URL]
                            [--push-timeout PUSH_TIMEOUT] [--model MODEL]
                            [--model-id MODEL_ID]
                            [--model-kind {huggingface,safetensors,vllm,mlx,mlx-lm}]
                            [--revision REVISION]
                            [--hf-cache-dir HF_CACHE_DIR] [--local-files-only]
                            [--weight-bits {4,8}] [--activation-bits {4,8}]
                            [--tenseal-path TENSEAL_PATH]
                            [--max-batch-size MAX_BATCH_SIZE]
                            [--max-wait-ms MAX_WAIT_MS] [--fixed-batch-wait]
                            [--allow-insecure-local-correlations]
                            [--engine-threads ENGINE_THREADS]
                            [--native-library NATIVE_LIBRARY]
                            [--compiled-cache-dir COMPILED_CACHE_DIR]
                            [--streaming-threshold-elements STREAMING_THRESHOLD_ELEMENTS]
                            [--quantization-chunk-rows QUANTIZATION_CHUNK_ROWS]
                            [--verification-component {none,pllm/freivalds-verify/v1}]
                            [--verification-target-failure-bits VERIFICATION_TARGET_FAILURE_BITS]
                            [--guard-max-rows-per-request GUARD_MAX_ROWS_PER_REQUEST]
                            [--guard-max-rows-per-stage GUARD_MAX_ROWS_PER_STAGE]
                            [--guard-max-requests-per-minute GUARD_MAX_REQUESTS_PER_MINUTE]
                            [--guard-output-dither GUARD_OUTPUT_DITHER]
                            [--rendezvous-timeout RENDEZVOUS_TIMEOUT]
                            [--rendezvous-capacity RENDEZVOUS_CAPACITY]
                            [--rendezvous-max-bytes RENDEZVOUS_MAX_BYTES]
                            [--prepared-session-capacity PREPARED_SESSION_CAPACITY]
                            [--prepared-session-idle PREPARED_SESSION_IDLE]

options:
  -h, --help            show this help message and exit
  --format {human,json,jsonl}
                        result format (default: human)
  --quiet               suppress non-error diagnostics
  --no-color            disable colored output
  --no-input            fail instead of prompting
  --dry-run             report work without persistent writes or service
                        startup
  --config CONFIG       JSON GatewayConfig file
  --experiment TARGET   Experiment .json/.yaml or explicit Python
                        path.py:object/module:object
  --factory             call the Python target as a zero-argument factory
  --trust-python        allow import and execution of the explicit Python
                        experiment target
  --privacy-mode, --mode {public,proprietary}
  --protocol, --proprietary-protocol {guarded,blinded,secure,direct}
  --host HOST
  --port PORT
  --inference-url INFERENCE_URL
  --push-timeout PUSH_TIMEOUT
  --model MODEL
  --model-id MODEL_ID
  --model-kind {huggingface,safetensors,vllm,mlx,mlx-lm}
  --revision REVISION
  --hf-cache-dir HF_CACHE_DIR
  --local-files-only
  --weight-bits {4,8}
  --activation-bits {4,8}
  --tenseal-path TENSEAL_PATH
  --max-batch-size MAX_BATCH_SIZE
  --max-wait-ms MAX_WAIT_MS
  --fixed-batch-wait
  --allow-insecure-local-correlations
  --engine-threads ENGINE_THREADS
  --native-library NATIVE_LIBRARY
  --compiled-cache-dir COMPILED_CACHE_DIR
  --streaming-threshold-elements STREAMING_THRESHOLD_ELEMENTS
  --quantization-chunk-rows QUANTIZATION_CHUNK_ROWS
  --verification-component {none,pllm/freivalds-verify/v1}
  --verification-target-failure-bits VERIFICATION_TARGET_FAILURE_BITS
  --guard-max-rows-per-request GUARD_MAX_ROWS_PER_REQUEST
  --guard-max-rows-per-stage GUARD_MAX_ROWS_PER_STAGE
  --guard-max-requests-per-minute GUARD_MAX_REQUESTS_PER_MINUTE
  --guard-output-dither GUARD_OUTPUT_DITHER
  --rendezvous-timeout RENDEZVOUS_TIMEOUT
  --rendezvous-capacity RENDEZVOUS_CAPACITY
  --rendezvous-max-bytes RENDEZVOUS_MAX_BYTES
  --prepared-session-capacity PREPARED_SESSION_CAPACITY
  --prepared-session-idle PREPARED_SESSION_IDLE