pllm gateway

Examples

Serve a Python experiment locally

Import the checked-in Experiment object and start both provider roles behind the trusted loopback gateway.

pllm gateway --local --experiment examples/composition.py:experiment --trust-python

Build an experiment with a Python factory

Use --factory when the selected Python object is a zero-argument callable rather than an existing Experiment.

pllm gateway --local --experiment examples/composition.py:build_experiment --factory --trust-python

Serve a declarative experiment

JSON and YAML targets are data, so they do not require --trust-python.

pllm gateway --local --experiment examples/pllm.yaml

Connect to separately operated providers

Set PLLM_INFERENCE_API_KEY and PLLM_PREPARATION_API_KEY in the environment; credentials should not appear in shell arguments.

pllm gateway --inference-url https://inference.example.com --preparation-url https://preparation.example.com

Options

usage: pllm gateway [-h] [--format {human,json,jsonl}] [--quiet] [--no-color]
                    [--no-input] [--dry-run] [--config CONFIG] [--host HOST]
                    [--port PORT] [--api-key API_KEY]
                    [--inference-url INFERENCE_URL]
                    [--inference-key INFERENCE_KEY]
                    [--preparation-url PREPARATION_URL]
                    [--preparation-key PREPARATION_KEY] [--experiment TARGET]
                    [--factory] [--trust-python] [--model MODEL]
                    [--model-id MODEL_ID] [--local]
                    [--transport {auto,http,websocket}]
                    [--correlation-mode {bfv,local-test}]
                    [--correlation-prefetch CORRELATION_PREFETCH]
                    [--prepared-inventory-rows PREPARED_INVENTORY_ROWS]
                    [--token-cache-size TOKEN_CACHE_SIZE]
                    [--bundle-cache-mode {read-write,read-only,refresh,off}]
                    [--bundle-cache-dir BUNDLE_CACHE_DIR]
                    [--tenseal-path TENSEAL_PATH] [--timeout TIMEOUT]
                    [--revision REVISION] [--hf-cache-dir HF_CACHE_DIR]
                    [--local-files-only] [--weight-bits {4,8}]
                    [--activation-bits {4,8}]

options:
  -h, --help            show this help message and exit
  --format {human,json,jsonl}
                        result format (default: human)
  --quiet               suppress non-error diagnostics
  --no-color            disable colored output
  --no-input            fail instead of prompting
  --dry-run             report work without persistent writes or service
                        startup
  --config CONFIG       client TOML file
  --host HOST
  --port PORT
  --api-key API_KEY
  --inference-url INFERENCE_URL
  --inference-key INFERENCE_KEY
  --preparation-url PREPARATION_URL
  --preparation-key PREPARATION_KEY
  --experiment TARGET   local Experiment .json/.yaml or explicit Python
                        path.py:object/module:object
  --factory             call the Python target as a zero-argument factory
  --trust-python        allow import and execution of the explicit Python
                        experiment target
  --model MODEL
  --model-id MODEL_ID
  --local               co-locate both server roles locally
  --transport {auto,http,websocket}
                        session transport
  --correlation-mode {bfv,local-test}
  --correlation-prefetch CORRELATION_PREFETCH
  --prepared-inventory-rows PREPARED_INVENTORY_ROWS
  --token-cache-size TOKEN_CACHE_SIZE
  --bundle-cache-mode {read-write,read-only,refresh,off}
  --bundle-cache-dir BUNDLE_CACHE_DIR
  --tenseal-path TENSEAL_PATH
  --timeout TIMEOUT
  --revision REVISION
  --hf-cache-dir HF_CACHE_DIR
  --local-files-only
  --weight-bits {4,8}
  --activation-bits {4,8}