Local gateway

The gateway is client-controlled and trusted. It binds to 127.0.0.1:8080 by default and uses the local API key local unless configured otherwise.

Run all roles locally

pllm gateway --local --model Qwen/Qwen2.5-0.5B-Instruct

This downloads or reuses the model, starts preparation and inference children on loopback, and keeps the gateway in the foreground. Stop the gateway to stop its children. This mode validates transport and development behavior only. Because both provider roles run under one operator, it does not establish non-collusion.

Connect separate roles

Create client.toml for the trusted gateway:

[client]
base_url = "https://inference.example"
api_key = "inference-secret"
model = "Qwen/Qwen2.5-0.5B-Instruct"
transport = "websocket"
preparation_base_url = "https://preparation.example"
preparation_api_key = "preparation-secret"

Then start the loopback gateway:

pllm gateway --config client.toml

Run provider roles in their respective environments:

pllm serve inference --config inference.json
pllm serve preparation --config preparation.json

A minimal inference.json is:

{
  "api_keys": ["inference-secret"],
  "provider_push_api_key": "push-secret",
  "engine_models": [
    {
      "engine": "masked-transformer-w4a4",
      "kind": "huggingface",
      "path": "/srv/models/Qwen2.5-0.5B-Instruct",
      "model_id": "Qwen/Qwen2.5-0.5B-Instruct"
    }
  ]
}

A minimal preparation.json is:

{
  "api_keys": ["preparation-secret"],
  "engine_models": [
    {
      "engine": "masked-transformer-w4a4",
      "kind": "huggingface",
      "path": "/srv/models/Qwen2.5-0.5B-Instruct",
      "model_id": "Qwen/Qwen2.5-0.5B-Instruct"
    }
  ]
}

Set PLLM_INFERENCE_URL and PLLM_PUSH_API_KEY for the preparation process. Their values must select the inference service and match its provider_push_api_key. Replace example URLs, credentials, and model paths with deployment values. Keep gateway credentials separate from both provider-role credentials. See deployment for remaining production limits.