# pllm.runtime Python API

Public objects, practical usage, and research context for pllm.runtime.

[View canonical HTML](https://pllm.run/sdk/reference/python/pllm/runtime/)

Document ID: `pllm.docs.reference.python.pllm.runtime`  
Release: `0.1.0`

Runtime APIs execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services. Presence documents API availability, not complete model, security, or deployment coverage; check [implementation status](/sdk/reference/status/) before relying on a path.

## Research context

Relevant design inputs: [Slalom](/research/papers/r07-slalom/), [DASH](/research/papers/r01-dash/). PLLM's implementation and evidence claims remain independent.

## Python SDK example

The example performs a bounded offline task and checks an observable result. It does not download a checkpoint or contact a provider.

```python
import pllm.runtime as runtime

app = runtime.create_app(runtime.GatewayConfig(api_keys=("local-test",)))
routes = {route.path for route in app.routes}
assert {"/health", "/v1/responses"} <= routes
```

API: [`pllm.runtime`](/sdk/reference/python/pllm/runtime/#objects-and-signatures)

## Objects and signatures

### `ExecutionBudget`

Immutable `ExecutionBudget` record carrying `requests`, `max_input_tokens`, `max_new_tokens` for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

- Canonical object: `pllm.configuration.ExecutionBudget`
- Kind: `class`
- Signature/type: `__init__(self, requests: int, max_input_tokens: int, max_new_tokens: int) -> None`
- Public exports: `pllm.ExecutionBudget`, `pllm.config.ExecutionBudget`, `pllm.runtime.ExecutionBudget`
- Public members: `requests: int`; `max_input_tokens: int`; `max_new_tokens: int`; `__init__(self, requests: int, max_input_tokens: int, max_new_tokens: int) -> None`; `get_params(self, deep: bool=True) -> dict[str, Any]`; `with_params(self, **changes: object) -> Self`; `to_spec(self) -> dict[str, Any]`

### `__version__`

Provides the public version value used where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

- Canonical object: `pllm.runtime.__version__`
- Kind: `str`
- Signature/type: `str`
- Public exports: `pllm.runtime.__version__`

### `AsyncOpenAI`

Asynchronous OpenAI-compatible client that targets the trusted local PLLM gateway.

- Canonical object: `pllm.runtime.client.AsyncOpenAI`
- Kind: `class`
- Signature/type: `(**kwargs: 'Any') -> 'None'`
- Public exports: `pllm.AsyncOpenAI`, `pllm.client.AsyncOpenAI`, `pllm.runtime.AsyncOpenAI`
- Public members: `close(self) -> 'None'`; `prepared_rows_for_response(self, input: 'str', max_output_tokens: 'int', *, model: 'str \| None' = None, instructions: 'str \| None' = None) -> 'int'`; `preprocess(self, model: 'str \| None' = None, *, count: 'int \| None' = None, stages: 'list[str] \| None' = None) -> 'dict[str, Any]'`; `property privacy_audit: 'PrivacyAudit'`

### `OpenAI`

Synchronous OpenAI-compatible client that targets the trusted local PLLM gateway.

- Canonical object: `pllm.runtime.client.OpenAI`
- Kind: `class`
- Signature/type: `(*, api_key: 'str \| None' = None, base_url: 'str \| None' = None, default_model: 'str \| None' = None, model: 'str \| None' = None, session_transport: 'str \| None' = None, correlation_mode: 'str \| None' = None, preparation_base_url: 'str \| None' = None, preparation_api_key: 'str \| None' = None, correlation_prefetch: 'int \| None' = None, prepared_inventory_rows: 'int \| None' = None, background_inventory_refill: 'bool' = True, token_cache_size: 'int \| None' = None, bundle_cache_mode: 'str \| None' = None, bundle_cache_dir: 'str \| Path \| None' = None, tenseal_path: 'str \| None' = None, timeout: 'float \| None' = None, http_client: 'httpx.Client \| None' = None, preparation_http_client: 'httpx.Client \| None' = None, experiment: 'Experiment \| ExperimentProfile \| None' = None) -> 'None'`
- Public exports: `pllm.OpenAI`, `pllm.client.OpenAI`, `pllm.runtime.OpenAI`
- Public members: `close(self) -> 'None'`; `prepared_inventory_status(self, model: 'str \| None' = None) -> 'dict[str, Any]'`; `prepared_rows_for_response(self, input: 'str', max_output_tokens: 'int', *, model: 'str \| None' = None, instructions: 'str \| None' = None) -> 'int'`; `preprocess(self, model: 'str \| None' = None, *, count: 'int \| None' = None, stages: 'list[str] \| None' = None) -> 'dict[str, Any]'`; `property privacy_audit: 'PrivacyAudit'`

### `GatewayConfig`

Immutable `GatewayConfig` record carrying `api_keys`, `privacy_mode`, `proprietary_protocol`, `backends`, `engine_models`, `max_batch_size`, `max_batch_wait_ms`, `adaptive_batching`, `response_retention_seconds`, `provider_push_api_key`, `rendezvous_timeout_seconds`, `rendezvous_capacity`, `rendezvous_max_bytes`, `rendezvous_max_attempts_per_session`, `prepared_session_capacity`, `prepared_session_idle_seconds`, `preparation_request_max_bytes`, `prepared_payload_max_bytes`, `prepared_tensor_max_elements`, `prepared_stage_batch_rows`, `preparation_inference_url`, `preparation_push_api_key`, `preparation_push_timeout_seconds`, `allow_insecure_local_correlations`, `tenseal_path`, `reference_model_id`, `reference_phrase`, `reference_alphabet` for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

- Canonical object: `pllm.runtime.config.GatewayConfig`
- Kind: `class`
- Signature/type: `(api_keys: 'tuple[str, ...]' = ('pllm-local',), privacy_mode: 'str' = 'public', proprietary_protocol: 'str' = 'guarded', backends: 'tuple[dict[str, Any], ...]' = (), engine_models: 'tuple[dict[str, Any], ...]' = (), max_batch_size: 'int' = 32, max_batch_wait_ms: 'float' = 0.25, adaptive_batching: 'bool' = True, response_retention_seconds: 'float' = 3600.0, provider_push_api_key: 'str \| None' = None, rendezvous_timeout_seconds: 'float' = 30.0, rendezvous_capacity: 'int' = 32768, rendezvous_max_bytes: 'int' = 268435456, rendezvous_max_attempts_per_session: 'int' = 33554432, prepared_session_capacity: 'int' = 4096, prepared_session_idle_seconds: 'float' = 300.0, preparation_request_max_bytes: 'int' = 16384, prepared_payload_max_bytes: 'int' = 268435456, prepared_tensor_max_elements: 'int' = 67108864, prepared_stage_batch_rows: 'int' = 4096, preparation_inference_url: 'str \| None' = None, preparation_push_api_key: 'str \| None' = None, preparation_push_timeout_seconds: 'float' = 10.0, allow_insecure_local_correlations: 'bool' = False, tenseal_path: 'str \| None' = None, reference_model_id: 'str' = 'pllm-bigram-demo', reference_phrase: 'str' = 'private\n', reference_alphabet: 'str' = 'private\\n') -> None`
- Public exports: `pllm.GatewayConfig`, `pllm.runtime.GatewayConfig`
- Public members: `classmethod from_dict(cls, value: 'dict[str, Any]') -> "'GatewayConfig'"`; `classmethod load(cls, path: 'str \| Path') -> "'GatewayConfig'"`; `to_dict(self) -> 'dict[str, Any]'`

### `CompiledRuntimeModel`

Opaque validated binding between a semantic model plan and a concrete runtime bundle.

- Canonical object: `pllm.runtime.model_binding.CompiledRuntimeModel`
- Kind: `class`
- Signature/type: `() -> 'None'`
- Public exports: `pllm.runtime.CompiledRuntimeModel`
- Public members: `canonical_bytes(self) -> 'bytes'`; `property bundle_fingerprint: 'str'`; `property complete: 'bool'`; `property completeness_scope: 'str'`; `property digest: 'str'`; `property local_operations: 'tuple[str, ...]'`; `property model_plan_digest: 'str'`; `property runtime_config_digest: 'str'`; `property runtime_schedule_digest: 'str'`; `property stage_bindings: 'tuple[RuntimeStageBinding, ...]'`; `property tokenizer_digest: 'str'`; `runtime(self, remote: 'Callable[[str, np.ndarray], np.ndarray]') -> 'MaskedTransformerClientRuntime'`; `session(self, remote: 'Callable[[str, np.ndarray], np.ndarray]') -> "'CompiledRuntimeSession'"`; `to_spec(self) -> 'dict[str, Any]'`; `validate(self) -> 'None'`

### `RuntimeBindingError`

Raised when a RuntimeBinding condition prevents the requested operation from completing.

- Canonical object: `pllm.runtime.model_binding.RuntimeBindingError`
- Kind: `class`
- Signature/type: `signature not exposed`
- Public exports: `pllm.runtime.RuntimeBindingError`

### `RuntimeStageBinding`

Immutable `RuntimeStageBinding` record carrying `stage_id`, `role`, `layer_index`, `semantic_operations`, `in_features`, `out_features`, `weight_bits`, `activation_bits`, `ring`, `modulus`, `wire_bits`, `weight_digest`, `weight_scales_digest`, `bias_digest`, `client_weight_layout`, `client_weight_digest`, `client_weight_scales_digest`, `client_aux_weight_digest`, `client_aux_scales_digest` for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

- Canonical object: `pllm.runtime.model_binding.RuntimeStageBinding`
- Kind: `class`
- Signature/type: `(stage_id: 'str', role: 'str', layer_index: 'int \| None', semantic_operations: 'tuple[str, ...]', in_features: 'int', out_features: 'int', weight_bits: 'int', activation_bits: 'int', ring: 'str', modulus: 'int', wire_bits: 'int', weight_digest: 'str', weight_scales_digest: 'str', bias_digest: 'str \| None', client_weight_layout: 'str \| None' = None, client_weight_digest: 'str \| None' = None, client_weight_scales_digest: 'str \| None' = None, client_aux_weight_digest: 'str \| None' = None, client_aux_scales_digest: 'str \| None' = None) -> None`
- Public exports: `pllm.runtime.RuntimeStageBinding`

### `compile_runtime_model`

Binds a semantic plan to a validated runtime bundle and fails on unresolved operations.

- Canonical object: `pllm.runtime.model_binding.compile_runtime_model`
- Kind: `function`
- Signature/type: `(plan: 'ModelPlan', bundle: 'ClientBundle', *, composition: 'Pipeline \| None' = None) -> 'CompiledRuntimeModel'`
- Public exports: `pllm.runtime.compile_runtime_model`

### `CompiledRuntimeSession`

Stateful execution session created from one validated compiled runtime model.

- Canonical object: `pllm.runtime.model_execution.CompiledRuntimeSession`
- Kind: `class`
- Signature/type: `() -> 'None'`
- Public exports: `pllm.runtime.CompiledRuntimeSession`
- Public members: `close(self) -> 'None'`; `decode_selected(self) -> 'np.ndarray'`; `finish(self) -> 'None'`; `generate_ids(self, token_ids: 'Sequence[int] \| np.ndarray', *, max_new_tokens: 'int \| None' = None) -> 'tuple[int, ...]'`; `prefill_ids(self, token_ids: 'Sequence[int] \| np.ndarray') -> 'np.ndarray'`; `property binding_digest: 'str'`; `property complete: 'bool'`; `property completeness_scope: 'str'`; `property generated_tokens: 'int'`; `property logits: 'np.ndarray'`; `property position: 'int'`; `property remaining_tokens: 'int'`; `property runtime_schedule_digest: 'str'`; `property status: 'str'`; `select_next(self) -> 'int'`

### `RuntimeExecutionError`

Raised when a RuntimeExecution condition prevents the requested operation from completing.

- Canonical object: `pllm.runtime.model_execution.RuntimeExecutionError`
- Kind: `class`
- Signature/type: `signature not exposed`
- Public exports: `pllm.runtime.RuntimeExecutionError`

### `PrivacyMode`

Defines the accepted values for `PrivacyMode`: `public`, `proprietary`.

- Canonical object: `pllm.runtime.privacy.PrivacyMode`
- Kind: `class`
- Signature/type: `(*values)`
- Public exports: `pllm.PrivacyMode`, `pllm.runtime.PrivacyMode`
- Public members: `classmethod parse(cls, value: "str \| 'PrivacyMode'") -> "'PrivacyMode'"`; `property description: 'str'`; `property protocol: 'str'`

### `ProprietaryProtocol`

Defines the accepted values for `ProprietaryProtocol`: `guarded`, `blinded`, `secure`, `direct`.

- Canonical object: `pllm.runtime.privacy.ProprietaryProtocol`
- Kind: `class`
- Signature/type: `(*values)`
- Public exports: `pllm.ProprietaryProtocol`, `pllm.runtime.ProprietaryProtocol`
- Public members: `classmethod parse(cls, value: "str \| 'ProprietaryProtocol'") -> "'ProprietaryProtocol'"`

### `create_app`

Builds the app ASGI application from validated runtime dependencies.

- Canonical object: `pllm.runtime.server.create_app`
- Kind: `function`
- Signature/type: `(config: 'GatewayConfig \| None' = None, *, backend_registry: 'BackendRegistry \| None' = None, private_models: 'dict[str, MaskedBigramModel] \| None' = None, engines: 'dict[str, InferenceEngine] \| None' = None, audit_hook: 'Callable[[str, bytes], None] \| None' = None) -> 'FastAPI'`
- Public exports: `pllm.create_app`, `pllm.runtime.create_app`, `pllm.server.create_app`

### `LocalTopology`

`LocalTopology` provides local topology behavior for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services. Public operations include `property model_id: str`, `property inference_url: str`, `property preparation_url: str`.

- Canonical object: `pllm.runtime.servers.LocalTopology`
- Kind: `class`
- Signature/type: `constructor not exposed`
- Public exports: `pllm.runtime.LocalTopology`
- Public members: `property model_id: str`; `property inference_url: str`; `property preparation_url: str`; `property started: bool`; `property closed: bool`; `property requires_preparation: bool`; `property statuses: tuple[RoleStatus, ...]`; `start(self) -> LocalTopology`; `is_healthy(self) -> bool`; `client(self, **overrides: Any) -> OpenAI`; `gateway_app(self, *, local_api_key: str, **overrides: Any) -> FastAPI`; `close(self) -> None`

### `RoleStatus`

Immutable `RoleStatus` record carrying `role`, `url`, `pid`, `running` for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

- Canonical object: `pllm.runtime.servers.RoleStatus`
- Kind: `class`
- Signature/type: `constructor not exposed`
- Public exports: `pllm.runtime.RoleStatus`
- Public members: `role: str`; `url: str`; `pid: int \| None`; `running: bool`

### `TopologyError`

Raised when a Topology condition prevents the requested operation from completing.

- Canonical object: `pllm.runtime.servers.TopologyError`
- Kind: `class`
- Signature/type: `constructor not exposed`
- Public exports: `pllm.runtime.TopologyError`

### `build_roles`

Starts the bounded local inference and preparation process topology.

- Canonical object: `pllm.runtime.servers.build_roles`
- Kind: `function`
- Signature/type: `build_roles(model: Model \| Pipeline \| Experiment \| str, *, model_id: str \| None=None, engine_threads: int \| None=None, weight_bits: int=8, activation_bits: int=8, correlation_mode: str='bfv', tenseal_path: str \| None=None, hf_cache_dir: str \| None=None, reserved_ports: Iterable[int]=(), rendezvous_capacity: int=262144, rendezvous_max_bytes: int=2147483648, log_dir: str \| Path \| None=None, telemetry_endpoint: str \| None=None, telemetry_token: str \| None=None, credential_prefix: str='local', startup_timeout: float=300.0, progress: Callable[[str], None] \| None=None) -> LocalTopology`
- Public exports: `pllm.runtime.build_roles`

### `serve_local`

Performs the serve local operation for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

- Canonical object: `pllm.runtime.servers.serve_local`
- Kind: `function`
- Signature/type: `serve_local(model: Model \| Pipeline \| Experiment \| str, *, model_id: str \| None=None, engine_threads: int \| None=None, weight_bits: int=8, activation_bits: int=8, correlation_mode: str='bfv', tenseal_path: str \| None=None, hf_cache_dir: str \| None=None, reserved_ports: Iterable[int]=(), rendezvous_capacity: int=262144, rendezvous_max_bytes: int=2147483648, log_dir: str \| Path \| None=None, telemetry_endpoint: str \| None=None, telemetry_token: str \| None=None, credential_prefix: str='local', startup_timeout: float=300.0, progress: Callable[[str], None] \| None=None) -> LocalTopology`
- Public exports: `pllm.runtime.serve_local`, `pllm.serve_local`

### `create_sidecar_app`

Builds the sidecar ASGI application from validated runtime dependencies.

- Canonical object: `pllm.runtime.sidecar.create_sidecar_app`
- Kind: `function`
- Signature/type: `(*, remote_base_url: 'str', remote_api_key: 'str', local_api_key: 'str' = 'local', client: 'OpenAI \| RuntimeClient \| None' = None, **client_kwargs: 'Any') -> 'FastAPI'`
- Public exports: `pllm.create_sidecar_app`, `pllm.runtime.create_sidecar_app`

### `AsyncPLLMTransport`

`AsyncPLLMTransport` provides async pllm transport behavior for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services. Public operations include `aclose`, `handle_async_request`, `preprocess`.

- Canonical object: `pllm.runtime.transport.AsyncPLLMTransport`
- Kind: `class`
- Signature/type: `(**kwargs: 'Any') -> 'None'`
- Public exports: `pllm.AsyncPLLMTransport`, `pllm.runtime.AsyncPLLMTransport`
- Public members: `aclose(self) -> 'None'`; `handle_async_request(self, request: 'Any') -> 'Any'`; `preprocess(self, model: 'str', *, count: 'int \| None' = None, stages: 'list[str] \| None' = None) -> 'dict[str, Any]'`

### `PLLMTransport`

Custom transport for the official OpenAI Python client.

- Canonical object: `pllm.runtime.transport.PLLMTransport`
- Kind: `class`
- Signature/type: `(*, gateway_url: 'str', api_key: 'str' = 'pllm-local', preparation_url: 'str \| None' = None, preparation_api_key: 'str \| None' = None, session_transport: 'str' = 'http', correlation_mode: 'str' = 'bfv', correlation_prefetch: 'int' = 4, prepared_inventory_rows: 'int' = 64, token_cache_size: 'int' = 512, bundle_cache_mode: 'str' = 'read-write', bundle_cache_dir: 'str \| None' = None, tenseal_path: 'str \| None' = None) -> 'None'`
- Public exports: `pllm.PLLMTransport`, `pllm.runtime.PLLMTransport`
- Public members: `close(self) -> 'None'`; `handle_request(self, request: 'Any') -> 'Any'`; `preprocess(self, model: 'str', *, count: 'int \| None' = None) -> 'dict[str, Any]'`