pllmPRIVATE INFERENCE
RUNTIME
pllm
SDK
Pythonpllm

pllm.runtime Python API

Public objects, practical usage, and research context for pllm.runtime.

PLLM documentationView Markdown ↗

Runtime APIs execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services. Presence documents API availability, not complete model, security, or deployment coverage; check implementation status before relying on a path.

Research context

Relevant design inputs: Slalom, DASH. PLLM's implementation and evidence claims remain independent.

Python SDK example

The example performs a bounded offline task and checks an observable result. It does not download a checkpoint or contact a provider.

import pllm.runtime as runtime

app = runtime.create_app(runtime.GatewayConfig(api_keys=("local-test",)))
routes = {route.path for route in app.routes}
assert {"/health", "/v1/responses"} <= routes

API: pllm.runtime

Objects and signatures

ExecutionBudget

Immutable ExecutionBudget record carrying requests, max_input_tokens, max_new_tokens for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

  • Canonical object: pllm.configuration.ExecutionBudget
  • Kind: class
  • Signature/type: __init__(self, requests: int, max_input_tokens: int, max_new_tokens: int) -> None
  • Public exports: pllm.ExecutionBudget, pllm.config.ExecutionBudget, pllm.runtime.ExecutionBudget
  • Public members: requests: int; max_input_tokens: int; max_new_tokens: int; __init__(self, requests: int, max_input_tokens: int, max_new_tokens: int) -> None; get_params(self, deep: bool=True) -> dict[str, Any]; with_params(self, **changes: object) -> Self; to_spec(self) -> dict[str, Any]

__version__

Provides the public version value used where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

  • Canonical object: pllm.runtime.__version__
  • Kind: str
  • Signature/type: str
  • Public exports: pllm.runtime.__version__

AsyncOpenAI

Asynchronous OpenAI-compatible client that targets the trusted local PLLM gateway.

  • Canonical object: pllm.runtime.client.AsyncOpenAI
  • Kind: class
  • Signature/type: (**kwargs: 'Any') -> 'None'
  • Public exports: pllm.AsyncOpenAI, pllm.client.AsyncOpenAI, pllm.runtime.AsyncOpenAI
  • Public members: close(self) -> 'None'; prepared_rows_for_response(self, input: 'str', max_output_tokens: 'int', *, model: 'str \| None' = None, instructions: 'str \| None' = None) -> 'int'; preprocess(self, model: 'str \| None' = None, *, count: 'int \| None' = None, stages: 'list[str] \| None' = None) -> 'dict[str, Any]'; property privacy_audit: 'PrivacyAudit'

OpenAI

Synchronous OpenAI-compatible client that targets the trusted local PLLM gateway.

  • Canonical object: pllm.runtime.client.OpenAI
  • Kind: class
  • Signature/type: (*, api_key: 'str \| None' = None, base_url: 'str \| None' = None, default_model: 'str \| None' = None, model: 'str \| None' = None, session_transport: 'str \| None' = None, correlation_mode: 'str \| None' = None, preparation_base_url: 'str \| None' = None, preparation_api_key: 'str \| None' = None, correlation_prefetch: 'int \| None' = None, prepared_inventory_rows: 'int \| None' = None, background_inventory_refill: 'bool' = True, token_cache_size: 'int \| None' = None, bundle_cache_mode: 'str \| None' = None, bundle_cache_dir: 'str \| Path \| None' = None, tenseal_path: 'str \| None' = None, timeout: 'float \| None' = None, http_client: 'httpx.Client \| None' = None, preparation_http_client: 'httpx.Client \| None' = None, experiment: 'Experiment \| ExperimentProfile \| None' = None) -> 'None'
  • Public exports: pllm.OpenAI, pllm.client.OpenAI, pllm.runtime.OpenAI
  • Public members: close(self) -> 'None'; prepared_inventory_status(self, model: 'str \| None' = None) -> 'dict[str, Any]'; prepared_rows_for_response(self, input: 'str', max_output_tokens: 'int', *, model: 'str \| None' = None, instructions: 'str \| None' = None) -> 'int'; preprocess(self, model: 'str \| None' = None, *, count: 'int \| None' = None, stages: 'list[str] \| None' = None) -> 'dict[str, Any]'; property privacy_audit: 'PrivacyAudit'

GatewayConfig

Immutable GatewayConfig record carrying api_keys, privacy_mode, proprietary_protocol, backends, engine_models, max_batch_size, max_batch_wait_ms, adaptive_batching, response_retention_seconds, provider_push_api_key, rendezvous_timeout_seconds, rendezvous_capacity, rendezvous_max_bytes, rendezvous_max_attempts_per_session, prepared_session_capacity, prepared_session_idle_seconds, preparation_request_max_bytes, prepared_payload_max_bytes, prepared_tensor_max_elements, prepared_stage_batch_rows, preparation_inference_url, preparation_push_api_key, preparation_push_timeout_seconds, allow_insecure_local_correlations, tenseal_path, reference_model_id, reference_phrase, reference_alphabet for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

  • Canonical object: pllm.runtime.config.GatewayConfig
  • Kind: class
  • Signature/type: (api_keys: 'tuple[str, ...]' = ('pllm-local',), privacy_mode: 'str' = 'public', proprietary_protocol: 'str' = 'guarded', backends: 'tuple[dict[str, Any], ...]' = (), engine_models: 'tuple[dict[str, Any], ...]' = (), max_batch_size: 'int' = 32, max_batch_wait_ms: 'float' = 0.25, adaptive_batching: 'bool' = True, response_retention_seconds: 'float' = 3600.0, provider_push_api_key: 'str \| None' = None, rendezvous_timeout_seconds: 'float' = 30.0, rendezvous_capacity: 'int' = 32768, rendezvous_max_bytes: 'int' = 268435456, rendezvous_max_attempts_per_session: 'int' = 33554432, prepared_session_capacity: 'int' = 4096, prepared_session_idle_seconds: 'float' = 300.0, preparation_request_max_bytes: 'int' = 16384, prepared_payload_max_bytes: 'int' = 268435456, prepared_tensor_max_elements: 'int' = 67108864, prepared_stage_batch_rows: 'int' = 4096, preparation_inference_url: 'str \| None' = None, preparation_push_api_key: 'str \| None' = None, preparation_push_timeout_seconds: 'float' = 10.0, allow_insecure_local_correlations: 'bool' = False, tenseal_path: 'str \| None' = None, reference_model_id: 'str' = 'pllm-bigram-demo', reference_phrase: 'str' = 'private\n', reference_alphabet: 'str' = 'private\\n') -> None
  • Public exports: pllm.GatewayConfig, pllm.runtime.GatewayConfig
  • Public members: classmethod from_dict(cls, value: 'dict[str, Any]') -> "'GatewayConfig'"; classmethod load(cls, path: 'str \| Path') -> "'GatewayConfig'"; to_dict(self) -> 'dict[str, Any]'

CompiledRuntimeModel

Opaque validated binding between a semantic model plan and a concrete runtime bundle.

  • Canonical object: pllm.runtime.model_binding.CompiledRuntimeModel
  • Kind: class
  • Signature/type: () -> 'None'
  • Public exports: pllm.runtime.CompiledRuntimeModel
  • Public members: canonical_bytes(self) -> 'bytes'; property bundle_fingerprint: 'str'; property complete: 'bool'; property completeness_scope: 'str'; property digest: 'str'; property local_operations: 'tuple[str, ...]'; property model_plan_digest: 'str'; property runtime_config_digest: 'str'; property runtime_schedule_digest: 'str'; property stage_bindings: 'tuple[RuntimeStageBinding, ...]'; property tokenizer_digest: 'str'; runtime(self, remote: 'Callable[[str, np.ndarray], np.ndarray]') -> 'MaskedTransformerClientRuntime'; session(self, remote: 'Callable[[str, np.ndarray], np.ndarray]') -> "'CompiledRuntimeSession'"; to_spec(self) -> 'dict[str, Any]'; validate(self) -> 'None'

RuntimeBindingError

Raised when a RuntimeBinding condition prevents the requested operation from completing.

  • Canonical object: pllm.runtime.model_binding.RuntimeBindingError
  • Kind: class
  • Signature/type: signature not exposed
  • Public exports: pllm.runtime.RuntimeBindingError

RuntimeStageBinding

Immutable RuntimeStageBinding record carrying stage_id, role, layer_index, semantic_operations, in_features, out_features, weight_bits, activation_bits, ring, modulus, wire_bits, weight_digest, weight_scales_digest, bias_digest, client_weight_layout, client_weight_digest, client_weight_scales_digest, client_aux_weight_digest, client_aux_scales_digest for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

  • Canonical object: pllm.runtime.model_binding.RuntimeStageBinding
  • Kind: class
  • Signature/type: (stage_id: 'str', role: 'str', layer_index: 'int \| None', semantic_operations: 'tuple[str, ...]', in_features: 'int', out_features: 'int', weight_bits: 'int', activation_bits: 'int', ring: 'str', modulus: 'int', wire_bits: 'int', weight_digest: 'str', weight_scales_digest: 'str', bias_digest: 'str \| None', client_weight_layout: 'str \| None' = None, client_weight_digest: 'str \| None' = None, client_weight_scales_digest: 'str \| None' = None, client_aux_weight_digest: 'str \| None' = None, client_aux_scales_digest: 'str \| None' = None) -> None
  • Public exports: pllm.runtime.RuntimeStageBinding

compile_runtime_model

Binds a semantic plan to a validated runtime bundle and fails on unresolved operations.

  • Canonical object: pllm.runtime.model_binding.compile_runtime_model
  • Kind: function
  • Signature/type: (plan: 'ModelPlan', bundle: 'ClientBundle', *, composition: 'Pipeline \| None' = None) -> 'CompiledRuntimeModel'
  • Public exports: pllm.runtime.compile_runtime_model

CompiledRuntimeSession

Stateful execution session created from one validated compiled runtime model.

  • Canonical object: pllm.runtime.model_execution.CompiledRuntimeSession
  • Kind: class
  • Signature/type: () -> 'None'
  • Public exports: pllm.runtime.CompiledRuntimeSession
  • Public members: close(self) -> 'None'; decode_selected(self) -> 'np.ndarray'; finish(self) -> 'None'; generate_ids(self, token_ids: 'Sequence[int] \| np.ndarray', *, max_new_tokens: 'int \| None' = None) -> 'tuple[int, ...]'; prefill_ids(self, token_ids: 'Sequence[int] \| np.ndarray') -> 'np.ndarray'; property binding_digest: 'str'; property complete: 'bool'; property completeness_scope: 'str'; property generated_tokens: 'int'; property logits: 'np.ndarray'; property position: 'int'; property remaining_tokens: 'int'; property runtime_schedule_digest: 'str'; property status: 'str'; select_next(self) -> 'int'

RuntimeExecutionError

Raised when a RuntimeExecution condition prevents the requested operation from completing.

  • Canonical object: pllm.runtime.model_execution.RuntimeExecutionError
  • Kind: class
  • Signature/type: signature not exposed
  • Public exports: pllm.runtime.RuntimeExecutionError

PrivacyMode

Defines the accepted values for PrivacyMode: public, proprietary.

  • Canonical object: pllm.runtime.privacy.PrivacyMode
  • Kind: class
  • Signature/type: (*values)
  • Public exports: pllm.PrivacyMode, pllm.runtime.PrivacyMode
  • Public members: classmethod parse(cls, value: "str \| 'PrivacyMode'") -> "'PrivacyMode'"; property description: 'str'; property protocol: 'str'

ProprietaryProtocol

Defines the accepted values for ProprietaryProtocol: guarded, blinded, secure, direct.

  • Canonical object: pllm.runtime.privacy.ProprietaryProtocol
  • Kind: class
  • Signature/type: (*values)
  • Public exports: pllm.ProprietaryProtocol, pllm.runtime.ProprietaryProtocol
  • Public members: classmethod parse(cls, value: "str \| 'ProprietaryProtocol'") -> "'ProprietaryProtocol'"

create_app

Builds the app ASGI application from validated runtime dependencies.

  • Canonical object: pllm.runtime.server.create_app
  • Kind: function
  • Signature/type: (config: 'GatewayConfig \| None' = None, *, backend_registry: 'BackendRegistry \| None' = None, private_models: 'dict[str, MaskedBigramModel] \| None' = None, engines: 'dict[str, InferenceEngine] \| None' = None, audit_hook: 'Callable[[str, bytes], None] \| None' = None) -> 'FastAPI'
  • Public exports: pllm.create_app, pllm.runtime.create_app, pllm.server.create_app

LocalTopology

LocalTopology provides local topology behavior for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services. Public operations include property model_id: str, property inference_url: str, property preparation_url: str.

  • Canonical object: pllm.runtime.servers.LocalTopology
  • Kind: class
  • Signature/type: constructor not exposed
  • Public exports: pllm.runtime.LocalTopology
  • Public members: property model_id: str; property inference_url: str; property preparation_url: str; property started: bool; property closed: bool; property requires_preparation: bool; property statuses: tuple[RoleStatus, ...]; start(self) -> LocalTopology; is_healthy(self) -> bool; client(self, **overrides: Any) -> OpenAI; gateway_app(self, *, local_api_key: str, **overrides: Any) -> FastAPI; close(self) -> None

RoleStatus

Immutable RoleStatus record carrying role, url, pid, running for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

  • Canonical object: pllm.runtime.servers.RoleStatus
  • Kind: class
  • Signature/type: constructor not exposed
  • Public exports: pllm.runtime.RoleStatus
  • Public members: role: str; url: str; pid: int \| None; running: bool

TopologyError

Raised when a Topology condition prevents the requested operation from completing.

  • Canonical object: pllm.runtime.servers.TopologyError
  • Kind: class
  • Signature/type: constructor not exposed
  • Public exports: pllm.runtime.TopologyError

build_roles

Starts the bounded local inference and preparation process topology.

  • Canonical object: pllm.runtime.servers.build_roles
  • Kind: function
  • Signature/type: build_roles(model: Model \| Pipeline \| Experiment \| str, *, model_id: str \| None=None, engine_threads: int \| None=None, weight_bits: int=8, activation_bits: int=8, correlation_mode: str='bfv', tenseal_path: str \| None=None, hf_cache_dir: str \| None=None, reserved_ports: Iterable[int]=(), rendezvous_capacity: int=262144, rendezvous_max_bytes: int=2147483648, log_dir: str \| Path \| None=None, telemetry_endpoint: str \| None=None, telemetry_token: str \| None=None, credential_prefix: str='local', startup_timeout: float=300.0, progress: Callable[[str], None] \| None=None) -> LocalTopology
  • Public exports: pllm.runtime.build_roles

serve_local

Performs the serve local operation for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services.

  • Canonical object: pllm.runtime.servers.serve_local
  • Kind: function
  • Signature/type: serve_local(model: Model \| Pipeline \| Experiment \| str, *, model_id: str \| None=None, engine_threads: int \| None=None, weight_bits: int=8, activation_bits: int=8, correlation_mode: str='bfv', tenseal_path: str \| None=None, hf_cache_dir: str \| None=None, reserved_ports: Iterable[int]=(), rendezvous_capacity: int=262144, rendezvous_max_bytes: int=2147483648, log_dir: str \| Path \| None=None, telemetry_endpoint: str \| None=None, telemetry_token: str \| None=None, credential_prefix: str='local', startup_timeout: float=300.0, progress: Callable[[str], None] \| None=None) -> LocalTopology
  • Public exports: pllm.runtime.serve_local, pllm.serve_local

create_sidecar_app

Builds the sidecar ASGI application from validated runtime dependencies.

  • Canonical object: pllm.runtime.sidecar.create_sidecar_app
  • Kind: function
  • Signature/type: (*, remote_base_url: 'str', remote_api_key: 'str', local_api_key: 'str' = 'local', client: 'OpenAI \| RuntimeClient \| None' = None, **client_kwargs: 'Any') -> 'FastAPI'
  • Public exports: pllm.create_sidecar_app, pllm.runtime.create_sidecar_app

AsyncPLLMTransport

AsyncPLLMTransport provides async pllm transport behavior for the workflow where runtime apis execute validated native kernels, bind compiled models, construct local role topologies, and expose trusted client-boundary services. Public operations include aclose, handle_async_request, preprocess.

  • Canonical object: pllm.runtime.transport.AsyncPLLMTransport
  • Kind: class
  • Signature/type: (**kwargs: 'Any') -> 'None'
  • Public exports: pllm.AsyncPLLMTransport, pllm.runtime.AsyncPLLMTransport
  • Public members: aclose(self) -> 'None'; handle_async_request(self, request: 'Any') -> 'Any'; preprocess(self, model: 'str', *, count: 'int \| None' = None, stages: 'list[str] \| None' = None) -> 'dict[str, Any]'

PLLMTransport

Custom transport for the official OpenAI Python client.

  • Canonical object: pllm.runtime.transport.PLLMTransport
  • Kind: class
  • Signature/type: (*, gateway_url: 'str', api_key: 'str' = 'pllm-local', preparation_url: 'str \| None' = None, preparation_api_key: 'str \| None' = None, session_transport: 'str' = 'http', correlation_mode: 'str' = 'bfv', correlation_prefetch: 'int' = 4, prepared_inventory_rows: 'int' = 64, token_cache_size: 'int' = 512, bundle_cache_mode: 'str' = 'read-write', bundle_cache_dir: 'str \| None' = None, tenseal_path: 'str \| None' = None) -> 'None'
  • Public exports: pllm.PLLMTransport, pllm.runtime.PLLMTransport
  • Public members: close(self) -> 'None'; handle_request(self, request: 'Any') -> 'Any'; preprocess(self, model: 'str', *, count: 'int \| None' = None) -> 'dict[str, Any]'

pllm.deployment Python API

Public objects, practical usage, and research context for pllm.deployment.

pllm.compiler Python API

Public objects, practical usage, and research context for pllm.compiler.

On this page

Research contextPython SDK exampleObjects and signaturesExecutionBudget__version__AsyncOpenAIOpenAIGatewayConfigCompiledRuntimeModelRuntimeBindingErrorRuntimeStageBindingcompile_runtime_modelCompiledRuntimeSessionRuntimeExecutionErrorPrivacyModeProprietaryProtocolcreate_appLocalTopologyRoleStatusTopologyErrorbuild_rolesserve_localcreate_sidecar_appAsyncPLLMTransportPLLMTransport

Loading documentation index…

Search runs locally in this browser.