Runtime

Understand which live state a PLLM runtime owns and how it differs from a reproducible plan.

A runtime session owns changing state such as inventory reservations, connections, scheduling, the KV cache, native executors, and cancellation. This state is not part of an immutable configuration or plan digest.

The Python binding validates input buffers, releases the Python interpreter lock while Rust runs, and returns immutable bytes. Python coordinates models and protocols. Runtime graph support and checkpoint loading remain separate from model planning.

Before execution, each profile enforces its runtime capabilities and verification contract; compiled paths additionally bind their plan and weights. Assurance records remain evidence rather than a universal execution prerequisite. The gateway and benchmark CLIs accept the same typed Experiment target and derive the required role topology from its profile; unsupported component combinations fail at resolution. See local gateway.

How runtime pieces fit together

The runtime turns immutable choices into live state. A local topology owns provider processes, the client owns the application-facing request lifecycle, role components record placement, state components describe persistent model state, and verification components govern output checks. None of these objects alone proves that a model and profile have complete executable coverage.

Use the client facade for application calls and build_roles only for local development and measurement. A loopback topology exercises real process boundaries, but does not establish independent non-colluding operators.

API: Runtime objects and signatures

Python SDK example

Build local provider roles with `build_roles`

build_roles converts a typed experiment or pipeline into a LocalTopology. Before the context manager starts, it is an inspectable process plan; entering it starts separate inference and preparation children and owns their shutdown.

from pllm import Deployment, ExecutionBudget, Experiment
from pllm.profiles import MaskedLinearCpu
from pllm.runtime import LocalTopology, build_roles
from pllm.sources import TinyModel

experiment = Experiment(
    name="runtime-topology",
    pipeline=MaskedLinearCpu(TinyModel(model_id="docs-tiny")),
    deployment=Deployment.local(root="local://runtime-topology"),
    budget=ExecutionBudget(requests=1, max_input_tokens=64, max_new_tokens=1),
)
topology = build_roles(experiment)
assert isinstance(topology, LocalTopology)
assert not topology.started
assert tuple(status.role for status in topology.statuses) == (
    "inference",
    "preparation",
)

Send a synchronous streaming request with `OpenAI`

OpenAI is the synchronous trusted client. A streaming call returns ResponseStream, which should be used as a context manager so cancellation burns reserved material and closes the connection. This tiny local-test run is an insecure development topology, not a deployment claim.

from pllm import Model
from pllm.client import OpenAI, ResponseStream
from pllm.profiles import MaskedLinearCpu
from pllm.runtime import build_roles

pipeline = MaskedLinearCpu(Model.tiny(model_id="docs-tiny"))
with build_roles(pipeline, correlation_mode="local-test") as local:
    with local.client(bundle_cache_mode="off") as client:
        assert isinstance(client, OpenAI)
        stream = client.responses.create(
            input="Keep this prompt private.",
            max_output_tokens=2,
            stream=True,
        )
        assert isinstance(stream, ResponseStream)
        with stream:
            events = list(stream)
        assert events[0].type == "response.created"
        assert events[-1].type in {"response.completed", "response.incomplete"}
        assert client.privacy_audit.plaintext_prompt_bytes_sent == 0

Configure the async lifecycle with `AsyncOpenAI`

AsyncOpenAI accepts the same connection settings and exposes async resources and cleanup. When stream=True, responses.create() returns an async iterator of response events rather than the synchronous ResponseStream wrapper.

import asyncio

from pllm.client import AsyncOpenAI

async def inspect_client() -> None:
    async with AsyncOpenAI(
        base_url="http://127.0.0.1:1",
        api_key="docs-only",
        default_model="docs-tiny",
    ) as client:
        assert client.responses is not None
        assert client.runtime.privacy_audit.plaintext_prompt_bytes_sent == 0

asyncio.run(inspect_client())

The setup example performs no request; use the same URL and credential issued by your trusted deployment boundary. See client integration for application-facing connection patterns.

Place work with `Inference`

Inference is configuration metadata selected by shipped profiles. It identifies the role that owns provider-side model work; it is not a running server process.

from pllm import Deployment, ExecutionBudget, Experiment
from pllm.profiles import MaskedLinearCpu
from pllm.roles import Inference
from pllm.sources import TinyModel

experiment = Experiment(
    name="inference-placement",
    pipeline=MaskedLinearCpu(
        TinyModel(model_id="role-tiny"),
        inference=Inference(),
    ),
    deployment=Deployment.local(root="local://inference-placement"),
    budget=ExecutionBudget(requests=1, max_input_tokens=64, max_new_tokens=1),
)
assert experiment.pipeline.to_spec()["components"]["inference"]["component"] == (
    "pllm/inference"
)

The parties and offline work guide maps this placement record to the preparation and inference service boundaries.

Describe client-owned KV state with `ClientLocalKv`

ClientLocalKv records that KV state remains inside the trusted client boundary. It does not make every cache topology executable; transformed MPCache state still requires compatible compiler and runtime coverage.

from pllm.state import ClientLocalKv

state = ClientLocalKv()
description = state.describe()
assert state.component == "pllm/client-local-kv"
assert description.capabilities == ("fixed-capacity-client-kv",)

Verify prepared execution with `FreivaldsVerify`

FreivaldsVerify adds the reviewed one-use native check to the prepared profile. The target failure bound is part of the immutable configuration and shared session accounting.

from pllm import Deployment, ExecutionBudget, Experiment
from pllm.profiles import VerifiedMaskedLinearCpu
from pllm.sources import TinyModel
from pllm.verification import FreivaldsVerify

experiment = Experiment(
    name="verified-runtime",
    pipeline=VerifiedMaskedLinearCpu(
        TinyModel(model_id="verified-tiny"),
        verification=FreivaldsVerify(target_failure_bits=48),
    ),
    deployment=Deployment.local(root="local://verified-runtime"),
    budget=ExecutionBudget(requests=1, max_input_tokens=64, max_new_tokens=1),
)
assert experiment.resolve().verification_target_failure_bits == 48

Save this example as experiment.py and run pllm gateway --local --experiment experiment.py:experiment --trust-python. Configuration does not itself perform verification; the verified profile binds authenticated one-use projection material when the gateway executes requests.

Exercise the lower-level `LinearIntegrity` policy

LinearIntegrity is an uncomposed lower-level policy for explicit integration work. The helpers below check one modular matrix result; a fixed seed is appropriate only for a deterministic test, not live verification material.

import numpy as np

from pllm.verification import (
    LinearIntegrity,
    check_linear_result,
    create_linear_check_key,
)

policy = LinearIntegrity()
weight = np.array([[2, 1], [1, 3]], dtype=np.int64)
inputs = np.array([[4, 5]], dtype=np.int64)
outputs = inputs @ weight.T % 65537
key = create_linear_check_key(weight, modulus=65537, checks=2, seed=7)
assert check_linear_result(inputs, outputs, key).valid
assert policy.component == "pllm/linear-integrity"

Selecting LinearIntegrity alone does not create verifier-backed model execution. Use the verified profile for the supported prepared composition.

On this page