pllm.metrics Python API

Metric definitions attach units and optimization direction to measurements so evidence and Pareto search do not guess whether larger or smaller is better. Presence documents API availability, not complete model, security, or deployment coverage; check implementation status before relying on a path.

Research context

This API is project infrastructure rather than an implementation of one specific paper. Relevant method pages link their primary sources separately.

Python SDK example

The example performs a bounded offline task and checks an observable result. It does not download a checkpoint or contact a provider.

import pllm.metrics as metrics

latency = metrics.Latency(statistic="p95", phase="online")
assert latency.to_spec()["params"] == {"phase": "online", "statistic": "p95"}

API: pllm.metrics

Objects and signatures

Accuracy

Selects component pllm/accuracy during benchmark; it provides ratio, higher-is-better.

• Canonical object: pllm.metrics.Accuracy
• Kind: class
• Signature/type: __init__(self, *, dataset: str, measure: str='task_score') -> None
• Public exports: pllm.metrics.Accuracy
• Public members: descriptor: ComponentDescriptor; __init__(self, *, dataset: str, measure: str='task_score') -> None; classmethod describe(cls) -> ComponentDescriptor

Communication

Selects component pllm/communication during benchmark; it provides bytes, lower-is-better.

• Canonical object: pllm.metrics.Communication
• Kind: class
• Signature/type: __init__(self, *, direction: str='total', phase: str='online') -> None
• Public exports: pllm.metrics.Communication
• Public members: descriptor: ComponentDescriptor; __init__(self, *, direction: str='total', phase: str='online') -> None; classmethod describe(cls) -> ComponentDescriptor

Cost

Selects component pllm/cost during benchmark; it provides currency, lower-is-better.

• Canonical object: pllm.metrics.Cost
• Kind: class
• Signature/type: __init__(self, *, currency: str='USD', basis: str='request') -> None
• Public exports: pllm.metrics.Cost
• Public members: descriptor: ComponentDescriptor; __init__(self, *, currency: str='USD', basis: str='request') -> None; classmethod describe(cls) -> ComponentDescriptor

Energy

Selects component pllm/energy during benchmark; it provides joules, lower-is-better.

• Canonical object: pllm.metrics.Energy
• Kind: class
• Signature/type: __init__(self, *, source: str='not_available') -> None
• Public exports: pllm.metrics.Energy
• Public members: descriptor: ComponentDescriptor; __init__(self, *, source: str='not_available') -> None; classmethod describe(cls) -> ComponentDescriptor

Latency

Selects component pllm/latency during benchmark; it provides seconds, lower-is-better.

• Canonical object: pllm.metrics.Latency
• Kind: class
• Signature/type: __init__(self, *, statistic: str='median', phase: str='online') -> None
• Public exports: pllm.metrics.Latency
• Public members: descriptor: ComponentDescriptor; __init__(self, *, statistic: str='median', phase: str='online') -> None; classmethod describe(cls) -> ComponentDescriptor

Memory

Selects component pllm/memory during benchmark; it provides bytes, lower-is-better.

• Canonical object: pllm.metrics.Memory
• Kind: class
• Signature/type: __init__(self, *, kind: str='peak_rss') -> None
• Public exports: pllm.metrics.Memory
• Public members: descriptor: ComponentDescriptor; __init__(self, *, kind: str='peak_rss') -> None; classmethod describe(cls) -> ComponentDescriptor

Metric

Immutable Metric record carrying component, params for the workflow where metric definitions attach units and optimization direction to measurements so evidence and pareto search do not guess whether larger or smaller is better.

• Canonical object: pllm.metrics.Metric
• Kind: class
• Signature/type: constructor not exposed
• Public exports: pllm.metrics.Metric

Perplexity

Selects component pllm/perplexity during benchmark; it provides ratio, lower-is-better.

• Canonical object: pllm.metrics.Perplexity
• Kind: class
• Signature/type: __init__(self, *, dataset: str) -> None
• Public exports: pllm.metrics.Perplexity
• Public members: descriptor: ComponentDescriptor; __init__(self, *, dataset: str) -> None; classmethod describe(cls) -> ComponentDescriptor

Throughput

Selects component pllm/throughput during benchmark; it provides per-second, higher-is-better.

• Canonical object: pllm.metrics.Throughput
• Kind: class
• Signature/type: __init__(self, *, basis: str='tokens') -> None
• Public exports: pllm.metrics.Throughput
• Public members: descriptor: ComponentDescriptor; __init__(self, *, basis: str='tokens') -> None; classmethod describe(cls) -> ComponentDescriptor