Evaluate streaming LLM output as it arrives

When to use this

Use the streaming endpoint when you want to block a response while it is still being generated - not after the whole response is already in the user's hands. Typical cases:

• Token-streaming chat UIs (OpenAI / Anthropic style).
• Long-form generation where waiting for evaluate/output after the full response would mean the user has already seen unsafe text.
• Multi-agent pipelines where a tool-call argument needs to be screened before the next tool fires.

Sliding window model

The request is a list of chunks (one per stream event). The engine concatenates them into a rolling buffer and evaluates every time the buffer reaches window_size characters, with overlap characters carried forward so phrases spanning two chunks still match.

Defaults (window_size=200, overlap=40) are tuned for Latin-script chat use. Tune up if you get false-positive detector fires at chunk boundaries.

Text is never returned before it has been evaluated. A chunk event is emitted only after the window covering that text has been evaluated and allowed, and always after that window's guardrail event. The overlap tail of each window is held back until the following window clears it, so a phrase straddling a window boundary cannot be forwarded ahead of the window that catches it. Anything still buffered when the chunk list ends is evaluated and released at the end of the stream.

The practical consequence: concatenating the text of every chunk event you receive, in order, gives you exactly the text that is safe to show. On a block, that concatenation stops at the last clean window.

Server-Sent Events

The response is text/event-stream with four event types:

event: guardrail
data: {"window_index": 1, "decision": "ALLOW", "risk_score": 12,
       "rule_hits": [], "latency_ms": 4, "text_preview": "Hello, let me help..."}

event: chunk
data: {"text": "Hello, let me help...", "chunk_index": 1}

event: block
data: {"window_index": 3, "decision": "BLOCK", "risk_score": 92,
       "rule_hits": [{"rule_id": "pii", "message": "Email address in output"}],
       "latency_ms": 6}

event: done
data: {"total_chunks": 7, "windows_evaluated": 5, "final_decision": "BLOCK",
       "max_risk_score": 92, "total_rule_hits": 2, "total_latency_ms": 28,
       "full_text_length": 1024, "released_text_length": 610}

Notes on the payloads:

• chunk_index counts releases, not request chunks. Windows do not line up with the chunk boundaries you sent, so use this to order the text you receive rather than to correlate back to your input.
• block carries no text_preview. The verdict event deliberately does not echo the text the stream is withholding, so forwarding or logging every SSE event cannot re-open the channel the block just closed. Correlate on window_index and rule_hits.
• done always arrives last. released_text_length is how much text the stream actually handed you; on an ALLOW it equals full_text_length, and on a BLOCK the difference is the text that was withheld.

Treat block as terminal: no further chunk events follow it.

Common pitfalls

• chunks takes strings, not raw token IDs. Decode upstream from whatever your LLM client yields.
• Render from chunk events, not from your own copy of the text. If you echo your own buffer to the user as you build the request, you have defeated the guardrail - the endpoint can only withhold text you have not already displayed.
• There is no inline policy field. The policy is resolved from the runtime's active bundle by tenant_id / app_id / agent_id / env, exactly as on /evaluate/output, so a caller cannot evaluate a stream against a weaker rule set than the one published for its scope. If the runtime is fail-closed and has no bundle loaded, the endpoint returns 503 rather than opening a stream it cannot police.
• This endpoint lives on the runtime, not the control plane. Point your client at your runtime's hostname, not api.znyx.ai.

Related

• POST /v1/evaluate/output - non-streaming equivalent, simpler to wire up.
• POST /v1/evaluate/input - screen user input before the LLM sees it.