Streaming

Set stream: true. The gateway relays the model server's SSE stream byte-for-byte and adds two things: a usage chunk and a final annotated chunk.

Usage

stream_options.include_usage is forced on upstream so every stream ends with a chunk carrying usage. That is what metering uses; you can read it too.

Final chunk

Before data: [DONE], one extra chunk has empty choices, the usage object, and the unrefused annotation. Output-stage decisions arrive here, because headers were sent before the model finished.

data: {"id":"chatcmpl-…","object":"chat.completion.chunk","choices":[],
       "usage":{"prompt_tokens":9,"completion_tokens":42,"total_tokens":51},
       "unrefused":{"request_id":"req_…","policy":"eval-harness@v14","input":"pass","output":"pass","retention":"none"}}

data: [DONE]

The response header x-unrefused-output reads pending on streams for this reason.

Blocking a stream

Bytes already delivered cannot be taken back. A profile can set output.buffer_tokens to hold back the first N tokens, run the output classifiers on them, and only then start flushing. If that check blocks, the client receives an error event instead of any content:

data: {"error":{"message":"Request blocked by policy (output): blocked before first flush",
       "type":"invalid_request_error","param":"output","code":"policy_blocked"}}

data: [DONE]

With buffer_tokens: 0 the stream starts immediately and output decisions are annotations only. Choose the trade-off per profile.

Disconnects and timeouts

• If your client disconnects, the upstream request is cancelled and the in-flight slot is released.
• If the model server stalls for longer than the idle timeout, you receive an upstream_timeout error event and [DONE].
• Usage for partial streams is recorded with status 499.