HTTP Operations - Durable Streams

The Durable Streams protocol defines six core HTTP operations for interacting with streams. All operations are applied to a stream URL, and the protocol is defined by the HTTP methods, query parameters, and headers used.

Create Stream

Creates a new stream at the specified URL. Request:

PUT {stream-url}

Idempotent Behavior: If the stream already exists, the server will:

Return 200 OK if the existing stream’s configuration (content type, TTL/expiry, and closure status) matches the request
Return 409 Conflict if configuration does not match

Request Headers

Content-Type

string

Sets the stream’s content type. If omitted, the server may default to application/octet-stream.

Stream-TTL

integer

Sets a relative time-to-live in seconds from creation. Must be a non-negative integer in decimal notation without leading zeros, plus signs, decimal points, or scientific notation (e.g., 3600 is valid; +3600, 03600, 3600.0, and 3.6e3 are not).

Stream-Expires-At

string

Sets an absolute expiry time as an RFC 3339 timestamp. If both Stream-TTL and Stream-Expires-At are supplied, servers should reject the request with 400 Bad Request.

Stream-Closed

boolean

When set to true, the stream is created in the closed state. Any body provided becomes the complete and final content of the stream. This enables atomic “create and close” semantics for single-message or empty streams.

Request Body

Optional initial stream bytes. If provided, these bytes form the first content of the stream.

Response Codes

201 Created: Stream created successfully
200 OK: Stream already exists with matching configuration (idempotent success)
409 Conflict: Stream already exists with different configuration
400 Bad Request: Invalid headers or parameters
429 Too Many Requests: Rate limit exceeded

Response Headers

Location

string

The stream URL (included on 201 Created)

Content-Type

string

The stream’s content type

Stream-Next-Offset

string

The tail offset after any initial content

Stream-Closed

boolean

Present when the stream was created in the closed state

Example

curl -X PUT https://your-server.com/v1/stream/my-stream \
  -H "Content-Type: application/json" \
  -H "Stream-TTL: 3600" \
  -d '{"event":"user.created","userId":"123"}'

Append to Stream

Appends bytes to the end of an existing stream. Request:

POST {stream-url}

Supports both full-body and streaming (chunked) append operations. Optionally closes the stream atomically with the append.

Request Headers

Content-Type

string

required

Must match the stream’s existing content type when a body is provided. May be omitted when the request body is empty (close-only requests with Stream-Closed: true).

Transfer-Encoding

string

Set to chunked for streaming bodies. Servers should support HTTP/1.1 chunked encoding and HTTP/2 streaming semantics.

Stream-Seq

string

A monotonic, lexicographic writer sequence number for coordination. If provided and less than or equal to the last appended sequence, the server returns 409 Conflict. Sequence numbers must be strictly increasing.

Stream-Closed

boolean

When set to true, the stream is closed after the append completes. This is an atomic operation: the body (if any) is appended as the final data, and the stream transitions to the closed state. Close-only requests (with empty body) are idempotent.

Producer-Id

string

Client-supplied stable identifier for idempotent producers (e.g., “order-service-1”, UUID). Must be provided together with Producer-Epoch and Producer-Seq.

Producer-Epoch

integer

Client-declared epoch for idempotent producers, starting at 0. Increment on producer restart to establish a new session. Must be a non-negative integer ≤ 2^53-1.

Producer-Seq

integer

Monotonically increasing sequence number per epoch for idempotent producers. Starts at 0 for each new epoch. Must be a non-negative integer ≤ 2^53-1.

Request Body

Bytes to append to the stream. Servers must reject POST requests with an empty body (Content-Length: 0 or no body) with 400 Bad Request, unless the Stream-Closed: true header is present.

Response Codes

204 No Content: Append successful (or stream already closed when closing idempotently)
200 OK: Append successful with idempotent producer (new data)
400 Bad Request: Malformed request (invalid header syntax, missing Content-Type, empty body without Stream-Closed: true)
404 Not Found: Stream does not exist
405 Method Not Allowed or 501 Not Implemented: Append not supported for this stream
409 Conflict: Content type mismatch, sequence regression, or stream is closed
413 Payload Too Large: Request body exceeds server limits
429 Too Many Requests: Rate limit exceeded
403 Forbidden: Stale producer epoch (with idempotent producer headers)

Response Headers

Stream-Next-Offset

string

The new tail offset after the append

Stream-Closed

boolean

Present when the stream is now closed (either by this request or previously)

Producer-Epoch

integer

Echoed back on success with idempotent producer, or current server epoch on stale epoch (403)

Producer-Seq

integer

On success with idempotent producer, the highest accepted sequence number for this (stream, producerId, epoch) tuple

Producer-Expected-Seq

integer

On 409 Conflict (sequence gap), the expected sequence

Producer-Received-Seq

integer

On 409 Conflict (sequence gap), the received sequence

Example

curl -X POST https://your-server.com/v1/stream/my-stream \
  -H "Content-Type: application/json" \
  -d '{"event":"user.updated","userId":"123"}'

Idempotent Producer Example

curl -X POST https://your-server.com/v1/stream/my-stream \
  -H "Content-Type: application/json" \
  -H "Producer-Id: service-1" \
  -H "Producer-Epoch: 0" \
  -H "Producer-Seq: 0" \
  -d '{"event":"order.created","orderId":"456"}'

Close Stream

Closes a stream without appending data. Request:

POST {stream-url}
Stream-Closed: true

Response Codes

204 No Content: Stream closed successfully (or already closed—idempotent)
404 Not Found: Stream does not exist
405 Method Not Allowed or 501 Not Implemented: Append/close not supported for this stream

Response Headers

Stream-Next-Offset

string

The tail offset (unchanged, since no data was appended)

Stream-Closed

boolean

Confirms the stream is now closed

Example

curl -X POST https://your-server.com/v1/stream/my-stream \
  -H "Stream-Closed: true"

Delete Stream

Deletes the stream and all its data. Request:

DELETE {stream-url}

Response Codes

204 No Content: Stream deleted successfully
404 Not Found: Stream does not exist
405 Method Not Allowed or 501 Not Implemented: Delete not supported for this stream

Example

curl -X DELETE https://your-server.com/v1/stream/my-stream

Stream Metadata

Checks stream existence and returns metadata without transferring data. Request:

HEAD {stream-url}

This is the canonical way to find the tail offset, TTL, expiry information, and closure status.

Response Codes

200 OK: Stream exists
404 Not Found: Stream does not exist
429 Too Many Requests: Rate limit exceeded

Response Headers

Content-Type

string

The stream’s content type

Stream-Next-Offset

string

The tail offset (next offset after the current end)

Stream-TTL

integer

Remaining time-to-live in seconds, if applicable

Stream-Expires-At

string

Absolute expiry time as RFC 3339 timestamp, if applicable

Stream-Closed

boolean

Present when the stream has been closed. Absence indicates the stream is still open.

Cache-Control

string

Servers should return no-store to avoid stale tail offsets and closure status

Example

curl -I https://your-server.com/v1/stream/my-stream

Read Stream - Catch-up

Returns bytes starting from the specified offset for historical replay. Request:

GET {stream-url}?offset=<offset>

Query Parameters

offset

string

Start offset token. If omitted, defaults to the stream start (offset -1). Use -1 to read from the beginning or now to start from the current tail.

Response Codes

200 OK: Data available (or empty body if offset equals tail)
400 Bad Request: Malformed offset or invalid parameters
404 Not Found: Stream does not exist
410 Gone: Offset is before the earliest retained position (retention/compaction)
429 Too Many Requests: Rate limit exceeded

Response Headers

Cache-Control

string

Derived from TTL/expiry. For shared streams: public, max-age=60, stale-while-revalidate=300. For user-specific streams: private, max-age=60, stale-while-revalidate=300.

ETag

string

Entity tag for cache validation, format: {internal_stream_id}:{start_offset}:{end_offset}

Stream-Cursor

string

Cursor to echo on subsequent long-poll requests for CDN collapsing. Optional for catch-up, required for live modes.

Stream-Next-Offset

string

The next offset to read from for subsequent requests

Stream-Up-To-Date

boolean

Must be true when the response includes all data available in the stream at the time the response was generated. Should not be present when returning partial data due to server-defined chunk size limits.

Stream-Closed

boolean

Must be present when the stream is closed and the client has reached the final offset. Indicates EOF (end-of-file).

Response Body

Bytes from the stream starting at the specified offset, up to a server-defined maximum chunk size.

Example

# Read from beginning
curl "https://your-server.com/v1/stream/my-stream?offset=-1"

# Resume from offset
curl "https://your-server.com/v1/stream/my-stream?offset=abc123xyz"

# Read from current tail (skip all existing data)
curl "https://your-server.com/v1/stream/my-stream?offset=now"

Read Stream - Live (Long-poll)

If no data is available at the specified offset, the server waits up to a timeout for new data to arrive. Request:

GET {stream-url}?offset=<offset>&live=long-poll[&cursor=<cursor>]

Query Parameters

offset

string

required

The offset to read from. Must be provided.

live

string

required

Set to long-poll to indicate long-polling mode.

cursor

string

Echo of the last Stream-Cursor header value from a previous response. Used for collapsing keys in CDN/proxy configurations.

Response Codes

200 OK: Data became available within the timeout
204 No Content: Timeout expired with no new data
400 Bad Request: Invalid parameters
404 Not Found: Stream does not exist
429 Too Many Requests: Rate limit exceeded

Response Headers (on 200)

Stream-Cursor

string

Servers must include this header for CDN collapsing

Stream-Next-Offset

string

The next offset to read from

Stream-Up-To-Date

boolean

Indicates the client is caught up with all available data

Stream-Closed

boolean

Must be present when the stream is closed (EOF signal)

Response Headers (on 204)

Stream-Next-Offset

string

The current tail offset

Stream-Up-To-Date

boolean

Must be true to indicate the client is caught up

Stream-Cursor

string

Must be included when the stream is open. May be omitted when Stream-Closed is true.

Stream-Closed

boolean

Must be present when the stream is closed. A 204 No Content with Stream-Closed: true indicates EOF.

Example

curl "https://your-server.com/v1/stream/my-stream?offset=abc123xyz&live=long-poll"

Read Stream - Live (SSE)

Returns data as a Server-Sent Events (SSE) stream. Request:

GET {stream-url}?offset=<offset>&live=sse

Query Parameters

offset

string

required

The offset to start reading from

live

string

required

Set to sse to indicate SSE streaming mode

Response Codes

200 OK: Streaming body (SSE format)
400 Bad Request: Invalid parameters
404 Not Found: Stream does not exist
429 Too Many Requests: Rate limit exceeded

Response Format

Data is emitted in Server-Sent Events format. The response uses Content-Type: text/event-stream. For streams with content-type text/* or application/json, data events carry UTF-8 text directly. For all other content types (binary streams), servers automatically base64-encode data events and include the response header Stream-SSE-Data-Encoding: base64. Events:

data: Emitted for each batch of data. For binary streams, the payload is base64-encoded per RFC 4648. For JSON streams, implementations may batch multiple messages into a single SSE data event by streaming a JSON array across multiple data: lines.
control: Emitted after every data event. Format is a JSON object with camelCase field names:
- streamNextOffset (required): The next offset to read from
- streamCursor (required when stream is open): Cursor for reconnection. May be omitted when streamClosed is true.
- upToDate (required when true): Set to true when the client is caught up with all available data
- streamClosed (required when true): Set to true when the stream is closed and all data has been sent

SSE Event Examples

Normal data event:

event: data
data: [
data: {"k":"v"},
data: {"k":"w"},
data: ]

event: control
data: {"streamNextOffset":"123456_789","streamCursor":"abc"}

Final data with stream closure:

event: data
data: [
data: {"k":"final"}
data: ]

event: control
data: {"streamNextOffset":"123456_999","streamClosed":true}

Binary stream with base64 encoding:

event: data
data: AQIDBAUG
data: BwgJCg==

event: control
data: {"streamNextOffset":"123456_789","streamCursor":"abc"}

Example

curl "https://your-server.com/v1/stream/my-stream?offset=-1&live=sse"

Stream Closure Behavior

When the stream is closed:

The final control event must include streamClosed: true
After emitting the final control event, servers must close the SSE connection
Clients receiving streamClosed: true must not attempt to reconnect

Connection Lifecycle

Server should close connections roughly every ~60 seconds to enable CDN collapsing
Client must reconnect using the last received streamNextOffset value from the control event
Client must not reconnect if the last control event included streamClosed: true

Client Libraries

Server

CLI

Protocol

Documentation Index

​Create Stream

​Request Headers

​Request Body

​Response Codes

​Response Headers

​Example

​Append to Stream

​Request Headers

​Request Body

​Response Codes

​Response Headers

​Example

​Idempotent Producer Example

​Close Stream

​Response Codes

​Response Headers

​Example

​Delete Stream

​Response Codes

​Example

​Stream Metadata

​Response Codes

​Response Headers

​Example

​Read Stream - Catch-up

​Query Parameters

​Response Codes

​Response Headers

​Response Body

​Example

​Read Stream - Live (Long-poll)

​Query Parameters

​Response Codes

​Response Headers (on 200)

​Response Headers (on 204)

​Example

​Read Stream - Live (SSE)

​Query Parameters

​Response Codes

​Response Format

​SSE Event Examples

​Example

​Stream Closure Behavior

​Connection Lifecycle

Create Stream

Request Headers

Request Body

Response Codes

Response Headers

Example

Append to Stream

Request Headers

Request Body

Response Codes

Response Headers

Example

Idempotent Producer Example

Close Stream

Response Codes

Response Headers

Example

Delete Stream

Response Codes

Example

Stream Metadata

Response Codes

Response Headers

Example

Read Stream - Catch-up

Query Parameters

Response Codes

Response Headers

Response Body

Example

Read Stream - Live (Long-poll)

Query Parameters

Response Codes

Response Headers (on 200)

Response Headers (on 204)

Example

Read Stream - Live (SSE)

Query Parameters

Response Codes

Response Format

SSE Event Examples

Example

Stream Closure Behavior

Connection Lifecycle