Server-Sent Events

The server can start sending the response to Lena’s POST before the whole reply is written, and send the rest as the model writes it. This is called streaming the response.

In the responses we have seen so far, the server has the whole body before it sends anything. It puts the body’s length in the Content-Length header, and the client reads that many bytes. A streamed response has no Content-Length, because the server does not know the length yet. Over HTTP/1.1, the server sends the header Transfer-Encoding: chunked instead, and then sends the body in chunks, one after another, as it has them. HTTP/2 sends every body in chunks anyway. Either way, the client can read each chunk as it arrives.

The client could add the text of each chunk to the screen as it arrives. But the response has to carry more than the reply’s text. At the end, the client needs the reply’s identifier. If the model fails partway, the client needs to know about the error. So the client has to tell one kind of data from another, and know where each one ends. Chunks do not help with this. The network can split the body anywhere, so a chunk can hold half of one part, or several parts. We need a format for the body.

The format

Server-sent events (SSE) is a standard format for a response body that the server sends as a series of events, over time. An event is one part of the body: a few lines of text, ended by a blank line. The response’s Content-Type is text/event-stream.

Here is the response to Lena’s POST:

HTTP/1.1 201 Created
Location: /conversations/31/messages/52817
Content-Type: text/event-stream
Cache-Control: no-store

event: text
data: {"text": "Because the shard"}

event: text
data: {"text": " of a link depends on N."}

event: text
data: {"text": " When N changes, ..."}

event: done
data: {"id": 52818}

Each line in an event is a field: a name, a colon, and a value.

  • data: holds the event’s content, as text. We put a small JSON object in it. The reply can contain line breaks, and JSON writes each one as \n, so a line break in the reply does not end the data: line.
  • event: names the event’s type. The client handles each type in its own way. Here there are two types: text carries the next few tokens of the reply, and done says the reply is complete and gives its identifier, 52818.

The server sends the status line and the headers right away. Then it sends a text event each time the model writes a few tokens, and a done event at the end. After done, the server ends the response. The client adds the text of each text event to the screen as it arrives. In the browser, fetch gives the client the body as it arrives, and a small library splits it into events. Lena sees the first words about one second after sending the question.

Cache-Control: no-store tells the browser and any cache along the way not to store the response, as we did for error responses earlier. A stored copy of this stream would not help the next request.

The model runs at a model provider, the company behind it, such as OpenAI or Anthropic. Our server sends the conversation to the provider in a request. The provider streams its response back in this same format, as server-sent events. Our server reads each of these events and writes a text event to Lena’s response.

Errors

The status line is the first thing the server sends, and once it is sent, the server cannot change it. So an error that the server finds before it starts the stream gets a status code, as usual:

  • 400 Bad Request: a message with an empty body.
  • 404 Not Found: a conversation that does not exist, or that belongs to another student.
  • 429 Too Many Requests: a student who sent too many messages in a short time. Rate limiting matters here, because the model provider charges for every token.

An error can also happen after the stream starts. For example, the model provider may fail halfway through the reply. By then, the response already says 201 Created, so the server sends the error as an event and then ends the response:

event: error
data: {"message": "The reply could not be completed."}

The client shows the error under the part of the reply it already has.

Proxies that buffer

Recall that a request can pass through proxies and load balancers on its way to the server. Some of them collect the whole response before they pass it on to the client. nginx, for example, does this by default. Then Lena’s client gets every event at once, when the reply is complete.

So we have to turn buffering off for streamed responses. With nginx, the server can turn buffering off for one response by sending the header X-Accel-Buffering: no.

What a stream costs the server

While the model writes the reply, the server holds Lena’s request and keeps its connection open, the same way it does for a long poll. A long poll waits up to 30 seconds. A reply takes about 10 seconds, and then the response ends. So at any moment, the server holds one open response for each student whose reply is being written.