Waiting for the Reply
When the server sends Lena’s conversation to the model, the model writes the reply one piece at a time. A piece is called a token, and it is usually a word or part of a word. To write each token, the model reads the conversation and the part of the reply it has written so far, and chooses the next token. So the model does not produce the reply all at once. The longer the reply, the more steps it takes.
Here are my assumptions about how fast the model writes the reply:
- The first words are ready about one second after the server sends the conversation to the model.
- After that, the model writes about 40 words a second.
- A reply is about 400 words long.
So a reply takes about 10 seconds to write. The server sends its response when the whole reply is written. For 10 seconds, Lena sees nothing. Then the whole reply appears at once.
The model writes faster than people read. A person reads about 4 words a second. If the reply appeared as the model wrote it, Lena could start reading after one second, and the text would stay ahead of Lena for the rest of the reply. So we add a requirement:
- Latency: CourseChat must show the first words of a reply within about one second of the model writing them, and show the rest as the model writes it.
Let’s try what we already know.
Polling
For polling, the POST responds at once, with the identifier of the reply the server has started, 52818. The server saves the reply as it grows: every few tokens, it updates the reply’s row with the text so far. The row also has a status: writing while the model writes, and complete at the end.
Lena’s client polls for this reply every half second:
GET /conversations/31/messages/52818
Each response holds the text written so far and the status. The client shows the text. When the status is complete, the client stops polling. The earlier replies in the conversation do not change, so the client does not poll for them.
A half-second interval meets the latency requirement. But a 10-second reply takes 20 polls. Suppose 100 students are waiting for a reply at the same moment. That is 200 requests a second. Each poll is a full HTTP request, with its headers and credentials, and the server authenticates it. The server also writes the growing reply to the database many times a second, only so that the polls can read it.
Long polling
Lena’s client could send a long poll instead. It asks for the text after the part it already has, for example after the first 120 characters:
GET /conversations/31/messages/52818?after=120
The server keeps the request until the model writes more, and then responds with the new tokens. When the response says the reply is complete, the client stops.
Recall the rule from our comparison of polling and long polling: if the data changes all the time, a long poll never waits. The model writes new tokens many times a second. So each long poll ends almost at once, and the client sends the next one. If new tokens reach the server 10 times a second, Lena’s client sends 10 requests a second. With 100 students waiting, that is 1,000 requests a second, five times as many as polling every half second. Each request has 700 to 800 bytes of headers, and each response carries only a few tokens.
A WebSocket
Lena’s client could open a WebSocket, and the server could send the reply on it a few tokens at a time, each time in one small frame.
But the reply is the response to Lena’s request. On a WebSocket, a message has no response, so we would have to build our own request and response on top of it. Each frame would carry a number that ties it to Lena’s message, because Lena may have more than one reply on its way, for example in two conversations. We would also need our own error codes and a message that says the reply is done. In the previous chapter, we kept posting a message on the REST API so that we would not rebuild request and response this way.