Event Streams

A topic’s log, as in Kafka, is often called an event stream:

• Event: each message records something that happened, such as “order 93 was placed”. It does not tell a service what to do, such as “ship order 93”.
• Stream: the messages keep coming and never end. The log keeps them in order, so a subscriber can read them again or start from an older one.

The messages we publish to order-placed are events. Each one says that an order was placed, and each subscriber decides what to do about it.

Stream processing

A subscriber can compute over the messages in an event stream as they arrive. One example is counting the orders placed each minute. This is called stream processing. Kafka calls itself an “event streaming platform”: it publishes, stores, and processes event streams. How stream processing works is beyond the scope of this course.

Stream and batch processing

Stream processing is usually compared with batch processing: collect the data first, then process all of it at once at set times. A report built every night from the orders in the database is batch processing.

• Stream processing: the count is ready within seconds of each order.
• Batch processing: the count is ready after the next run. It is simpler, because all the data is already there when the run starts.

You have seen the idea behind batch processing in the materialized view. A materialized view stores a result that is computed ahead of time from the tables. We rebuild it on each refresh, so between refreshes it is stale. A nightly report is computed ahead of time in the same way, so it is stale between runs.