Internet Engineering

NATS101

Introduction

Fall 2026 · Amirkabir University of Technology
@1995parham

History

  • NATS is a high-performance messaging system created by Derek Collison in 2010 written in Ruby.
  • It was originally built to serve as the message bus for Cloud Foundry
  • Handling internal communication among components of the system

It Addresses

  • Service Discovery
  • Low latency communication
  • Load balancing
  • Notification and events handling

Overview

  1. Client publishes message on foo subject
  2. Only clients subscribed to foo receive the message
A publisher sends PUB foo 5 hello. nats-server delivers MSG foo 1 5 and MSG foo 7 5 to the two foo subscribers; the bar subscriber gets nothing

How Simple!


INFO {"server_id":"NCNS32OEKOMMHTBVUYC7QWASFAP3BPEKGZG3QDIYRCBUL23WE3OVVBBO","server_name":"NCNS32OEKOMMHTBVUYC7QWASFAP3BPEKGZG3QDIYRCBUL23WE3OVVBBO","version":"2.1.7","proto":1,"git_commit":"bf0930e","go":"go1.13.10","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":1,"client_ip":"172.17.0.1"}

SUB greetings 1
+OK

PUB greetings 12
Hello World!
+OK

MSG greetings 1 12
Hello World!

Request/Response

A request is a publish with a reply subject: a unique _INBOX subject that only the requester listens on. A wildcard audit subscriber sees both the request and the reply.

Sequence: the requester subscribes to _INBOX.5qlz….*, publishes PUB help _INBOX.5qlz….Oi9iNmkp 5; the server delivers MSG help to the worker and to the audit subscriber on >; the worker publishes I can help on the inbox; the server delivers it to the requester and to audit

Request/Response (cont'd)

  • A publisher sends a message on a particular subject by attaching a reply subject name along with it.
  • Then, the subscribers who receive this message send the response to the reply subject that was specified in the request message.
  • Reply subjects have a unique name called inbox and the NATS server will send the message coming into these subjects to the relevant publisher.

Queue

Workers that subscribe to help with the same queue name form a group: each message goes to one random member of it.

900 requests on help: the queue group workers splits them 302, 303 and 295 across three workers, while a plain subscriber on help gets all 900

NATS As an Always Available Dial Tone

  • The main design constraints that define the style of the NATS project are:
    • Simplicity
    • Performance
    • Reliability
  • It does not offer any Persistence or Buffering
  • It is true Fire and Forget

Availability

  • NATS will try to protect itself at all costs to be available for all users
  • NATS client libraries, internally try to have an always established connection to one of the available NATS servers
  • In case a server fails, NATS will reconnect to another available server in the pool
  • NATS supports high-availability via a clustering mode that is set up as a full-mesh of the servers

Cluster Failover

Three servers n0, n1, n2 joined by routes in a full mesh. The client on n0 gets EOF when n0 is killed and reconnects to n2 after 0.61 s; the subscriber on n2 gets all 40 messages

Slow Consumer

The server will not buffer without limit for a client that does not read: it disconnects it instead.

The server queues pending bytes for a subscriber that is not reading. Past max_pending (64 MB) or a write_deadline (10 s) it closes the connection; the log says Slow Consumer Detected: MaxPending of 67108864 Exceeded, and the client sees only the TCP close, no -ERR

Ping/Pong

The server PINGs every client on an interval; a client that stops answering PONG is closed as stale.

Two timelines. A client that answers every PING with PONG stays connected. A silent client gets PINGs at 2.3 s and 5.3 s, then -ERR 'Stale Connection' at 8.3 s and the connection is closed

Subject Names

  • The dot character (.) has special meaning as part of the subject.
  • When using it as part of the subject, we can create namespaces (hierarchy) that can be later matched via wildcards.
SubjectMatches
foo.*.barfoo.hello.bar ✅, foo.hi.bar ✅
foo.*foo.hello ✅, foo.hi.bar ❌
foo.>foo.hello.bar ✅, foo.hi.bar ✅, foo.hello ✅

Wildcards, Token by Token

The subject foo.hello.bar is three tokens. foo.*.bar matches: star takes hello. foo.* does not match: bar is left over. foo.> matches: > takes hello.bar. *.hello.> matches. Star is exactly one token anywhere; > is one or more tokens and only last, so foo.> does not match foo

Queue Subscriptions: Hands-on

  • Open two terminals and subscribe both to requests, with the same queue group
  • Publish a few messages, and watch where each one lands
  • The point to look for: each message goes to exactly one worker, not to both

telnet localhost 4222
SUB requests workers 5

Worker 1 👷


INFO {"server_id":"NARXYLU3XEZSFYCFRV5WJ63GQS5WDV3WC4WAKCYDYY2IBGIETWJADZVS","server_name":"NARXYLU3XEZSFYCFRV5WJ63GQS5WDV3WC4WAKCYDYY2IBGIETWJADZVS","version":"2.1.8","proto":1,"git_commit":"c0b574f","go":"go1.14.8","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":4,"client_ip":"172.22.0.1"}
SUB requests workers 5
+OK

MSG requests 5 6
second

Worker 2 👷


INFO {"server_id":"NARXYLU3XEZSFYCFRV5WJ63GQS5WDV3WC4WAKCYDYY2IBGIETWJADZVS","server_name":"NARXYLU3XEZSFYCFRV5WJ63GQS5WDV3WC4WAKCYDYY2IBGIETWJADZVS","version":"2.1.8","proto":1,"git_commit":"c0b574f","go":"go1.14.8","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":5,"client_ip":"172.22.0.1"}
SUB requests workers 55
+OK

MSG requests 55 5
first

Publisher ✍️


INFO {"server_id":"NARXYLU3XEZSFYCFRV5WJ63GQS5WDV3WC4WAKCYDYY2IBGIETWJADZVS","server_name":"NARXYLU3XEZSFYCFRV5WJ63GQS5WDV3WC4WAKCYDYY2IBGIETWJADZVS","version":"2.1.8","proto":1,"git_commit":"c0b574f","go":"go1.14.8","host":"0.0.0.0","port":4222,"max_payload":1048576,"client_id":6,"client_ip":"172.22.0.1"}

PUB requests 5
first
+OK

PUB requests 6
second
+OK

Even More Features!

  • JetStream 🚀
  • Clustering ✨
  • Super-Cluster with Gateways 🗺️

JetStream

  • JetStream is a NATS built-in distributed persistence system.
  • JetStream is built-in to nats-server and you only need 1 (or 3 or 5 if you want fault-tolerance against 1 or 2 simultaneous NATS server failures) of your NATS server(s) to be JetStream enabled for it to be available to all the client applications.

Functionalities enabled by JetStream

  • Streaming: temporal decoupling between the publishers and subscribers
  • Replay policies
  • Retention policies and limits
  • Persistent distributed storage
  • Stream replication factor
  • Mirroring between streams
  • De-coupled flow control
  • Exactly once semantics

Streaming

temporal decoupling between the publishers and subscribers

Subscribers only receive the messages that are published when they are actively connected to the messaging system (i.e. they do not receive messages that are published while they are not subscribing or not running or disconnected).

However, nowadays a new way to provide this temporal de-coupling has been devised and has become 'mainstream': streaming.

Streams capture and store messages published on one (or more) subject and allow client applications to create subscribers (i.e. JetStream consumers) at any time to replay (or consume) all or some of the messages stored in the stream

Replay policies

JetStream consumers support multiple replay policies.

  • all of the messages currently stored in the stream, meaning a complete replay and you can select the replay policy(i.e. the speed of the replay) to be either:
    • instant
    the messages are delivered to the consumer as fast as it can take them.
    original
    the messages are delivered to the consumer at the rate they were published into the stream, which can be very useful for example for staging production traffic.
  • the last message stored in the stream, or the last message for each subject (as streams can capture more than one subject).
  • starting from a specific sequence number.
  • starting from a specific start time.

Retention policies and limits

Practically speaking, streams cannot always just keep growing forever and therefore JetStream supports multiple retention policies as well as the ability to impose size limits on streams.

limit

You can impose the following limits on a stream

  • Maximum message age
  • Maximum total stream size (in bytes)
  • Maximum number of messages in the stream
  • Maximum individual message size
  • You can also set limits on the number of consumers that can be defined for the stream at any given point in time.

Discard policy

What happens once the stream has reached one of its limits and a new message is published.

A stream with max_msgs 3 holds messages 1, 2 and 3 when message 4 arrives. With discard old, the default, message 1 is removed and the stream holds 2, 3 and 4. With discard new, the stream keeps 1, 2 and 3 and the publisher gets the error maximum messages exceeded

Retention policy

You choose one per stream; limits is the default.

The same three messages under each policy. Limits: all three are kept whatever was acked, until a limit on age, bytes or count. Interest: message 1, acked by both consumers a and b, is removed; message 2, acked by a only, and message 3 are kept; with no consumers a message is not kept at all. Workqueue: messages 1 and 2, acked by the one consumer w, are removed; message 3 is kept

Regardless of the retention policy selected, the limits (and the discard policy) always apply.

Persistent distributed storage

You can choose the durability as well as the resilience of the message store according to your needs.

  • Memory storage
  • File storage
  • Replication (1 (none), 2, 3) between nats servers for Fault Tolerance

JetStream uses a NATS optimized Raft consensus algorithm to distribute the persistence service between nats servers in a cluster while maintaining immediate consistency even when servers crash or the network drops messages.

Raft tolerates crash and omission faults. It is not Byzantine fault tolerant — it assumes that a server which does answer, answers honestly.

In JetStream the configuration for storing messages is defined separately from how they are consumed.

Storage is defined in a Stream and consuming messages is defined by multiple Consumers.

Stream replication factor

A stream's replication factor (R, often referred to as the number Replicas) determines how many places it is stored allowing you to tune to balance risk with resource usage and performance.

A stream that is easily rebuilt or temporary might be memory based with a R=1. A stream you cannot afford to lose should be file based with R=3, which survives losing one server.

ReplicasDescription
1Cannot operate during an outage of the server servicing the stream. Highly performant.
2No significant benefit at this time. We recommend using Replicas=3 instead.
3Can tolerate loss of one server servicing the stream. An ideal balance between risk and performance.
4No significant benefit over Replicas=3 except marginally in a 5 node cluster.
5Can tolerate simultaneous loss of two servers servicing the stream. Mitigates risk at the expense of performance.

Mirroring between streams

JetStream also allows server administrators to easily mirror streams, for example between different JetStream domains in order to offer disaster recovery.

You can also define a stream as one of the sources for another stream.

De-coupled flow control

JetStream provides de-coupled flow control over streams, the flow control is not 'end to end' where the publisher(s) are limited to publish no faster than the slowest of all the consumers (i.e. the lowest common denominator) can receive, but is instead happening individually between each client application (publishers or consumers) and the nats server.

Exactly once semantics

Because publications to streams using the JetStream publish calls are acknowledged by the server, the base quality of service offered by streams is at least once.

Meaning that while reliable and normally duplicate free there are some specific failure scenarios:

  • That could result in a publishing application believing (wrongly) that a message was not published successfully and therefore publishing it again.
  • That could result in a client application's consumption acknowledgement getting lost and therefore the message being re-sent to the consumer by the server.

Therefore, JetStream also offers exactly once semantics.

Publish side: the publisher sends orders.new.6 with header Nats-Msg-Id order-6; the ack stream ORDERS seq 6 is lost, so it publishes again; the server has seen the id inside the duplicate window, 2 minutes by default, and answers seq 6 duplicate true without storing it again. Consume side: the server delivers stream seq 6, the consumer sends +ACK as a request, and the server's empty reply confirms the ack, so the message is not sent again

Consumers

A consumer is a view on a stream, with its own position in it.

A publisher writes orders.new.1 to 6 into the stream ORDERS, subjects orders.>, which keeps messages 1 to 6. The durable pull consumer workers starts at seq 1 and is shared by worker 1, which gets 1 3 5, and worker 2, which gets 2 4 6. The consumer audit starts at seq 4 and its app gets 4 5 6

Fast push consumers

Client applications can choose to use fast un-acknowledged push (ordered) consumers to receive messages as fast as possible (for the selected replay policy) on a specific delivery subject or to an inbox.

Those consumers are meant to be used to replay rather than consume the messages in a stream.

Horizontally scalable pull consumers with batching

Client applications can also use and share pull consumers that are demand-driven, support batching and must explicitly acknowledge message reception and processing which means that they can be used to consume (i.e. use the stream as a distributed queue) as well as process the messages in a stream.

Pull consumers can and are meant to be shared between applications (just like queue groups) in order to provide easy and transparent horizontal scalability of the processing or consumption of messages in a stream without having (for example) to worry about having to define partitions or worry about fault-tolerance.

References 📚

Fork me on GitHub