Low Water Mark and High Water Mark in System Design

When designing large-scale distributed systems, we deal with thresholds all the time — memory, queues, replication logs, caches, and network buffers. Two key concepts that help maintain stability and efficiency are High Water Mark (HWM) and Low Water Mark (LWM).

They are critical in replication protocols, messaging queues, and quorum systems.

🔹 High Water Mark (HWM)

In system design, HWM is the maximum threshold at which the system takes corrective action to avoid overload or inconsistency.

  • It prevents overflow, ensures stability, and protects resources.

🔹 Low Water Mark (LWM)

The LWM is the safe lower threshold that signals the system has recovered enough to resume normal operations.

  • It prevents rapid toggling between "block/unblock" states (avoids thrashing).

🔹 Why Do We Need Both?

The gap between HWM and LWM (called hysteresis) ensures smooth control.

  • Without it, systems would keep toggling rapidly whenever they touch the threshold.

🔹 Applications in System Design

1. Replication (Databases & Logs)

In replication protocols like Raft or Kafka log replication:

  • High Water Mark (HWM):
    The highest log entry known to be safely replicated across a quorum of followers.

    • Leaders commit up to this point.

    • Clients only see data beyond HWM as "committed."

  • Low Water Mark (LWM):
    The oldest log entry that is still retained across replicas before it can be truncated or garbage collected.

    • Ensures lagging replicas can still catch up.

👉 Example:
In Kafka:

  • HWM = last offset replicated to all in-sync replicas (ISR).

  • LWM = earliest offset that is still required (older logs may be deleted once all replicas pass this point).

2. Message Queues (Kafka, RabbitMQ, JMS)

Queues need flow control between producers and consumers.

  • HWM (queue length threshold):
    If the queue size reaches this point, producers are throttled or blocked to avoid memory overflow.

  • LWM (queue recovery point):
    Once the queue drains below this level, producers are unblocked and can resume publishing freely.

👉 Example:

  • Queue size = 1000 messages.

  • HWM = 800 → producers slow down/backpressure.

  • LWM = 400 → safe to resume.

3. Quorum Systems (Consensus Protocols)

In distributed systems (Raft, Paxos, Zookeeper, Cassandra), quorum decisions often depend on watermarks:

  • High Water Mark (HWM):
    The highest operation acknowledged by a quorum of nodes (majority).

    • Ensures durability and strong consistency.

    • Example: Raft leader commits log entries only once HWM is replicated to a majority.

  • Low Water Mark (LWM):
    The minimum point at which enough replicas have acknowledged operations.

    • Used for safe cleanup of logs or state machine snapshots.

    • Prevents removing entries still needed by lagging nodes.

👉 Example:
In Zookeeper/ZAB protocol:

  • HWM = last committed transaction agreed by quorum.

  • LWM = last transaction safe to garbage collect.

🔹 Visualization

Replication Logs Example
--------------------------------------------------
0   10   20   30   40   50   60   70   80   90  100
^                 ^                     ^
|                 |                     |
LWM           Safe Zone              HWM
(earliest     Operations        (highest fully
 kept logs)   in-flight range   committed entry)

🔹 Key Takeaways

  • Replication:

    • HWM = last safely replicated log entry.

    • LWM = earliest retained log for lagging nodes.

  • Queues:

    • HWM = max queue threshold → apply backpressure.

    • LWM = min queue threshold → resume producers.

  • Quorum:

    • HWM = operation confirmed by majority → commit point.

    • LWM = safe cleanup threshold.

👉 In short:

  • HWM protects against overload/inconsistency.

  • LWM ensures recovery & smooth operation.

  • Together, they balance safety, performance, and stability in distributed systems.

Would you like me to draw parallels between Kafka’s LWM/HWM and Raft’s quorum-based watermarks (side-by-side comparison), so the blog has a deeper distributed-systems flavor?

Comments

Post a Comment

Popular posts from this blog

Ensuring Missing Resources Are Created Automatically in a Spring Boot Project

Image
In a Spring Boot application , many resources—like directories, files, database tables, or cloud resources—are essential for the app to function correctly. But what if these resources don’t exist? Without them, your application may fail or behave unexpectedly. The good news: Spring Boot provides mechanisms to automatically create missing resources at startup . Let’s explore how to handle different scenarios. 1. Automatically Creating Directories Applications often need directories for file uploads, logs, or temporary storage. You can ensure these directories exist using Spring’s lifecycle hooks : import jakarta.annotation.PostConstruct ; import org.springframework.stereotype.Component ; import java.io.File ; @Component public class DirectoryInitializer { private static final String UPLOAD_DIR = "uploads"; @PostConstruct public void init() { File dir = new File(UPLOAD_DIR); if (!dir.exists()) { boolean created = dir.mkdirs(); ...
Read more

Singular Update Queue: A Smarter Way to Handle Updates

In modern systems, efficiency and consistency are key. Whether you are building a database engine, a high-performance backend service, or a user-facing application, the way you handle updates can make a huge difference in system stability and responsiveness. One elegant pattern that often emerges is the Singular Update Queue . What is a Singular Update Queue? A Singular Update Queue is a mechanism where multiple update requests targeting the same resource are funneled into a single queue , ensuring updates are: Serialized – executed in strict order, preventing race conditions. Consolidated – redundant or frequent updates are batched together. Efficient – avoids re-processing the same data multiple times. Instead of having multiple threads or services updating a resource simultaneously, a single queue ensures that only one update is processed at a time . Why Use a Singular Update Queue? Avoid Race Conditions Concurrent writes can corrupt state. A queue ensure...
Read more

WebSocket vs gRPC: A Detailed Guide to Modern Communication Protocols

In today’s world of distributed applications, microservices, and real-time user experiences, communication protocols are the backbone of scalable systems. Two prominent technologies stand out for client-server and service-to-service communication: WebSockets and gRPC . While both aim to improve data exchange efficiency, they differ fundamentally in architecture, performance, and ideal use cases. Let’s dive deep into both. 🔹 What is WebSocket? WebSocket is a full-duplex communication protocol built over TCP, enabling persistent connections between client and server. Unlike HTTP (request-response), WebSocket allows real-time bidirectional communication . Key Characteristics: Protocol : Works over TCP (ws:// or wss:// for secure). Persistent connection : Once established, the connection remains open. Full-duplex : Both client and server can send messages anytime. Text & Binary : Supports JSON, text, and binary data. Browser-friendly : Widely supported in modern ...
Read more