Complete Guide to External State Stores: Implementation and Best Practices

Scaibu

Published in

DevOps.dev

17 min readNov 17, 2024

Introduction
Fundamental Concepts
State Store Technologies

1. Introduction

1.1 What are External State Stores?

External state stores are persistent storage systems that exist outside your microservice’s runtime environment but typically within the same network boundary. They serve as the source of truth for your application’s state.

1.2 Core Concepts

// TypeScript interface defining core state store operations
interface StateStore<T> {
  get(key: string): Promise<T | null>;
  set(key: string, value: T): Promise<void>;
  delete(key: string): Promise<void>;
  exists(key: string): Promise<boolean>;
}

2. Fundamental Concepts

2.1 State Types

Think of a state as a piece of information that a program uses while it’s running. For example, when you shop online, your cart (what you’ve added) is part of the state. Now, depending on how long this information is needed, we can categorize it into three types:

Ephemeral State

What it is: Temporary information that only exists while the program is running. Once you close or restart the app, it’s gone.

Example:

A shopping cart that resets when you leave the website.
A calculator app showing the current result but forgetting it when you close it.

Key Idea: Ephemeral = “Temporary and forgettable.”

Persistent State: Information that the app saves permanently, so you can use it later even after closing or restarting the app.

Example:

Your saved user profile in a social media app.
A to-do list that remains saved until you delete it.

Key Idea: Persistent = “Saved forever until you delete it.”

Cached State

What it is: Temporary information stored to save time. Cached data is like a shortcut — if the app already has the data, it doesn’t fetch or calculate it again. But it has an expiry time (TTL, or Time-to-Live) and will be discarded after that.

Example:

A news app saves the latest headlines temporarily so it doesn’t load them again.
A weather app caching the forecast for a few hours.

Key Idea: Cached = “Temporary but efficient.”

State Metadata: When working with a state, we also store some extra details about it. These details, called metadata, describe things like:

When it was created (created_at): The time when the state was first stored.
When it was updated (updated_at): The last time you changed the state.
Version (version): A way to track changes. For example, if the way you save data changes, the version helps keep things organized.
Type (type): Whether it’s ephemeral, persistent, or cached.
TTL (ttl): For cached data, how long it should last before it’s considered expired?

The State Class: This is a simple way to organize a state. Imagine you’re labelling a box to store something:

key: The label on the box (e.g., "user_cart" or "weather_data").
value: The actual content inside the box (e.g., a list of items in the cart or the forecast details).
is_expired(): A quick check to see if the box’s content is outdated (based on TTL). If the TTL is not set, the data never expires.

from enum import Enum
from dataclasses import dataclass
from typing import Any, Dict, Optional

class StateType(Enum):
    EPHEMERAL = "ephemeral"
    PERSISTENT = "persistent"
    CACHED = "cached"

@dataclass
class StateMetadata:
    created_at: float
    updated_at: float
    version: int
    type: StateType
    ttl: Optional[int] = None

class State:
    def __init__(self, key: str, value: Any, metadata: StateMetadata):
        self.key = key
        self.value = value
        self.metadata = metadata

    def is_expired(self) -> bool:
        if self.metadata.ttl is None:
            return False
        return (time.time() - self.metadata.updated_at) > self.metadata.ttl

2.2 State Serialization

Now, let’s imagine you need to save or send your state (the box and its contents) to another place. To do that, you need to serialize it. Serialization is like turning the box into a file format that can be stored or sent through the internet.

Why is Serialization Important?

Storing Data: Save the state permanently in a database or file.
Sharing Data: Send it across systems or devices (like sending your state to a cloud server).
Efficiency: Compress the data into a smaller format to save time and space.

JSON Serialization

What it does: Converts the state into a readable text format (JSON), which is easy to store and share.

Example: Imagine your state looks like this:

{
  "key": "user_cart",
  "value": ["item1", "item2"],
  "type": "ephemeral"
}

How it works:

Serialize: Convert the object to a JSON string and store it in a file or send it.

Deserialize: Read the string and turn it back into the original object.

Advantages:

Human-readable (you can open the file and understand it).
Easy to debug.

Disadvantages:

Larger in size and slower than other methods.
Not ideal for big systems that handle a lot of data.

Protobuf Serialization

What it does: Converts the state into a compact, binary format that’s faster and smaller than JSON.
Example: Instead of saving the state as readable text, Protobuf turns it into something like this: 010100110101. This is much smaller and faster to process.

How it works:

Serialize: Use the schema to turn the state into a compact binary format.

Deserialize: Use the same schema to decode the binary format back into the original object.

Advantages:

Very fast and efficient.
Ideal for systems that need to handle a lot of data, like large apps or real-time services.

Disadvantages:

Not human-readable (you can’t easily open it to see what’s inside).
Requires the schema, which adds complexity.


// TypeScript implementation of state serialization
interface Serializer<T> {
  serialize(data: T): Buffer;
  deserialize(data: Buffer): T;
}

class JSONSerializer<T> implements Serializer<T> {
  serialize(data: T): Buffer {
    return Buffer.from(JSON.stringify(data));
  }

  deserialize(data: Buffer): T {
    return JSON.parse(data.toString());
  }
}

class ProtobufSerializer<T> implements Serializer<T> {
  private readonly schema: protobuf.Type;

  constructor(schema: protobuf.Type) {
    this.schema = schema;
  }

  serialize(data: T): Buffer {
    return this.schema.encode(data).finish();
  }

  deserialize(data: Buffer): T {
    return this.schema.decode(data);
  }
}

How These Concepts Work Together

State: You define the data you want to store or manage (e.g., your shopping cart or user profile).
Serialization: You package the state into a format that’s easy to save, share, or process (like saving a file or sending data over the internet).
Efficiency: By choosing the right serialization method (JSON for simplicity or Protobuf for performance), you make your app faster and more reliable.

This simplifies how apps store, transfer, and retrieve data, ensuring a smooth experience for users.

3. State Store Technologies

State store technologies manage and persist application states for distributed systems. These are particularly useful for applications that require sharing, consistency, and fault tolerance for state data. Two common implementations for state stores are:

In-memory databases like Redis for fast, low-latency state management.
Relational databases like PostgreSQL for durable and transactional state management.

Below is a detailed explanation of both implementations using Python and TypeScript.

3.1 Redis Implementation

Scenario: Online Ticket Booking System

Imagine you’re running an online platform for booking movie tickets. Tickets sell out fast, and multiple users are trying to book at the same time. You need a system to manage temporary data like user reservations before confirming payments.

How Redis Fits:

get:

Use Case: A user selects seats in the booking system. You store the selected seat information temporarily in Redis with a unique key (e.g., user_1234_booking).
When the user revisits their selection, you retrieve this data using get. The system shows an empty selection if no data exists (e.g., the user took too long and the session expired).

set:

Use Case: Once the user selects seats, you store the reservation data in Redis and set a TTL (e.g., 15 minutes) to automatically release the seats if the user doesn’t proceed to payment.

atomic_update:

Use Case: While booking tickets, you must ensure no one else selects the same seats simultaneously.
Redis uses the watch feature to monitor the availability of those seats.
If no changes occur to the seat data, the transaction succeeds, and the booking is confirmed.
If another user books the same seats first, Redis detects the conflict and retries until the operation is safe.

Why Redis? Redis is perfect for this scenario because it’s fast, supports temporary data with TTLs, and ensures atomic updates to avoid double bookings.

from typing import Optional
import redis
import json

class RedisStateStore:
    def __init__(self, host: str, port: int, db: int = 0):
        self.redis = redis.Redis(host=host, port=port, db=db)
        
    async def get(self, key: str) -> Optional[dict]:
        value = await self.redis.get(key)
        return json.loads(value) if value else None
        
    async def set(self, key: str, value: dict, ttl: Optional[int] = None) -> None:
        serialized = json.dumps(value)
        if ttl:
            await self.redis.setex(key, ttl, serialized)
        else:
            await self.redis.set(key, serialized)
            
    async def atomic_update(self, key: str, update_fn) -> None:
        async with self.redis.pipeline() as pipe:
            while True:
                try:
                    # Watch for changes on key
                    await pipe.watch(key)
                    
                    # Get current value
                    current = await self.get(key)
                    
                    # Calculate new value
                    new_value = update_fn(current)
                    
                    # Try to execute transaction
                    pipe.multi()
                    pipe.set(key, json.dumps(new_value))
                    await pipe.execute()
                    break
                except redis.WatchError:
                    # Key changed, retry operation
                    continue

3.2 PostgreSQL Implementation

Scenario: Online Ticket Booking System

How Redis Fits:

get:

Use Case: A user selects seats in the booking system. You store the selected seat information temporarily in Redis with a unique key (e.g., user_1234_booking).
When the user revisits their selection, you retrieve this data using get. The system shows an empty selection if no data exists (e.g., the user took too long and the session expired).

set:

Use Case: Once the user selects seats, you store the reservation data in Redis and set a TTL (e.g., 15 minutes) to automatically release the seats if the user doesn’t proceed to payment.

atomic_update:

Use Case: While booking tickets, you need to ensure no one else selects the same seats simultaneously.
Redis uses the watch feature to monitor the availability of those seats.
If no changes occur to the seat data, the transaction succeeds, and the booking is confirmed.
If another user books the same seats first, Redis detects the conflict and retries until the operation is safe.

Why Redis? Redis is perfect for this scenario because it’s fast, supports temporary data with TTLs, and ensures atomic updates to avoid double bookings.

import { Pool } from 'pg';

class PostgresStateStore<T> implements StateStore<T> {
  private pool: Pool;
  private serializer: Serializer<T>;

  constructor(connectionString: string, serializer: Serializer<T>) {
    this.pool = new Pool({ connectionString });
    this.serializer = serializer;
  }

  async initialize(): Promise<void> {
    await this.pool.query(`
      CREATE TABLE IF NOT EXISTS state_store (
        key TEXT PRIMARY KEY,
        value BYTEA,
        version INTEGER,
        created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
        updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
      )
    `);
  }

  async get(key: string): Promise<T | null> {
    const result = await this.pool.query(
      'SELECT value FROM state_store WHERE key = $1',
      [key]
    );

    if (result.rows.length === 0) {
      return null;
    }

    return this.serializer.deserialize(result.rows[0].value);
  }

  async set(key: string, value: T): Promise<void> {
    const serialized = this.serializer.serialize(value);

    await this.pool.query(`
      INSERT INTO state_store (key, value, version)
      VALUES ($1, $2, 1)
      ON CONFLICT (key) 
      DO UPDATE SET 
        value = $2,
        version = state_store.version + 1,
        updated_at = CURRENT_TIMESTAMP
    `, [key, serialized]);
  }

  async atomic_update(key: string, updateFn: (current: T | null) => T): Promise<void> {
    await this.pool.query('BEGIN');

    try {
      const current = await this.get(key);
      const newValue = updateFn(current);
      await this.set(key, newValue);
      await this.pool.query('COMMIT');
    } catch (error) {
      await this.pool.query('ROLLBACK');
      throw error;
    }
  }
}

4. Implementation Patterns

4.1 Event Sourcing Pattern

What is Event Sourcing?

Event sourcing is a design pattern where changes to the application state are stored as a series of events. Instead of saving the current state directly, every action (or event) that changes the state is recorded in sequence. The current state is derived by replaying these events.

How It Works:

appendEvent:

Use Case: When a customer creates an order:
An ORDER_CREATED event is generated and stored.
The aggregateId links all events for the same order, so you can easily track the order's history.
Events are appended to the event store without modifying previous events, preserving a complete history.

getEvents:

Use Case: To view an order’s history:
Fetch all events for the order ID (aggregateId).
For example, if an order were created, paid for, and cancelled, these events would show the full sequence.

Aggregate (e.g., OrderAggregate):

Use Case: Rebuild the current state of an order:
Retrieve all events for the order.
Replay them sequentially to determine the order’s final state.
Example: If the last event is, the order status becomes "cancelled".

Why Event Sourcing?

Auditability: You can see exactly how and why the state changed over time.
Error Recovery: If a bug corrupts the current state, you can rebuild it by replaying events.

interface Event {
  id: string;
  type: string;
  payload: any;
  timestamp: number;
  aggregateId: string;
  version: number;
}

class EventStore {
  private store: StateStore<Event[]>;

  constructor(store: StateStore<Event[]>) {
    this.store = store;
  }

  async appendEvent(event: Event): Promise<void> {
    await this.store.atomic_update(
      event.aggregateId,
      (events: Event[] | null) => {
        const currentEvents = events || [];
        return [...currentEvents, event];
      }
    );
  }

  async getEvents(aggregateId: string): Promise<Event[]> {
    return await this.store.get(aggregateId) || [];
  }
}

class OrderAggregate {
  private events: Event[] = [];
  private state: OrderState;

  constructor(private eventStore: EventStore) {}

  async load(orderId: string): Promise<void> {
    this.events = await this.eventStore.getEvents(orderId);
    this.state = this.events.reduce(this.apply, new OrderState());
  }

  private apply(state: OrderState, event: Event): OrderState {
    switch (event.type) {
      case 'ORDER_CREATED':
        return { ...state, status: 'created' };
      case 'ORDER_PAID':
        return { ...state, status: 'paid', paidAt: event.payload.paidAt };
      // ... other event handlers
    }
    return state;
  }
}

4.2 CQRS Pattern

CQRS (Command Query Responsibility Segregation) separates commands (actions that change state) from queries (actions that read state). It often works alongside Event Sourcing to provide fast reads and writes

Scenario: Customer Order Processing

Continuing with the e-commerce example, you also need to provide customers with real-time views of their orders. However, writing and reading data efficiently requires different approaches.

Key Components:

Command Side:

Handles actions that modify state, like creating or updating orders.
Example: A customer places an order, which creates an ORDER_CREATED event.

Query Side:

Handles actions that read state, like fetching an order summary.
Example: A customer views the order details, which are retrieved from a query model.

from abc import ABC, abstractmethod
from dataclasses import dataclass
from typing import List

@dataclass
class OrderCommand:
    order_id: str
    user_id: str
    items: List[dict]

class CommandHandler(ABC):
    @abstractmethod
    async def handle(self, command: OrderCommand) -> None:
        pass

class OrderCommandHandler(CommandHandler):
    def __init__(self, event_store: EventStore, query_store: StateStore):
        self.event_store = event_store
        self.query_store = query_store
    
    async def handle(self, command: OrderCommand) -> None:
        # Create event
        event = {
            'type': 'ORDER_CREATED',
            'aggregate_id': command.order_id,
            'payload': {
                'user_id': command.user_id,
                'items': command.items
            }
        }
        
        # Store event
        await self.event_store.append_event(event)
        
        # Update query model
        query_model = {
            'order_id': command.order_id,
            'status': 'created',
            'user_id': command.user_id,
            'items': command.items,
            'total': sum(item['price'] for item in command.items)
        }
        await self.query_store.set(f"order:{command.order_id}", query_model)

Common Pitfalls in Application Development

Do’s (Things You Should Always Do)

1. Implement Proper Error Handling and Retries

What It Means: Always anticipate that things can go wrong. Network failures, database crashes, or external API timeouts are common. Use error-handling mechanisms to gracefully recover and, where possible, retry failed operations.

Why It’s Important: This prevents system crashes and provides a better user experience.

Real-World Example: Imagine you’re ordering a pizza online. If the payment gateway times out, the app should:

Show an error message (error handling).
Automatically retry the payment request a few times before giving up (retry mechanism).

2. Use Circuit Breakers for External Dependencies

What It Means: A circuit breaker is a design pattern that prevents a system from repeatedly calling a failing service. If an external dependency (e.g., a payment gateway) is down, the circuit breaker temporarily “trips” and stops making calls until the dependency recovers.

Why It’s Important: This avoids overwhelming failing systems and improves the reliability of your application.

Real-World Example: Think of a coffee machine. If the grinder fails, the machine stops working until the grinder is fixed, rather than continuing to attempt grinding and burning out other components.

3. Implement Proper Monitoring and Alerting

What It Means: Use tools to monitor the health of your application and alert your team when something goes wrong (e.g., high latency, memory leaks, or crashes).

Real-World Example: A smoke detector in your house monitors for fire and immediately alerts you if it detects smoke.

Why It’s Important: Early detection of issues minimizes downtime and helps maintain user trust.

4. Use Appropriate Caching Strategies

What It Means: Cache frequently accessed data (e.g., product catalogues, user profiles) to reduce database load and improve performance. However, cache expiration and invalidation should be carefully managed to avoid serving stale data.

Real-World Example: A coffee shop pre-brews coffee during peak hours to serve customers quickly. However, they must throw out the pre-brewed coffee if it sits too long (cache invalidation).

Why It’s Important: Proper caching reduces latency and improves user experience but requires careful planning to avoid outdated or inconsistent data.

5. Implement Proper Security Measures

What It Means: Protect your system from unauthorized access, data breaches, and other security threats by encrypting sensitive data, using secure communication protocols (e.g., HTTPS), and employing strong authentication mechanisms.

Real-World Example: Locking your house, installing security cameras, and encrypting your Wi-Fi password to ensure that only authorized individuals can access your property.

Why It’s Important: A security breach can damage your reputation and result in legal consequences.

6. Plan for Scaling from the Start

What It Means: Design your application to handle increased traffic and data volumes as your user base grows. Use scalable infrastructure, database sharding, and load balancers.

Real-World Example: Imagine hosting a party. If you expect a large crowd, you rent a bigger venue, hire more staff, and stock more supplies from the beginning.

Why It’s Important: Poor scalability leads to crashes and slow performance as your system grows.

Don’ts (Things You Should Avoid)

1. Don’t Share State Stores Between Microservices

What It Means: Each microservice should have its own database or state store. Sharing state stores creates tight coupling between services, making them harder to scale and maintain.

Real-World Example: Think of roommates sharing a single toothbrush. If one roommate takes it, the others can’t use it until it’s returned. This creates unnecessary dependency.

Why It’s Important: Independent state stores ensure microservices remain autonomous and can scale independently.

2. Don’t Implement Complex Business Logic in State Store Operations

What It Means: State stores should only handle simple read/write operations. Complex logic (e.g., calculating discounts or applying business rules) should be implemented in application services.

Real-World Example: A bank teller should only record deposits and withdrawals. The bank manager decides interest rates and policies (business logic).

Why It’s Important: Keeping business logic in the application layer improves maintainability and avoids performance issues in state stores.

3. Don’t Ignore Proper Error Handling

What It Means: Never assume operations will always succeed. Failing to handle errors properly can lead to unhandled exceptions, data corruption, or system crashes.

Real-World Example: A car without airbags assumes no accidents will happen. If a collision occurs, the occupants face severe consequences.

Why It’s Important: Error handling ensures system stability and improves user trust by gracefully managing failures.

4. Don’t Store Sensitive Data Without Encryption

What It Means: Always encrypt sensitive information (e.g., passwords, credit card numbers) at rest and in transit. Use strong encryption algorithms and avoid hardcoding secrets.

Real-World Example: Storing cash in a safe versus leaving it on a table. The safe adds an extra layer of security.

Why It’s Important: Data breaches can expose sensitive information, leading to financial and reputational damage.

5. Don’t Ignore Performance Monitoring

What It Means: Continuously measure the performance of your application (e.g., response times, throughput, memory usage) to identify bottlenecks and plan for optimization.

Real-World Example: A factory without regular maintenance inspections might not notice that machinery is slowing down until production stops entirely.

Why It’s Important: Ignoring performance monitoring leads to slow applications and frustrated users, especially under heavy traffic.

Conclusion

External state stores are indispensable for managing data in today’s distributed systems, where multiple services or components interact to deliver seamless user experiences. These systems rely on external state stores to maintain critical information that individual microservices cannot handle effectively. However, implementing external state stores successfully requires meticulous planning, thoughtful design, and adherence to best practices.

1. Careful Consideration of Technology Choices

Selecting the right state store technology is foundational. Not all state stores are created equal, and each comes with trade-offs. For instance:

Redis: Known for its high-speed, in-memory data storage, Redis is ideal for caching, session management, and real-time analytics. Its limitations include memory constraints and higher costs for large datasets.
PostgreSQL: A robust relational database, PostgreSQL is suitable for systems requiring strong consistency, complex queries, and transactional support. It also offers JSON support, making it versatile for hybrid use cases.
Cassandra or DynamoDB: These NoSQL databases shine in scenarios requiring horizontal scalability and high availability, such as applications with global user bases.

The choice depends on factors like data access patterns, latency requirements, scalability needs, and the nature of the workload. For example, an e-commerce application might use Redis for product catalogue caching and PostgreSQL for storing customer orders due to the transactional nature of orders.

2. Proper Implementation of Patterns

Implementing design patterns like Event Sourcing and CQRS (Command Query Responsibility Segregation) is essential for scalable and maintainable state management.

Event Sourcing: In this pattern, every state change is recorded as an immutable event. This allows reconstructing the system’s state at any point in time by replaying these events. For example, in a banking application, recording events like “Account Created” or “Funds Deposited” makes it easier to audit transactions, debug issues, or build new features without altering existing data.

Advantages:

Provides a complete history of changes (audit trail).
Enables time travel by replaying events to a specific state.
Simplifies debugging by analyzing event streams.

Challenges:

Increased storage requirements.
Complexity in managing event versioning.

CQRS: This pattern separates commands (write operations) from queries (read operations), optimizing each independently. For example, a ride-hailing app like Uber might use CQRS to handle booking requests efficiently while maintaining a separate query layer for tracking driver locations in real-time.

Advantages:

Enhances scalability by isolating read and write workloads.
Improves performance for high-read systems like dashboards.

Challenges:

Adds complexity by requiring two separate data models.
Ensures consistency between the write and read sides.

3. Robust Error Handling

Distributed systems are prone to failures. Networks can go down, databases might crash, or third-party APIs could timeout. Robust error handling ensures the system can recover from such issues without data corruption or service disruption.

Retry Mechanisms: Automatically reattempt operations like saving data to a state store or fetching updates from an API. Use exponential backoff strategies to avoid overwhelming the system.
Circuit Breakers: Prevent repeated calls to failing services by temporarily halting requests. For example, if a payment gateway is unresponsive, the system should pause payment attempts and redirect users to an alternate method.
Fallback Strategies: Provide alternative solutions when primary operations fail. For instance, if fetching real-time stock prices fails, display cached data instead.

According to studies, robust error handling can reduce downtime by up to 50% in complex systems.

4. Comprehensive Monitoring

Monitoring and observability are non-negotiable in distributed systems. Effective monitoring involves tracking both system health and business metrics, such as latency, error rates, and throughput.

Proactive Detection: Tools like Prometheus and Grafana help monitor critical metrics. For example, tracking Redis memory usage can alert developers before the cache is full, avoiding system slowdowns.
Alerting: Alerts ensure the right teams are notified of issues. For example, a sudden spike in query response times might indicate database contention or traffic surges.
Distributed Tracing: Tools like Jaeger or Zipkin enable tracing requests across microservices, helping identify bottlenecks in complex workflows.

Businesses lose an estimated $5600 per minute during IT outages, making proactive monitoring crucial.

5. Proper Security Measures

State stores often handle sensitive data, such as user credentials, payment information, or personal details. Implementing proper security measures is critical for compliance and trust.

Encryption: Use AES-256 for encrypting data at rest and TLS 1.3 for secure data transmission. This protects against unauthorized access, even if storage media is compromised.
Authentication and Authorization: Implement role-based access control (RBAC) to ensure that only authorized users or services can access state stores.
Secrets Management: Avoid hardcoding credentials or API keys. Use tools like AWS Secrets Manager or HashiCorp Vault to store secrets securely.

With over 22 billion records exposed in 2023 due to data breaches, ensuring data security is a fundamental requirement.

6. Scalability Planning

As user bases grow, systems must handle increased traffic and data volumes without degrading performance. Scalability should be a primary consideration during the design phase.

Horizontal Scaling: Add more servers or instances to handle traffic spikes. For example, e-commerce platforms like Amazon scale horizontally during Black Friday sales, accommodating up to 2000% traffic surges.
Sharding: Split databases into smaller, more manageable pieces to distribute workloads. For instance, sharding a user database by geographical region ensures faster access for regional users.
Replication: Use replicas to distribute read workloads and improve availability. PostgreSQL, for example, supports read replicas that can handle query-heavy operations without affecting the primary database.

Failing to plan for scalability can lead to service outages, frustrated users, and revenue loss.

External state stores are the backbone of modern distributed systems, enabling reliable state management across microservices and other components. Their successful implementation hinges on careful technology choices, effective design patterns, error handling, monitoring, security, and scalability planning. By following these principles, developers can build systems that are not only robust and efficient but also capable of evolving to meet future demands. The patterns and implementations discussed here provide a solid foundation for creating high-quality, maintainable state management solutions that align with the complexities of modern applications.