The Anatomy of a Cascading Outage

In a distributed microservices ecosystem, service dependencies resemble a directed graph. When a single downstream dependency experiences latency spikes, resource exhaustion, or transient deadlocks, upstream clients often amplify the problem. A standard HTTP client configured with naive retries will repeatedly slam an already struggling database or third-party payment gateway. Incoming user traffic queues up, connection pools deplete across thread pools, main threads stall waiting on I/O responses, and within seconds, an isolated hiccup propagates upstream into a system-wide catastrophic outage.

The traditional mitigation is the Circuit Breaker pattern. Inspired by electrical safety switches, a software circuit breaker wraps remote calls and monitors failures. When errors cross a predetermined threshold, the breaker trips, instantaneously short-circuiting outbound requests without touching the network. However, simply throwing a 503 Service Unavailable error degrades the end-user experience. A resilient architecture couples the breaker state machine with an intelligent stale fallback cache, allowing applications to degrade gracefully rather than fail outright.

The Circuit Breaker Finite State Machine

A resilient circuit breaker operates across three primary states:

  • CLOSED: Normal operation. Every request passes through to the downstream service. Failures and successes are tracked within a rolling time window. If the failure rate exceeds the configured threshold, the breaker transitions to OPEN.
  • OPEN: The downstream service is deemed unhealthy. Requests are intercepted immediately before making network I/O. The breaker throws a fast failure or routes execution directly to an auxiliary fallback handler. A reset timer dictates how long the breaker remains in this state.
  • HALF-OPEN: Once the reset cooldown expires, the breaker transitions to HALF-OPEN, permitting a limited probe trial of concurrent requests through to the downstream service. If these canary probes succeed, the breaker resets to CLOSED. If even a single canary probe encounters a server-side fault, the breaker returns to OPEN with an exponentially scaled cooldown.

Implementing a Production-Ready Breaker in TypeScript

Below is a clean, dependency-free implementation of a state-driven Circuit Breaker featuring rolling execution metrics and cached fallback recovery.

type BreakerState = 'CLOSED' | 'OPEN' | 'HALF_OPEN';

interface CircuitBreakerOptions<T> {
  failureThresholdPercentage: number;
  minimumRequests: number;
  recoveryTimeoutMs: number;
  samplingWindowMs: number;
  fallbackFn?: (error: Error) => Promise<T>;
}

interface WindowBucket {
  timestamp: number;
  successes: number;
  failures: number;
}

export class CircuitBreaker<T> {
  private state: BreakerState = 'CLOSED';
  private buckets: WindowBucket[] = [];
  private lastStateChange: number = Date.now();
  private halfOpenSuccessfulProbes: number = 0;
  private readonly requiredProbes: number = 3;

  constructor(
    private readonly action: () => Promise<T>,
    private readonly options: CircuitBreakerOptions<T>
  ) {}

  public async execute(): Promise<T> {
    this.evaluateStateTransitions();

    if (this.state === 'OPEN') {
      return this.handleFallback(
        new Error('CircuitBreaker is OPEN: Fast-failing request to protect downstream service.')
      );
    }

    try {
      const result = await this.action();
      this.recordSuccess();
      return result;
    } catch (err) {
      const error = err instanceof Error ? err : new Error(String(err));
      this.recordFailure();
      return this.handleFallback(error);
    }
  }

  private evaluateStateTransitions(): void {
    const now = Date.now();

    if (this.state === 'OPEN') {
      // Check if cooldown timer has elapsed
      if (now - this.lastStateChange >= this.options.recoveryTimeoutMs) {
        this.transitionTo('HALF_OPEN');
      }
    }

    this.pruneExpiredBuckets(now);
  }

  private recordSuccess(): void {
    if (this.state === 'HALF_OPEN') {
      this.halfOpenSuccessfulProbes += 1;
      if (this.halfOpenSuccessfulProbes >= this.requiredProbes) {
        this.transitionTo('CLOSED');
      }
      return;
    }

    this.getCurrentBucket().successes += 1;
  }

  private recordFailure(): void {
    if (this.state === 'HALF_OPEN') {
      this.transitionTo('OPEN');
      return;
    }

    this.getCurrentBucket().failures += 1;
    this.checkThresholds();
  }

  private checkThresholds(): void {
    const { totalSuccesses, totalFailures } = this.calculateTotals();
    const totalRequests = totalSuccesses + totalFailures;

    if (totalRequests < this.options.minimumRequests) {
      return;
    }

    const failurePercentage = (totalFailures / totalRequests) * 100;
    if (failurePercentage >= this.options.failureThresholdPercentage) {
      this.transitionTo('OPEN');
    }
  }

  private transitionTo(newState: BreakerState): void {
    this.state = newState;
    this.lastStateChange = Date.now();
    this.halfOpenSuccessfulProbes = 0;
    if (newState === 'CLOSED') {
      this.buckets = [];
    }
  }

  private getCurrentBucket(): WindowBucket {
    const now = Date.now();
    const current = this.buckets[this.buckets.length - 1];

    if (!current || now - current.timestamp > 1000) {
      const newBucket: WindowBucket = { timestamp: now, successes: 0, failures: 0 };
      this.buckets.push(newBucket);
      return newBucket;
    }

    return current;
  }

  private pruneExpiredBuckets(now: number): void {
    const cutoff = now - this.options.samplingWindowMs;
    this.buckets = this.buckets.filter((b) => b.timestamp >= cutoff);
  }

  private calculateTotals(): { totalSuccesses: number; totalFailures: number } {
    return this.buckets.reduce(
      (acc, bucket) => ({
        totalSuccesses: acc.totalSuccesses + bucket.successes,
        totalFailures: acc.totalFailures + bucket.failures,
      }),
      { totalSuccesses: 0, totalFailures: 0 }
    );
  }

  private async handleFallback(error: Error): Promise<T> {
    if (this.options.fallbackFn) {
      return await this.options.fallbackFn(error);
    }
    throw error;
  }

  public getState(): BreakerState {
    return this.state;
  }
}

Combining with Stale-While-Revalidate Caching

A circuit breaker without a fallback mechanism turns an internal degradation into an external client failure. For read-heavy systems, combining the breaker with an asynchronous stale fallback cache guarantees high availability. When the breaker opens, the service serves slightly expired cache data from Redis or in-process memory rather than returning an unhandled exception.

interface UserProfile {
  id: string;
  name: string;
  tier: string;
}

class ResilientUserService {
  private localCache = new Map<string, { data: UserProfile; cachedAt: number }>();

  async fetchUserProfile(userId: string): Promise<UserProfile> {
    const breaker = new CircuitBreaker<UserProfile>(
      async () => {
        // Simulated network call to upstream user microservice
        const response = await fetch(`https://api.internal.local/v1/users/${userId}`, {
          signal: AbortSignal.timeout(1200),
        });

        if (!response.ok) {
          throw new Error(`Upstream returned HTTP ${response.status}`);
        }

        const data: UserProfile = await response.json();
        // Update local stale cache on success
        this.localCache.set(userId, { data, cachedAt: Date.now() });
        return data;
      },
      {
        failureThresholdPercentage: 50,
        minimumRequests: 10,
        recoveryTimeoutMs: 15000,
        samplingWindowMs: 30000,
        fallbackFn: async (err: Error) => {
          const cached = this.localCache.get(userId);
          if (cached) {
            // Log graceful degradation event for metrics
            return cached.data;
          }
          // If no fallback data exists, propagate structured error
          throw new Error(`Service temporarily unavailable. Root cause: ${err.message}`);
        },
      }
    );

    return breaker.execute();
  }
}

Mitigating the Thundering Herd with Jittered Canary Probes

When dozens of server instances transition from OPEN to HALF_OPEN simultaneously at the exact same 15-second boundary, they unleash a synchronized burst of canary requests upon the recovering dependency. This is the classic thundering herd problem.

To avoid knocking the downstream system back into failure, apply decorrelated jitter to the recovery interval:

function calculateJitteredBackoff(baseDelayMs: number, maxJitterMs: number): number {
  const jitter = Math.floor(Math.random() * maxJitterMs);
  return baseDelayMs + jitter;
}

// In production, instantiate options dynamically:
const recoveryTimeout = calculateJitteredBackoff(15000, 5000); // 15s to 20s window

Production Pitfalls and Anti-Patterns

Implementing circuit breakers in production requires addressing subtle operational failure modes:

  • Global Shared Breaker Instances: Wrapping multiple distinct endpoints into a single circuit breaker instance is an anti-pattern. If a non-critical endpoint (e.g., user avatar upload) fails, a shared breaker will trip and unnecessarily block critical paths (e.g., authentication or checkout). Breaker instances must be scoped per upstream dependency and protocol endpoint.
  • Unbounded Bucket Allocations: Creating unbounded timestamps inside the metrics collector introduces memory leaks under high throughput. Always bucket metrics into discrete time windows (such as 1-second chunks) and purge entries older than the rolling window threshold.
  • Ignoring Error Classifications: Not all errors indicate an unhealthy downstream server. Client-side HTTP 4xx errors (e.g., 400 Bad Request or 404 Not Found) represent invalid input, not server failure. Including 4xx status codes in the breaker's failure metrics will trip the breaker during validation attacks or client bugs. Only record network timeouts, connection drops, and HTTP 5xx responses.

Key Takeaways

  • Circuit breakers protect system stability by failing fast when downstream services become unresponsive, preventing connection pool exhaustion and thread starvation.
  • A rolling time window with discrete buckets allows for statistically sound error rate calculations rather than reacting prematurely to isolated transient glitches.
  • Pairing a tripped breaker with a stale fallback cache enables graceful degradation, delivering usable data to end users instead of raw errors.
  • Incorporate decorrelated jitter into recovery timeouts across distributed instances to prevent canary probes from overwhelming recovering services.