What RDS Multi-AZ Failover Looks Like in an Incident

Mahesh Bahir•

Most teams enable RDS Multi-AZ, see the high availability checkbox ticked, and move on. What they do not see is what actually happens during a failover event - the sequence AWS runs through, how long it takes, and what their application experiences during every second of that window. The documentation calls it automatic. Production calls it something else entirely.

1. What Multi-AZ Actually Is and What Teams Think It Is

RDS Multi-AZ maintains a synchronous standby instance in a separate Availability Zone. Every write committed on the primary is replicated to the standby before the primary acknowledges the transaction. When the primary becomes unhealthy, AWS promotes the standby automatically without manual intervention.

That is what Multi-AZ does. What it does not do is equally important.

Multi-AZ is not a read replica. The standby does not serve read traffic under normal operation. It is a warm standby that exists solely to be promoted during a failure event. It is not instant - promotion takes time, DNS propagation takes time, and application reconnection takes time. Most critically, Multi-AZ is not invisible to applications. Every application connecting through a standard connection pool will experience a disruption during failover. How long that disruption lasts depends on factors entirely outside the database configuration.

2. What Triggers a Multi-AZ Failover in Production

Failovers fall into two categories with meaningfully different behavior.

Unplanned failovers occur when AWS detects the primary is unhealthy - an AZ outage, hardware failure, network loss, or database engine crash. Detection is automatic, and promotion begins without human intervention.

Planned failovers occur during maintenance operations - engine upgrades, instance class changes, and manual reboots with failover selected. These are predictable and schedulable, but the application still experiences the same connection disruption as an unplanned event.

A common misconception is that degraded performance or slow queries on a running primary will trigger failover. AWS initiates failover only when the primary is confirmed unhealthy by its monitoring systems.

3. The Failover Sequence AWS Actually Runs Through

Each step in the sequence runs in order - none are skippable, and each consumes time the application experiences as unavailability.

StageWhat HappensTypical DurationWhere Delays Occur
DetectionAWS monitors detect the primary is unhealthy10 – 30 secondsHealth check interval configuration
DecisionFailover initiated to standby5 – 10 secondsInternal AWS coordination
PromotionStandby promoted to new primary10 – 30 secondsInstance class and storage type
DNS UpdateCNAME flips to new primary endpoint10 – 30 secondsDNS propagation delay
App ReconnectionApplication reconnects to new primary10 – 60 secondsDNS TTL cache and pool behavior
Total WindowFirst error to first successful query60 – 180 secondsEvery stage above compounds

AWS reports failover time from detection through DNS update. The app reconnection row is covered in detail in the sections that follow. The number to internalize is the total window - 60 to 180 seconds is what the application experiences, not what the AWS event log reports.

4. What Your Application Experiences During Failover

From the application's perspective, a Multi-AZ failover begins with connection errors. Active connections to the primary drop when AWS takes it offline. New connection attempts fail during the promotion window until the new primary is reachable.

In-flight transactions do not survive failover. Any transaction that had not committed before the failure is rolled back. The standby promotes with the last committed transaction it received through synchronous replication. Applications without explicit transaction retry logic will surface this as data loss or user-facing errors.

Read queries that were in progress return errors and can be safely retried against the new primary. Write operations require idempotency handling at the application layer before resubmission.

Read replicas attached to the original primary continue replicating after promotion - now from the new primary. They may experience brief replication lag during the transition. Applications reading from replicas may observe stale data during this period.

5. DNS TTL and Why It Is the Hidden Variable Nobody Tunes

RDS instances are accessed through a DNS endpoint - not a fixed IP. When failover completes, AWS updates the CNAME record to point to the new primary. Applications reconnect once their DNS cache expires and resolves the updated record.

The problem is client-side DNS caching. Operating systems, JVM instances, and database drivers each maintain their own DNS cache. An application whose JVM cached the old primary IP continues attempting connections to the failed instance until that cache expires - regardless of what the RDS endpoint now resolves to.

AWS sets a low TTL on RDS DNS records to support rapid failover. The issue is not the AWS-side TTL but client-side caching in the application stack. JVM DNS caching holds records indefinitely by default unless explicitly configured. Setting networkaddress.cache.ttl to between 1 and 5 seconds in the JVM security properties ensures DNS resolution reflects the updated endpoint promptly after failover.

This single configuration detail frequently determines the difference between a 90-second recovery and a 5-minute outage - with no other infrastructure change required. For non-JVM stacks, the equivalent check is DNS TTL configuration in the database driver and any intermediate caching layer in the network stack.

6. Connection Pool Behavior During and After Failover

When the primary fails, connection pools do not automatically discard existing connections. Each connection remains stale until it is used and the error is detected, or until the pool's configured lifetime expires. Under load, a pool holding stale connections can exhaust retry attempts before reconnecting to the new primary, extending the application-observed outage beyond the AWS-reported window.

Pool behavior during failover varies significantly by library and configuration.

Connection PoolAuto ReconnectDNS Cache BehaviorManual Config NeededRecommended Setting
JDBC defaultNo - throws exceptionCaches old IP by defaultYesEnable reconnect with retry logic
HikariCPYes - built-inRespects system TTLMinimalSet connectionTimeout and maxLifetime
pgBouncerYes - pool levelResolves per connectionMinimalSet server_lifetime below DNS TTL
RDS ProxyHandled at proxy layerNot applicableNoneRecommended for Lambda and serverless
Node.js mysql2No by defaultCaches old IPYesEnable reconnect flag explicitly
Sequelize ORMPartialDepends on driverYesConfigure retry and dialectOptions

Default configuration is rarely sufficient for clean failover recovery across any pool type. Pool-level reconnection and DNS resolution are two separate problems that compound each other during a failover event. Solving one without the other still leaves a recovery gap. RDS Proxy addresses both at the proxy layer - covered in the next section.

7. What RDS Proxy Changes About Failover Behavior

RDS Proxy sits between the application and RDS, maintaining its own persistent connection pool to the database. Applications connect to the proxy endpoint rather than the RDS endpoint directly. During failover, the proxy detects the primary change internally, reconnects its pool to the new primary, and continues serving application connections with a significantly shorter interruption.

AWS documentation indicates failover time with RDS Proxy is typically under 35 seconds for MySQL and PostgreSQL, compared to 60 to 120 seconds for a direct connection. The improvement comes from the proxy resolving DNS and managing pool reconnection internally, removing both variables from the application-observed recovery time.

For Lambda functions and serverless workloads that open many short-lived connections, RDS Proxy prevents connection exhaustion on the database during traffic spikes that coincide with a failover event.

The tradeoff is cost. RDS Proxy is priced per vCPU of the underlying database instance. For high-traffic production workloads where database availability directly affects users, the reduced failover window justifies the cost. For low-traffic internal services with a tolerant recovery time objective, the added complexity may not be warranted.

8. How to Validate Failover Behavior Before an Incident

RDS supports manual failover through the AWS console and CLI using the reboot with failover option on a Multi-AZ instance. Running this in a non-production environment that mirrors production configuration is the most reliable way to understand how a failover affects a specific application stack.

During a failover test, measure application-observed recovery time - from the first connection error logged by the application to the first successful database query after reconnection. That number reflects real user impact and accounts for the full application-layer recovery sequence specific to the stack being tested.

Run the test under realistic load. A failover test at minimal traffic does not surface pool exhaustion behavior that only appears under production conditions. Run it during business hours at least once before relying on the configuration during a real incident.

9. Configuration Decisions That Affect Failover Outcome

Instance class affects promotion time. Larger instances generally promote faster. For workloads where recovery time is critical, sizing the standby appropriately is part of the high availability configuration.

Storage type affects failover performance. Provisioned IOPS storage completes the promotion phase faster than general-purpose SSD under heavy I/O workloads. For write-heavy databases, storage type selection directly influences how quickly the standby is ready to serve production traffic.

RDS Multi-AZ DB Cluster, available for MySQL and PostgreSQL, maintains two readable standby instances across three Availability Zones rather than one non-readable standby. Failover time is significantly shorter - often under 35 seconds - and readable standbys provide additional read capacity under normal operation. For workloads with strict recovery time requirements, Multi-AZ DB Cluster is worth evaluating against standard Multi-AZ.

10. What Good Multi-AZ Failover Behavior Actually Looks Like

A well-configured Multi-AZ setup has three measurable properties during a failover event.

Application retry logic handles connection errors without surfacing them to users. This is visible in error rate metrics - a working setup shows a brief spike during the promotion window that returns to baseline without manual intervention.

Connection pools recover automatically without manual intervention. No pool restarts, application restarts, or connection clearing should be required. This is visible in connection pool health metrics and database connection count climbing to normal levels on the new primary within the expected window.

Application-observed recovery time matches what the team measured during failover testing. If the tested recovery was 90 seconds and the production incident took 4 minutes, the gap points to a difference between test and production conditions - load, pool size, or dependency behavior under real traffic.

The honest tradeoff with RDS Multi-AZ is this: it doubles instance cost, adds complexity to the connection layer, and still delivers recovery measured in minutes rather than seconds. For workloads where database availability directly determines user experience, that is the correct investment. For workloads where brief degradation is acceptable, it is a deliberate architectural decision - not a default checkbox. Enable Multi-AZ because the recovery time it provides fits the workload's requirements, validate it under realistic conditions, and configure the application layer to handle the window it creates.

11. Frequently Asked Questions (FAQ)

Q1: AWS says failover takes 60 to 120 seconds. Why did our outage last longer?

AWS measures detection to DNS update. Your application measures from the first connection error to the first successful query. DNS TTL caching and stale connection pool recovery add time on top of what AWS reports. That gap is where most production surprises live.

Q2: Why did our app throw errors during a planned maintenance window?

Planned maintenance - engine upgrades, instance class changes - triggers a failover. The application still experiences the connection interruption during the DNS flip and pool reconnection. Adding retry logic with exponential backoff at the database connection layer resolves this.

Q3: What is the difference between Multi-AZ and a read replica for availability?

Multi-AZ maintains a synchronous standby that promotes automatically on failure. A read replica uses asynchronous replication for read scaling - not automatic failover. They serve different purposes and are not interchangeable for high availability.

Q4: Does RDS Proxy eliminate application downtime during failover?

It reduces the window to under 35 seconds by handling DNS resolution and pool reconnection at the proxy layer. It does not eliminate downtime entirely. In-flight transactions that did not commit still roll back regardless of RDS Proxy.

Tags
RDS Multi-AZ failoverAWS RDSMulti-AZRDS ProxyDNS TTLRDS high availabilityDB ClusterConnection Pool
Maximize Your Cloud Potential
Streamline your cloud infrastructure for cost-efficiency and enhanced security.
Discover how CloudOptimo optimize your AWS and Azure services.
Request a Demo