In a hands-on experiment, a developer deliberately shut down the primary database in a read-write separation setup to observe cluster behavior. Contrary to expectations, the system did not fail over cleanly; instead, it exhibited latency spikes, partial write failures, and inconsistent state propagation. The post details the exact sequence of events, including how the replica handled (or failed to handle) the transition. Key takeaways include the need for proper health check intervals, connection pool timeouts, and application-level retry logic. This real-world test underscores that high-availability configurations require rigorous validation beyond documentation assumptions. For teams relying on MySQL or similar architectures, this is a cautionary tale worth studying.
An engineer tests high-availability read-write separation by stopping the primary database, discovering unexpected cluster responses.