Have you ever seen node flakiness when a Cassandra cluster is expanded across different domains through a Virtual Private Network (VPN)? This Cassandra cluster is expanded across two domains through VPN, phi_convict_threshold is set to 10; it was upgraded a few days back. But this flakiness existed before the upgrade as well. The instance types and architectures are the same and even the config parameters are consistent across the cluster. What are your first guesses about the flakiness shown in the following screenshot? Bad calculation in the metric system? A bad network-layer integration? A bad VPN?
Figure 7.39: Nodes going ...
