I've set up fault tolerance, what's a good way of testing it?

Pull some network cables out of the front-end machines! But first, check if the balancer thinks everything is fine before any cables get removed. Go to the admin server web pages, and check that both load balancers are shown and are not marked red.

For more detail of the load balancer's status, you can check their log files. On each machine, go to the directory where the load balancers are installed. Then, run the command:

tail -f balancer/log/flipper

This command will follow the log, printing the last few lines. At the bottom, the log should read:

INFO:State: Normal, Current Code: 0x3f ( Fine )

This signifies that everything is normal and the two balancers both are working correctly. Now, try pulling one or more of the network cables out of a balancer. On the still-connected balancer, a message like the following should appear:

INFO:State: Switched, Current Code: 0x26 ( Remote Machine Power failure )

The actual code and diagnosis may be different (here, both network cables on the other machine have been removed). The log line shows that the network has switched (so this balancer now will hold all the traffic IP addresses), and the log also offers a possible explanation of what caused the problem.

If you then plug the other balancer back in, and wait a short while, the log should then print out:

INFO:State: Normal, Current Code: 0x3f ( Fine )

This means that the balancer has noticed that the other machine is back up and running.

Content Manager [Administrator] 12 August 2005