18.6.6 Responses to Failure Detection and Network Partitioning

When a suspicion is created, it times out immediately (its lifetime is set to 0), so the suspected member is expelled as soon as the expired suspicion is identified. The member could potentially survive for a further few seconds after the timeout because the check for expired suspicions is carried out periodically.

If an expelled member resumes communication and realises that it was expelled, it does not try to rejoin the group and accepts its expulsion.

When an expelled member accepts its expulsion, it switches to super read only mode and awaits operator attention. (The exception is in releases from MySQL 8.0.12 to 8.0.15, where the default was for the member to shut itself down. From MySQL 8.0.16, the behavior was changed to match the behavior in MySQL 5.7.)

You can use the group_replication_member_expel_timeout system variable, which is available from MySQL 8.0.13, to allow additional time between the creation of a suspicion and the expulsion of the suspect member. You can set the lifetime of the suspicion up to 3600 seconds (one hour) before it times out. (The 5-second detection period before a suspicion is created is not configurable.) Suspect members in this state are listed as UNREACHABLE, but are not removed from the group's membership list.

Bear in mind that while a group has unreachable members, you cannot add or remove any other members or elect a new primary. If you do want to take one of these actions and you cannot make the suspect member active again, you can force the suspicion to time out by changing group_replication_member_expel_timeout on any online member to a value less than the time that has already elapsed since the suspicion was created.

You can use the group_replication_autorejoin_tries system variable, which is available from MySQL 8.0.16, to make an expelled member that is able to resume communication automatically try to rejoin the group. You can specify a number of attempts that the member makes to rejoin the group, instead of just accepting its expulsion as soon as it resumes communication. When the member's expulsion or unreachable majority timeout is reached, it makes an attempt to rejoin (using the current plugin option values), then continues to make further auto-rejoin attempts up to the specified number of tries. After an unsuccessful auto-rejoin attempt, the member waits 5 minutes before the next try. During the auto-rejoin procedure, the expelled member remains in super read only mode and displays an ERROR state on its view of the replication group.

Bear in mind that while a member remains in this mode, although writes cannot be made on the member, reads can, with an increasing likelihood of stale reads over time. If you do want to intervene to take the member offline, the member can be stopped manually at any time by using a STOP GROUP_REPLICATION statement or shutting down the server. You can monitor the auto-rejoin procedure using the Performance Schema. While an auto-rejoin procedure is taking place, the Performance Schema table events_stages_current shows the event “Undergoing auto-rejoin procedure”, with the number of retries that have been attempted so far during this instance of the procedure (in the WORK_COMPLETED field). The events_stages_summary_global_by_event_name table shows the number of times the server instance has initiated the auto-rejoin procedure (in the COUNT_STAR field). The events_stages_history_long table shows the time each of these auto-rejoin procedures was completed (in the TIMER_END field).

You can use the group_replication_exit_state_action system variable, which is available from MySQL 8.0.12 and MySQL 5.7.24, to choose whether an expelled member that fails to rejoin (or does not try) shuts down MySQL Server or switches itself to super read only mode. As with the auto-rejoin process, if the member goes to super read only mode, there is a probability of stale reads which increases over time. Instructing the member to shut itself down ends this situation and means that you do not need to pro-actively monitor the servers for failures, but it means that the MySQL Server instance is unavailable and must be restarted. Operator intervention is required whatever exit action is set, as an ex-member that has exhausted its auto-rejoin attempts (or never had any) and has been expelled from the group is not allowed to rejoin without a restart of Group Replication.