其他分享
首页 > 其他分享> > MHA实战案例

MHA实战案例

作者:互联网

一、机器环境准备

MHA:192.168.247.150  2vcpu 2G  centos7

master: 192.168.247.151 2vcpu 4G rocky8.6  mysql8.0.26

slave-01: 192.168.247.152 2vcpu 4G rocky8.6  mysql8.0.26

slave-02: 192.168.247.153 2vcpu 4G rocky8.6  mysql8.0.26

二、mha配置管理

软件包下载
wget  https://github.com/yoshinorim/mha4mysql-manager/releases/download/v0.58/mha4mysql-manager-0.58-0.el7.centos.noarch.rpm
wget https://github.com/yoshinorim/mha4mysql-node/releases/download/v0.58/mha4mysql-node-0.58-0.el7.centos.noarch.rpm

yum localinstall mha4mysql-node-0.58-0.el7.centos.noarch.rpm -y
yum localinstall mha4mysql-manager-0.58-0.el7.centos.noarch.rpm -y
yum install -y rsync

在所有mysql 节点安装mha4mysql-node
yum install -y rsync
scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm root@192.168.247.151:/root
scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm root@192.168.247.152:/root
scp mha4mysql-node-0.58-0.el7.centos.noarch.rpm root@192.168.247.153:/root
yum localinstall mha4mysql-node-0.58-0.el7.centos.noarch.rpm -y

创建密钥互信

[root@mha-manager ~]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:hNohAruUsgQWaRoW7QKQCX+R/LaNng1LuHuzBjFOjHA root@mha
The key's randomart image is:
+---[RSA 2048]----+
|B*=...           |
|OBE.o. .         |
|BB+oo.o .        |
|=+.+=+oo         |
|o .o.=.+S        |
|    + = .        |
|     = =         |
|    . B .        |
|    .+.o         |
+----[SHA256]-----+
[root@mha ~]# 
[root@mha ~]# ssh-copy-id 127.0.0.1
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
The authenticity of host '127.0.0.1 (127.0.0.1)' can't be established.
ECDSA key fingerprint is SHA256:XgCQNiZS5ZJTc9ejJyeI6j2vKF+JaHyBoSwK0Bq9P6w.
ECDSA key fingerprint is MD5:db:b0:b1:41:79:0a:b9:94:69:f2:a7:49:d8:78:fb:be.
Are you sure you want to continue connecting (yes/no)? yes
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
/usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys
root@127.0.0.1's password: 

Number of key(s) added: 1

Now try logging into the machine, with:   "ssh '127.0.0.1'"
and check to make sure that only the key(s) you wanted were added.

[root@mha ~]# 
[root@mha ~]# rsync -av .ssh 192.168.247.151:/root
root@192.168.247.151's password: 
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(226) [sender=3.1.2]
[root@mha ~]# rsy
rsync                  rsyslogd               rsyslog-recover-qi.pl  
[root@mha ~]# rsync -av .ssh 192.168.247.151:/root
root@192.168.247.151's password: 
bash: rsync: command not found
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: remote command not found (code 127) at io.c(226) [sender=3.1.2]
[root@mha ~]# rsync -av .ssh 192.168.247.151:/root
root@192.168.247.151's password: 
sending incremental file list
.ssh/
.ssh/authorized_keys
.ssh/id_rsa
.ssh/id_rsa.pub
.ssh/known_hosts

sent 3,497 bytes  received 102 bytes  1,439.60 bytes/sec
total size is 3,157  speedup is 0.88
[root@mha ~]# rsync -av .ssh 192.168.247.152:/root
root@192.168.247.152's password: 
Permission denied, please try again.
root@192.168.247.152's password: 
sending incremental file list
.ssh/
.ssh/authorized_keys
.ssh/id_rsa
.ssh/id_rsa.pub
.ssh/known_hosts

sent 3,497 bytes  received 102 bytes  423.41 bytes/sec
total size is 3,157  speedup is 0.88
[root@mha ~]# rsync -av .ssh 192.168.247.153:/root
root@192.168.247.153's password: 
sending incremental file list
.ssh/
.ssh/authorized_keys
.ssh/id_rsa
.ssh/id_rsa.pub
.ssh/known_hosts

sent 3,497 bytes  received 102 bytes  1,439.60 bytes/sec
total size is 3,157  speedup is 0.88
[root@mha ~]# 

配置mha

[root@mha-manager ~]# mkdir /etc/mastermha
[root@mha-manager ~]# cd /etc/mastermha/
[root@mha-manager mastermha]# ls
[root@mha-manager mastermha]# vi app1.cnf
[root@mha-manager mastermha]# cat app1.cnf 
[server default]
check_repl_delay=0
manager_log=/data/mastermha/app1/manager.log
manager_workdir=/data/mastermha/app1/
master_binlog_dir=/data/mysql/
master_ip_failover_script=/usr/local/bin/master_ip_failover
password=magedu
ping_interval=1
remote_workdir=/data/mastermha/app1/
repl_password=magedu
repl_user=repluser
report_script=/usr/local/bin/sendmail.sh
ssh_user=root
user=mhauser

[server1]
candidate_master=1
hostname=192.168.247.151

[server2]
hostname=192.168.247.152

[server3]
candidate_master=1
hostname=192.168.247.153

[root@mha-manager mastermha]# 
[root@mha-manager ~]# cat /usr/local/bin/master_ip_failover
#!/usr/bin/env perl
# Copyright (C) 2011 DeNA Co.,Ltd.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; either version 2 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc.,
# 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA
## Note: This is a sample script and is not complete. Modify the script based on your environment.
use strict;
use warnings FATAL => 'all';

use Getopt::Long;
use MHA::DBHelper;

my (
  $command, $ssh_user, $orig_master_host,
  $orig_master_ip, $orig_master_port, $new_master_host,
  $new_master_ip, $new_master_port, $new_master_user,
  $new_master_password
);

#执行时必须删除下面三行注释
my $vip = '192.168.247.100/24'; #设置Virtual IP
my $key = "1"; #指定VIP所在网卡的别名
my $ssh_start_vip = "/sbin/ifconfig eth0:$key $vip"; #指定VIP所在网卡
my $ssh_stop_vip = "/sbin/ifconfig eth0:$key down";

GetOptions(
  'command=s' => \$command,
  'ssh_user=s' => \$ssh_user,
  'orig_master_host=s' => \$orig_master_host,
  'orig_master_ip=s' => \$orig_master_ip,
  'orig_master_port=i' => \$orig_master_port,
  'new_master_host=s' => \$new_master_host,
  'new_master_ip=s' => \$new_master_ip,
  'new_master_port=i' => \$new_master_port,
  'new_master_user=s' => \$new_master_user,
  'new_master_password=s' => \$new_master_password,
);

exit &main();

sub main {
  if ( $command eq "stop" || $command eq "stopssh" ) {

    # $orig_master_host, $orig_master_ip, $orig_master_port are passed.
    # If you manage master ip address at global catalog database,
    # invalidate orig_master_ip here.
    my $exit_code = 1;
    eval {

      # updating global catalog, etc
      $exit_code = 0;
    };
    if ($@) {
      warn "Got Error: $@\n";
      exit $exit_code;
    }
    exit $exit_code;
}
    elsif ( $command eq "start" ) {

        # all arguments are passed.
        # If you manage master ip address at global catalog database,
        # activate new_master_ip here.
        # You can also grant write access (create user, set read_only=0, etc)here.
        my $exit_code = 10;
        eval {
            print "Enabling the VIP - $vip on the new master - $new_master_host\n";
            &start_vip();
            &stop_vip();
            $exit_code = 0;
        };
        if ($@) {
            warn $@;
            exit $exit_code;
        }
        exit $exit_code;
  }
  elsif ( $command eq "status" ) {
      print "Checking the Status of the script.. OK \n";
      `ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
      exit 0;
  }
  else {
      &usage();
      exit 1;
  }
}

sub start_vip() {
    `ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
    `ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}
sub usage {
  print
"Usage: master_ip_failover --command=start|stop|stopssh|status --
orig_master_host=host --orig_master_ip=ip --orig_master_port=port --
new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}
[root@mha-manager ~]# 
[root@mha-manager ~]# chmod +x /usr/local/bin/master_ip_failover

三、设置邮件报警

安装mailx包
yum install mailx -y

配置邮件发送内容 [root@mha-manager ~]# cat /usr/local/bin/sendmail.sh #!/bin/bash echo "MHA is failover!" | mail -s "MHA Warning" 281442855@qq.com [root@mha-manager ~]#
增加发送端邮件授权码 cat >> /etc/mail.rc <<EOF set from=xxxxxx@163.com set smtp=smtp.163.com set smtp-auth-user=xxxxxxx@163.com set smtp-auth-password=xxxxxxzzzzzyyyyy #发送端邮箱授权码 EOF

四、mysql主从配置

mysql配置

yum install mysql-server -y

[root@mysql-master ~]# cat /etc/my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[mysqld]
server_id=151
log-bin=/data/mysql/mysql-bin
skip_name_resolve=1
general_log
[root@mysql-master ~]# systemctl enable --now mysqld
创建mysql bin-log日志目录并指定用户组权限为mysql
mkdir /data/mysql chown -R mysql:mysql /data/mysql systemctl enable --now mysqld 在master创建账号并授权 create user repluser@'192.168.247.%' identified by 'magedu'; grant replication slave on *.* to repluser@'192.168.247.%'; create user mhauser@'192.168.247.%' identified by 'magedu'; grant all on *.* to mhauser@'192.168.247.%';
[root@mysql-master mysql]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.26 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> show master logs;
+------------------+-----------+-----------+
| Log_name         | File_size | Encrypted |
+------------------+-----------+-----------+
| mysql-bin.000001 |       156 | No        |
+------------------+-----------+-----------+
1 row in set (0.00 sec)

mysql> create user repluser@'192.168.247.%' identified by 'magedu';
Query OK, 0 rows affected (0.01 sec)

mysql> grant replication slave on *.* to repluser@'192.168.247.%';
Query OK, 0 rows affected (0.00 sec)

mysql> create user mhauser@'192.168.247.%' identified by 'magedu';
Query OK, 0 rows affected (0.00 sec)

mysql> grant all on *.* to mhauser@'192.168.247.%';
Query OK, 0 rows affected (0.00 sec)

mysql> flush privileges;
Query OK, 0 rows affected (0.00 sec)

mysql> 
配置vip
[root@mysql-master ~]# ifconfig eth0:1 192.168.247.100/24 配置slave mkdir /data/mysql chown -R mysql:mysql /data/mysql vi /etc/my.cnf [mysqld] server_id=152 log-bin=/data/mysql/mysql-bin read_only relay_log_purge=0 skip_name_resolve=1 general_log
systemctl enable --now mysqld

[root@mysql-slave-01 ~]# mysql Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8 Server version: 8.0.26 Source distribution Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. mysql> CHANGE MASTER TO -> MASTER_HOST='192.168.247.151', -> MASTER_USER='repluser', -> MASTER_PASSWORD='magedu', -> MASTER_LOG_FILE='mysql-bin.000001', -> MASTER_LOG_POS=156; Query OK, 0 rows affected, 8 warnings (0.01 sec) mysql> start slave; Query OK, 0 rows affected, 1 warning (0.00 sec) mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: 192.168.247.151 Master_User: repluser Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 1388 Relay_Log_File: mysql-slave-01-relay-bin.000002 Relay_Log_Pos: 1556 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1388 Relay_Log_Space: 1774 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 151 Master_UUID: 7bf8f5a4-2bff-11ed-824b-000c29aae8d7 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: 1 row in set, 1 warning (0.01 sec) mysql>

slave-02配置

[root@mysql-slave-02 ~]# vi /etc/my.cnf
[root@mysql-slave-02 ~]# 
[root@mysql-slave-02 ~]# cat /etc/my.cnf
#
# This group is read both both by the client and the server
# use it for options that affect everything
#
[client-server]

#
# include all files from the config directory
#
!includedir /etc/my.cnf.d

[mysqld]
server_id=153
log-bin=/data/mysql/mysql-bin
read_only
relay_log_purge=0
skip_name_resolve=1
general_log

[root@mysql-slave-02 ~]# mkdir -p /data/mysql/
[root@mysql-slave-02 ~]# chown -R mysql.mysql /data/mysql/
[root@mysql-slave-02 ~]# 
[root@mysql-slave-02 ~]# 
[root@mysql-slave-02 ~]# systemctl enable --now mysqld
Created symlink /etc/systemd/system/multi-user.target.wants/mysqld.service → /usr/lib/systemd/system/mysqld.service.
[root@mysql-slave-02 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.26 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> CHANGE MASTER TO -> MASTER_HOST='192.168.247.151', -> MASTER_USER='repluser', -> MASTER_PASSWORD='magedu', -> MASTER_LOG_FILE='mysql-bin.000001', -> MASTER_LOG_POS=156; Query OK, 0 rows affected, 8 warnings (0.01 sec) mysql> start slave; Query OK, 0 rows affected, 1 warning (0.01 sec) mysql> show slave status\G *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: 192.168.247.151 Master_User: repluser Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-bin.000001 Read_Master_Log_Pos: 1388 Relay_Log_File: mysql-slave-02-relay-bin.000002 Relay_Log_Pos: 1556 Relay_Master_Log_File: mysql-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 1388 Relay_Log_Space: 1774 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 151 Master_UUID: 7bf8f5a4-2bff-11ed-824b-000c29aae8d7 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: Executed_Gtid_Set: Auto_Position: 0 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: 1 row in set, 1 warning (0.00 sec) mysql>

五、启动MHA服务

检查mha环境

masterha_check_ssh --conf=/etc/mastermha/app1.cnf
masterha_check_repl --conf=/etc/mastermha/app1.cnf
masterha_check_status --conf=/etc/mastermha/app1.cnf
[root@mha-manager ~]# masterha_check_ssh --conf=/etc/mastermha/app1.cnf 
Sun Sep  4 03:14:53 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 03:14:53 2022 - [info] Reading application default configuration from /etc/mastermha/app1.conf..
Sun Sep  4 03:14:53 2022 - [info] Reading server configuration from /etc/mastermha/app1.conf..
Sun Sep  4 03:14:53 2022 - [info] Starting SSH connection tests..
Sun Sep  4 03:14:54 2022 - [debug] 
Sun Sep  4 03:14:53 2022 - [debug]  Connecting via SSH from root@192.168.247.151(192.168.247.151:22) to root@192.168.247.152(192.168.247.152:22)..
Sun Sep  4 03:14:54 2022 - [debug]   ok.
Sun Sep  4 03:14:54 2022 - [debug]  Connecting via SSH from root@192.168.247.151(192.168.247.151:22) to root@192.168.247.153(192.168.247.153:22)..
Sun Sep  4 03:14:54 2022 - [debug]   ok.
Sun Sep  4 03:14:55 2022 - [debug] 
Sun Sep  4 03:14:54 2022 - [debug]  Connecting via SSH from root@192.168.247.152(192.168.247.152:22) to root@192.168.247.151(192.168.247.151:22)..
Sun Sep  4 03:14:54 2022 - [debug]   ok.
Sun Sep  4 03:14:54 2022 - [debug]  Connecting via SSH from root@192.168.247.152(192.168.247.152:22) to root@192.168.247.153(192.168.247.153:22)..
Sun Sep  4 03:14:55 2022 - [debug]   ok.
Sun Sep  4 03:14:56 2022 - [debug] 
Sun Sep  4 03:14:54 2022 - [debug]  Connecting via SSH from root@192.168.247.153(192.168.247.153:22) to root@192.168.247.151(192.168.247.151:22)..
Sun Sep  4 03:14:55 2022 - [debug]   ok.
Sun Sep  4 03:14:55 2022 - [debug]  Connecting via SSH from root@192.168.247.153(192.168.247.153:22) to root@192.168.247.152(192.168.247.152:22)..
Sun Sep  4 03:14:55 2022 - [debug]   ok.
Sun Sep  4 03:14:56 2022 - [info] All SSH connection tests passed successfully.
[root@mha-manager ~]# 
[root@mha-manager ~]# 
[root@mha-manager ~]# masterha_check_repl --conf=/etc/mastermha/app1.cnf 
Sun Sep  4 03:15:00 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 03:15:00 2022 - [info] Reading application default configuration from /etc/mastermha/app1.conf..
Sun Sep  4 03:15:00 2022 - [info] Reading server configuration from /etc/mastermha/app1.conf..
Sun Sep  4 03:15:00 2022 - [info] MHA::MasterMonitor version 0.58.
Sun Sep  4 03:15:01 2022 - [info] GTID failover mode = 0
Sun Sep  4 03:15:01 2022 - [info] Dead Servers:
Sun Sep  4 03:15:01 2022 - [info] Alive Servers:
Sun Sep  4 03:15:01 2022 - [info]   192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:15:01 2022 - [info]   192.168.247.152(192.168.247.152:3306)
Sun Sep  4 03:15:01 2022 - [info]   192.168.247.153(192.168.247.153:3306)
Sun Sep  4 03:15:01 2022 - [info] Alive Slaves:
Sun Sep  4 03:15:01 2022 - [info]   192.168.247.152(192.168.247.152:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 03:15:01 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:15:01 2022 - [info]   192.168.247.153(192.168.247.153:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 03:15:01 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:15:01 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 03:15:01 2022 - [info] Current Alive Master: 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:15:01 2022 - [info] Checking slave configurations..
Sun Sep  4 03:15:01 2022 - [info]  read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 03:15:01 2022 - [warning]  relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 03:15:01 2022 - [info] Checking replication filtering settings..
Sun Sep  4 03:15:01 2022 - [info]  binlog_do_db= , binlog_ignore_db= 
Sun Sep  4 03:15:01 2022 - [info]  Replication filtering check ok.
Sun Sep  4 03:15:01 2022 - [info] GTID (with auto-pos) is not supported
Sun Sep  4 03:15:01 2022 - [info] Starting SSH connection tests..
Sun Sep  4 03:15:04 2022 - [info] All SSH connection tests passed successfully.
Sun Sep  4 03:15:04 2022 - [info] Checking MHA Node version..
Sun Sep  4 03:15:05 2022 - [info]  Version check ok.
Sun Sep  4 03:15:05 2022 - [info] Checking SSH publickey authentication settings on the current master..
Sun Sep  4 03:15:05 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable.
Sun Sep  4 03:15:05 2022 - [info] Master MHA Node version is 0.58.
Sun Sep  4 03:15:05 2022 - [info] Checking recovery script configurations on 192.168.247.151(192.168.247.151:3306)..
Sun Sep  4 03:15:05 2022 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000001 
Sun Sep  4 03:15:05 2022 - [info]   Connecting to root@192.168.247.151(192.168.247.151:22).. 
  Creating /data/mastermha/app1 if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /data/mysql/, up to mysql-bin.000001
Sun Sep  4 03:15:06 2022 - [info] Binlog setting check done.
Sun Sep  4 03:15:06 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Sun Sep  4 03:15:06 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.152 --slave_ip=192.168.247.152 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-01-relay-bin.000002  --slave_pass=xxx
Sun Sep  4 03:15:06 2022 - [info]   Connecting to root@192.168.247.152(192.168.247.152:22).. 
  Checking slave recovery environment settings..
    Relay log found at /var/lib/mysql, up to mysql-slave-01-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-slave-01-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Sun Sep  4 03:15:06 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.153 --slave_ip=192.168.247.153 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-02-relay-bin.000002  --slave_pass=xxx
Sun Sep  4 03:15:06 2022 - [info]   Connecting to root@192.168.247.153(192.168.247.153:22).. 
  Checking slave recovery environment settings..
    Relay log found at /var/lib/mysql, up to mysql-slave-02-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-slave-02-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Sun Sep  4 03:15:06 2022 - [info] Slaves settings check done.
Sun Sep  4 03:15:06 2022 - [info] 
192.168.247.151(192.168.247.151:3306) (current master)
 +--192.168.247.152(192.168.247.152:3306)
 +--192.168.247.153(192.168.247.153:3306)

Sun Sep  4 03:15:06 2022 - [info] Checking replication health on 192.168.247.152..
Sun Sep  4 03:15:06 2022 - [info]  ok.
Sun Sep  4 03:15:06 2022 - [info] Checking replication health on 192.168.247.153..
Sun Sep  4 03:15:06 2022 - [info]  ok.
Sun Sep  4 03:15:06 2022 - [info] Checking master_ip_failover_script status:
Sun Sep  4 03:15:06 2022 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 
Checking the Status of the script.. OK 
Sun Sep  4 03:15:07 2022 - [info]  OK.
Sun Sep  4 03:15:07 2022 - [warning] shutdown_script is not defined.
Sun Sep  4 03:15:07 2022 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.
[root@mha-manager ~]# 
[root@mha-manager ~]# 
[root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.cnf 
app1 is stopped(2:NOT_RUNNING).
[root@mha-manager ~]# 
#开启MHA,默认是前台运行,生产环境一般为后台执行
nohup masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover &> /dev/null

#测试环境:
#masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover

#如果想停止后台执行的MHA,可以执行下面命令
[root@mha-master ~]#masterha_stop --conf=/etc/mastermha/app1.cnf

#查看状态
masterha_check_status --conf=/etc/mastermha/app1.cnf

查看到健康性检查

 

 

 六、故障模拟与恢复

停掉主库

[root@mysql-master ~]# systemctl stop mysqld

mha机器守护进程自动跳出

[root@mha-manager ~]# masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover
Sun Sep  4 03:23:52 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 03:23:52 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 03:23:52 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf..
  Creating /data/mastermha/app1 if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /data/mysql/, up to mysql-bin.000001


  Creating /data/mastermha/app1 if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /data/mysql/, up to mysql-bin.000001
Sun Sep  4 05:42:25 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 05:42:25 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 05:42:25 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf..
[root@mha-manager ~]# 
[root@mha-manager ~]# 
[root@mha-manager ~]# cat /data/mastermha/app1/manager.log 
Sun Sep  4 02:49:41 2022 - [info] MHA::MasterMonitor version 0.58.
Sun Sep  4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/Server.pm, ln180] Got MySQL error when connecting 192.168.247.151(192.168.247.151:3306) :1045:Access denied for user 'mhauser'@'192.168.247.150' (using password: YES), but this is not a MySQL crash. Check MySQL server settings.
Sun Sep  4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301]  at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297.
Sun Sep  4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/Server.pm, ln180] Got MySQL error when connecting 192.168.247.152(192.168.247.152:3306) :1045:Access denied for user 'mhauser'@'192.168.247.150' (using password: YES), but this is not a MySQL crash. Check MySQL server settings.
Sun Sep  4 02:49:41 2022 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301]  at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line 297.
Sun Sep  4 02:49:42 2022 - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln309] Got fatal error, stopping operations
Sun Sep  4 02:49:42 2022 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln427] Error happened on checking configurations.  at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line 329.
Sun Sep  4 02:49:42 2022 - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln525] Error happened on monitoring servers.
Sun Sep  4 02:49:42 2022 - [info] Got exit code 1 (Not master dead).
Sun Sep  4 03:23:52 2022 - [info] MHA::MasterMonitor version 0.58.
Sun Sep  4 03:23:53 2022 - [info] GTID failover mode = 0
Sun Sep  4 03:23:53 2022 - [info] Dead Servers:
Sun Sep  4 03:23:53 2022 - [info] Alive Servers:
Sun Sep  4 03:23:53 2022 - [info]   192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:23:53 2022 - [info]   192.168.247.152(192.168.247.152:3306)
Sun Sep  4 03:23:53 2022 - [info]   192.168.247.153(192.168.247.153:3306)
Sun Sep  4 03:23:53 2022 - [info] Alive Slaves:
Sun Sep  4 03:23:53 2022 - [info]   192.168.247.152(192.168.247.152:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 03:23:53 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:23:53 2022 - [info]   192.168.247.153(192.168.247.153:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 03:23:53 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:23:53 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 03:23:53 2022 - [info] Current Alive Master: 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 03:23:53 2022 - [info] Checking slave configurations..
Sun Sep  4 03:23:53 2022 - [info]  read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 03:23:53 2022 - [warning]  relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 03:23:53 2022 - [info] Checking replication filtering settings..
Sun Sep  4 03:23:53 2022 - [info]  binlog_do_db= , binlog_ignore_db= 
Sun Sep  4 03:23:53 2022 - [info]  Replication filtering check ok.
Sun Sep  4 03:23:53 2022 - [info] GTID (with auto-pos) is not supported
Sun Sep  4 03:23:53 2022 - [info] Starting SSH connection tests..
Sun Sep  4 03:23:55 2022 - [info] All SSH connection tests passed successfully.
Sun Sep  4 03:23:55 2022 - [info] Checking MHA Node version..
Sun Sep  4 03:23:56 2022 - [info]  Version check ok.
Sun Sep  4 03:23:56 2022 - [info] Checking SSH publickey authentication settings on the current master..
Sun Sep  4 03:23:56 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable.
Sun Sep  4 03:23:57 2022 - [info] Master MHA Node version is 0.58.
Sun Sep  4 03:23:57 2022 - [info] Checking recovery script configurations on 192.168.247.151(192.168.247.151:3306)..
Sun Sep  4 03:23:57 2022 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000001 
Sun Sep  4 03:23:57 2022 - [info]   Connecting to root@192.168.247.151(192.168.247.151:22).. 
  Creating /data/mastermha/app1 if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /data/mysql/, up to mysql-bin.000001
Sun Sep  4 03:23:57 2022 - [info] Binlog setting check done.
Sun Sep  4 03:23:57 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Sun Sep  4 03:23:57 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.152 --slave_ip=192.168.247.152 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-01-relay-bin.000002  --slave_pass=xxx
Sun Sep  4 03:23:57 2022 - [info]   Connecting to root@192.168.247.152(192.168.247.152:22).. 
  Checking slave recovery environment settings..
    Relay log found at /var/lib/mysql, up to mysql-slave-01-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-slave-01-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Sun Sep  4 03:23:57 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.153 --slave_ip=192.168.247.153 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-02-relay-bin.000002  --slave_pass=xxx
Sun Sep  4 03:23:57 2022 - [info]   Connecting to root@192.168.247.153(192.168.247.153:22).. 
  Checking slave recovery environment settings..
    Relay log found at /var/lib/mysql, up to mysql-slave-02-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-slave-02-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Sun Sep  4 03:23:58 2022 - [info] Slaves settings check done.
Sun Sep  4 03:23:58 2022 - [info] 
192.168.247.151(192.168.247.151:3306) (current master)
 +--192.168.247.152(192.168.247.152:3306)
 +--192.168.247.153(192.168.247.153:3306)

Sun Sep  4 03:23:58 2022 - [info] Checking master_ip_failover_script status:
Sun Sep  4 03:23:58 2022 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 
Checking the Status of the script.. OK 
Sun Sep  4 03:23:58 2022 - [info]  OK.
Sun Sep  4 03:23:58 2022 - [warning] shutdown_script is not defined.
Sun Sep  4 03:23:58 2022 - [info] Set master ping interval 1 seconds.
Sun Sep  4 03:23:58 2022 - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Sun Sep  4 03:23:58 2022 - [info] Starting ping health check on 192.168.247.151(192.168.247.151:3306)..
Sun Sep  4 03:23:58 2022 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Sun Sep  4 05:37:45 2022 - [warning] Got timeout on MySQL Ping(SELECT) child process and killed it! at /usr/share/perl5/vendor_perl/MHA/HealthCheck.pm line 432.
Sun Sep  4 05:37:47 2022 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin
Sun Sep  4 05:37:47 2022 - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Sun Sep  4 05:37:48 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable.
Sun Sep  4 05:42:21 2022 - [warning] Got error on MySQL select ping: 1053 (Server shutdown in progress)
Sun Sep  4 05:42:21 2022 - [info] Executing SSH check script: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --binlog_prefix=mysql-bin
Sun Sep  4 05:42:22 2022 - [info] HealthCheck: SSH to 192.168.247.151 is reachable.
Sun Sep  4 05:42:22 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.247.151' (111))
Sun Sep  4 05:42:22 2022 - [warning] Connection failed 2 time(s)..
Sun Sep  4 05:42:23 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.247.151' (111))
Sun Sep  4 05:42:23 2022 - [warning] Connection failed 3 time(s)..
Sun Sep  4 05:42:24 2022 - [warning] Got error on MySQL connect: 2003 (Can't connect to MySQL server on '192.168.247.151' (111))
Sun Sep  4 05:42:24 2022 - [warning] Connection failed 4 time(s)..
Sun Sep  4 05:42:24 2022 - [warning] Master is not reachable from health checker!
Sun Sep  4 05:42:24 2022 - [warning] Master 192.168.247.151(192.168.247.151:3306) is not reachable!
Sun Sep  4 05:42:24 2022 - [warning] SSH is reachable.
Sun Sep  4 05:42:24 2022 - [info] Connecting to a master server failed. Reading configuration file /etc/masterha_default.cnf and /etc/mastermha/app1.cnf again, and trying to connect to all servers to check server status..
Sun Sep  4 05:42:24 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 05:42:24 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 05:42:24 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 05:42:25 2022 - [info] GTID failover mode = 0
Sun Sep  4 05:42:25 2022 - [info] Dead Servers:
Sun Sep  4 05:42:25 2022 - [info]   192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:25 2022 - [info] Alive Servers:
Sun Sep  4 05:42:25 2022 - [info]   192.168.247.152(192.168.247.152:3306)
Sun Sep  4 05:42:25 2022 - [info]   192.168.247.153(192.168.247.153:3306)
Sun Sep  4 05:42:25 2022 - [info] Alive Slaves:
Sun Sep  4 05:42:25 2022 - [info]   192.168.247.152(192.168.247.152:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:25 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:25 2022 - [info]   192.168.247.153(192.168.247.153:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:25 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:25 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 05:42:25 2022 - [info] Checking slave configurations..
Sun Sep  4 05:42:25 2022 - [info]  read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 05:42:25 2022 - [warning]  relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 05:42:25 2022 - [info] Checking replication filtering settings..
Sun Sep  4 05:42:25 2022 - [info]  Replication filtering check ok.
Sun Sep  4 05:42:25 2022 - [info] Master is down!
Sun Sep  4 05:42:25 2022 - [info] Terminating monitoring script.
Sun Sep  4 05:42:25 2022 - [info] Got exit code 20 (Master dead).
Sun Sep  4 05:42:26 2022 - [info] MHA::MasterFailover version 0.58.
Sun Sep  4 05:42:26 2022 - [info] Starting master failover.
Sun Sep  4 05:42:26 2022 - [info] 
Sun Sep  4 05:42:26 2022 - [info] * Phase 1: Configuration Check Phase..
Sun Sep  4 05:42:26 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] GTID failover mode = 0
Sun Sep  4 05:42:27 2022 - [info] Dead Servers:
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:27 2022 - [info] Checking master reachability via MySQL(double check)...
Sun Sep  4 05:42:27 2022 - [info]  ok.
Sun Sep  4 05:42:27 2022 - [info] Alive Servers:
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.152(192.168.247.152:3306)
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.153(192.168.247.153:3306)
Sun Sep  4 05:42:27 2022 - [info] Alive Slaves:
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.152(192.168.247.152:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:27 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.153(192.168.247.153:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:27 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:27 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 05:42:27 2022 - [info] Starting Non-GTID based failover.
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] ** Phase 1: Configuration Check Phase completed.
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] * Phase 2: Dead Master Shutdown Phase..
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] Forcing shutdown so that applications never connect to the current master..
Sun Sep  4 05:42:27 2022 - [info] Executing master IP deactivation script:
Sun Sep  4 05:42:27 2022 - [info]   /usr/local/bin/master_ip_failover --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 --command=stopssh --ssh_user=root  
Sun Sep  4 05:42:27 2022 - [info]  done.
Sun Sep  4 05:42:27 2022 - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Sun Sep  4 05:42:27 2022 - [info] * Phase 2: Dead Master Shutdown Phase completed.
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] * Phase 3: Master Recovery Phase..
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] * Phase 3.1: Getting Latest Slaves Phase..
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] The latest binary log file/position on all slaves is mysql-bin.000001:1388
Sun Sep  4 05:42:27 2022 - [info] Latest slaves (Slaves that received relay log files to the latest):
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.152(192.168.247.152:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:27 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.153(192.168.247.153:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:27 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:27 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 05:42:27 2022 - [info] The oldest binary log file/position on all slaves is mysql-bin.000001:1388
Sun Sep  4 05:42:27 2022 - [info] Oldest slaves:
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.152(192.168.247.152:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:27 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:27 2022 - [info]   192.168.247.153(192.168.247.153:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:27 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:27 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:27 2022 - [info] * Phase 3.2: Saving Dead Master's Binlog Phase..
Sun Sep  4 05:42:27 2022 - [info] 
Sun Sep  4 05:42:28 2022 - [info] Fetching dead master's binary logs..
Sun Sep  4 05:42:28 2022 - [info] Executing command on the dead master 192.168.247.151(192.168.247.151:3306): save_binary_logs --command=save --start_file=mysql-bin.000001  --start_pos=1388 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog --handle_raw_binlog=1 --disable_log_bin=0 --manager_version=0.58
  Creating /data/mastermha/app1 if not exists..    ok.
 Concat binary/relay logs from mysql-bin.000001 pos 1388 to mysql-bin.000001 EOF into /data/mastermha/app1//saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog ..
 Binlog Checksum enabled
  Dumping binlog format description event, from position 0 to 156.. ok.
  No need to dump effective binlog data from /data/mysql//mysql-bin.000001 (pos starts 1388, filesize 1388). Skipping.
 Binlog Checksum enabled
 /data/mastermha/app1//saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog has no effective data events.
Event not exists.
Sun Sep  4 05:42:29 2022 - [info] Additional events were not found from the orig master. No need to save.
Sun Sep  4 05:42:29 2022 - [info] 
Sun Sep  4 05:42:29 2022 - [info] * Phase 3.3: Determining New Master Phase..
Sun Sep  4 05:42:29 2022 - [info] 
Sun Sep  4 05:42:29 2022 - [info] Finding the latest slave that has all relay logs for recovering other slaves..
Sun Sep  4 05:42:29 2022 - [info] All slaves received relay logs to the same position. No need to resync each other.
Sun Sep  4 05:42:29 2022 - [info] Searching new master from slaves..
Sun Sep  4 05:42:29 2022 - [info]  Candidate masters from the configuration file:
Sun Sep  4 05:42:29 2022 - [info]   192.168.247.153(192.168.247.153:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 05:42:29 2022 - [info]     Replicating from 192.168.247.151(192.168.247.151:3306)
Sun Sep  4 05:42:29 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 05:42:29 2022 - [info]  Non-candidate masters:
Sun Sep  4 05:42:29 2022 - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Sun Sep  4 05:42:29 2022 - [info] New master is 192.168.247.153(192.168.247.153:3306)
Sun Sep  4 05:42:29 2022 - [info] Starting master failover..
Sun Sep  4 05:42:29 2022 - [info] 
From:
192.168.247.151(192.168.247.151:3306) (current master)
 +--192.168.247.152(192.168.247.152:3306)
 +--192.168.247.153(192.168.247.153:3306)

To:
192.168.247.153(192.168.247.153:3306) (new master)
 +--192.168.247.152(192.168.247.152:3306)
Sun Sep  4 05:42:29 2022 - [info] 
Sun Sep  4 05:42:29 2022 - [info] * Phase 3.4: New Master Diff Log Generation Phase..
Sun Sep  4 05:42:29 2022 - [info] 
Sun Sep  4 05:42:29 2022 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Sun Sep  4 05:42:29 2022 - [info] 
Sun Sep  4 05:42:29 2022 - [info] * Phase 3.5: Master Log Apply Phase..
Sun Sep  4 05:42:29 2022 - [info] 
Sun Sep  4 05:42:29 2022 - [info] *NOTICE: If any error happens from this phase, manual recovery is needed.
Sun Sep  4 05:42:29 2022 - [info] Starting recovery on 192.168.247.153(192.168.247.153:3306)..
Sun Sep  4 05:42:29 2022 - [info]  This server has all relay logs. Waiting all logs to be applied.. 
Sun Sep  4 05:42:29 2022 - [info]   done.
Sun Sep  4 05:42:29 2022 - [info]  All relay logs were successfully applied.
Sun Sep  4 05:42:29 2022 - [info] Getting new master's binlog name and position..
Sun Sep  4 05:42:29 2022 - [info]  mysql-bin.000002:1423
Sun Sep  4 05:42:29 2022 - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.247.153', MASTER_PORT=3306, MASTER_LOG_FILE='mysql-bin.000002', MASTER_LOG_POS=1423, MASTER_USER='repluser', MASTER_PASSWORD='xxx';
Sun Sep  4 05:42:29 2022 - [info] Executing master IP activate script:
Sun Sep  4 05:42:29 2022 - [info]   /usr/local/bin/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.247.151 --orig_master_ip=192.168.247.151 --orig_master_port=3306 --new_master_host=192.168.247.153 --new_master_ip=192.168.247.153 --new_master_port=3306 --new_master_user='mhauser'   --new_master_password=xxx
Enabling the VIP - 192.168.247.100/24 on the new master - 192.168.247.153
Sun Sep  4 05:42:30 2022 - [info]  OK.
Sun Sep  4 05:42:30 2022 - [info] Setting read_only=0 on 192.168.247.153(192.168.247.153:3306)..
Sun Sep  4 05:42:30 2022 - [info]  ok.
Sun Sep  4 05:42:30 2022 - [info] ** Finished master recovery successfully.
Sun Sep  4 05:42:30 2022 - [info] * Phase 3: Master Recovery Phase completed.
Sun Sep  4 05:42:30 2022 - [info] 
Sun Sep  4 05:42:30 2022 - [info] * Phase 4: Slaves Recovery Phase..
Sun Sep  4 05:42:30 2022 - [info] 
Sun Sep  4 05:42:30 2022 - [info] * Phase 4.1: Starting Parallel Slave Diff Log Generation Phase..
Sun Sep  4 05:42:30 2022 - [info] 
Sun Sep  4 05:42:30 2022 - [info] -- Slave diff file generation on host 192.168.247.152(192.168.247.152:3306) started, pid: 20985. Check tmp log /data/mastermha/app1//192.168.247.152_3306_20220904054225.log if it takes time..
Sun Sep  4 05:42:31 2022 - [info] 
Sun Sep  4 05:42:31 2022 - [info] Log messages from 192.168.247.152 ...
Sun Sep  4 05:42:31 2022 - [info] 
Sun Sep  4 05:42:30 2022 - [info]  This server has all relay logs. No need to generate diff files from the latest slave.
Sun Sep  4 05:42:31 2022 - [info] End of log messages from 192.168.247.152.
Sun Sep  4 05:42:31 2022 - [info] -- 192.168.247.152(192.168.247.152:3306) has the latest relay log events.
Sun Sep  4 05:42:31 2022 - [info] Generating relay diff files from the latest slave succeeded.
Sun Sep  4 05:42:31 2022 - [info] 
Sun Sep  4 05:42:31 2022 - [info] * Phase 4.2: Starting Parallel Slave Log Apply Phase..
Sun Sep  4 05:42:31 2022 - [info] 
Sun Sep  4 05:42:31 2022 - [info] -- Slave recovery on host 192.168.247.152(192.168.247.152:3306) started, pid: 20987. Check tmp log /data/mastermha/app1//192.168.247.152_3306_20220904054225.log if it takes time..
Sun Sep  4 05:42:32 2022 - [info] 
Sun Sep  4 05:42:32 2022 - [info] Log messages from 192.168.247.152 ...
Sun Sep  4 05:42:32 2022 - [info] 
Sun Sep  4 05:42:31 2022 - [info] Starting recovery on 192.168.247.152(192.168.247.152:3306)..
Sun Sep  4 05:42:31 2022 - [info]  This server has all relay logs. Waiting all logs to be applied.. 
Sun Sep  4 05:42:31 2022 - [info]   done.
Sun Sep  4 05:42:31 2022 - [info]  All relay logs were successfully applied.
Sun Sep  4 05:42:31 2022 - [info]  Resetting slave 192.168.247.152(192.168.247.152:3306) and starting replication from the new master 192.168.247.153(192.168.247.153:3306)..
Sun Sep  4 05:42:31 2022 - [info]  Executed CHANGE MASTER.
Sun Sep  4 05:42:31 2022 - [info]  Slave started.
Sun Sep  4 05:42:32 2022 - [info] End of log messages from 192.168.247.152.
Sun Sep  4 05:42:32 2022 - [info] -- Slave recovery on host 192.168.247.152(192.168.247.152:3306) succeeded.
Sun Sep  4 05:42:32 2022 - [info] All new slave servers recovered successfully.
Sun Sep  4 05:42:32 2022 - [info] 
Sun Sep  4 05:42:32 2022 - [info] * Phase 5: New master cleanup phase..
Sun Sep  4 05:42:32 2022 - [info] 
Sun Sep  4 05:42:32 2022 - [info] Resetting slave info on the new master..
Sun Sep  4 05:42:32 2022 - [info]  192.168.247.153: Resetting slave info succeeded.
Sun Sep  4 05:42:32 2022 - [info] Master failover to 192.168.247.153(192.168.247.153:3306) completed successfully.
Sun Sep  4 05:42:32 2022 - [info] Deleted server1 entry from /etc/mastermha/app1.cnf .
Sun Sep  4 05:42:32 2022 - [info] 

----- Failover Report -----

app1: MySQL Master failover 192.168.247.151(192.168.247.151:3306) to 192.168.247.153(192.168.247.153:3306) succeeded

Master 192.168.247.151(192.168.247.151:3306) is down!

Check MHA Manager logs at mha-manager:/data/mastermha/app1/manager.log for details.

Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.247.151(192.168.247.151:3306)
The latest slave 192.168.247.152(192.168.247.152:3306) has all relay logs for recovery.
Selected 192.168.247.153(192.168.247.153:3306) as a new master.
192.168.247.153(192.168.247.153:3306): OK: Applying all logs succeeded.
192.168.247.153(192.168.247.153:3306): OK: Activated master IP address.
192.168.247.152(192.168.247.152:3306): This host has the latest relay log events.
Generating relay diff files from the latest slave succeeded.
192.168.247.152(192.168.247.152:3306): OK: Applying all logs succeeded. Slave started, replicating from 192.168.247.153(192.168.247.153:3306)
192.168.247.153(192.168.247.153:3306): Resetting slave info succeeded.
Master failover to 192.168.247.153(192.168.247.153:3306) completed successfully.
Sun Sep  4 05:42:32 2022 - [info] Sending mail..
[root@mha-manager ~]# 

此时也收到邮件告警提醒

 mha节点服务状态节点检查异常

 

 vip也自动漂到指定的从节点

app1.cnf配置文件自动剔除manager 151节点信息

 

如果要再次运行MHA,需要先删除下面文件

因为MHA只能漂移一次,如果多次使用必须删除以下文件,要不MHA不可重用

[root@mha-manager ~]# rm -fr /data/mastermha/app1/*
[root@mysql-master ~]# ls /data/mastermha/app1/
saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog
[root@mysql-master ~]#
[root@mysql-master ~]# rm -fr /data/mastermha/app1/*
[root@mysql-master ~]#

将原有的master当slave加入

[root@mysql-slave-02 ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 65
Server version: 8.0.26 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show master logs;
+------------------+-----------+-----------+
| Log_name         | File_size | Encrypted |
+------------------+-----------+-----------+
| mysql-bin.000001 |       179 | No        |
| mysql-bin.000002 |     12487 | No        |
+------------------+-----------+-----------+
2 rows in set (0.00 sec)

mysql> 

将旧的master当成slave加入集群

[root@mysql-master ~]# ls /data/mastermha/app1/
saved_master_binlog_from_192.168.247.151_3306_20220904054225.binlog
[root@mysql-master ~]# 
[root@mysql-master ~]# rm -fr /data/mastermha/app1/*
[root@mysql-master ~]# 
[root@mysql-master ~]# systemctl start mysqld
[root@mysql-master ~]# mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 8
Server version: 8.0.26 Source distribution

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> CHANGE MASTER TO 
    -> MASTER_HOST='192.168.247.153', 
    -> MASTER_USER='repluser',
    -> MASTER_PASSWORD='magedu', 
    -> MASTER_LOG_FILE='mysql-bin.000002',
    -> MASTER_LOG_POS=12487;
Query OK, 0 rows affected, 8 warnings (0.02 sec)

mysql> start slave;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for source to send event
                  Master_Host: 192.168.247.153
                  Master_User: repluser
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000002
          Read_Master_Log_Pos: 12487
               Relay_Log_File: mysql-master-relay-bin.000002
                Relay_Log_Pos: 324
        Relay_Master_Log_File: mysql-bin.000002
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: 
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 12487
              Relay_Log_Space: 540
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 153
                  Master_UUID: f8c35181-2c01-11ed-b28f-000c29d095cd
             Master_Info_File: mysql.slave_master_info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
       Master_public_key_path: 
        Get_master_public_key: 0
            Network_Namespace: 
1 row in set, 1 warning (0.00 sec)

mysql> 

在次检查并启动mha

[root@mha-manager ~]# masterha_check_ssh --conf=/etc/mastermha/app1.cnf
Sun Sep  4 07:49:06 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 07:49:06 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 07:49:06 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 07:49:06 2022 - [info] Starting SSH connection tests..
Sun Sep  4 07:49:07 2022 - [debug] 
Sun Sep  4 07:49:06 2022 - [debug]  Connecting via SSH from root@192.168.247.153(192.168.247.153:22) to root@192.168.247.152(192.168.247.152:22)..
Sun Sep  4 07:49:07 2022 - [debug]   ok.
Sun Sep  4 07:49:07 2022 - [debug]  Connecting via SSH from root@192.168.247.153(192.168.247.153:22) to root@192.168.247.151(192.168.247.151:22)..
Sun Sep  4 07:49:07 2022 - [debug]   ok.
Sun Sep  4 07:49:08 2022 - [debug] 
Sun Sep  4 07:49:07 2022 - [debug]  Connecting via SSH from root@192.168.247.152(192.168.247.152:22) to root@192.168.247.153(192.168.247.153:22)..
Sun Sep  4 07:49:07 2022 - [debug]   ok.
Sun Sep  4 07:49:07 2022 - [debug]  Connecting via SSH from root@192.168.247.152(192.168.247.152:22) to root@192.168.247.151(192.168.247.151:22)..
Sun Sep  4 07:49:07 2022 - [debug]   ok.
Sun Sep  4 07:49:09 2022 - [debug] 
Sun Sep  4 07:49:07 2022 - [debug]  Connecting via SSH from root@192.168.247.151(192.168.247.151:22) to root@192.168.247.153(192.168.247.153:22)..
Sun Sep  4 07:49:08 2022 - [debug]   ok.
Sun Sep  4 07:49:08 2022 - [debug]  Connecting via SSH from root@192.168.247.151(192.168.247.151:22) to root@192.168.247.152(192.168.247.152:22)..
Sun Sep  4 07:49:08 2022 - [debug]   ok.
Sun Sep  4 07:49:09 2022 - [info] All SSH connection tests passed successfully.
[root@mha-manager ~]# 
[root@mha-manager ~]# 
[root@mha-manager ~]# masterha_check_repl --conf=/etc/mastermha/app1.cnf 
Sun Sep  4 07:49:12 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 07:49:12 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 07:49:12 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 07:49:12 2022 - [info] MHA::MasterMonitor version 0.58.
Sun Sep  4 07:49:13 2022 - [info] GTID failover mode = 0
Sun Sep  4 07:49:13 2022 - [info] Dead Servers:
Sun Sep  4 07:49:13 2022 - [info] Alive Servers:
Sun Sep  4 07:49:13 2022 - [info]   192.168.247.153(192.168.247.153:3306)
Sun Sep  4 07:49:13 2022 - [info]   192.168.247.152(192.168.247.152:3306)
Sun Sep  4 07:49:13 2022 - [info]   192.168.247.151(192.168.247.151:3306)
Sun Sep  4 07:49:13 2022 - [info] Alive Slaves:
Sun Sep  4 07:49:13 2022 - [info]   192.168.247.152(192.168.247.152:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 07:49:13 2022 - [info]     Replicating from 192.168.247.153(192.168.247.153:3306)
Sun Sep  4 07:49:13 2022 - [info]   192.168.247.151(192.168.247.151:3306)  Version=8.0.26 (oldest major version between slaves) log-bin:enabled
Sun Sep  4 07:49:13 2022 - [info]     Replicating from 192.168.247.153(192.168.247.153:3306)
Sun Sep  4 07:49:13 2022 - [info]     Primary candidate for the new Master (candidate_master is set)
Sun Sep  4 07:49:13 2022 - [info] Current Alive Master: 192.168.247.153(192.168.247.153:3306)
Sun Sep  4 07:49:13 2022 - [info] Checking slave configurations..
Sun Sep  4 07:49:13 2022 - [info]  read_only=1 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 07:49:13 2022 - [warning]  relay_log_purge=0 is not set on slave 192.168.247.152(192.168.247.152:3306).
Sun Sep  4 07:49:13 2022 - [info] Checking replication filtering settings..
Sun Sep  4 07:49:13 2022 - [info]  binlog_do_db= , binlog_ignore_db= 
Sun Sep  4 07:49:13 2022 - [info]  Replication filtering check ok.
Sun Sep  4 07:49:13 2022 - [info] GTID (with auto-pos) is not supported
Sun Sep  4 07:49:13 2022 - [info] Starting SSH connection tests..
Sun Sep  4 07:49:15 2022 - [info] All SSH connection tests passed successfully.
Sun Sep  4 07:49:15 2022 - [info] Checking MHA Node version..
Sun Sep  4 07:49:16 2022 - [info]  Version check ok.
Sun Sep  4 07:49:16 2022 - [info] Checking SSH publickey authentication settings on the current master..
Sun Sep  4 07:49:16 2022 - [info] HealthCheck: SSH to 192.168.247.153 is reachable.
Sun Sep  4 07:49:17 2022 - [info] Master MHA Node version is 0.58.
Sun Sep  4 07:49:17 2022 - [info] Checking recovery script configurations on 192.168.247.153(192.168.247.153:3306)..
Sun Sep  4 07:49:17 2022 - [info]   Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/data/mysql/ --output_file=/data/mastermha/app1//save_binary_logs_test --manager_version=0.58 --start_file=mysql-bin.000002 
Sun Sep  4 07:49:17 2022 - [info]   Connecting to root@192.168.247.153(192.168.247.153:22).. 
  Creating /data/mastermha/app1 if not exists..    ok.
  Checking output directory is accessible or not..
   ok.
  Binlog found at /data/mysql/, up to mysql-bin.000002
Sun Sep  4 07:49:17 2022 - [info] Binlog setting check done.
Sun Sep  4 07:49:17 2022 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Sun Sep  4 07:49:17 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.152 --slave_ip=192.168.247.152 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-slave-01-relay-bin.000002  --slave_pass=xxx
Sun Sep  4 07:49:17 2022 - [info]   Connecting to root@192.168.247.152(192.168.247.152:22).. 
  Checking slave recovery environment settings..
    Relay log found at /var/lib/mysql, up to mysql-slave-01-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-slave-01-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Sun Sep  4 07:49:17 2022 - [info]   Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.247.151 --slave_ip=192.168.247.151 --slave_port=3306 --workdir=/data/mastermha/app1/ --target_version=8.0.26 --manager_version=0.58 --relay_dir=/var/lib/mysql --current_relay_log=mysql-master-relay-bin.000002  --slave_pass=xxx
Sun Sep  4 07:49:17 2022 - [info]   Connecting to root@192.168.247.151(192.168.247.151:22).. 
  Checking slave recovery environment settings..
    Relay log found at /var/lib/mysql, up to mysql-master-relay-bin.000002
    Temporary relay log file is /var/lib/mysql/mysql-master-relay-bin.000002
    Checking if super_read_only is defined and turned on.. not present or turned off, ignoring.
    Testing mysql connection and privileges..
mysql: [Warning] Using a password on the command line interface can be insecure.
 done.
    Testing mysqlbinlog output.. done.
    Cleaning up test file(s).. done.
Sun Sep  4 07:49:18 2022 - [info] Slaves settings check done.
Sun Sep  4 07:49:18 2022 - [info] 
192.168.247.153(192.168.247.153:3306) (current master)
 +--192.168.247.152(192.168.247.152:3306)
 +--192.168.247.151(192.168.247.151:3306)

Sun Sep  4 07:49:18 2022 - [info] Checking replication health on 192.168.247.152..
Sun Sep  4 07:49:18 2022 - [info]  ok.
Sun Sep  4 07:49:18 2022 - [info] Checking replication health on 192.168.247.151..
Sun Sep  4 07:49:18 2022 - [info]  ok.
Sun Sep  4 07:49:18 2022 - [info] Checking master_ip_failover_script status:
Sun Sep  4 07:49:18 2022 - [info]   /usr/local/bin/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.247.153 --orig_master_ip=192.168.247.153 --orig_master_port=3306 
Checking the Status of the script.. OK 
Sun Sep  4 07:49:18 2022 - [info]  OK.
Sun Sep  4 07:49:18 2022 - [warning] shutdown_script is not defined.
Sun Sep  4 07:49:18 2022 - [info] Got exit code 0 (Not master dead).

MySQL Replication Health is OK.
[root@mha-manager ~]# 
[root@mha-manager ~]# masterha_check_status --conf=/etc/mastermha/app1.cnf 
app1 is stopped(2:NOT_RUNNING).
[root@mha-manager ~]# 
[root@mha-manager ~]# 
[root@mha-manager ~]# masterha_manager --conf=/etc/mastermha/app1.cnf --remove_dead_master_conf --ignore_last_failover
Sun Sep  4 07:49:54 2022 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sun Sep  4 07:49:54 2022 - [info] Reading application default configuration from /etc/mastermha/app1.cnf..
Sun Sep  4 07:49:54 2022 - [info] Reading server configuration from /etc/mastermha/app1.cnf..

此时监控启动正常,故障模拟恢复成功

 

标签:实战,info,Sep,Sun,05,192.168,MHA,案例,2022
来源: https://www.cnblogs.com/cyh00001/p/16654088.html