Node-1 ====== 2017-07-05 00:26:23 4092 [Warning] Aborted connection 1133883 to db: 'qauser' user: 'qauser' host: 'MySqlM1.info.in' (Got an error reading communication packets) 2017-07-05 01:42:40 4092 [Warning] Aborted connection 1138196 to db: 'qauser' user: 'qauser' host: 'MySqlM1.info.in' (Got an error reading communication packets) 2017-07-05 06:02:29 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.107:4567 2017-07-05 06:02:30 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') connection established to 451ca788 tcp://10.1.3.107:4567 2017-07-05 06:02:33 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-07-05 06:53:32 4092 [Warning] Aborted connection 1159129 to db: 'qauser' user: 'qauser' host: 'MySqlM1.info.in' (Got an error reading communication packets) 2017-07-05 06:55:24 4092 [Warning] Aborted connection 1159256 to db: 'qauser' user: 'qauser' host: 'MySqlM1.info.in' (Got an error reading communication packets) 2017-07-05 08:00:18 4092 [Warning] Aborted connection 1163639 to db: 'qauser' user: 'qauser' host: 'MySqlM1.info.in' (Got an error reading communication packets) 2017-07-05 09:05:31 4092 [Warning] Aborted connection 1167492 to db: 'unconnected' user: 'qauserqa' host: 'MySqlM1.info.in' (Got an error reading communication packets) 2017-07-05 10:12:00 4092 [Warning] Aborted connection 1173622 to db: 'qauser' user: 'qauser' host: 'MySqlM1.info.in' (Got an error reading communication packets) 2017-07-05 12:10:42 4092 [Warning] IP address '10.1.3.107' could not be resolved: Name or service not known 2017-07-05 12:15:50 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') connection established to 451ca788 tcp://10.1.3.107:4567 2017-07-05 12:15:50 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2017-07-05 12:15:51 4092 [Note] WSREP: declaring 0b83c48f at tcp://10.1.3.102:4567 stable 2017-07-05 12:15:51 4092 [Note] WSREP: Node 0b83c48f state prim 2017-07-05 12:15:51 4092 [Note] WSREP: declaring 0b83c48f at tcp://10.1.3.102:4567 stable 2017-07-05 12:15:51 4092 [Note] WSREP: declaring 451ca788 at tcp://10.1.3.107:4567 stable 2017-07-05 12:15:51 4092 [Note] WSREP: Node 0b83c48f state prim 2017-07-05 12:15:51 4092 [Note] WSREP: view(view_id(PRIM,0b83c48f,204) memb { 0b83c48f,0 451ca788,0 aec83d42,0 } joined { } left { } partitioned { }) 2017-07-05 12:15:51 4092 [Note] WSREP: save pc into disk 2017-07-05 12:15:51 4092 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 2, memb_num = 3 2017-07-05 12:15:51 4092 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2017-07-05 12:15:51 4092 [Note] WSREP: STATE EXCHANGE: sent state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 2017-07-05 12:15:51 4092 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 0 (MySQLM2.info.in) 2017-07-05 12:15:51 4092 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 1 (MySQLM3.info.in) 2017-07-05 12:15:51 4092 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 2 (MySqlM1.info.in) 2017-07-05 12:15:51 4092 [Note] WSREP: Quorum results: version = 4, component = PRIMARY, conf_id = 115, members = 2/3 (joined/total), act_id = 47076814, last_appl. = 47076759, protocols = 0/7/3 (gcs/repl/appl), group UUID = fed83039-1a7a-11e7-b295-5fd619b022d9 2017-07-05 12:15:51 4092 [Note] WSREP: Flow-control interval: [28, 28] 2017-07-05 12:15:51 4092 [Note] WSREP: New cluster view: global state: fed83039-1a7a-11e7-b295-5fd619b022d9:47076814, view# 116: Primary, number of nodes: 3, my index: 2, protocol version 3 2017-07-05 12:15:51 4092 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2017-07-05 12:15:51 4092 [Note] WSREP: REPL Protocols: 7 (3, 2) 2017-07-05 12:15:51 4092 [Note] WSREP: Assign initial position for certification: 47076814, protocol version: 3 2017-07-05 12:15:51 4092 [Note] WSREP: Service thread queue flushed. 2017-07-05 12:15:51 4092 [ERROR] WSREP: Certification failed for TO isolated action: source: 0b83c48f-4768-11e7-a3f0-a309c3ecfc8c version: 3 local: 0 state: CERTIFYING flags: 65 conn_id: 2651462 trx_id: -1 seqnos (l: 27832949, g: 47076815, s: 47076808, d: -1, ts: 11915704329178894) 2017-07-05 12:15:51 4092 [Note] WSREP: Closing send monitor... 2017-07-05 12:15:51 4092 [Note] WSREP: Closed send monitor. 2017-07-05 12:15:51 4092 [Note] WSREP: gcomm: terminating thread 2017-07-05 12:15:51 4092 [Note] WSREP: gcomm: joining thread 2017-07-05 12:15:51 4092 [Note] WSREP: gcomm: closing backend 2017-07-05 12:15:51 4092 [Note] WSREP: Member 1.0 (MySQLM3.info.in) requested state transfer from '*any*'. Selected 2.0 (MySqlM1.info.in)(SYNCED) as donor. 2017-07-05 12:15:51 4092 [Note] WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 47076815) 2017-07-05 12:15:51 4092 [Note] WSREP: declaring 451ca788 at tcp://10.1.3.107:4567 stable 2017-07-05 12:15:51 4092 [Note] WSREP: forgetting 0b83c48f (tcp://10.1.3.102:4567) 2017-07-05 12:15:51 4092 [Note] WSREP: Node 451ca788 state prim 2017-07-05 12:15:53 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-07-05 12:15:55 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') connection to peer 451ca788 with addr tcp://10.1.3.107:4567 timed out, no messages seen in PT3S 2017-07-05 12:15:55 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.107:4567 2017-07-05 12:15:56 4092 [Note] WSREP: (aec83d42, 'tcp://0.0.0.0:4567') reconnecting to 451ca788 (tcp://10.1.3.107:4567), attempt 0 2017-07-05 12:15:56 4092 [Warning] WSREP: TO isolation failed for: 3, schema: qauser, sql: DROP TABLE IF EXISTS tmp_ldspassage. Check wsrep connection state and retry the query. 2017-07-05 12:15:56 4092 [Warning] WSREP: TO isolation failed for: 3, schema: qauser, sql: DROP TABLE IF EXISTS tmp_ldspassage. Check wsrep connection state and retry the query. 2017-07-05 12:15:57 4092 [Note] WSREP: cleaning up 0b83c48f (tcp://10.1.3.102:4567) 2017-07-05 12:15:57 4092 [Warning] WSREP: TO isolation failed for: 3, schema: qauser, sql: DROP TABLE IF EXISTS tmp_ldspassage. Check wsrep connection state and retry the query. 2017-07-05 12:15:57 4092 [Note] WSREP: evs::proto(aec83d42, LEAVING, view_id(REG,451ca788,205)) suspecting node: 451ca788 2017-07-05 12:15:57 4092 [Note] WSREP: evs::proto(aec83d42, LEAVING, view_id(REG,451ca788,205)) suspected node without join message, declaring inactive 2017-07-05 12:15:57 4092 [Note] WSREP: view(view_id(NON_PRIM,451ca788,205) memb { aec83d42,0 } joined { } left { } partitioned { 0b83c48f,0 451ca788,0 }) 2017-07-05 12:15:57 4092 [Note] WSREP: view((empty)) 2017-07-05 12:15:57 4092 [Note] WSREP: gcomm: closed 2017-07-05 12:15:57 4092 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:15:57 4092 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:15:57 4092 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:15:57 4092 [Note] WSREP: Shifting DONOR/DESYNCED -> OPEN (TO: 47076815) 2017-07-05 12:15:57 4092 [Note] WSREP: Received self-leave message. 2017-07-05 12:15:57 4092 [Note] WSREP: Flow-control interval: [0, 0] 2017-07-05 12:15:57 4092 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2017-07-05 12:15:57 4092 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 47076815) 2017-07-05 12:15:57 4092 [Note] WSREP: RECV thread exiting 0: Success 2017-07-05 12:15:57 4092 [Note] WSREP: recv_thread() joined. 2017-07-05 12:15:57 4092 [Note] WSREP: Closing replication queue. 2017-07-05 12:15:57 4092 [Note] WSREP: Closing slave action queue. 2017-07-05 12:15:57 4092 [Note] WSREP: /usr/sbin/mysqld: Terminated. Node-2 ====== 2017-07-05 00:38:15 17027 [Warning] Aborted connection 2604295 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 04:56:38 17027 [Warning] Aborted connection 2620939 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 05:48:01 17027 [Warning] Aborted connection 2624872 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 06:14:00 17027 [Warning] Aborted connection 2573183 to db: 'qausertest' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 07:57:32 17027 [Warning] Aborted connection 2630564 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 08:00:20 17027 [Warning] Aborted connection 2632129 to db: 'unconnected' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 08:32:49 17027 [Warning] Aborted connection 2635022 to db: 'unconnected' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 09:00:55 17027 [Warning] Aborted connection 2636493 to db: 'unconnected' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 09:46:33 17027 [Warning] Aborted connection 2640126 to db: 'unconnected' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 09:47:33 17027 [Warning] Aborted connection 2635852 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 09:51:44 17027 [Warning] Aborted connection 2639752 to db: 'qausertest' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 10:27:07 17027 [Warning] Aborted connection 2643690 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 10:32:02 17027 [Warning] Aborted connection 2642725 to db: 'qausertest' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 11:03:15 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.107:4567 2017-07-05 11:03:16 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') reconnecting to 451ca788 (tcp://10.1.3.107:4567), attempt 0 2017-07-05 11:03:16 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') connection established to 451ca788 tcp://10.1.3.107:4567 2017-07-05 11:03:19 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-07-05 11:09:51 17027 [Warning] Aborted connection 2646274 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 11:23:57 17027 [Warning] Aborted connection 2647147 to db: 'qausertest' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 11:29:02 17027 [Warning] Aborted connection 2645144 to db: 'qauser' user: 'qauser' host: '10.1.3.107' (Got an error reading communication packets) 2017-07-05 11:51:03 17027 [Warning] Aborted connection 2649131 to db: 'unconnected' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 12:15:48 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.107:4567 2017-07-05 12:15:49 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') reconnecting to 451ca788 (tcp://10.1.3.107:4567), attempt 0 2017-07-05 12:15:49 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') connection established to 451ca788 tcp://10.1.3.107:4567 2017-07-05 12:15:50 17027 [Note] WSREP: (0b83c48f, 'tcp://0.0.0.0:4567') connection established to 451ca788 tcp://10.1.3.107:4567 2017-07-05 12:15:51 17027 [Note] WSREP: declaring aec83d42 at tcp://10.1.3.101:4567 stable 2017-07-05 12:15:51 17027 [Note] WSREP: Node 0b83c48f state prim 2017-07-05 12:15:51 17027 [Warning] WSREP: 0b83c48f sending install message failed: Resource temporarily unavailable 2017-07-05 12:15:51 17027 [Note] WSREP: declaring 451ca788 at tcp://10.1.3.107:4567 stable 2017-07-05 12:15:51 17027 [Note] WSREP: declaring aec83d42 at tcp://10.1.3.101:4567 stable 2017-07-05 12:15:51 17027 [Note] WSREP: Node 0b83c48f state prim 2017-07-05 12:15:51 17027 [Note] WSREP: view(view_id(PRIM,0b83c48f,204) memb { 0b83c48f,0 451ca788,0 aec83d42,0 } joined { } left { } partitioned { }) 2017-07-05 12:15:51 17027 [Note] WSREP: save pc into disk 2017-07-05 12:15:51 17027 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 0, memb_num = 3 2017-07-05 12:15:51 17027 [Note] WSREP: STATE_EXCHANGE: sent state UUID: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 2017-07-05 12:15:51 17027 [Note] WSREP: STATE EXCHANGE: sent state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 2017-07-05 12:15:51 17027 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 0 (MySQLM2.info.in) 2017-07-05 12:15:51 17027 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 1 (MySQLM3.info.in) 2017-07-05 12:15:51 17027 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 2 (MySQLM1.info.in) 2017-07-05 12:15:51 17027 [Note] WSREP: Quorum results: version = 4, component = PRIMARY, conf_id = 115, members = 2/3 (joined/total), act_id = 47076814, last_appl. = 47076759, protocols = 0/7/3 (gcs/repl/appl), group UUID = fed83039-1a7a-11e7-b295-5fd619b022d9 2017-07-05 12:15:51 17027 [Note] WSREP: Flow-control interval: [28, 28] 2017-07-05 12:15:51 17027 [Note] WSREP: New cluster view: global state: fed83039-1a7a-11e7-b295-5fd619b022d9:47076814, view# 116: Primary, number of nodes: 3, my index: 0, protocol version 3 2017-07-05 12:15:51 17027 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2017-07-05 12:15:51 17027 [Note] WSREP: REPL Protocols: 7 (3, 2) 2017-07-05 12:15:51 17027 [Note] WSREP: Assign initial position for certification: 47076814, protocol version: 3 2017-07-05 12:15:51 17027 [Note] WSREP: Service thread queue flushed. 2017-07-05 12:15:51 17027 [ERROR] WSREP: Certification failed for TO isolated action: source: 0b83c48f-4768-11e7-a3f0-a309c3ecfc8c version: 3 local: 1 state: CERTIFYING flags: 65 conn_id: 2651462 trx_id: -1 seqnos (l: 48752470, g: 47076815, s: 47076808, d: -1, ts: 11915704329178894) 2017-07-05 12:15:51 17027 [Note] WSREP: Closing send monitor... 2017-07-05 12:15:51 17027 [Note] WSREP: Closed send monitor. 2017-07-05 12:15:51 17027 [Note] WSREP: gcomm: terminating thread 2017-07-05 12:15:51 17027 [Note] WSREP: gcomm: joining thread 2017-07-05 12:15:51 17027 [Note] WSREP: gcomm: closing backend 2017-07-05 12:15:51 17027 [Note] WSREP: Member 1.0 (MySQLM3.info.in) requested state transfer from '*any*'. Selected 2.0 (MySQLM1.info.in)(SYNCED) as donor. 2017-07-05 12:15:51 17027 [Note] WSREP: view(view_id(NON_PRIM,0b83c48f,204) memb { 0b83c48f,0 } joined { } left { } partitioned { 451ca788,0 aec83d42,0 }) 2017-07-05 12:15:51 17027 [Note] WSREP: view((empty)) 2017-07-05 12:15:51 17027 [Note] WSREP: gcomm: closed 2017-07-05 12:15:51 17027 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:15:51 17027 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:15:51 17027 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:15:51 17027 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 47076815) 2017-07-05 12:15:51 17027 [Note] WSREP: Received self-leave message. 2017-07-05 12:15:51 17027 [Note] WSREP: Flow-control interval: [0, 0] 2017-07-05 12:15:51 17027 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2017-07-05 12:15:51 17027 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 47076815) 2017-07-05 12:15:51 17027 [Note] WSREP: RECV thread exiting 0: Success 2017-07-05 12:15:51 17027 [Note] WSREP: recv_thread() joined. 2017-07-05 12:15:51 17027 [Note] WSREP: Closing replication queue. 2017-07-05 12:15:51 17027 [Note] WSREP: Closing slave action queue. 2017-07-05 12:15:51 17027 [Note] WSREP: /usr/sbin/mysqld: Terminated. Node-3 ====== 2017-07-05 02:35:30 28984 [Warning] Aborted connection 2744508 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 02:47:05 28984 [Warning] Aborted connection 2744616 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 03:51:37 28984 [Warning] Aborted connection 2748819 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 06:01:10 28984 [Warning] Aborted connection 2757265 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 06:02:29 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection to peer aec83d42 with addr tcp://10.1.3.101:4567 timed out, no messages seen in PT3S 2017-07-05 06:02:29 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.101:4567 2017-07-05 06:02:30 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') reconnecting to aec83d42 (tcp://10.1.3.101:4567), attempt 0 2017-07-05 06:02:30 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection established to aec83d42 tcp://10.1.3.101:4567 2017-07-05 06:02:34 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-07-05 06:13:53 28984 [Warning] Aborted connection 2704189 to db: 'qausertest' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 07:06:12 28984 [Warning] Aborted connection 2761515 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 07:23:15 28984 [Warning] Aborted connection 2760552 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 07:47:15 28984 [Warning] Aborted connection 2760473 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 08:04:41 28984 [Warning] Aborted connection 2764885 to db: 'qausertest' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 08:10:38 28984 [Warning] Aborted connection 2765871 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 08:16:45 28984 [Warning] Aborted connection 2765690 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 08:22:07 28984 [Warning] Aborted connection 2766614 to db: 'unconnected' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 09:21:39 28984 [Warning] Aborted connection 2770870 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 09:28:05 28984 [Warning] Aborted connection 2767060 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 09:31:03 28984 [Warning] Aborted connection 2770114 to db: 'mysql' user: 'qauserqa' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 10:15:22 28984 [Warning] Aborted connection 2776050 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 10:56:19 28984 [Warning] Aborted connection 2778087 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 11:03:15 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection to peer 0b83c48f with addr tcp://10.1.3.102:4567 timed out, no messages seen in PT3S 2017-07-05 11:03:15 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.102:4567 2017-07-05 11:03:16 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection established to 0b83c48f tcp://10.1.3.102:4567 2017-07-05 11:03:19 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-07-05 11:30:03 28984 [Warning] Aborted connection 2779291 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 11:31:04 28984 [Warning] Aborted connection 2779915 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 11:32:26 28984 [Warning] Aborted connection 2780464 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 11:57:28 28984 [Warning] Aborted connection 2782242 to db: 'unconnected' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 12:15:47 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection to peer 0b83c48f with addr tcp://10.1.3.102:4567 timed out, no messages seen in PT3S 2017-07-05 12:15:47 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection to peer aec83d42 with addr tcp://10.1.3.101:4567 timed out, no messages seen in PT3S 2017-07-05 12:15:47 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.101:4567 tcp://10.1.3.102:4567 2017-07-05 12:15:48 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') reconnecting to 0b83c48f (tcp://10.1.3.102:4567), attempt 0 2017-07-05 12:15:49 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') reconnecting to aec83d42 (tcp://10.1.3.101:4567), attempt 0 2017-07-05 12:15:49 28984 [Note] WSREP: evs::proto(451ca788, OPERATIONAL, view_id(REG,0b83c48f,202)) suspecting node: 0b83c48f 2017-07-05 12:15:49 28984 [Note] WSREP: evs::proto(451ca788, OPERATIONAL, view_id(REG,0b83c48f,202)) suspected node without join message, declaring inactive 2017-07-05 12:15:49 28984 [Note] WSREP: evs::proto(451ca788, OPERATIONAL, view_id(REG,0b83c48f,202)) suspecting node: aec83d42 2017-07-05 12:15:49 28984 [Note] WSREP: evs::proto(451ca788, OPERATIONAL, view_id(REG,0b83c48f,202)) suspected node without join message, declaring inactive 2017-07-05 12:15:49 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection established to 0b83c48f tcp://10.1.3.102:4567 2017-07-05 12:15:50 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection established to 0b83c48f tcp://10.1.3.102:4567 2017-07-05 12:15:50 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection established to aec83d42 tcp://10.1.3.101:4567 2017-07-05 12:15:50 28984 [Note] WSREP: view(view_id(NON_PRIM,0b83c48f,202) memb { 451ca788,0 } joined { } left { } partitioned { 0b83c48f,0 aec83d42,0 }) 2017-07-05 12:15:50 28984 [Note] WSREP: view(view_id(NON_PRIM,451ca788,203) memb { 451ca788,0 } joined { } left { } partitioned { 0b83c48f,0 aec83d42,0 }) 2017-07-05 12:15:50 28984 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:15:50 28984 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:15:50 28984 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:15:50 28984 [Note] WSREP: Shifting SYNCED -> OPEN (TO: 47076807) 2017-07-05 12:15:50 28984 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:15:50 28984 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:15:50 28984 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:15:50 28984 [Note] WSREP: New cluster view: global state: fed83039-1a7a-11e7-b295-5fd619b022d9:47076807, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 2017-07-05 12:15:50 28984 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2017-07-05 12:15:50 28984 [Note] WSREP: New cluster view: global state: fed83039-1a7a-11e7-b295-5fd619b022d9:47076807, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 2017-07-05 12:15:50 28984 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2017-07-05 12:15:51 28984 [Note] WSREP: declaring 0b83c48f at tcp://10.1.3.102:4567 stable 2017-07-05 12:15:51 28984 [Note] WSREP: declaring aec83d42 at tcp://10.1.3.101:4567 stable 2017-07-05 12:15:51 28984 [Note] WSREP: Node 0b83c48f state prim 2017-07-05 12:15:51 28984 [Note] WSREP: view(view_id(PRIM,0b83c48f,204) memb { 0b83c48f,0 451ca788,0 aec83d42,0 } joined { } left { } partitioned { }) 2017-07-05 12:15:51 28984 [Note] WSREP: save pc into disk 2017-07-05 12:15:51 28984 [Note] WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 1, memb_num = 3 2017-07-05 12:15:51 28984 [Note] WSREP: STATE EXCHANGE: Waiting for state UUID. 2017-07-05 12:15:51 28984 [Note] WSREP: STATE EXCHANGE: sent state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 2017-07-05 12:15:51 28984 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 0 (MySQLM2.info.in) 2017-07-05 12:15:51 28984 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 1 (MySQLM3.info.in) 2017-07-05 12:15:51 28984 [Note] WSREP: STATE EXCHANGE: got state msg: ec3f0f65-616a-11e7-a8cd-3a87d6a4de00 from 2 (MySQLM1.info.in) 2017-07-05 12:15:51 28984 [Note] WSREP: Quorum results: version = 4, component = PRIMARY, conf_id = 115, members = 2/3 (joined/total), act_id = 47076814, last_appl. = 47076759, protocols = 0/7/3 (gcs/repl/appl), group UUID = fed83039-1a7a-11e7-b295-5fd619b022d9 2017-07-05 12:15:51 28984 [Note] WSREP: Flow-control interval: [28, 28] 2017-07-05 12:15:51 28984 [Note] WSREP: Shifting OPEN -> PRIMARY (TO: 47076814) 2017-07-05 12:15:51 28984 [Note] WSREP: State transfer required: Group state: fed83039-1a7a-11e7-b295-5fd619b022d9:47076814 Local state: fed83039-1a7a-11e7-b295-5fd619b022d9:47076807 2017-07-05 12:15:51 28984 [Note] WSREP: New cluster view: global state: fed83039-1a7a-11e7-b295-5fd619b022d9:47076814, view# 116: Primary, number of nodes: 3, my index: 1, protocol version 3 2017-07-05 12:15:51 28984 [Warning] WSREP: Gap in state sequence. Need state transfer. 2017-07-05 12:15:51 28984 [Note] WSREP: You have configured 'xtrabackup-v2' state snapshot transfer method which cannot be performed on a running server. Wsrep provider won't be able to fall back to it if other means of state transfer are unavailable. In that case you will need to restart the server. 2017-07-05 12:15:51 28984 [Note] WSREP: wsrep_notify_cmd is not defined, skipping notification. 2017-07-05 12:15:51 28984 [Note] WSREP: REPL Protocols: 7 (3, 2) 2017-07-05 12:15:51 28984 [Note] WSREP: Assign initial position for certification: 47076814, protocol version: 3 2017-07-05 12:15:51 28984 [Note] WSREP: Service thread queue flushed. 2017-07-05 12:15:51 28984 [Note] WSREP: IST receiver addr using tcp://10.1.3.107:4568 2017-07-05 12:15:51 28984 [Note] WSREP: Prepared IST receiver, listening at: tcp://10.1.3.107:4568 2017-07-05 12:15:51 28984 [Note] WSREP: Member 1.0 (MySQLM3.info.in) requested state transfer from '*any*'. Selected 2.0 (MySQLM1.info.in)(SYNCED) as donor. 2017-07-05 12:15:51 28984 [Note] WSREP: Shifting PRIMARY -> JOINER (TO: 47076815) 2017-07-05 12:15:51 28984 [Note] WSREP: Requesting state transfer: success, donor: 2 2017-07-05 12:15:51 28984 [Note] WSREP: GCache history reset: old(fed83039-1a7a-11e7-b295-5fd619b022d9:47076807) -> new(fed83039-1a7a-11e7-b295-5fd619b022d9:47076814) 2017-07-05 12:15:51 28984 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): discarded 2140077359 bytes 2017-07-05 12:15:51 28984 [Note] WSREP: GCache DEBUG: RingBuffer::seqno_reset(): found 1/2 locked buffers 2017-07-05 12:15:51 28984 [Note] WSREP: declaring aec83d42 at tcp://10.1.3.101:4567 stable 2017-07-05 12:15:51 28984 [Note] WSREP: forgetting 0b83c48f (tcp://10.1.3.102:4567) 2017-07-05 12:15:51 28984 [Note] WSREP: Node 451ca788 state prim 2017-07-05 12:15:51 28984 [Warning] WSREP: 451ca788 sending install message failed: Resource temporarily unavailable 2017-07-05 12:15:51 28984 [Note] WSREP: view(view_id(NON_PRIM,451ca788,205) memb { 451ca788,0 } joined { } left { } partitioned { 0b83c48f,0 aec83d42,0 }) 2017-07-05 12:15:51 28984 [Note] WSREP: forgetting aec83d42 (tcp://10.1.3.101:4567) 2017-07-05 12:15:51 28984 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:15:51 28984 [Note] WSREP: view(view_id(NON_PRIM,451ca788,206) memb { 451ca788,0 } joined { } left { } partitioned { 0b83c48f,0 aec83d42,0 }) 2017-07-05 12:15:51 28984 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:15:51 28984 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:15:51 28984 [Note] WSREP: Shifting JOINER -> OPEN (TO: 47076815) 2017-07-05 12:15:51 28984 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:15:51 28984 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:15:51 28984 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:15:51 28984 [Note] WSREP: Receiving IST: 7 writesets, seqnos 47076807-47076814 2017-07-05 12:15:53 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-07-05 12:15:54 28984 [Warning] Aborted connection 2782392 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 12:15:56 28984 [Warning] Aborted connection 2782451 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 12:15:56 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection established to aec83d42 tcp://10.1.3.101:4567 2017-07-05 12:15:56 28984 [Warning] WSREP: discarding established (time wait) aec83d42 (tcp://10.1.3.101:4567) 2017-07-05 12:15:57 28984 [Note] WSREP: cleaning up aec83d42 (tcp://10.1.3.101:4567) 2017-07-05 12:15:57 28984 [Note] WSREP: cleaning up 0b83c48f (tcp://10.1.3.102:4567) 2017-07-05 12:16:32 28984 [Warning] Aborted connection 2576499 to db: 'qauser' user: 'qauser' host: '10.1.3.101' (Got an error reading communication packets) 2017-07-05 12:39:29 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') connection established to 3979fe73 tcp://10.1.3.101:4567 2017-07-05 12:39:29 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: 2017-07-05 12:39:29 28984 [Note] WSREP: declaring 3979fe73 at tcp://10.1.3.101:4567 stable 2017-07-05 12:39:29 28984 [Note] WSREP: view(view_id(NON_PRIM,3979fe73,207) memb { 3979fe73,0 451ca788,0 } joined { } left { } partitioned { 0b83c48f,0 aec83d42,0 }) 2017-07-05 12:39:29 28984 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 2 2017-07-05 12:39:29 28984 [Note] WSREP: Flow-control interval: [23, 23] 2017-07-05 12:39:29 28984 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:39:32 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting off 2017-07-05 12:40:00 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.1.3.101:4567 2017-07-05 12:40:01 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') reconnecting to 3979fe73 (tcp://10.1.3.101:4567), attempt 0 2017-07-05 12:40:04 28984 [Note] WSREP: evs::proto(451ca788, OPERATIONAL, view_id(REG,3979fe73,207)) suspecting node: 3979fe73 2017-07-05 12:40:04 28984 [Note] WSREP: evs::proto(451ca788, OPERATIONAL, view_id(REG,3979fe73,207)) suspected node without join message, declaring inactive 2017-07-05 12:40:05 28984 [Note] WSREP: view(view_id(NON_PRIM,3979fe73,207) memb { 451ca788,0 } joined { } left { } partitioned { 0b83c48f,0 3979fe73,0 aec83d42,0 }) 2017-07-05 12:40:05 28984 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:40:05 28984 [Note] WSREP: view(view_id(NON_PRIM,451ca788,208) memb { 451ca788,0 } joined { } left { } partitioned { 0b83c48f,0 3979fe73,0 aec83d42,0 }) 2017-07-05 12:40:05 28984 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:40:05 28984 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:40:05 28984 [Note] WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 2017-07-05 12:40:05 28984 [Note] WSREP: Flow-control interval: [16, 16] 2017-07-05 12:40:05 28984 [Note] WSREP: Received NON-PRIMARY. 2017-07-05 12:40:47 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') reconnecting to 3979fe73 (tcp://10.1.3.101:4567), attempt 30 2017-07-05 12:41:32 28984 [Note] WSREP: (451ca788, 'tcp://0.0.0.0:4567') reconnecting to 3979fe73 (tcp://10.1.3.101:4567), attempt 60 2017-07-05 12:42:15 28984 [Note] /usr/sbin/mysqld: Normal shutdown 2017-07-05 12:42:15 28984 [Note] WSREP: Stop replication 2017-07-05 12:42:15 28984 [Note] WSREP: Closing send monitor... 2017-07-05 12:42:15 28984 [Note] WSREP: Closed send monitor. 2017-07-05 12:42:15 28984 [Note] WSREP: gcomm: terminating thread 2017-07-05 12:42:15 28984 [Note] WSREP: gcomm: joining thread 2017-07-05 12:42:15 28984 [Note] WSREP: gcomm: closing backend 2017-07-05 12:42:15 28984 [Note] WSREP: view((empty)) 2017-07-05 12:42:15 28984 [Note] WSREP: gcomm: closed 2017-07-05 12:42:15 28984 [Note] WSREP: Received self-leave message. 2017-07-05 12:42:15 28984 [Note] WSREP: Flow-control interval: [0, 0] 2017-07-05 12:42:15 28984 [Note] WSREP: Received SELF-LEAVE. Closing connection. 2017-07-05 12:42:15 28984 [Note] WSREP: Shifting OPEN -> CLOSED (TO: 47076815) 2017-07-05 12:42:15 28984 [Note] WSREP: RECV thread exiting 0: Success 2017-07-05 12:42:15 28984 [Note] WSREP: recv_thread() joined. 2017-07-05 12:42:15 28984 [Note] WSREP: Closing replication queue. 2017-07-05 12:42:15 28984 [Note] WSREP: Closing slave action queue. 2017-07-05 12:42:17 28984 [Note] WSREP: IST received: fed83039-1a7a-11e7-b295-5fd619b022d9:47076807 2017-07-05 12:42:17 28984 [ERROR] WSREP: Failed to JOIN the cluster after SST 2017-07-05 12:42:17 28984 [ERROR] WSREP: Certification failed for TO isolated action: source: 0b83c48f-4768-11e7-a3f0-a309c3ecfc8c version: 3 local: 0 state: CERTIFYING flags: 65 conn_id: 2651462 trx_id: -1 seqnos (l: 48749969, g: 47076815, s: 47076808, d: -1, ts: 11915704329178894) 2017-07-05 12:42:17 28984 [Note] WSREP: /usr/sbin/mysqld: Terminated.