Oct 27 15:23:37 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 6569ade9 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:37 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') turning message relay requesting on, nonlive peers: tcp://10.30.30.12:4567 Oct 27 15:23:39 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') reconnecting to 6569ade9 (tcp://10.30.30.12:4567), attempt 0 Oct 27 15:23:39 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 8cc5e0ca with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:40 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') reconnecting to 8cc5e0ca (tcp://10.30.30.11:4567), attempt 0 Oct 27 15:23:42 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:43 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:46 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:46 db1.p10clt mysqld[2339]: WSREP: declaring node with index 0 suspected, timeout PT10S (evs.suspect_timeout) Oct 27 15:23:46 db1.p10clt mysqld[2339]: WSREP: declaring node with index 3 suspected, timeout PT10S (evs.suspect_timeout) Oct 27 15:23:46 db1.p10clt mysqld[2339]: WSREP: evs::proto(950b8795, GATHER, view_id(REG,6569ade9,646)) suspecting node: 6569ade9 Oct 27 15:23:46 db1.p10clt mysqld[2339]: WSREP: evs::proto(950b8795, GATHER, view_id(REG,6569ade9,646)) suspected node without join message, declaring inactive Oct 27 15:23:46 db1.p10clt mysqld[2339]: WSREP: evs::proto(950b8795, GATHER, view_id(REG,6569ade9,646)) suspecting node: 8cc5e0ca Oct 27 15:23:46 db1.p10clt mysqld[2339]: WSREP: evs::proto(950b8795, GATHER, view_id(REG,6569ade9,646)) suspected node without join message, declaring inactive Oct 27 15:23:47 db1.p10clt mysqld[2339]: WSREP: declaring node with index 0 inactive (evs.inactive_timeout) Oct 27 15:23:47 db1.p10clt mysqld[2339]: WSREP: declaring node with index 3 inactive (evs.inactive_timeout) Oct 27 15:23:47 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:50 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:51 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:54 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:56 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:23:58 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:24:00 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: evs::proto(950b8795, GATHER, view_id(REG,6569ade9,646)) install timer expired Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: no install message received Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Current view of cluster as seen by this node view (view_id(NON_PRIM,6569ade9,646) memb {     950b8795,2     } joined {     } left {     } partitioned {     6569ade9,1     677ffd76,2     70217988,3     8cc5e0ca,1     } ) Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Flow-control interval: [128, 128] Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Trying to continue unpaused monitor Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Received NON-PRIMARY. Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Current view of cluster as seen by this node view (view_id(NON_PRIM,950b8795,647) memb {     950b8795,2     } joined {     } left {     } partitioned {     6569ade9,1     677ffd76,2     70217988,3     8cc5e0ca,1     } ) Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Shifting SYNCED -> OPEN (TO: 227126141) Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: New cluster view: global state: dc431075-80ab-11e4-8343-d3d0bd92519f:227126141, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Setting wsrep_ready to false Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: wsrep_notify_cmd is not defined, skipping notification. Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 0, memb_num = 1 Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Flow-control interval: [128, 128] Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Trying to continue unpaused monitor Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Received NON-PRIMARY. Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: New cluster view: global state: dc431075-80ab-11e4-8343-d3d0bd92519f:227126141, view# -1: non-Primary, number of nodes: 1, my index: 0, protocol version 3 Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Setting wsrep_ready to false Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: wsrep_notify_cmd is not defined, skipping notification. Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: declaring 677ffd76 at tcp://10.30.40.12:4567 stable Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Current view of cluster as seen by this node view (view_id(NON_PRIM,677ffd76,648) memb {     677ffd76,2     950b8795,2     } joined {     } left {     } partitioned {     6569ade9,1     70217988,3     8cc5e0ca,1     } ) Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 1, memb_num = 2 Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Flow-control interval: [181, 181] Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Trying to continue unpaused monitor Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Received NON-PRIMARY. Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: New cluster view: global state: dc431075-80ab-11e4-8343-d3d0bd92519f:227126141, view# -1: non-Primary, number of nodes: 2, my index: 1, protocol version 3 Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: Setting wsrep_ready to false Oct 27 15:24:01 db1.p10clt mysqld[2339]: WSREP: wsrep_notify_cmd is not defined, skipping notification. Oct 27 15:24:02 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:24:05 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: declaring 677ffd76 at tcp://10.30.40.12:4567 stable Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: declaring 70217988 at tcp://10.30.50.11:4567 stable Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: Current view of cluster as seen by this node view (view_id(NON_PRIM,677ffd76,649) memb {     677ffd76,2     70217988,3     950b8795,2     } joined {     } left {     } partitioned {     6569ade9,1     8cc5e0ca,1     } ) Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: New COMPONENT: primary = no, bootstrap = no, my_idx = 2, memb_num = 3 Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: Flow-control interval: [222, 222] Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: Trying to continue unpaused monitor Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: Received NON-PRIMARY. Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: New cluster view: global state: dc431075-80ab-11e4-8343-d3d0bd92519f:227126141, view# -1: non-Primary, number of nodes: 3, my index: 2, protocol version 3 Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: Setting wsrep_ready to false Oct 27 15:24:07 db1.p10clt mysqld[2339]: WSREP: wsrep_notify_cmd is not defined, skipping notification. Oct 27 15:24:09 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) ... Oct 27 15:32:29 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.11:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:32:29 db1.p10clt mysqld[2339]: WSREP: (950b8795, 'tcp://0.0.0.0:4567') connection to peer 00000000 with addr tcp://10.30.30.12:4567 timed out, no messages seen in PT3S (gmcast.peer_timeout) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: declaring 6569ade9 at tcp://10.30.30.12:4567 stable Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: declaring 677ffd76 at tcp://10.30.40.12:4567 stable Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: declaring 70217988 at tcp://10.30.50.11:4567 stable Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: declaring 8cc5e0ca at tcp://10.30.30.11:4567 stable Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: re-bootstrapping prim from partitioned components Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Current view of cluster as seen by this node view (view_id(PRIM,6569ade9,650) memb {     6569ade9,1     677ffd76,2     70217988,3     8cc5e0ca,1     950b8795,2     } joined {     } left {     } partitioned {     } ) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Save the discovered primary-component to disk Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: New COMPONENT: primary = yes, bootstrap = no, my_idx = 4, memb_num = 5 Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: STATE EXCHANGE: Waiting for state UUID. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: STATE EXCHANGE: sent state msg: 0c646fea-da1f-11e8-87be-260754eede86 Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: STATE EXCHANGE: got state msg: 0c646fea-da1f-11e8-87be-260754eede86 from 0 (db2.p10jax) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: STATE EXCHANGE: got state msg: 0c646fea-da1f-11e8-87be-260754eede86 from 1 (db2.p10clt) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: STATE EXCHANGE: got state msg: 0c646fea-da1f-11e8-87be-260754eede86 from 3 (db1.p10jax) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: STATE EXCHANGE: got state msg: 0c646fea-da1f-11e8-87be-260754eede86 from 4 (db1.p10clt) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: STATE EXCHANGE: got state msg: 0c646fea-da1f-11e8-87be-260754eede86 from 2 (garb) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Quorum: No node with complete state: Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Partial re-merge of primary e62c9d7d-da18-11e8-a3eb-d2da62d5c26d found: 4 of 5. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Quorum results:     version    = 4,     component  = PRIMARY,     conf_id    = 590,     members    = 4/5 (primary/total),     act_id     = 227126141,     last_appl. = 227126118,     protocols  = 0/9/3 (gcs/repl/appl),     group UUID = dc431075-80ab-11e4-8343-d3d0bd92519f Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Flow-control interval: [256, 256] Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Trying to continue unpaused monitor Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Restored state OPEN -> SYNCED (227126141) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: REPL Protocols: 9 (4, 2) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: New cluster view: global state: dc431075-80ab-11e4-8343-d3d0bd92519f:227126141, view# 591: Primary, number of nodes: 5, my index: 4, protocol version 3 Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Setting wsrep_ready to true Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Auto Increment Offset/Increment re-align with cluster membership change (Offset: 5 -> 5) (Increment: 5 -> 5) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: wsrep_notify_cmd is not defined, skipping notification. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Assign initial position for certification: 227126141, protocol version: 4 Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Service thread queue flushed. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Synchronized with group, ready for connections Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Setting wsrep_ready to true Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: wsrep_notify_cmd is not defined, skipping notification. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Member 2.3 (garb) requested state transfer from '*any*'. Selected 4.2 (db1.p10clt)(SYNCED) as donor. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Shifting SYNCED -> DONOR/DESYNCED (TO: 227126141) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: 2.3 (garb): State transfer from 4.2 (db1.p10clt) complete. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: 4.2 (db1.p10clt): State transfer to 2.3 (garb) complete. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Shifting DONOR/DESYNCED -> JOINED (TO: 227126141) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Member 2.3 (garb) synced with group. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Member 4.2 (db1.p10clt) synced with group. Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Shifting JOINED -> SYNCED (TO: 227126141) Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Synchronized with group, ready for connections Oct 27 15:32:31 db1.p10clt mysqld[2339]: WSREP: Setting wsrep_ready to true