190716 16:16:28 InnoDB: Assertion failure in thread 140135523792640 in file dict0dict.c line 2697 InnoDB: Failing assertion: UT_LIST_GET_LEN(table->referenced_list) == rbt_size(table->referenced_rbt) InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 14:16:28 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. We will try our best to scrape up some info that will hopefully help diagnose the problem, but since we have already crashed, something is definitely wrong and this may fail. Please help us make Percona XtraDB Cluster better by reporting any bugs at https://bugs.launchpad.net/percona-xtradb-cluster key_buffer_size=16777216 read_buffer_size=262144 max_used_connections=131 max_threads=302 thread_count=46 connection_count=46 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 252174 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7cdaf9c0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f73d81dae48 thread_stack 0x30000 /usr/sbin/mysqld(my_print_stacktrace+0x20)[0x7a3aa0] /usr/sbin/mysqld(handle_fatal_signal+0x36f)[0x692daf] /lib/x86_64-linux-gnu/libpthread.so.0(+0x10340)[0x7f88d97d2340] /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x39)[0x7f88d87efcc9] /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7f88d87f30d8] /usr/sbin/mysqld[0x85b963] /usr/sbin/mysqld[0x86664e] /usr/sbin/mysqld[0x86a3f8] /usr/sbin/mysqld[0x7e4773] /usr/sbin/mysqld[0x7bef5d] /usr/sbin/mysqld(_Z18mysql_rename_tableP10handlertonPKcS2_S2_S2_j+0x117)[0x5f67c7] /usr/sbin/mysqld(_Z17mysql_alter_tableP3THDPcS1_P24st_ha_create_informationP10TABLE_LISTP10Alter_infojP8st_orderb+0x2996)[0x5fb3e6] /usr/sbin/mysqld(_Z20mysql_recreate_tableP3THDP10TABLE_LIST+0x1a1)[0x5fc131] /usr/sbin/mysqld[0x63a6c0] /usr/sbin/mysqld(_ZN24Optimize_table_statement7executeEP3THD+0xcc)[0x63b32c] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THD+0x37e7)[0x599237] /usr/sbin/mysqld(_Z11mysql_parseP3THDPcjP12Parser_state+0x2ab)[0x59d6eb] /usr/sbin/mysqld[0x59df14] /usr/sbin/mysqld(_Z16dispatch_command19enum_server_commandP3THDPcj+0x2125)[0x5a0605] /usr/sbin/mysqld(_Z10do_commandP3THD+0x148)[0x5a0f58] /usr/sbin/mysqld(_Z24do_handle_one_connectionP3THD+0x296)[0x62fd96] /usr/sbin/mysqld(handle_one_connection+0x42)[0x62fe32] /lib/x86_64-linux-gnu/libpthread.so.0(+0x8182)[0x7f88d97ca182] /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7f88d88b347d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f707c944177): is an invalid pointer Connection ID (thread ID): 109109378 Status: NOT_KILLED You may download the Percona XtraDB Cluster operations manual by visiting http://www.percona.com/software/percona-xtradb-cluster/. You may find information in the manual which will help you identify the cause of the crash. 190716 16:16:32 mysqld_safe Number of processes running now: 0 190716 16:16:32 mysqld_safe mysqld restarted 190716 16:16:32 mysqld_safe Skipping wsrep-recover for 4544482d-7860-11e4-b362-7738976f33e0:439360963 pair 190716 16:16:32 mysqld_safe Assigning 4544482d-7860-11e4-b362-7738976f33e0:439360963 to wsrep_start_position 190716 16:16:32 [Warning] Using unique option prefix key_buffer instead of key_buffer_size is deprecated and will be removed in a future release. Please use the full name instead. 190716 16:16:32 [Note] WSREP: wsrep_start_position var submitted: '4544482d-7860-11e4-b362-7738976f33e0:439360963' 190716 16:16:32 [Warning] Using unique option prefix myisam-recover instead of myisam-recover-options is deprecated and will be removed in a future release. Please use the full name instead. 190716 16:16:32 [Note] Plugin 'FEDERATED' is disabled. 190716 16:16:32 InnoDB: The InnoDB memory heap is disabled 190716 16:16:32 InnoDB: Mutexes and rw_locks use GCC atomic builtins 190716 16:16:32 InnoDB: Compressed tables use zlib 1.2.8 190716 16:16:32 InnoDB: Using Linux native AIO 190716 16:16:32 InnoDB: Initializing buffer pool, size = 80.0G 190716 16:16:37 InnoDB: Completed initialization of buffer pool 190716 16:16:37 InnoDB: highest supported file format is Barracuda. InnoDB: Log scan progressed past the checkpoint lsn 34034818642982 190716 16:16:37 InnoDB: Database was not shut down normally! InnoDB: Starting crash recovery. InnoDB: Reading tablespace information from the .ibd files... InnoDB: Restoring possible half-written data pages from the doublewrite InnoDB: buffer... InnoDB: Doing recovery: scanned up to log sequence number 34034823885824 InnoDB: Doing recovery: scanned up to log sequence number 34034829128704 InnoDB: Doing recovery: scanned up to log sequence number 34034834371584 InnoDB: Doing recovery: scanned up to log sequence number 34034839614464 InnoDB: Doing recovery: scanned up to log sequence number 34034844857344 InnoDB: Doing recovery: scanned up to log sequence number 34034850100224 InnoDB: Doing recovery: scanned up to log sequence number 34034855343104 InnoDB: Doing recovery: scanned up to log sequence number 34034860585984 InnoDB: Doing recovery: scanned up to log sequence number 34034865828864 InnoDB: Doing recovery: scanned up to log sequence number 34034871071744 InnoDB: Doing recovery: scanned up to log sequence number 34034876314624 InnoDB: Doing recovery: scanned up to log sequence number 34034881557504 InnoDB: Doing recovery: scanned up to log sequence number 34034886800384 InnoDB: Doing recovery: scanned up to log sequence number 34034892043264 InnoDB: Doing recovery: scanned up to log sequence number 34034897286144 InnoDB: Doing recovery: scanned up to log sequence number 34034902529024 InnoDB: Doing recovery: scanned up to log sequence number 34034907771904 InnoDB: Doing recovery: scanned up to log sequence number 34034913014784 InnoDB: Doing recovery: scanned up to log sequence number 34034918257664 InnoDB: Doing recovery: scanned up to log sequence number 34034923500544 InnoDB: Doing recovery: scanned up to log sequence number 34034928743424 InnoDB: Doing recovery: scanned up to log sequence number 34034933986304 InnoDB: Doing recovery: scanned up to log sequence number 34034939229184 InnoDB: Doing recovery: scanned up to log sequence number 34034944472064 InnoDB: Doing recovery: scanned up to log sequence number 34034949714944 InnoDB: Doing recovery: scanned up to log sequence number 34034954957824 InnoDB: Doing recovery: scanned up to log sequence number 34034960200704 InnoDB: Doing recovery: scanned up to log sequence number 34034965443584 InnoDB: Doing recovery: scanned up to log sequence number 34034970686464 InnoDB: Doing recovery: scanned up to log sequence number 34034975929344 InnoDB: Doing recovery: scanned up to log sequence number 34034981172224 InnoDB: Doing recovery: scanned up to log sequence number 34034986415104 InnoDB: Doing recovery: scanned up to log sequence number 34034991657984 InnoDB: Doing recovery: scanned up to log sequence number 34034996900864 InnoDB: Doing recovery: scanned up to log sequence number 34035002143744 InnoDB: Doing recovery: scanned up to log sequence number 34035007386624 InnoDB: Doing recovery: scanned up to log sequence number 34035012629504 InnoDB: Doing recovery: scanned up to log sequence number 34035017872384 InnoDB: Doing recovery: scanned up to log sequence number 34035023115264 InnoDB: Doing recovery: scanned up to log sequence number 34035028358144 InnoDB: Doing recovery: scanned up to log sequence number 34035033601024 InnoDB: Doing recovery: scanned up to log sequence number 34035038843904 InnoDB: Doing recovery: scanned up to log sequence number 34035044086784 InnoDB: Doing recovery: scanned up to log sequence number 34035049329664 InnoDB: Doing recovery: scanned up to log sequence number 34035054572544 InnoDB: Doing recovery: scanned up to log sequence number 34035059815424 InnoDB: Doing recovery: scanned up to log sequence number 34035065058304 InnoDB: Doing recovery: scanned up to log sequence number 34035070301184 InnoDB: Doing recovery: scanned up to log sequence number 34035075544064 InnoDB: Doing recovery: scanned up to log sequence number 34035080786944 InnoDB: Doing recovery: scanned up to log sequence number 34035086029824 InnoDB: Doing recovery: scanned up to log sequence number 34035091272704 InnoDB: Doing recovery: scanned up to log sequence number 34035096515584 InnoDB: Doing recovery: scanned up to log sequence number 34035101758464 InnoDB: Doing recovery: scanned up to log sequence number 34035107001344 InnoDB: Doing recovery: scanned up to log sequence number 34035112244224 InnoDB: Doing recovery: scanned up to log sequence number 34035117487104 InnoDB: Doing recovery: scanned up to log sequence number 34035122729984 InnoDB: Doing recovery: scanned up to log sequence number 34035127972864 InnoDB: Doing recovery: scanned up to log sequence number 34035133215744 InnoDB: Doing recovery: scanned up to log sequence number 34035138458624 InnoDB: Doing recovery: scanned up to log sequence number 34035143701504 InnoDB: Doing recovery: scanned up to log sequence number 34035148944384 InnoDB: Doing recovery: scanned up to log sequence number 34035154187264 InnoDB: Doing recovery: scanned up to log sequence number 34035159430144 InnoDB: Doing recovery: scanned up to log sequence number 34035164673024 InnoDB: Doing recovery: scanned up to log sequence number 34035169915904 InnoDB: Doing recovery: scanned up to log sequence number 34035175158784 InnoDB: Doing recovery: scanned up to log sequence number 34035180401664 InnoDB: Doing recovery: scanned up to log sequence number 34035185644544 InnoDB: Doing recovery: scanned up to log sequence number 34035185955609 InnoDB: 1 transaction(s) which must be rolled back or cleaned up InnoDB: in total 15 row operations to undo InnoDB: Trx id counter is 553250800 190716 16:16:47 InnoDB: Starting an apply batch of log records to the database... InnoDB: Progress in percents: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 InnoDB: Apply batch completed InnoDB: In a MySQL replication slave the last master binlog file InnoDB: position 16627982, file name mysql-bin.003831 InnoDB: and relay log file InnoDB: position 692444, file name /var/lib/mysql/binlog/mysqld-relay-bin.000069 InnoDB: Last MySQL binlog file position 0 0, file name /var/lib/mysql/binlog/mysql-bin.032770 190716 16:17:05 InnoDB: Error: table 'dbname/#sql-70b6_680e082' InnoDB: in InnoDB data dictionary has tablespace id 63476110, InnoDB: but the tablespace with that id has name ./dbname/tablename.ibd. InnoDB: Have you deleted or moved .ibd files? InnoDB: Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting-datadict.html InnoDB: for how to resolve the issue. InnoDB: We removed now the InnoDB internal data dictionary entry InnoDB: of table "tmp"."#sql70b6_680e082_0". InnoDB: We removed now the InnoDB internal data dictionary entry InnoDB: of table "tmp"."#sql70b6_6821c28_a8". InnoDB: Starting in background the rollback of uncommitted transactions 190716 16:17:05 InnoDB: Rolling back trx with id 553250680, 15 rows to undo 190716 16:17:05 InnoDB: Waiting for the background threads to start InnoDB: Rolling back of trx id 553250680 completed 190716 16:17:05 InnoDB: Rollback of non-prepared transactions completed 190716 16:17:06 Percona XtraDB (http://www.percona.com) 5.5.39-36.0 started; log sequence number 34035185955609 190716 16:17:06 [Note] Recovering after a crash using /var/lib/mysql/binlog/mysql-bin 190716 16:17:06 [Note] Starting crash recovery... 190716 16:17:06 [Note] Crash recovery finished. 190716 16:17:06 InnoDB: Error: table "tmp"."#sql70b6_6821c28_a8" does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html 190716 16:17:06 InnoDB: Error: table "tmp"."#sql70b6_680e082_0" does not exist in the InnoDB internal InnoDB: data dictionary though MySQL is trying to drop it. InnoDB: Have you copied the .frm file of the table to the InnoDB: MySQL database directory from another database? InnoDB: You can look for further help from InnoDB: http://dev.mysql.com/doc/refman/5.5/en/innodb-troubleshooting.html 190716 16:17:06 [Note] Event Scheduler: Loaded 0 events 190716 16:17:06 [Note] WSREP: Initial position: 4544482d-7860-11e4-b362-7738976f33e0:439360963 190716 16:17:06 [Note] WSREP: wsrep_load(): loading provider library 'none' 190716 16:17:06 [Note] /usr/sbin/mysqld: ready for connections. Version: '5.5.39-36.0-55-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona XtraDB Cluster (GPL), Release rel36.0, Revision 816, WSREP version 25.11, wsrep_25.11.r4023