早上8点种线上服务器大量报出以下异常
java.sql.SQLTransientConnectionException: MyHikariCP - Connection is not available, request timed out after 30000ms
show processlist后发现大量线程处于Wating For table flush状态
状态分析
官方文档对Waiting for table flush状态的解释
The thread is executing FLUSH TABLES and is waiting for all threads to close their tables, or the thread got a notification that the underlying structure for a table has changed and it needs to reopen the table to get the new structure. However, to reopen the table, it must wait until all other threads have closed the table in question.
This notification takes place if another thread has used FLUSH TABLES or one of the following statements on the table in question: FLUSH TABLES tbl_name, ALTER TABLE, RENAME TABLE, REPAIR TABLE, ANALYZE TABLE, or OPTIMIZE TABLE
以下场景会导致线程处于Waiting for table flush状态
1 线程执行 FLUSH TABLES，等待其他线程关闭相应flush的表。
2 其他线程执行如 FLUSH TABLES tbl_name, ALTER TABLE..等操作，该线程也需要等相应的表关闭。
模拟场景
| Time | session1 | session2 | Session3 |
|---|---|---|---|
| T1 | select *, sleep(1000) from t1; | ||
| T2 | flush table t1; | ||
| T3 | select * from t1; |
session1在t1时间查询表t1,这里模拟一个慢查询。
session2在t2时间flush table t1
session3在t3时间查询表t1.
show processlist看下现在三个线程状态
可以看到由于慢查询导致t1表没关闭，后面的flush table等待，然后导致所有查询t1表的线程都会处于Wating for table flush状态。
问题回顾
经过和DBA复盘整个流程，DBA定时执行Dump备份，而Dump命令会导致flush tables,由于这个时候慢sql还没有执行结束，所以导致flush table等待，从而导致后续sql也处于waiting for table flush.
解决方案
kill掉慢查询sql,如示例显示就是id = 2的线程。
kill掉后flush table完成，后续sql也执行正常。
事后重点处理慢sql的优化。
