Communications link failure错误分析

1.异常描述

Java代码  

1. Exception:  
2. com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure  
3.   
4. The last packet successfully received from the server was 7 milliseconds ago.  The last packet sent successfully to the server was 1,023,250 milliseconds ago.  
5.     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)  
6.     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)  
7.     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)  
8.     at java.lang.reflect.Constructor.newInstance(Constructor.java:513)  
9.     at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)  
10.     at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1116)  
11.     at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3589)  
12.     at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3478)  
13.     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4019)  
14.     at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:951)  
15.     at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1929)  
16.     at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:414)  
17.     at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:393)  
18.     at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:6883)  
19.     at com.seven.migrate.MySQLJdbcTest.main(MySQLJdbcTest.java:58)  
20. Caused by: java.io.EOFException: Can not read response from server. Expected to read 18 bytes, read 14 bytes before connection was unexpectedly lost.  
21.     at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3039)  
22.     at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3545)  
23.     ... 8 more  

 2. 问题分析：

 

从网上分析得知，此问题一般是MySQL端数据发送timeout所致。MySQL中可以设置net_write_timeout这个变量，来调整IO写的时候的超时时间。

 

connect_timeout

The number of seconds that the mysqld server waits for a connect packet before responding with Bad handshake.

interactive_timeout

The number of seconds the server waits for activity on an interactive connection before closing it.

wait_timeout

The number of seconds the server waits for activity on a noninteractive connection before closing it.

net_read_timeout

The number of seconds to wait for more data from a connection before aborting the read.

net_write_timeout

The number of seconds to wait for a block to be written to a connection before aborting the write.

 

由此可知，当客户端长时间不接收数据的时候，就会断开连接。接下来我们尝试在MySQL这边设置net_write_timeout,但发现还是无效，百思不得其解。

查看jdbc驱动发现，当调用PreparedStatement的executeQuery（）方法的时候，如果我们是去获取流式resultset的话，就会默认执行SET net_write_timeout= ？ 这个命令去重新设置timeout时间。源代码如下：

 

Java代码  

1. if (doStreaming && this.connection.getNetTimeoutForStreamingResults() > 0) {  
2.                   
3.                 java.sql.Statement stmt = null;  
4.                   
5.                 try {  
6.                     stmt = this.connection.createStatement();  
7.                       
8.                     ((com.mysql.jdbc.StatementImpl)stmt).executeSimpleNonQuery(this.connection, "SET net_write_timeout="   
9.                             + this.connection.getNetTimeoutForStreamingResults());  
10.                 } finally {  
11.                     if (stmt != null) {  
12.                         stmt.close();  
13.                     }  
14.                 }  
15.             }  

 因此我们需要为MySQL驱动的connection这个类设置NetTimeoutForStreamingResults。当不设置NetTimeoutForStreamingResults这个变量的时候，默认是600秒。

 

接下来查看MySQL的源代码，发现最后影响的是sockopt的SO_SNDTIMEO。

 

C代码  

1. void vio_timeout(Vio *vio, uint which, uint timeout)  
2. {  
3.   r= setsockopt(vio->sd, SOL_SOCKET, which ? SO_SNDTIMEO : SO_RCVTIMEO,  
4.                 IF_WIN(const char*, const void*)&wait_timeout,  
5.                 sizeof(wait_timeout));  
6. }  

 在msdn上有如下解释

 

setsockopt(SO_RCVTIMEO) "If a send or receive operation times out on a 
socket, the socket state is indeterminate, and should not be used; TCP 
sockets in this state have a potential for data loss, since the 
operation could be canceled at the same moment the operation was to be 
completed." 

 

3.问题重现

在类ReadAheadInputStream中的fill方法中的this.underlyingStream.read处设置一个断点。停留长于NetTimeoutForStreamingResults的时间，这个问题就会重现了。MySQL驱动最终用来读取网络数据的对象是underlyingStream。

 

 

Java代码  

1. private void fill(int readAtLeastTheseManyBytes) throws IOException {  
2.         checkClosed();  
3.   
4.         this.currentPosition = 0; /* no mark: throw away the buffer */  
5.   
6.         this.endOfCurrentData = currentPosition;  
7.   
8.         // Read at least as many bytes as the caller wants, but don't  
9.         // block to fill the whole buffer (like java.io.BufferdInputStream  
10.         // does)  
11.   
12.         int bytesToRead = Math.min(this.buf.length - currentPosition,  
13.                 readAtLeastTheseManyBytes);  
14.   
15.         int bytesAvailable = this.underlyingStream.available();  
16.   
17.         if (bytesAvailable > bytesToRead) {  
18.   
19.             // Great, there's more available, let's grab those  
20.             // bytes too! (read-ahead)  
21.   
22.             bytesToRead = Math.min(this.buf.length - currentPosition,  
23.                     bytesAvailable);  
24.         }  
25.   
26.         if (this.doDebug) {  
27.             StringBuffer debugBuf = new StringBuffer();  
28.             debugBuf.append("  ReadAheadInputStream.fill(");  
29.             debugBuf.append(readAtLeastTheseManyBytes);  
30.             debugBuf.append("), buffer_size=");  
31.             debugBuf.append(this.buf.length);  
32.             debugBuf.append(", current_position=");  
33.             debugBuf.append(currentPosition);  
34.             debugBuf.append(", need to read ");  
35.             debugBuf.append(Math.min(this.buf.length - currentPosition,  
36.                     readAtLeastTheseManyBytes));  
37.             debugBuf.append(" bytes to fill request,");  
38.   
39.             if (bytesAvailable > 0) {  
40.                 debugBuf.append(" underlying InputStream reports ");  
41.                 debugBuf.append(bytesAvailable);  
42.   
43.                 debugBuf.append(" total bytes available,");  
44.             }  
45.   
46.             debugBuf.append(" attempting to read ");  
47.             debugBuf.append(bytesToRead);  
48.             debugBuf.append(" bytes.");  
49.   
50.             if (this.log != null) {  
51.                 this.log.logTrace(debugBuf.toString());  
52.             } else {  
53.                 System.err.println(debugBuf.toString());  
54.             }  
55.         }  
56.   
57.         int n = this.underlyingStream.read(this.buf, currentPosition,  
58.                 bytesToRead);  
59.   
60.         if (n > 0) {  
61.             endOfCurrentData = n + currentPosition;  
62.         }  
63.     }  

 

http://frankfan915.iteye.com/blog/1672465