webmagic爬虫自学（二）遇到问题收集

版权声明：本文为博主原创文章，请尊重原创，未经博主允许禁止转载，保留追究权 https://blog.csdn.net/qq_29914837/article/details/88936084

一、webMagic-0.7.3出现javax.net.ssl.SSLException: Received fatal alert: protocol_version错误

javax.net.ssl.SSLException: Received fatal alert: protocol_version
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
	at sun.security.ssl.Alerts.getSSLException(Alerts.java:154)
	at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023)
	at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)

原因：WebMagic-0.7.3版本默认的HttpClient只会用TLSv1去请求，对于某些只支持TLS1.2的站点，就会报错。

解决方案：
一、目前版本的webmagic已解决此问题，引用新版本即可

二、

1、首先复制源码中的 HttpClientGenerator 与 HttpClientDownloader 到自己的项目中。

2、修改 HttpClientGenerator 类代码中的buildSSLConnectionSocketFactory 方法，该为如下即可

private SSLConnectionSocketFactory buildSSLConnectionSocketFactory() {
	try {
		return new SSLConnectionSocketFactory(createIgnoreVerifySSL(), new String[]{"SSLv3", "TLSv1", "TLSv1.1", "TLSv1.2"},
                null,
                new DefaultHostnameVerifier()); // 优先绕过安全证书
	} catch (KeyManagementException e) {
        logger.error("ssl connection fail", e);
    } catch (NoSuchAlgorithmException e) {
        logger.error("ssl connection fail", e);
    }
	return SSLConnectionSocketFactory.getSocketFactory();
}

3、修改 HttpClientDownloader 中引用的 HttpClientGenerator 为你修改后的类。
4、设置爬虫 Spider 的 Downloader 为 你修改的 HttpClientDownloader如下：

Spider.create(new GithubRepoPageProcessor()).setDownloader(new HttpClientDownloader()).addUrl("https://github.com/code4craft").thread(5).run();

二、爬出网页超长文本数据：Cause: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column ‘content’ at row 1

### Error updating database.  Cause: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'content' at row 1
### The error may involve demo.blog.csdn.net.dao.CsdnDAO.add-Inline
### The error occurred while setting parameters
### SQL: insert into csdn (`article`,`time`,`nick_name`,`read_count`,`label`,`category`,`content`,`url`,`collect_time`) values (?,?,?,?,?,?,?,?,?)
### Cause: com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'content' at row 1
; SQL []; Data truncation: Data too long for column 'content' at row 1; nested exception is com.mysql.jdbc.MysqlDataTruncation: Data truncation: Data too long for column 'content' at row 1
	at org.springframework.jdbc.support.SQLStateSQLExceptionTranslator.doTranslate(SQLStateSQLExceptionTranslator.java:101)

解决：字段长度不够，即使设置为了text还是不够，因为text类型是可变长度的字符串，最多65535个字符，所有，最好是把字段类型设置为longtext，最多存放4294967295个字符