【Java爬虫】HttpClient快速入门

HttpClient是Java的爬虫库，其使用起来挺方便的，一般步骤可以总计为：

1）实例化HttpClient对象

2）实例化HttpGet对象

3）设置请求头

4）执行get请求拿到数据

5）获取实体

6）解析实体

7)  对解析内容进行处理

8）关闭资源

HttpClient的maven坐标如下

        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpcore</artifactId>
            <version>4.4.10</version>
        </dependency>

        <!-- https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient -->
        <dependency>
            <groupId>org.apache.httpcomponents</groupId>
            <artifactId>httpclient</artifactId>
            <version>4.5.6</version>
        </dependency>

基础的使用代码如下

public class HttpTest {
    public static void main(String[] args) {
        //创建HttpClient对象
        CloseableHttpClient request = HttpClients.createDefault();
        //使用HttpGet请求
        HttpGet httpGet = new HttpGet("https://www.tuicool.com/");
        //伪装一下
        httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36");
        //获得response
        CloseableHttpResponse response = null;
        try {
            response = request.execute(httpGet);

            //解析响应
            if(response.getStatusLine().getStatusCode() == 200){
                String content = EntityUtils.toString(response.getEntity(), "GBK");
                System.out.println(content);
            }


        } catch (IOException e) {
            e.printStackTrace();
        }finally {
            //关闭response
            try {
                response.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
            try {
                request.close();
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }
}

当然我们也可以用爬虫下载图片，我们可以使用commons-io来简化io操作，其maven坐标如下

<!-- https://mvnrepository.com/artifact/commons-io/commons-io -->
        <dependency>
            <groupId>commons-io</groupId>
            <artifactId>commons-io</artifactId>
            <version>2.5</version>
        </dependency>

下载图片的代码如下

public class pictureSpider {
    public static void main(String[] args) throws IOException {
        CloseableHttpClient client = HttpClients.createDefault();
        String url = "https://c-ssl.duitang.com/uploads/item/201504/15/20150415H2548_4vjy3.jpeg";
        HttpGet httpGet = new HttpGet(url);
        httpGet.setHeader("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.122 Safari/537.36");
        CloseableHttpResponse response = client.execute(httpGet);
        HttpEntity entity = response.getEntity();

        //下载图片
        if(entity != null){
            System.out.println("ContentType:" + entity.getContentType().getValue());
            InputStream input = entity.getContent();
            FileUtils.copyToFile(input, new File("pic.jpg"));
        }

        response.close();
        client.close();
    }
}