正则表达式——抓取网页中的http及网络图片

**

正则表达式——抓取网页中的http及网络图片

**
最近网上有些朋友问我怎么用JAVA抓起网页中的image地址，我现在就给大家写一个实例，希望大家能通过这个实例学习到一点东西

需要抓取网页中http链接地址首先还需准备一个网站地址，我们就以hao123为例
https://www.hao123.com/

在这个网页中，我们要抓取这里面的href地址首先需要解析这个网站中的内容，把它转换成String字符串

 @Test
    public void parsing_URL() {
        System.out.println(getHTMLText("https://www.hao123.com/","utf-8"));
    }

    /**
     * 获取网页文本
     * */
    public String getHTMLText(String htmlUrl,String charset){

        StringBuilder sb = new StringBuilder();
        try {
            URL url = new URL(htmlUrl);
            BufferedReader br = new BufferedReader(new InputStreamReader(url.openStream(),charset));
            String result = "";
            while ((result = br.readLine()) != null){
                sb.append(result).append("\n");
            }
        } catch (MalformedURLException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        }

        return sb.toString();
    }

打印结果如下

然后我们在通过正则表达式来解析网站中存在的href对应的地址

 @Test
    public void parsing_URL() {
        String content = getHTMLText("https://www.hao123.com/","gbk");
        String regular = "href=\"([\\w\\s./:]+?)\"";
        List<String> list = regularParsingResult(content,regular);
        for (String str:list) {
            System.out.println(str);
        }
    }


    /**
     * 正则解析结果
     */
    public List<String> regularParsingResult(String content,String regular){
        Pattern pattern = Pattern.compile(regular);
        Matcher matcher = pattern.matcher(content);
        List<String>  list = new ArrayList<>();
        while (matcher.find()){
            list.add(matcher.group(1));
        }
        return list;
    }

打印结果如下

现在我们来获取一些网页中的img标签下的图片地址

@Test
    public void parsing_URL() {
        String content = getHTMLText("https://www.hao123.com/","utf-8");
        String regular = "<img[\\s\\S]+?src=\"(.+?)\"";
        List<String> list = regularParsingResult(content,regular);
        for (String str:list) {
            System.out.println(str);
        }
    }



    /**
     * 正则解析结果
     */
    public List<String> regularParsingResult(String content,String regular){
        Pattern pattern = Pattern.compile(regular);
        Matcher matcher = pattern.matcher(content);
        List<String>  list = new ArrayList<>();
        while (matcher.find()){
            list.add(matcher.group(1));
        }
        return list;
    }

打印结果

获取网页完整代码

public void getHTMLTextComplete(String htmlUrl) throws InterruptedException {
        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_10);
        //设置webClient的相关参数
        webClient.getOptions().setJavaScriptEnabled(true);
        webClient.getOptions().setCssEnabled(false);
        webClient.setAjaxController(new NicelyResynchronizingAjaxController());
        //webClient.getOptions().setTimeout(50000);
        webClient.getOptions().setThrowExceptionOnScriptError(false);
        //模拟浏览器打开一个目标网址
        HtmlPage rootPage = webClient.getPage(htmlUrl);
        System.out.println("为了获取js执行的数据 线程开始沉睡等待");
        Thread.sleep(3000);//主要是这个线程的等待 因为js加载也是需要时间的
        System.out.println("线程结束沉睡");
        String html = rootPage.asText();
        System.out.println(html);
    }