docker- vibrato web-side data "docker combat chapter" python crawl (19)

Original article, welcome to reprint. Reproduced please specify: Reprinted from IT people story , thank you!
Original link address: docker- vibrato web-side data "docker combat chapter" python crawl (19)

Vibrato grab combat, why not grab data? For example: There is a fresh electricity providers Internet company, the boss would like to put on some traffic advertising, increasing the company's products by way of exposure, marketing, put on the choice he discovered the vibrato, vibrato has a very large data flow, try to run ads on the vibrato, to see if there are profits and income effects. They analyzed data vibrato, the vibrato portrait of analyzing user, user groups and companies to determine the degree of matching, take vibrato fans, like the number of points, concerned about the number, nickname. User preferences by the company's products into the video, to better promote the company's products. Some public relations firms can find these data network red horse, marketing packaging. Source: https://github.com/limingios/dockerpython.git (douyin)

Vibrato Share page

Introduction

https://www.douyin.com/share/user/ user ID, user ID in the source code through txt acquired, and then by way of a link can open web pages corresponding end. By then end of the web page. Crawling basic information .

docker- vibrato web-side data "docker combat chapter" python crawl (19)

Install Google xpath helper tool

Source acquire crx

docker- vibrato web-side data "docker combat chapter" python crawl (19)

Google browser enter: chrome: // extensions /

docker- vibrato web-side data "docker combat chapter" python crawl (19)

Xpath-helper.crx directly dragged into the interface chrome: // extensions /

After a successful installation

docker- vibrato web-side data "docker combat chapter" python crawl (19)

Shortcut key ctrl + shift + x start xpath, usually Google's developer tools used in conjunction with f12.

docker- vibrato web-side data "docker combat chapter" python crawl (19)

Python crawling vibrato start sharing website data

Share Analysis page https://www.douyin.com/share/user/76055758243

1.抖音做了反派机制，抖音ID中的数字变成了字符串，进行替换。

{'name':[' &#xe603; ',' &#xe60d; ',' &#xe616; '],'value':0},
        {'name':[' &#xe602; ',' &#xe60e; ',' &#xe618; '],'value':1},
        {'name':[' &#xe605; ',' &#xe610; ',' &#xe617; '],'value':2},
        {'name':[' &#xe604; ',' &#xe611; ',' &#xe61a; '],'value':3},
        {'name':[' &#xe606; ',' &#xe60c; ',' &#xe619; '],'value':4},
        {'name':[' &#xe607; ',' &#xe60f; ',' &#xe61b; '],'value':5},
        {'name':[' &#xe608; ',' &#xe612; ',' &#xe61f; '],'value':6},
        {'name':[' &#xe60a; ',' &#xe613; ',' &#xe61c; '],'value':7},
        {'name':[' &#xe60b; ',' &#xe614; ',' &#xe61d; '],'value':8},
        {'name':[' &#xe609; ',' &#xe615; ',' &#xe61e; '],'value':9},

docker- vibrato web-side data "docker combat chapter" python crawl (19)

2.获取需要的节点的的xpath

# 昵称
//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()

#抖音ID
//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()

#工作
//div[@class='personal-card']/div[@class='info2']/div[@class='verify-info']/span[@class='info']/text()

#描述
//div[@class='personal-card']/div[@class='info2']/p[@class='signature']/text()

#地址
//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[1]/text()

#星座
//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[2]/text()

#关注数
//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='focus block']//i[@class='icon iconfont follow-num']/text()

#粉丝数
//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']//i[@class='icon iconfont follow-num']/text()

#赞数
//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']/span[@class='num']/text()

docker- vibrato web-side data "docker combat chapter" python crawl (19)

完整代码


import re
import requests
import time
from lxml import etree

def handle_decode(input_data,share_web_url,task):
search_douyin_str = re.compile(r'抖音ID：')
regex_list = [
{'name':['  ','  ','  '],'value':0},
{'name':['  ','  ','  '],'value':1},
{'name':['  ','  ','  '],'value':2},
{'name':['  ','  ','  '],'value':3},
{'name':['  ','  ','  '],'value':4},
{'name':['  ','  ','  '],'value':5},
{'name':['  ','  ','  '],'value':6},
{'name':['  ','  ','  '],'value':7},
{'name':['  ','  ','  '],'value':8},
{'name':['  ','  ','  '],'value':9},
]

for i1 in regex_list:
    for i2 in i1['name']:
        input_data = re.sub(i2,str(i1['value']),input_data)
share_web_html = etree.HTML(input_data)
douyin_info = {}
douyin_info['nick_name'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()")[0]
douyin_id = ''.join(share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']/p[@class='shortid']/i/text()"))
douyin_info['douyin_id'] = re.sub(search_douyin_str,'',share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()")[0]).strip() + douyin_id

try:
    douyin_info['job'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/div[@class='verify-info']/span[@class='info']/text()")[0].strip()
except:
    pass
douyin_info['describe'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='signature']/text()")[0].replace('\n',',')
douyin_info['location'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[1]/text()")[0]
douyin_info['xingzuo'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[2]/text()")[0]
douyin_info['follow_count'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='focus block']//i[@class='icon iconfont follow-num']/text()")[0].strip()
fans_value = ''.join(share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']//i[@class='icon iconfont follow-num']/text()"))
unit = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']/span[@class='num']/text()")
if unit[-1].strip() == 'w':
    douyin_info['fans'] = str((int(fans_value)/10))+'w'
like = ''.join(share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']//i[@class='icon iconfont follow-num']/text()"))
unit = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']/span[@class='num']/text()")
if unit[-1].strip() == 'w':
    douyin_info['like'] = str(int(like)/10)+'w'
douyin_info['from_url'] = share_web_url

print(douyin_info)

def handle_douyin_web_share(share_id):
share_web_url = 'https://www.douyin.com/share/user/'+share_id
print(share_web_url)
share_web_header = {
'user-agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36'
}
share_web_response = requests.get(url=share_web_url,headers=share_web_header)
handle_decode(share_web_response.text,share_web_url,share_id)

if name == 'main':
while True:
share_id = "76055758243"
if share_id == None:
print('当前处理task为:%s'%share_id)
break
else:
print('当前处理task为:%s'%share_id)
handle_douyin_web_share(share_id)
time.sleep(2)


![](https://upload-images.jianshu.io/upload_images/11223715-651b910cb91c1c8d.png?imageMogr2/auto-orient/strip%7CimageView2/2/w/1240)

#### mongodb
>通过vagrant 生成虚拟机创建mongodb，具体查看
「docker实战篇」python的docker爬虫技术-python脚本app抓取（13）

``` bash
su -
#密码：vagrant
docker

>https://hub.docker.com/r/bitnami/mongodb
>默认端口：27017
``` bash
docker pull bitnami/mongodb:latest
mkdir bitnami
cd bitnami
mkdir mongodb
docker run -d -v /path/to/mongodb-persistence:/root/bitnami -p 27017:27017 bitnami/mongodb:latest

#关闭防火墙
systemctl stop firewalld

docker- vibrato web-side data "docker combat chapter" python crawl (19)

操作mongodb

读txt文件获取userId的编号。

#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2019/1/30 19:35
# @Author  : Aries
# @Site    : 
# @File    : handle_mongo.py.py
# @Software: PyCharm

import pymongo
from pymongo.collection import Collection

client = pymongo.MongoClient(host='192.168.66.100',port=27017)
db = client['douyin']

def handle_init_task():
    task_id_collections = Collection(db, 'task_id')
    with open('douyin_hot_id.txt','r') as f:
        f_read = f.readlines()
        for i in f_read:
            task_info = {}
            task_info['share_id'] = i.replace('\n','')
            task_id_collections.insert(task_info)

def handle_get_task():
    task_id_collections = Collection(db, 'task_id')
    # return task_id_collections.find_one({})
    return task_id_collections.find_one_and_delete({})

#handle_init_task()

修改python程序调用


import re
import requests
import time
from lxml import etree
from handle_mongo import handle_get_task
from handle_mongo import handle_insert_douyin

for i1 in regex_list:
    for i2 in i1['name']:
        input_data = re.sub(i2,str(i1['value']),input_data)
share_web_html = etree.HTML(input_data)
douyin_info = {}
douyin_info['nick_name'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()")[0]
douyin_id = ''.join(share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']/p[@class='shortid']/i/text()"))
douyin_info['douyin_id'] = re.sub(search_douyin_str,'',share_web_html.xpath("//div[@class='personal-card']/div[@class='info1']//p[@class='nickname']/text()")[0]).strip() + douyin_id

try:
    douyin_info['job'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/div[@class='verify-info']/span[@class='info']/text()")[0].strip()
except:
    pass
douyin_info['describe'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='signature']/text()")[0].replace('\n',',')
douyin_info['location'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[1]/text()")[0]
douyin_info['xingzuo'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='extra-info']/span[2]/text()")[0]
douyin_info['follow_count'] = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='focus block']//i[@class='icon iconfont follow-num']/text()")[0].strip()
fans_value = ''.join(share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']//i[@class='icon iconfont follow-num']/text()"))
unit = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='follower block']/span[@class='num']/text()")
if unit[-1].strip() == 'w':
    douyin_info['fans'] = str((int(fans_value)/10))+'w'
like = ''.join(share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']//i[@class='icon iconfont follow-num']/text()"))
unit = share_web_html.xpath("//div[@class='personal-card']/div[@class='info2']/p[@class='follow-info']//span[@class='liked-num block']/span[@class='num']/text()")
if unit[-1].strip() == 'w':
    douyin_info['like'] = str(int(like)/10)+'w'
douyin_info['from_url'] = share_web_url

print(douyin_info)
handle_insert_douyin(douyin_info)

def handle_douyin_web_share(task):
share_web_url = 'https://www.douyin.com/share/user/'+task["share_id"]
print(share_web_url)
share_web_header = {
'user-agent':'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.75 Safari/537.36'
}
share_web_response = requests.get(url=share_web_url,headers=share_web_header)
handle_decode(share_web_response.text,share_web_url,task["share_id"])

if name == 'main':
while True:
task=handle_get_task()
handle_douyin_web_share(task)
time.sleep(2)


* mongodb字段
>handle_init_task 是将txt存入mongodb中
>handle_get_task 查出来一条然后删除一条，因为txt是存在的，所以删除根本没有关系

``` python
#!/usr/bin/env python
# -*- coding: utf-8 -*-
# @Time    : 2019/1/30 19:35
# @Author  : Aries
# @Site    : 
# @File    : handle_mongo.py.py
# @Software: PyCharm

import pymongo
from pymongo.collection import Collection

client = pymongo.MongoClient(host='192.168.66.100',port=27017)
db = client['douyin']

def handle_init_task():
    task_id_collections = Collection(db, 'task_id')
    with open('douyin_hot_id.txt','r') as f:
        f_read = f.readlines()
        for i in f_read:
            task_info = {}
            task_info['share_id'] = i.replace('\n','')
            task_id_collections.insert(task_info)

def handle_insert_douyin(douyin_info):
    task_id_collections = Collection(db, 'douyin_info')
    task_id_collections.insert(douyin_info)

def handle_get_task():
    task_id_collections = Collection(db, 'task_id')
    # return task_id_collections.find_one({})
    return task_id_collections.find_one_and_delete({})

handle_init_task()

docker- vibrato web-side data "docker combat chapter" python crawl (19)

PS: text text data in 1000 was not enough to climb too little, in fact, app end and end with the pc to crawling, pc side is responsible for initializing the data, obtained by userID to the fan list and then stop the cycle by crawling, so it is not possible to obtain very large amounts of data.