"Docker actual articles" docker- end multi-device concurrent python crawling fans vibrato data (23)

Original article, welcome to reprint. Reproduced please specify: Reprinted from IT people story , thank you!
Original link address: "docker combat chapter" docker- end multi-device concurrent python crawling fans vibrato data (23)

Engaged in the previous article, vibrato and vibrato app crawl end web end user information a fan crawl. Grab a piece of equipment vibrato fan of data is really slow, this time to demonstrate our multi-simulator while crawling information. Source: https://github.com/limingios/dockerpython.git (Source / "docker combat chapter" python end of docker- multi-device concurrent crawl fans vibrato data (22))

Before copy tape simulator xponsed

• 1. The desktop has a god simulator to open

• 2. Select, click Copy, copy may look out of three, we do not need so many deleted two on it, if you have a computer powerful enough to copy multiple, thinking the following code is the same.

• 3. After completion of copying, the change alias differentiate between

• 4. Start two simulator.

Multitasking crawl

Normally the general operation of the Internet, if you want to multi-task crawl basically use containerized done, but there is little direct say docker too early, but this series is certainly to be multi-device docker crawled, where the first python talk about the way to use multiple processes to complete.

• 1. View a god simulator port, start cmd, enter

  adb devices

• 2. If you do not enter adb devices List

  In the simulator to start the night when God adb not start. That is, a god simulator than adb first start, solutions.
  1. Open the Task Manager to see a god simulator PID, by clicking Status - select the PID

2. Just to see the PID

netstat -ano | findstr "452"

netstat -ano | findstr "16712"

3. Yagami simulator port is regular, a first simulator port 62001, 62025 is the second, the third 62025 + 1, 2 + 4 62025

4. The manual command devices connected

   adb connect 127.0.0.1:62025

5.appium【客户端】需要设置udid，在appium里面识别就是udid，因为之前是一台设备所以不需要指定udid，光指定deviceName就可以了。

6. appium【服务端】需要设置bootstrapPort，服务端进行设置，设备和appium通信的端口。

7.appium的2台的属性设置
1.douyin1 port=4723 bootstrapPort=4724
2.douyin1 port=4725 bootstrapPort=4726

• 3. 报错解决方案

selenium.common.exceptions.WebDriverException: Message: An unknown server-side error occurred while processing the command. Original error: Failed to Dump Window Hierarchy

解决方案
在含有Emoji特殊符号的页面中,爆出Failed to Dump Window Hierarchy
https://github.com/appium/appium/issues/4151
http://blog.csdn.net/soslinken/article/details/50126477
won’t be solved until the new android driver is completed. This is a known bug in uiautomator v1
此问题是uiautomator自身bug，换用Android5.1以上的系统

源码

#!/usr/bin/env python

import time
from appium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
import multiprocessing

def get_size(driver):
    x = driver.get_window_size()['width']
    y = driver.get_window_size()['height']
    return (x, y)

def handle_douyin(driver):
    while True:
        #定位搜索框
        if WebDriverWait(driver,60).until(lambda x:x.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[2]/android.widget.EditText[1]")):
            #获取douyin_id进行搜索
            driver.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[2]/android.widget.EditText[1]").send_keys('1860719705')
            while driver.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[2]/android.widget.EditText[1]").text != '1860719705':
                driver.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[2]/android.widget.EditText[1]").send_keys('1860719705')
                time.sleep(0.1)
        #点击搜索
        driver.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[1]/android.widget.TextView[1]").click()

        #点击用户标签
        if WebDriverWait(driver,30).until(lambda x:x.find_element_by_xpath("//android.widget.TextView[@text='用户']")):
            driver.find_element_by_xpath("//android.widget.TextView[@text='用户']").click()
        #点击头像
        if WebDriverWait(driver,30).until(lambda x:x.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.support.v4.view.ViewPager[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[2]/android.view.View[1]/android.support.v7.widget.RecyclerView[1]/android.widget.RelativeLayout[1]/android.widget.RelativeLayout[1]/android.widget.ImageView[1]")):
            driver.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.support.v4.view.ViewPager[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[2]/android.view.View[1]/android.support.v7.widget.RecyclerView[1]/android.widget.RelativeLayout[1]/android.widget.RelativeLayout[1]/android.widget.ImageView[1]").click()
        #点击粉丝按钮
        if WebDriverWait(driver,30).until(lambda x:x.find_element_by_xpath("//android.widget.TextView[@text='粉丝']")):
            driver.find_element_by_xpath("//android.widget.TextView[@text='粉丝']").click()

            x1 = int(driver.get_window_size()['width']*0.5)
            y1 = int(driver.get_window_size()['height']*0.75)
            y2 = int(driver.get_window_size()['height']*0.25)
            while True:
                time.sleep(3)
                if '没有更多了' in driver.page_source:
                    break
                elif 'TA还没有粉丝' in  driver.page_source:
                    break
                else:
                    driver.swipe(x1,y1,x1,y2)
                    time.sleep(0.5)

            #返回
            driver.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[1]/android.widget.RelativeLayout[1]/android.widget.FrameLayout[1]/android.widget.ImageView[1]").click()
            #返回
            driver.find_element_by_id("com.ss.android.ugc.aweme:id/jk").click()
            #重新清空用户id
            driver.find_element_by_xpath("//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[2]/android.widget.EditText[1]").clear()

def handle_appium(device,port):
    cap = {
        "platformName": "Android",
        "platformVersion": "4.4.2",
        "deviceName": device,
        "udid":device,
        # 真机的
        # "platformName": "Android",
        # "platformVersion": "7.1.2",
        # "deviceName": "10d4e4387d74",
        "appPackage": "com.ss.android.ugc.aweme",
        "appActivity": "com.ss.android.ugc.aweme.main.MainActivity",
        "noReset": True,
        "unicodeKeyboard": True,
        "resetkeyboard": True
    }

    driver = webdriver.Remote("http://localhost:"+str(port)+"/wd/hub", cap)

    try:
        # 点击搜索
        print('点击搜索')
        if WebDriverWait(driver, 60).until(lambda x: x.find_element_by_xpath(
                "//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.support.v4.view.ViewPager[1]/android.widget.FrameLayout[1]/android.support.v4.view.ViewPager[1]/android.widget.TabHost[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.view.View[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[2]/android.widget.FrameLayout[1]/android.widget.ImageView[1]")):
            driver.find_element_by_xpath(
                "//android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.support.v4.view.ViewPager[1]/android.widget.FrameLayout[1]/android.support.v4.view.ViewPager[1]/android.widget.TabHost[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.widget.FrameLayout[1]/android.view.View[1]/android.widget.FrameLayout[1]/android.widget.LinearLayout[1]/android.widget.FrameLayout[1]/android.widget.RelativeLayout[1]/android.widget.LinearLayout[2]/android.widget.FrameLayout[1]/android.widget.ImageView[1]").click()
    except:
        # [26,76][115,165]
        driver.tap([(26, 76), (115, 165)], 500)

    handle_douyin(driver)

if __name__ == '__main__':
    m_list = []
    #定义了2台虚拟设备，夜神模拟器
    devices_list = ["127.0.0.1:62001","127.0.0.1:62025"]
    for device in range(len(devices_list)):
        port = 4723 + 2 * device
        m_list.append(multiprocessing.Process(target=handle_appium,args=(devices_list[device],port,)))

    for m1 in m_list:
        m1.start()

    for m2 in m_list:
        m2.join()

启动步骤

• 1.启动2个appium

  之前上边已经提到过的端口（4723 4724 ）（4725 4726）

• 2. Start python code to see the effect

  Too much debugging test code number, douyin asking me to login, ha ha

Camouflage reptile

First appeared because too many access number of books, douyin identified over there, so need to be addressed. Acting way through.

• Ip 1. purchasing agents, or with our old friend Abu [cloud] https://www.abuyun.com/

• 2. Log after Abu [cloud], using mitmdump way, proxy settings Abu cloud

  Start cmd, enter
  mitmdump -s test.py -p 8889 --mode upstream: HTTP Tunnel Server: Port --upstream-auth pass books: access keys

mitmdump -s test.py -p 8889 --mode upstream:http://http-pro.abuyun.com:9010 --upstream-auth HS821YO6BA7D6M8P:75021E5CF3AB82EE

• 3. simulator wifi also set the corresponding agent, said before the repeated under.

• 4. This way of using a proxy. All requests is the agent of the Abu cloud.

Thanks to the old iron recommends that all of the source code after the series is to organize in accordance with the corresponding article, the original source code are in accordance with my development process, so not suitable for intermediate involved in the old iron.

PS: debugging process, a god simulator, appium, python plugin code is not a problem, the program appears in the course of running up to the problem or issue xpath location for the python code is actually well understood. Also mentioned in the article noted that you want to use Android 5.1 or higher or because Emoji page contains special symbols, broke Failed to Dump Window Hierarchy. udid for more time to start the simulator must be set.