【python爬虫自学笔记】-------自动化测试工具selenium用法

简介

是一个用于Web应用程序测试的工具。直接运行在浏览器中，

主要功能：测试与浏览器的兼容性---测试你的应用程序看是否能够很好工作在不同浏览器和操作系统上；测试系统功能--创建回归测试检验软件功能和用户需求。

用于爬虫，主要解决javascript渲染的问题。

安装：

安装selenium，和python中安装第三方库的方法相同。

安装chromedriver，参照http://blog.csdn.net/qq_29186489/article/details/78581249 

用法详解：

1、声明浏览器对象

测试chromedriver，安装成功后显示如下。

from selenium import webdriver
browser = webdriver.Chrome()

2、测试在百度输入关键词并进行搜索

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
import time
browser = webdriver.Chrome()
try:
    browser.get("http://www.baidu.com")
    input = browser.find_element_by_id("kw")
    input.send_keys("selenium")
    input.send_keys(Keys.ENTER)
    wait = WebDriverWait(browser,10)
    wait.until(EC.presence_of_element_located((By.ID,"content_left")))
    print(browser.current_url)
    print(browser.get_cookies())
    print(browser.page_source)
    time.sleep(10)
finally:
    browser.close()

https://www.baidu.com/s?ie=utf-8&f=8&rsv_bp=0&rsv_idx=1&tn=baidu&wd=selenium&rsv_pq=8a96c8120006578e&rsv_t=db9c2d9ujFATDPl%2BJurR6B5Jyz7kJJt%2FRmESUGxm%2BM%2BGVRFmUIP3uoMVGjw&rqlang=cn&rsv_enter=1&rsv_sug3=8&rsv_sug2=0&inputT=245&rsv_sug4=245

查找单个元素

browser = webdriver.Chrome()
browser.get("http://www.taobao.com")
test1 = browser.find_element_by_id("q")
test2 = browser.find_element_by_css_selector("#q")
test3 = browser.find_element(By.ID,"q")
print(test1,test2,test3)

<selenium.webdriver.remote.webelement.WebElement (session="2f468267d20e962da9ce518188e6e09a", element="0.511632817011686-1")> <selenium.webdriver.remote.webelement.WebElement (session="2f468267d20e962da9ce518188e6e09a", element="0.511632817011686-1")> <selenium.webdriver.remote.webelement.WebElement (session="2f468267d20e962da9ce518188e6e09a", element="0.511632817011686-1")>

查找多个元素

browser = webdriver.Chrome()
browser.get("http://www.taobao.com")
lis = browser.find_element_by_css_selector("li")
lis_c = browser.find_element(By.CSS_SELECTOR,"li")
print(lis,lis_c)
browser.close()

D:\develop\Anaconda3\python.exe D:/thislove/pythonworkspace/blogspark/taobaofood.py
<selenium.webdriver.remote.webelement.WebElement (session="79de473a0294cf9557e550d390384432", element="0.8140376499603905-1")> <selenium.webdriver.remote.webelement.WebElement (session="79de473a0294cf9557e550d390384432", element="0.8140376499603905-1")>

Process finished with exit code 0

元素的交互操作：在淘宝搜索中输入关键字并进行搜索

browser = webdriver.Chrome()
browser.get("http://www.taobao.com")
input = browser.find_element_by_id("q")
input.send_keys("iPhone")
time.sleep(10)
input.clear()
input.send_keys("iPad")
button = browser.find_element_by_class_name("btn-search")
button.click()
time.sleep(10)
browser.close()

交互动作：把动作附加到交互链中

from selenium.webdriver import ActionChains
import time
browser = webdriver.Chrome()
url="http://www.runoob.com/try/try.php?filename=jqueryui-api-droppable"
browser.get(url)
browser.switch_to.frame("iframeResult")
source = browser.find_element_by_id("draggable")
target = browser.find_element_by_id("droppable")
actions = ActionChains(browser)
actions.drag_and_drop(source,target)
actions.perform()
browser.switch_to.alert()
time.sleep(10)
browser.close()

执行javascript

browser = webdriver.Chrome()
browser.get("https://www.zhihu.com/explore")
browser.execute_script("window.scrollTo(0,document.body.scrollHeight)")
browser.execute_script("alert('To Button')")
# browser.close()

获取元素的信息

获取属性

browser = webdriver.Chrome()
browser.get("https://www.zhihu.com/explore")
logo = browser.find_element_by_id("zh-top-link-logo")
print(logo)
print(logo.get_attribute("class"))

D:\develop\Anaconda3\python.exe D:/thislove/pythonworkspace/blogspark/taobaofood.py
<selenium.webdriver.remote.webelement.WebElement (session="754ab2fd598085506fbc0b18cd5ab9ad", element="0.9086647244209951-1")>
zu-top-link-logo

Process finished with exit code 0

获取文本值

browser = webdriver.Chrome()
browser.get("https://www.zhihu.com/explore")
logo = browser.find_element_by_id("zh-top-link-logo")
print(logo)
print(logo.text)

D:\develop\Anaconda3\python.exe D:/thislove/pythonworkspace/blogspark/taobaofood.py
<selenium.webdriver.remote.webelement.WebElement (session="06bab951211cc8f8e6c7d39898b9b771", element="0.3505634118116907-1")>
知乎

Process finished with exit code 0

获取其他属性

browser = webdriver.Chrome()
browser.get("https://www.zhihu.com/explore")
logo = browser.find_element_by_id("zh-top-link-logo")
print(logo)
print(logo.id)
print(logo.location)
print(logo.tag_name)
print(logo.size)

D:\develop\Anaconda3\python.exe D:/thislove/pythonworkspace/blogspark/taobaofood.py
<selenium.webdriver.remote.webelement.WebElement (session="7bd2b084242253c32fa19e057166d7ca", element="0.00473317282732566-1")>
0.00473317282732566-1
{'x': 28, 'y': 0}
a
{'height': 45, 'width': 61}

Process finished with exit code 0

等待

显式等待

browser.get("http://www.taobao.com")
wait = WebDriverWait(browser,10)
input = wait.until(EC.presence_of_element_located((By.ID,"q")))
button = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,".btn-search")))
print(input,button)

D:\develop\Anaconda3\python.exe D:/thislove/pythonworkspace/blogspark/taobaofood.py
<selenium.webdriver.remote.webelement.WebElement (session="ef4aa986c988b7cd897fa767b10c402f", element="0.5417483220523784-1")> <selenium.webdriver.remote.webelement.WebElement (session="ef4aa986c988b7cd897fa767b10c402f", element="0.5417483220523784-2")>

Process finished with exit code 0

隐式等待：如果webdriver没有在DOM中找到元素，将继续等待，超过设定的时间后则抛出找不到元素的异常，即当查找元素或者元素并没有立即出现的时候，隐式等待将等待一段时间再查找DOM，默认时间为0.

browser.get("https://www.zhihu.com/explore")
wait = WebDriverWait(browser,10)
browser.implicitly_wait(10)
logo = browser.find_element_by_id("zh-top-link-logo")
print(logo)

D:\develop\Anaconda3\python.exe D:/thislove/pythonworkspace/blogspark/taobaofood.py
<selenium.webdriver.remote.webelement.WebElement (session="a65e16b18761a122fff7bf5ec4728cc4", element="0.739579951812422-1")>

Process finished with exit code 0

浏览器前进后退

browser.get("https://www.zhihu.com/explore")
browser.get("https://www.taobao.com")
browser.get("https://www.python.org")
browser.back()
time.sleep(1)
browser.forward()
browser.close()

cookies的处理

browser.get("https://www.zhihu.com/explore")
print(browser.get_cookies())
#添加
browser.add_cookie({"name":"name","domain":"www.zhihu.com","value":"germey"})
print(browser.get_cookies())
#删除
browser.delete_all_cookies()
print(browser.get_cookies())
browser.close()

选项卡管理

browser.get("https://www.zhihu.com/explore")
browser.execute_script("window.open()")
print(browser.window_handles)
browser.switch_to.window(browser.window_handles[1])
browser.get("https://www.taobao.com")
time.sleep(1)
browser.switch_to.window(browser.window_handles[0])
browser.get("https://python.org")
browser.close()

异常处理

browser = webdriver.Chrome()
try:
    browser.get("https://www.zhihu.com/explore")
except TimeoutException:
    print("Time out")
try:
    browser.find_element_by_id("hello")
except NoSuchElementException:
    print("No element")
finally:
    browser.close()

D:\develop\Anaconda3\python.exe D:/thislove/pythonworkspace/blogspark/taobaofood.py
No element

Process finished with exit code 0

https://blog.csdn.net/qq_29186489/article/details/78661008