Buried remember once information server statistics

background:

Allow testers to test the device information Buried reporting cases.

The equipment required statistical fields are as follows:

name	Explanation
platform	Operating system platform
adblock	Anti-adware: by judge whether the insertion dom
cpuCoresCpu	cpu cores: The navigator determines (cpu type ie able to obtain, not only processes)
audio	Fingerprint card: judging by window
WebglVendorAndRenderer	Graphics models: obtained by calling canvas
ip	Public network, local area network ip
webglKey	webgl (different browsers is not uniform)
touchSupport	Whether to support touch: judging by the navigator, (the number of touch points, touch events)
hasLiedLanguages	Whether to modify the language: judging by navigator
scrDeviceXDPI	DPI devices: ie only able to get to, not treated
pixelDepth	Color Resolution
colorDepth	Color Depth
doNotTrack	Whether to allow tracking
HasLiedResolution	Whether to modify the screen resolution
cookieEnabled	Whether the cookie is enabled
BrowserName	Browser name
timeZone	Time zone
MimeTypes	Supported MIME types of arrays (no phone)
systemLanguage	System language (ie only)
BrowserLanguage	Browser Language
Online	The system is in offline mode
wh	Screen resolution: W * H
brand	Mobile phone brand (brand coverage test)

carried out:

Access Address: https://testwx.aikaixin.com/vue/go

View server will see a record as follows:

10.10.15.9 - - [11/Feb/2020:09:30:05 +0800] "GET /s.png?Adblock=false&Audio=48000_0_1_0_2_explicit_speakers&BrowserLanguage=zh-CN&BrowserName&CookieEnabled=true&DoNotTrack=false&HasLiedLanguages=true&MimeTypes=&Online=true&TouchSupport=5_true&WebglFp=eff1c22f7a276449cca05e9be289b9ee&WebglVendorAndRenderer=Apple%20Inc.~Apple%20GPU&brand=iPhone&platform=iPhone&startTime=1581384605010&wh=375%2A667 HTTP/1.0" 200 70 "https://testwx.aikaixin.com/vue/go" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 AliApp(DingTalk/4.7.25) com.laiwang.DingTalk/12472299 Channel/201200 language/zh-Hans-CN UT4Aplus/0.0.6 WK" "49.67.212.46"

Product Manager up to ask: What time will probably test the test is completed?

I? What is meant by the test is completed yet.

The current in the home office, limited machine on hand, how to measure the results they want, and how to export data to EXCEL product manager can see it.

First look at the top of this data generated by the server, you will find some places were url encoded:

To Webmaster Tools http://tool.chinaz.com/tools/urlencode.aspx inside to explain them code:

10.10.15.9 - - [11/Feb/2020:09:30:05 0800] "GET /s.png?Adblock=false&Audio=48000_0_1_0_2_explicit_speakers&BrowserLanguage=zh-CN&BrowserName&CookieEnabled=true&DoNotTrack=false&HasLiedLanguages=true&MimeTypes=&Online=true&TouchSupport=5_true&WebglFp=eff1c22f7a276449cca05e9be289b9ee&WebglVendorAndRenderer=Apple Inc.~Apple GPU&brand=iPhone&platform=iPhone&startTime=1581384605010&wh=375*667 HTTP/1.0" 200 70 "https://testwx.aikaixin.com/vue/go" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 AliApp(DingTalk/4.7.25) com.laiwang.DingTalk/12472299 Channel/201200 language/zh-Hans-CN UT4Aplus/0.0.6 WK" "49.67.212.46"

可以看到现在都显示正常了。

当然可以手动将数据填到excel表里，为此还真尝试了一下，填了两组数据之后-------眼疼。

对在同一台手机上，在浏览器，qq，微信，钉钉上打开同一个链接，经肉眼观察，数据相差不大，可以初步假设其他手机上也是如此，那假如要大量的数据，用云真机去访问下这个链接，我再到服务器日志里将数据捞出来不就搞定收工了？

于是我百度了下，首先用众测试了下，新用户有免费30分钟，但是用过之后发现巨卡。后来还是用了wetest，巨贵，扣起时间来也毫不手软，30分钟要19.8，然后花了两个19.8，选用了主流品牌的几款高端机型点击了一下链接。

数据有了，接下来就是将数据导出来：cat access.log |grep "GET /s.png" > 5.log

随便取两条

10.10.15.9 - - [11/Feb/2020:09:30:36 +0800] "GET /s.png?Adblock=false&Audio=48000_0_1_0_2_explicit_speakers&BrowserLanguage=zh-CN&BrowserName=WeChat&CookieEnabled=true&DoNotTrack=false&HasLiedLanguages=true&MimeTypes=&Online=true&TouchSupport=5_true&WebglFp=eff1c22f7a276449cca05e9be289b9ee&WebglVendorAndRenderer=Apple%20Inc.~Apple%20GPU&brand=iPhone&platform=iPhone&startTime=1581384636223&wh=375%2A667 HTTP/1.0" 200 70 "https://testwx.aikaixin.com/vue/go" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 MicroMessenger/7.0.10(0x17000a21) NetType/WIFI Language/zh_CN" "49.67.212.46"

第一步现将数据url解码，将导出来的日志文件解码后存入到新的文件里面

def filedecode():
    #导出log文件，通过这种命令导出cat access.log |grep "GET /s.png" > 5.log
    with open('5.log','r') as f:
        content = f.readlines()

    #将导出的文件解码，解码后的文件存到6.txt
    with open('6.txt','w') as f:
        for line in content:
            contentdecode = parse.unquote(line)
            f.write(contentdecode)

第二步对每一行的数据通过“&”分段，用到字符串处理split函数

datalist = line.split('&')

数据处理之后大概是这样子的一个列表：

['10.10.15.9 - - [11/Feb/2020:09:30:05 +0800] "GET /s.png?Adblock=false','Audio=48000_0_1_0_2_explicit_speakers'.......,'wh=375*667 HTTP/1.0" 200 70 "https://testwx.aikaixin.com/vue/go" "Mozilla/5.0 (iPhone; CPU iPhone OS 13_3_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148 AliApp(DingTalk/4.7.25) com.laiwang.DingTalk/12472299 Channel/201200 language/zh-Hans-CN UT4Aplus/0.0.6 WK" "49.67.212.46"']

可以看到列表的第一个元素和最后一个元素需要特殊处理，中间的字段是 A=B的形式。

处理第一个元素，获取Adblock=flase：

datalist[0] = datalist[0].split('?')[1]

处理最后一个元素,获取wh

lastdatalist = datalist[-1].split(' ')

datalist[-1] = lastdatalist[0]

我们的最终目的是将是将数据处理成一条条的字典，然后再将这些字典写入到excel的指定位置。

将处理完成的数据转化为字典将A=B的这种，A作为字典的key值，B作为value值，因为数据里有这种A=,或者A的情况，没有B所以他的value将没有值，这里需要特殊处理一下，将没有值的给他一个默认值“无数据”

dlist = d.split('=')

if len(dlist) == 2 and dlist[1] != '':

dkey = dlist[0]

dvalue = dlist[1]

else:

dkey = dlist[0]

dvalue = '无数据'

datadict[dkey] = dvalue

数据处理完成后，就是将其写入到EXCEL表中，这里用到的是openpyxl

表格样式如下：

为了使得数据灵活一些，这里直接使用表格中第二行的值作为key值，如果这个key值在我们处理的数据中不存在，那么就给他默认“无数据”

这个就比较简单了。

详细代码如下：

from urllib import parse
import openpyxl
from openpyxl.styles import colors,PatternFill
#读取文件信息，进行url解码

def filedecode():
    #导出log文件，通过这种命令导出cat access.log |grep "GET /s.png" > 5.log
    with open('5.log','r') as f:
        content = f.readlines()

    #将导出的文件解码，解码后的文件存到6.txt
    with open('6.txt','w') as f:
        for line in content:
            contentdecode = parse.unquote(line)
            f.write(contentdecode)


#对数据文件进行处理
def dealdata():
    #对文件解码
    filedecode()
    #读取需要处理的数据文件
    with open('6.txt','r') as f:
        data = f.readlines()

    #定义一个数组用于存储处理完成的字典数据
    datadonelist = []
    #对每一行数据进行处理
    for line in data:
        #将一行的数据通过&和?符号分隔开来
        datalist = line.split('&')

        #第一次处理后，数组里的第一个元素和最后一个元素需要进行进一步处理，获取准确的数据
        #数组里的第一个元素获取Adblock
        datalist[0] = datalist[0].split('?')[1]

        #数组里最后一个元素取wh和ip地址
        lastdatalist = datalist[-1].split(' ')
        datalist[-1] = lastdatalist[0]
        #取ip，然后将其加到数组里去
        lastdata = "ip="+lastdatalist[-1]
        datalist.append(lastdata)

        #将数据处理成字典格式
        #1.先通过“=”分隔数据，左侧为key，右侧为value
        #2.定义一个字典用于存储数据
        datadict = {}
        for d in datalist:
            #因为有的字段为空值，所以要判断数组长度
            dlist = d.split('=')
            if len(dlist) == 2 and dlist[1] != '':
                dkey = dlist[0]
                dvalue = dlist[1]
            else:
                dkey = dlist[0]
                dvalue = '无数据'
            datadict[dkey] = dvalue
        datadonelist.append(datadict)               

    return datadonelist

#将处理好的数据写入excel表
def writexls():
    #1.读取excel表格数据
    wb = openpyxl.load_workbook('data.xlsx')
    #2.定位表格
    sheet1 = wb['Sheet1']
    #3.获取最大列
    maxcolumn = sheet1.max_column
    #获取处理好的数据
    data = dealdata()
    print(len(data))
    #从第三行开始写入excel表，根据第二行的值来写
    for i in range(3,len(data)+3):
        for j in range(1,maxcolumn+1):
            titlekey = sheet1.cell(2,j).value
            #需要判断键值是否在字典中存在，因为存在键值不存在的情况
            sheet1.cell(i,j).value = data[i-3].get(titlekey,'无数据')
    
    #对结果数据进行标识处理,将无数据的标黄
    fillcolor = PatternFill(fill_type='solid',fgColor=colors.YELLOW)
    #遍历表格数据，如果为无数据，将它标黄
    #获取最大行
    maxrow = sheet1.max_row
    for i in range(1,maxrow+1):
        for j in range(1,maxcolumn+1):
            if sheet1.cell(i,j).value == '无数据':
                sheet1.cell(i,j).fill = fillcolor
    #保存表格
    wb.save('data.xlsx')


writexls()

最终处理结果如下：

第一阶段测试搞定收工！后续就等开发他们优化，增加了字段之后，只要小改代码就行了

joey_2018_

发布了23 篇原创文章 · 获赞 15 · 访问量 2万+

私信关注

Buried remember once information server statistics

Guess you like