pcap处理
我们在使用pcap文件作为训练集时，都需要进行预处理，比如在使用CNN模型的时候，需要将pcap转为灰度图像来作为输入，我这里使用USTC-TK2016开源工具来进行处理。
首先我们从github下载USTC-TK2016的工具以及USTC-TFC2016的数据集
https://github.com/yungshenglu/USTC-TK2016
https://github.com/yungshenglu/USTC-TFC2016
然后将按照github文档上写的，将USTC-TFC2016数据集移动到1_Pcap\目录下
（温馨提示：如果使用pycharm打开的项目，请把数据集所在目录以及按session分割后的文件夹1_Pcap和2_Session文件夹先排除，防止你的pycharm会变得非常卡！！！）
然后执行1_Pcap2Session.ps1，但是这里出现问题：
看一下代码是什么问题
# Wei Wang ([email protected])
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file, You
# can obtain one at http://mozilla.org/MPL/2.0/.
# ==============================================================================
foreach($f in gci 1_Pcap *.pcap)
{
0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -o 2_Session\AllLayers\$($f.BaseName)-ALL
# 0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -s flow -o 2_Session\AllLayers\$($f.BaseName)-ALL
gci 2_Session\AllLayers\$($f.BaseName)-ALL | ?{
$_.Length -eq 0} | del
0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -o 2_Session\L7\$($f.BaseName)-L7 -y L7
# 0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -s flow -o 2_Session\L7\$($f.BaseName)-L7 -y L7
gci 2_Session\L7\$($f.BaseName)-L7 | ?{
$_.Length -eq 0} | del
}
0_Tool\finddupe -del 2_Session\AllLayers
0_Tool\finddupe -del 2_Session\L7
这段代码是一个powershell脚本，通过powershell执行一些列操作
首先foreach()这里是一个循环语句，遍历在当前目录下名为1_Pcap的文件夹中的所有以 .pcap扩展名结尾的文件。每次迭代，$f都会被赋值为一个文件对象。那么我们就知道问题就是我们将下载下来的USTC-TFC2016文件夹放进1_Pcap后，目录中没有.pcap文件
需要将目录改到benign和malware下，并且我们可以看到，直接下载的USTC-TFC2016数据集中，还有很多压缩文件，我们需要将它们解压缩才可以正常使用。
代码修改为：
# Wei Wang ([email protected])
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file, You
# can obtain one at http://mozilla.org/MPL/2.0/.
# ==============================================================================
foreach($f in gci 1_Pcap\USTC-TFC2016\Benign *.pcap)
{
0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -o 2_Session\AllLayers\$($f.BaseName)-ALL
# 0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -s flow -o 2_Session\AllLayers\$($f.BaseName)-ALL
gci 2_Session\AllLayers\$($f.BaseName)-ALL | ?{
$_.Length -eq 0} | del
0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -o 2_Session\L7\$($f.BaseName)-L7 -y L7
# 0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -s flow -o 2_Session\L7\$($f.BaseName)-L7 -y L7
gci 2_Session\L7\$($f.BaseName)-L7 | ?{
$_.Length -eq 0} | del
}
foreach($f in gci 1_Pcap\USTC-TFC2016\Malware *.pcap)
{
0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -o 2_Session\AllLayers\$($f.BaseName)-ALL
# 0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -s flow -o 2_Session\AllLayers\$($f.BaseName)-ALL
gci 2_Session\AllLayers\$($f.BaseName)-ALL | ?{
$_.Length -eq 0} | del
0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -o 2_Session\L7\$($f.BaseName)-L7 -y L7
# 0_Tool\SplitCap_2-1\SplitCap -p 50000 -b 50000 -r $f.FullName -s flow -o 2_Session\L7\$($f.BaseName)-L7 -y L7
gci 2_Session\L7\$($f.BaseName)-L7 | ?{
$_.Length -eq 0} | del
}
0_Tool\finddupe -del 2_Session\AllLayers
0_Tool\finddupe -del 2_Session\L7
这样修改之后，再次运行1_PcapSeesion，还是会报错：
然后去搜了一下原因，发现是splitcap文件的版本问题，所以要去官网下载一个最新的替换掉0_Tool目录下SplitCaP_2-1中的可执行程序，下载地址：https://www.netresec.com/index.ashx?page=SplitCap
替换后再执行1_Pcap2Session.ps1就可以了
运行结果：
（跑完发现，把数据集全解压出来的话，太大了，我这个跑了好几个小时，建议只想试一下代码的话单独拿一个pcap文件跑一下就行）
紧接着执行2_ProcessSession.ps1，又会报错：
显然是因为路径错误的问题，我们看一下2_ProcessSession.ps1：
# Wei Wang ([email protected])
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file, You
# can obtain one at http://mozilla.org/MPL/2.0/.
# ==============================================================================
$SESSIONS_COUNT_LIMIT_MIN = 0
$SESSIONS_COUNT_LIMIT_MAX = 60000
$TRIMED_FILE_LEN = 784
$SOURCE_SESSION_DIR = "Flow\L7"
echo "If Sessions more than $SESSIONS_COUNT_LIMIT_MAX we only select the largest $SESSIONS_COUNT_LIMIT_MAX."
echo "Finally Selected Sessions:"
$dirs = gci $SOURCE_SESSION_DIR -Directory
foreach($d in $dirs)
{
$files = gci $d.FullName
$count = $files.count
if($count -gt $SESSIONS_COUNT_LIMIT_MIN)
{
echo "$($d.Name) $count"
if($count -gt $SESSIONS_COUNT_LIMIT_MAX)
{
$files = $files | sort Length -Descending | select -First $SESSIONS_COUNT_LIMIT_MAX
$count = $SESSIONS_COUNT_LIMIT_MAX
}
$files = $files | resolve-path
$test = $files | get-random -count ([int]($count/10))
$train = $files | ?{
$_ -notin $test}
$path_test = "3_ProcessedSession\FilteredSession\Test\$($d.Name)"
$path_train = "3_ProcessedSession\FilteredSession\Train\$($d.Name)"
ni -Path $path_test -ItemType Directory -Force
ni -Path $path_train -ItemType Directory -Force
cp $test -destination $path_test
cp $train -destination $path_train
}
}
echo "All files will be trimed to $TRIMED_FILE_LEN length and if it's even shorter we'll fill the end with 0x00..."
$paths = @(('3_ProcessedSession\FilteredSession\Train', '3_ProcessedSession\TrimedSession\Train'), ('3_ProcessedSession\FilteredSession\Test', '3_ProcessedSession\TrimedSession\Test'))
foreach($p in $paths)
{
foreach ($d in gci $p[0] -Directory)
{
ni -Path "$($p[1])\$($d.Name)" -ItemType Directory -Force
foreach($f in gci $d.fullname)
{
$content = [System.IO.File]::ReadAllBytes($f.FullName)
$len = $f.length - $TRIMED_FILE_LEN
if($len -gt 0)
{
$content = $content[0..($TRIMED_FILE_LEN-1)]
}
elseif($len -lt 0)
{
$padding = [Byte[]] (,0x00 * ([math]::abs($len)))
$content = $content += $padding
}
Set-Content -value $content -encoding byte -path "$($p[1])\$($d.Name)\$($f.Name)"
}
}
}
修改为$SOURCE_SESSION_DIR = "2_Session\L7"就可以了，我们的文件夹没有Flow这个目录，再次运行2_ProcessSession.ps1，发现成功运行并在2_ProcessedSession目录下生成如下的目录：
+---FilteredSession
| +---Test
| | +---BitTorrent-L7
| | +---Cridex-L7
| | +---Facetime-L7
| | +---FTP-L7
| | +---Geodo-L7
| | +---Gmail-L7
| | +---Htbot-L7
| | +---Miuref-L7
| | +---MySQL-L7
| | +---Neris-L7
| | +---Nsis-ay-L7
| | +---Outlook-L7
| | +---Shifu-L7
| | +---Skype-L7
| | +---Tinba-L7
| | +---Virut-L7
| | +---WorldOfWarcraft-L7
| | \---Zeus-L7
| \---Train
| +---BitTorrent-L7
| +---Cridex-L7
| +---Facetime-L7
| +---FTP-L7
| +---Geodo-L7
| +---Gmail-L7
| +---Htbot-L7
| +---Miuref-L7
| +---MySQL-L7
| +---Neris-L7
| +---Nsis-ay-L7
| +---Outlook-L7
| +---Shifu-L7
| +---Skype-L7
| +---Tinba-L7
| +---Virut-L7
| +---WorldOfWarcraft-L7
| \---Zeus-L7
\---TrimedSession
+---Test
| +---BitTorrent-L7
| +---Cridex-L7
| +---Facetime-L7
| +---FTP-L7
| +---Geodo-L7
| +---Gmail-L7
| +---Htbot-L7
| +---Miuref-L7
| +---MySQL-L7
| +---Neris-L7
| +---Nsis-ay-L7
| +---Outlook-L7
| +---Shifu-L7
| +---Skype-L7
| +---Tinba-L7
| +---Virut-L7
| +---WorldOfWarcraft-L7
| \---Zeus-L7
\---Train
+---BitTorrent-L7
+---Cridex-L7
+---Facetime-L7
+---FTP-L7
+---Geodo-L7
+---Gmail-L7
+---Htbot-L7
+---Miuref-L7
+---MySQL-L7
+---Neris-L7
+---Nsis-ay-L7
+---Outlook-L7
+---Shifu-L7
+---Skype-L7
+---Tinba-L7
+---Virut-L7
+---WorldOfWarcraft-L7
\---Zeus-L7
还是因为之前把所有的数据集都解压了，所以就会很慢。然后我们用python运行3_Session2Png.py，发现报错：
查看代码：
# -*- coding: utf-8 -*-
# Wei Wang ([email protected])
#
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this file, You
# can obtain one at http://mozilla.org/MPL/2.0/.
# ==============================================================================
import numpy
from PIL import Image
import binascii
import errno
import os
PNG_SIZE = 28
def getMatrixfrom_pcap(filename,width):
with open(filename, 'rb') as f:
content = f.read()
hexst = binascii.hexlify(content)
fh = numpy.array([int(hexst[i:i+2],16) for i in range(0, len(hexst), 2)])
rn = len(fh)/width
fh = numpy.reshape(fh[:rn*width],(-1,width))
fh = numpy.uint8(fh)
return fh
def mkdir_p(path):
try:
os.makedirs(path)
except OSError as exc: # Python >2.5
if exc.errno == errno.EEXIST and os.path.isdir(path):
pass
else:
raise
paths = [['3_ProcessedSession\TrimedSession\Train', '4_Png\Train'],['3_ProcessedSession\TrimedSession\Test', '4_Png\Test']]
for p in paths:
for i, d in enumerate(os.listdir(p[0])):
dir_full = os.path.join(p[1], str(i))
mkdir_p(dir_full)
for f in os.listdir(os.path.join(p[0], d)):
bin_full = os.path.join(p[0], d, f)
im = Image.fromarray(getMatrixfrom_pcap(bin_full,PNG_SIZE))
png_full = os.path.join(dir_full, os.path.splitext(f)[0]+'.png')
im.save(png_full)
发现第22行，有一个
rn = len(fh)/width #python2
修改为
rn = len(fh)//width #python3
再次执行
发现已经生成了灰度图像：
到这里整个流程已经跑通了，后面有时间会写一下具体的代码详解。欢迎同样在做深度学习流量检测的大佬们交流，私信会很快回复！