First Benpian explain that crawl home to brush the user's video users concerned about the user's information
The tools used have environmental structures
1.appium1.7.2 : automated testing tools (also can be run directly on the phone In this part)
2.Appium-desktop appium Desktop Visualization Tools
3.fiddler4 data capture tool for grabbing vibrato returned by the server
4.jsoup reptiles and appium tool for java-client interaction of java jar
<dependency>
<groupId>io.appium</groupId>
<artifactId>java-client</artifactId>
<version>5.0.4</version>
</dependency>
<dependency>
<groupId>org.jsoup</groupId>
<artifactId>jsoup</artifactId>
<version>1.11.3</version>
</dependency>5. Android sdk environmental needs
6.node6.11.3 appium environment needs
appium-doctor and do not install python
(Problems can not write a detailed message, just write a rough idea of what the code too much for yourself in the project)
First few pictures
This video is crawling general process (too much garbage company computer simulator run)
This is crawled to each video information (unfinished) in order to facilitate his cause display TXT files, in practical use is preferably stored in the database (with built database)
The basic idea is the same first capture obtain a fiddler url Analysis
Links: This article required information Password: po4y
nox night God simulator appium and Android sdk jdk also did not say when the installation remember to use the database changed
fiddler reference will not use this article
Then open vibrato into the user's home page
You can see there are two home url first is to get basic user information
The main parameters : the two most important link is the user_id other can copy and paste
user_id : real user ID vibrato get all the basic data of the most important is user_id
TS : time stamp of the current time in units of seconds
AS , cp , MAS : these are the three big brothers can try to crack the encryption parameters about these three parameters copy it if only for a single user to crawl video then you can use (currently vibrato 3.0 version 2018.11. Number 5)
The data returned:
Interception only some of the key data:
total_favorited : 获赞数
following_count : 关注数
mplatform_followers_count :粉丝数
aweme_count : 作品数
dongtai_count : 动态数
favoriting_count :喜欢数
avatar_medium :用户头像
short_id : 用户抖音号
province : 省份
city : 城市
country : 国家
uid: 用户uid
nickname : 昵称
signature : 用户简介
With this url so you know the user's user_id after the basic data the user can obtain the url (not including video information)
The second is to get the next url user video information
The basic parameters can refer to the first link, one thing should be noted that max_cursor and count these two parameters
max_cursor : This parameter allows the user to return to the video list, the first visit, this parameter is 0, it will return the latest video of 20 (count was 20)
The data returned in a url parameter max_cursor likewise, and this data is acquired as the first video max_cursor 21-40
And so on,
count : the number returned by the default video like 20, if you change the number, then also returns corresponding according to your count
Returned by the second parsing data obtained url
aweme_list : 视频列表
has_more : 是否还有视频 , 1有 , 0没有
list的每个子项 :
aweme_id : 视频id
desc : 视频标题
share_count : 分享次数
digg_count : 点赞次数
comment_count : 评论数
forward_count : 转发数
video : 视频的其他信息
video的每个子项:
radio : 清晰度
dynamic_cover : 封面gif(数组 每一项都是)
origin_cover : 封面图片(数组 每一项都是)
download_addr : 视频下载地址(数组 每一项都是)
play_addr : 也可以用来下载So both url on completion of the analysis, with these two url, then know the user's user_id, you can put a user's entire video download
One thing to note is the need for an analog phone requests, set up under the user-Agent header from the request when the browser debugging find themselves just fine
Mozilla/5.0 (Linux; Android 5.0; SM-G900P Build/LRX21T) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Mobile Safari/537.36The next addition to the remaining two interfaces interfaces are time-sensitive, and all can not just execute a program in java
Benpian Solution: Use data appium automated testing tools simulate user behavior handling simulator night of God, then fiddler preservation request data, obtained by parsing with java
After complete environment to build, configure a god simulator and appium
Press and modify the network configuration according to their own ip and fiddler
fiddler automatically save the configuration data
This article is the required configuration:
static function OnBeforeResponse(oSession: Session) {
if (m_Hide304s && oSession.responseCode == 304) {
oSession["ui-hide"] = "true";
}
//aweme 用户主页视频详情
if ((oSession.HostnameIs("api.amemv.com")|| oSession.HostnameIs("aweme.snssdk.com")) && oSession.uriContains("/aweme/v1/aweme/post")) {
oSession.utilDecodeResponse();
//消除保存的请求可能存在乱码的情况
var url = oSession.url.ToString();
var demo1 = getAllUrlParams(url);
var fso;
var file;
fso = new ActiveXObject("Scripting.FileSystemObject");
//文件保存路径，可自定义
file = fso.OpenTextFile("D:/douyin/request/" + demo1.user_id +"_aweme.txt",8 ,true,true);
file.writeLine( oSession.GetResponseBodyAsString());
file.close();
}
//用户主页信息详情
if ((oSession.HostnameIs("api.amemv.com")|| oSession.HostnameIs("aweme.snssdk.com")) && oSession.uriContains("/aweme/v1/user")) {
oSession.utilDecodeResponse();
//消除保存的请求可能存在乱码的情况
var url = oSession.url.ToString();
var demo1 = getAllUrlParams(url);
var fso;
var file;
fso = new ActiveXObject("Scripting.FileSystemObject");
//文件保存路径，可自定义
file = fso.OpenTextFile("D:/douyin/request/" + demo1.user_id +"_user.txt",8 ,true , true);
file.writeLine( oSession.GetResponseBodyAsString());
file.close();
}
//用户关注人详情
if ((oSession.HostnameIs("api.amemv.com")|| oSession.HostnameIs("aweme.snssdk.com"))&& oSession.uriContains("/aweme/v1/user/following/list")) {
oSession.utilDecodeResponse();
//消除保存的请求可能存在乱码的情况
var url = oSession.url.ToString();
var demo1 = getAllUrlParams(url);
var fso;
var file;
fso = new ActiveXObject("Scripting.FileSystemObject");
//文件保存路径，可自定义
file = fso.OpenTextFile("D:/douyin/request/" + demo1.user_id +"_following.txt",8 ,true , true);
file.writeLine( oSession.GetResponseBodyAsString());
file.close();
}
}
static function getAllUrlParams(url) {
var queryString = url ? url.split('?')[1] : "";
var obj = {};
if (queryString) {
queryString = queryString.split('#')[0];
var arr = queryString.split('&');
for (var i = 0; i < arr.length; i++) {
var a = arr[i].split('=');
var paramNum = undefined;
var paramName = a[0].replace(/\[\d*\]/, function (v) {
paramNum = v.slice(1, -1);
return '';
});
var paramValue = typeof(a[1]) === 'undefined' ? true : a[1];
if (obj[paramName]) {
if (typeof obj[paramName] === 'string') {
obj[paramName] = [obj[paramName]];
}
if (typeof paramNum === 'undefined') {
obj[paramName].push(paramValue);
}
else {
obj[paramName][paramNum] = paramValue;
}
}
else {
obj[paramName] = paramValue;
}
}
}
return obj;
}In the D drive to create a new folder douyin
Next, using java script writing appium start vibrato
AndroidDriver driver;
DesiredCapabilities cap = new DesiredCapabilities();
cap.setCapability("automationName", "Appium");//appium做自动化
// cap.setCapability("app", "C:\\software\\jrtt.apk");//安装apk
// cap.setCapability("browserName", "chrome");//设置HTML5的自动化，打开谷歌浏览器
cap.setCapability("deviceName", "127.0.0.1:62001");//设备名称
cap.setCapability("platformName", "Android"); //安卓自动化还是IOS自动化
cap.setCapability("platformVersion", "4.4.2"); //安卓操作系统版本
//cap.setCapability("automationName", "UIAutomator2"); //支持新版安卓在手机上运行记//得打开
//cap.setCapability("udid", "127.0.0.1:62001"); //设备的udid (adb devices 查看到的)
cap.setCapability("appPackage", "com.ss.android.ugc.aweme");//被测app的包名
cap.setCapability("appActivity", "com.ss.android.ugc.aweme.splash.SplashActivity");//被测app的入口Activity名称
//cap.setCapability("unicodeKeyboard", "True"); //支持中文输入
// cap.setCapability("resetKeyboard", "True"); //支持中文输入，必须两条都配置
cap.setCapability("noSign", "True"); //不重新签名apk
cap.setCapability("noReset", "True"); //不重新签名apk
//cap.setCapability("newCommandTimeout", "30"); //没有新命令，appium30秒退出
URL url = new URL("http://127.0.0.1:4723/wd/hub");
driver = new AndroidDriver(new URL("http://127.0.0.1:4723/wd/hub"), cap);//把以上配置传到appium服务端并连接手机
driver.manage().timeouts().implicitlyWait(10, TimeUnit.SECONDS);//隐式等待
Thread.sleep(10000);//休息10s避免广告deviceName look through adb devices command
If you encounter the following problems:
.. 1 the adb absent: in sdk / platform-tools path environment variable added to the
Add the bin a god simulator to the path environment variable
2 . adb server version (31) doesn't match this client (36)
Adb sdk inconsistent and night to night copy of God of God in sdk installation directory
Find the installation path: \ Nox \ bin, which has a nox_adb.exe, in fact, adb.exe, in order to avoid conflicts nox which changed the name.
Then copy the android-sdk inside adb.exe version comes out, then a change of name is nox_adb.exe, replacing nox installation path: \ Nox \ nox_adb.exe file in the bin on the line
3 . adb server version (31) doesn't match this client (40); killing...
could not read ok from ADB Server
* failed to start daemon
error: cannot connect to daemon
Prove port is occupied most of the 360 security guards can also modify the port
That is what I encountered the pit
app package name and through inlet aapt dump badging douyin3.0.apk
包名:
Entrance:
Specific logic: slip on the home page and then slip into the left this time fiddler Home users will automatically get to these two files
Then click concern view
view acquired by viewing appium-desktop
Open appium click on start button and then click on the map at
Configuration is as follows
{
"platformName": "Android",
"deviceName": "127.0.0.1:62001",
"appPackage": "com.ss.android.ugc.aweme",
"appActivity": "com.ss.android.ugc.aweme.splash.SplashActivity",
"platformVersion": "4.4.2"
}Then click on the bottom right to start start start session following the completion of the interface
Click on the top
Found attention view 
On the right will be displayed
This is the view of a7t ID can also get the following xpath path (not recommended with bad often can not find)
then click 
can be seen
java script acquisition mode
Click on
Xy coordinates can be obtained 
Then you can write a script
//获取用户的关注信息
public static void getConcernUser(AndroidDriver driver) throws InterruptedException {
//获取关注的view
WebElement webElement = getElementById("a7t", driver);
if (webElement != null ){
//获取关注数量view
WebElement nums = getElementById("a7u", driver);
String text = nums.getAttribute("text");
if (StringUtils.isNotBlank( text )) {
int count = (Integer.parseInt(text)- 10 ) / 5;
//点击关注页
webElement.click();
for (int i = 0 ; i <= count ; i ++){
//屏幕上滑
Swipe.swipeUp(driver);
}
//加载完所有关注用户信息 返回用户视频首页
Swipe.back(driver);//点击返回
}
}
//右滑 进入视频首页
Swipe.back(driver);//点击返回
}
public static WebElement getElementById(String id , AndroidDriver driver){
List<WebElement> list = driver.findElements(By.id(id));
if (list != null && !list.isEmpty()){
return list.get(0);
}
return null;
}
//下滑进入用户主页
public static void upSelectUser(AndroidDriver driver) throws InterruptedException {
//下滑两次
Swipe.swipeUp(driver);
/*WebElement arw = getElementById("arw", driver);
if (arw != null )
arw.click(); //进入用户主页*/
Swipe.swipeLeft(driver);
}
public static void selectUser(String shortId , AndroidDriver driver){
//右滑搜索
Swipe.swipeRight(driver);
WebElement a65 = driver.findElementById("a65");
a65.click();
//driver.pressKeyCode(8);
a65.sendKeys( shortId);
//搜索按钮点击
driver.findElementById("a67").click();
//用户头像 点击
List<WebElement> atk = driver.findElements(By.id("atk"));
if (atk != null && !atk.isEmpty()){
atk.get(0).click();
}else{
//头像点击
atk = driver.findElements(By.id("arw"));
if ( atk != null && ! atk.isEmpty())
atk.get(0).click();
}
}These are part of the package the next good script can be run directly
Resolve fiddler to get the file DouYinVideoUtl class in the project
Project files are douyin.zip
Database file: douyin.sql
It can also automatically search ah various functions we can study his run directly run the main main idea to start springboot
Packaged into a jar by running java -jar command
