APP reptiles (2) the little sister of pictures down down

APP reptiles (1) want to learn a new language, and no power, how do?

Book connected to the text. Use appium draw action on the list of posts analog interface, capturing less than the law of the list. After the draw, the list can only get part of the screen to appear, and the subscript is zero-based.

According to father many years experience in yogurt code code, the post is certainly not a one-time loading is complete, and there will be paging, on-drag operation to somewhere APP will request API interface. Then we listen APP network requests can not get to post content yet!

Under charles mac using this software.

charles Profile

Charles configured network monitoring stumbling block encountered after requesting APP is https protocol, charles listening to the messages are garbled. I'm on Google in Baidu a bit.

Charles Mac use for HTTPS packet capture

Finally able to see the plaintext network requests, so the law can be summed up by a network message page loaded in charles request. Such yogurt Dad can write your own program, the cycle call API request to fetch the data. Here I encountered stumbling block (Why should I add "and" word). API request parameters do signature verification, each caught a request from charles, aged only 5 minutes.

If the crawler can mimic user behavior as much as possible, then we will break all the services side of the barrier.

Use appium zoned time, listening to the network using charles request a virtual machine on post details page every 3 seconds simulated user. This will be a steady stream of charles capture API request message paging. Here I encountered stumbling block (I do not like "bar", but I like "Land Rover"). And the little sisters of the photo or in the charles years, but also are the url string.

Charles using auto-save feature, the timing storage API response results. Charles menu bar in the "Tools" -> "Auto Save". Under the configuration according to FIG., "Save Type" select "HTTP Archive". Once configured, charles crawl to the API response will be saved as a text file format HAR.

HAR format specification

到这里总算是拨开云雾见青天,har文件记录是每次API请求的全过程,有url、参数、响应时间和返回值等等,而且还是标准的json格式。我们可以用新学的go语言写一个控制台程序来解析har文件,并将解析后的结果整理归类,帖子内容存入MySQL数据库,小姐姐们的图片curl下载到文件系统。

新语言的学习并不都是新知识的学习,更多的是编程习惯的迁移。

酸奶爸爸本来是做PHP开发的,现在用GO语言写这个,本次迁移的技能如下:

  • 日志记录
  • MySQL数据库读写
  • 断点调试
  • 文本文件读取
  • json解析
  • 配置文件的解析
  • curl下载文件

所以新语言的学习,第一步只要掌握语法,后续其实就是探索新语言所引入的新特性而已,以go语言为例,它对高并发有独到的设计。至于我们在旧语言所掌握的一些通用技能,一并迁移过来就好。go语言如何连接redis、RabbitMQ,这等通用的功能无非就是用go语言的方式引用一下包而已。

所以你看,离开舒适区其实也挺容易的吧。

总结,这里才是干货

  • 学习新技能要给自己设立一个目标作为奖励,遇到困难的时候,想想我们的达成目标后的奖励,是不是又有动力了。哪怕仅仅以看视频教程的方式学新语言,也坚持不了多久。
  • 我们掌握的每一项技能都是一个工具,遇到问题的时候,回顾一下我们的工具箱,将其中某几种工具组合起来就能解决问题。所以平时要多听多看多思考,这样我们才能向工具箱里装足够多的工具。遇到问题的时候,才有足够多的工具以供组装起来解决问题。
  • 旧语言与新语言重叠的功能,这些是时间积淀的经典,需要夯实;新语言的新特性需要格外关注,无论是找工作还是装13都用得到;旧语言的旧功能,就让它随风而去吧。

Guess you like

Origin www.cnblogs.com/lts8989/p/11239398.html