Based on the Internet worry-free future employment data
by wooden Ye

This article Department of Public elective course "Introduction to Data Science" large operations, as a sophomore non-computer student, and this job is a complete exam week, time is tight, if not the rigorous, but also look understanding.

This article will briefly, crawling data, data cleaning, data visualization analysis, results are presented to show five aspects.

A brief introduction

Background to the issue
of the Internet industry as currently accepted high-paying jobs, internet jobs also have a large market demand, many graduates enter the job each year Internet industry
issues elaborated
more on what the city needs Internet career salary? Salary offered more attractive?

Which programming language to master an advantage in employment, a higher salary?

Education, work experience for employment compensation they have the kind of impact?
Starting from these practical problems, Analysis of Internet employment.

Data collection: from 51job.com fetch data were analyzed statistically.

The main use: scrapy, regular expressions, SVM and random forest predict, Kmeans clustering, BIRCH clustering, word cloud, visualization methods

II. Data crawling

Use crawlers to collect relevant information from 51job.com (worry-free future) website.
(URL link is: https: //search.51job.com/)

He has tried crawling a pull hook net, BOSS network and 51job.com in the three large domestic recruitment website.

After several rebuffed found BOSS pull hook web and the desired information is located, or because the js, or as anti-crawler crawler mechanism easily lead to too strong such that IP is closed, and 51job.com in trans crawler mechanism is much weaker than the pull hook BOSS net, the relative difficulty in terms of data collection, but because it can crawl data is not neat, so clean data requires more work.

Reptiles the following steps:

1. First of https://search.51job.com/ climb to get spider1.csv, because the information this site contains only a small amount, and the remaining information is hidden in each position corresponding recruitment of small pages. So start with this website crawling specific positions corresponding to the detailed description of the URL of the page

2. crawling with small pages, get spider2.csv, examples of sites such as: https:? //Jobs.51job.com/beijing/119317470.html s = 01 & t = 0

3. The two csv file merged data set derived data.csv

Crawling a hundred thousand data sets involving Beijing, Shanghai, Guangzhou, Shenzhen, Hangzhou, Chengdu, Chongqing, Harbin, Xi'an, Wuhan ten cities, are shown below:

III. Data Cleansing

1. Original crawling data have job (job title), web (site), company recruitment company, companyweb (details Company website), place (place of work), salary (wage compensation), time (Published), jtap ( entry, which contains the location, work experience, academic requirements, the number of recruits etc.), text (employee benefits introduced), key (keyword), duty (duties and job description)

2. Due to limitations of the original site, work experience, academic requirements, the number of recruits are all located jtap one, with a "|" separated, with regular expressions to extract keywords

3. The irregular original compensation data into a uniform format, and extract a number, Wan / year, yuan / day, thousand / month unified converted for the million / month

4. From the Job (job name), Key (keyword positions), Duty (description of the job needs) matching the required programming language, and stored in different columns

The deleting redundant web, companyweb other column, make some adjustments format

After finishing data are as follows , education for the academic requirements, money to pay (a unified format for the month / year), city to city, num is the number of recruits, exp for the work experience requirements, a total of 19 languages after the adjacency matrix, a representative of the Jobs there is a demand for such language.

Matching language there: 'c ++', 'java ', 'c #', 'python', 'go', 'php', 'matlab', 'swift', 'lua', 'perl', 'delphi', 'kotlin ',' ruby ',' typescript ',' vba ',' rust ',' haskell ',' visual basic ',' sql '

IV. Visual analysis

1. explore different cities the demand for professional Internet

As can be seen from the following histogram, Shanghai, Shenzhen, Guangzhou is the largest city in the Internet professional needs, compared with only 1/20 of Harbin and demand. These three cities in
big cities, there are indeed more opportunity

2. explore employment requirements for academic qualifications

It can be seen clearly requires very little doctorate, master's degree also explicitly require only about 3% of graduate will be able to step into the most professional.

And the following college, middle school, secondary technical secondary school in the Internet occupations than difficult.

3. Study of Employment Remuneration different cities of
Beijing highest average salary employment, followed by Shanghai, Shenzhen, Hangzhou, Guangzhou.

Harbin employment pay significantly lower than other cities.