任务1：论文数据统计
1.1 任务说明
- 任务主题：论⽂文数量量统计，即统计2019年全年计算机各个⽅向论⽂文数量；
- 任务内容：赛题的理解、使用 Pandas 读取数据并进行统计；
- 任务成果：学习 Pandas 的基础操作；
- 可参考的学习料：开源组织Datawhale joyful-pandas项目
1.2 数据集介绍
- 数据集来源：数据集链接
- 数据集的格式如下：
- id ：arXiv ID，可⽤用于访问论⽂文；
- submitter ：论⽂文提交者；
- authors ：论⽂文作者；
- title ：论⽂文标题；
- comments ：论⽂文⻚页数和图表等其他信息；
- journal-ref ：论⽂文发表的期刊的信息；
- doi ：数字对象标识符，https://www.doi.org；
- report-no ：报告编号；
- categories ：论⽂文在 arXiv 系统的所属类别或标签；
- license ：⽂文章的许可证；
- abstract ：论⽂文摘要；
- versions ：论⽂文版本；
- authors_parsed ：作者的信息。
- 数据集实例例：
"""
"root":{
"id":string"0704.0001"
"submitter":string"Pavel Nadolsky"
"authors":string"C. Bal\'azs, E. L. Berger, P. M. Nadolsky, C.-P. Yuan"
"title":string"Calculation of prompt diphoton production cross sections at
Tevatron and LHC energies"
"comments":string"37 pages, 15 figures; published version"
"journal-ref":string"Phys.Rev.D76:013009,2007"
"doi":string"10.1103/PhysRevD.76.013009"
"report-no":string"ANL-HEP-PR-07-12"
"categories":string"hep-ph"
"license":NULL
"abstract":string" A fully differential calculation in perturbative
quantum chromodynamics is presented for the production of massive photon pairs
at hadron colliders. All next-to-leading order perturbative contributions from
quark-antiquark, gluon-(anti)quark, and gluon-gluon subprocesses are included,
as well as all-orders resummation of initial-state gluon radiation valid at
next-to-next-to leading logarithmic accuracy. The region of phase space is
specified in which the calculation is most reliable. Good agreement is
demonstrated with data from the Fermilab Tevatron, and predictions are made for
more detailed tests with CDF and DO data. Predictions are shown for
distributions of diphoton pairs produced at the energy of the Large Hadron
Collider (LHC). Distributions of the diphoton pairs from the decay of a Higgs
boson are contrasted with those produced from QCD processes at the LHC, showing
that enhanced sensitivity to the signal can be obtained with judicious
selection of events."
"versions":[
0:{
"version":string"v1"
"created":string"Mon, 2 Apr 2007 19:18:42 GMT"
}
1:{
"version":string"v2"
"created":string"Tue, 24 Jul 2007 20:10:27 GMT"
}]
"update_date":string"2008-11-26"
"authors_parsed":[
0:[
0:string"Balázs"
1:string"C."
2:string""]
1:[
0:string"Berger"
1:string"E. L."
2:string""]
2:[
0:string"Nadolsky"
1:string"P. M."
2:string""]
3:[
0:string"Yuan"
1:string"C. -P."
2:string""]]
"""
'\n"root":{\n"id":string"0704.0001"\n"submitter":string"Pavel Nadolsky"\n"authors":string"C. Bal\'azs, E. L. Berger, P. M. Nadolsky, C.-P. Yuan"\n"title":string"Calculation of prompt diphoton production cross sections at\nTevatron and LHC energies"\n"comments":string"37 pages, 15 figures; published version"\n"journal-ref":string"Phys.Rev.D76:013009,2007"\n"doi":string"10.1103/PhysRevD.76.013009"\n"report-no":string"ANL-HEP-PR-07-12"\n"categories":string"hep-ph"\n"license":NULL\n"abstract":string" A fully differential calculation in perturbative\nquantum chromodynamics is presented for the production of massive photon pairs\nat hadron colliders. All next-to-leading order perturbative contributions from\nquark-antiquark, gluon-(anti)quark, and gluon-gluon subprocesses are included,\nas well as all-orders resummation of initial-state gluon radiation valid at\nnext-to-next-to leading logarithmic accuracy. The region of phase space is\nspecified in which the calculation is most reliable. Good agreement is\ndemonstrated with data from the Fermilab Tevatron, and predictions are made for\nmore detailed tests with CDF and DO data. Predictions are shown for\ndistributions of diphoton pairs produced at the energy of the Large Hadron\nCollider (LHC). Distributions of the diphoton pairs from the decay of a Higgs\nboson are contrasted with those produced from QCD processes at the LHC, showing\nthat enhanced sensitivity to the signal can be obtained with judicious\nselection of events."\n"versions":[\n0:{\n"version":string"v1"\n"created":string"Mon, 2 Apr 2007 19:18:42 GMT"\n}\n1:{\n"version":string"v2"\n"created":string"Tue, 24 Jul 2007 20:10:27 GMT"\n}]\n"update_date":string"2008-11-26"\n"authors_parsed":[\n0:[\n0:string"Balázs"\n1:string"C."\n2:string""]\n1:[\n0:string"Berger"\n1:string"E. L."\n2:string""]\n2:[\n0:string"Nadolsky"\n1:string"P. M."\n2:string""]\n3:[\n0:string"Yuan"\n1:string"C. -P."\n2:string""]]\n'
1.3 arxiv论⽂文类别介绍
我们从arxiv官⽹网，查询到论⽂文的类别名称以及其解释如下。
链接：https://arxiv.org/help/api/user-manual 的 5.3 ⼩小节的 Subject Classifications 的部分，或
https://arxiv.org/category_taxonomy， 具体的153种paper的类别部分如下：
''''
astro-ph': 'Astrophysics',
'astro-ph.CO': 'Cosmology and Nongalactic Astrophysics',
'astro-ph.EP': 'Earth and Planetary Astrophysics',
'astro-ph.GA': 'Astrophysics of Galaxies',
'cs.AI': 'Artificial Intelligence',
'cs.AR': 'Hardware Architecture',
'cs.CC': 'Computational Complexity',
'cs.CE': 'Computational Engineering, Finance, and Science',
'cs.CV': 'Computer Vision and Pattern Recognition',
'cs.CY': 'Computers and Society',
'cs.DB': 'Databases',
'cs.DC': 'Distributed, Parallel, and Cluster Computing',
'cs.DL': 'Digital Libraries',
'cs.NA': 'Numerical Analysis',
'cs.NE': 'Neural and Evolutionary Computing',
'cs.NI': 'Networking and Internet Architecture',
'cs.OH': 'Other Computer Science',
'cs.OS': 'Operating Systems',
'''
"'\nastro-ph': 'Astrophysics',\n'astro-ph.CO': 'Cosmology and Nongalactic Astrophysics',\n'astro-ph.EP': 'Earth and Planetary Astrophysics',\n'astro-ph.GA': 'Astrophysics of Galaxies',\n'cs.AI': 'Artificial Intelligence',\n'cs.AR': 'Hardware Architecture',\n'cs.CC': 'Computational Complexity',\n'cs.CE': 'Computational Engineering, Finance, and Science',\n'cs.CV': 'Computer Vision and Pattern Recognition',\n'cs.CY': 'Computers and Society',\n'cs.DB': 'Databases',\n'cs.DC': 'Distributed, Parallel, and Cluster Computing',\n'cs.DL': 'Digital Libraries',\n'cs.NA': 'Numerical Analysis',\n'cs.NE': 'Neural and Evolutionary Computing',\n'cs.NI': 'Networking and Internet Architecture',\n'cs.OH': 'Other Computer Science',\n'cs.OS': 'Operating Systems',\n"
1.4 具体代码实现以及讲解
1.4.1 导⼊入package并读取原始数据
# 导⼊入所需的package
import seaborn as sns #用于画图
from bs4 import BeautifulSoup # 爬取数据
import re #用于正则表达式，匹配字符串模式
import requests #用于网络链接，发送网络请求，使用域名获取对应信息
import json #读取数据，
import pandas as pd
import matplotlib.pyplot as plt #画图工具
# 读取数据
data = [] # 初始化
#使⽤用with语句句优势：1.⾃自动关闭⽂文件句句柄；2.⾃自动显示（处理理）⽂文件读取数据异常
with open('arxiv-metadata-oai-2019.json/arxiv-metadata-oai-2019.json', 'r') as f:
for line in f:
data.append(json.loads(line))
data = pd.DataFrame(data)#将list变为dataframe格式，⽅方便便使⽤用pandas进⾏行行分析
data.shape #显示数据大小
(170618, 14)
#其中的1778381表示数据总量量，14表示特征数，对应我们1.2节说明的论⽂文的14种信息。
1.4.2 数据预处理理
首先我们先来粗略略统计论⽂文的种类信息：
'''
count：⼀一列列数据的元素个数；
unique：⼀一列列数据中元素的种类；
top：⼀一列列数据中出现频率最⾼高的元素；
freq：⼀一列列数据中出现频率最⾼高的元素的个数；
'''
data['categories'].describe()
count 170618
unique 15592
top cs.CV
freq 5559
Name: categories, dtype: object
以上的结果表面:共有170618个数据，有15592个子类（因为有论⽂文的类别是多个，例例如⼀一篇paper的类别是CS.AI & CS.MM和⼀一篇paper的类别是CS.AI & CS.OS属于不不同的⼦子类别，这⾥里里仅仅是粗略略统计）
其中出现频率最高的是cs.CV
由于部分论⽂文的类别不不⽌止⼀一种，所以下⾯面我们判断在本数据集中共出现了了多少种独⽴立的数据集。
# 所有的种类(独⽴立的)
unique_categories = set([i for l in [x.split(' ') for x in data['categories']] for i in l])
len(unique_categories)
unique_categories
#这⾥里里使⽤用了了 split 函数将多类别使⽤用 “ ”（空格）分开，组成list，并使⽤用 for 循环将独⽴立出现的类别找出
#来，并使⽤用 set 类别，将重复项去除得到最终所有的独⽴立paper种类。
{'acc-phys',
'adap-org',
'alg-geom',
'astro-ph',
'astro-ph.CO',
'astro-ph.EP',
'astro-ph.GA',
'astro-ph.HE',
'astro-ph.IM',
'astro-ph.SR',
'chao-dyn',
'chem-ph',
'cmp-lg',
'comp-gas',
'cond-mat',
'cond-mat.dis-nn',
'cond-mat.mes-hall',
'cond-mat.mtrl-sci',
'cond-mat.other',
'cond-mat.quant-gas',
'cond-mat.soft',
'cond-mat.stat-mech',
'cond-mat.str-el',
'cond-mat.supr-con',
'cs.AI',
'cs.AR',
'cs.CC',
'cs.CE',
'cs.CG',
'cs.CL',
'cs.CR',
'cs.CV',
'cs.CY',
'cs.DB',
'cs.DC',
'cs.DL',
'cs.DM',
'cs.DS',
'cs.ET',
'cs.FL',
'cs.GL',
'cs.GR',
'cs.GT',
'cs.HC',
'cs.IR',
'cs.IT',
'cs.LG',
'cs.LO',
'cs.MA',
'cs.MM',
'cs.MS',
'cs.NA',
'cs.NE',
'cs.NI',
'cs.OH',
'cs.OS',
'cs.PF',
'cs.PL',
'cs.RO',
'cs.SC',
'cs.SD',
'cs.SE',
'cs.SI',
'cs.SY',
'dg-ga',
'econ.EM',
'econ.GN',
'econ.TH',
'eess.AS',
'eess.IV',
'eess.SP',
'eess.SY',
'funct-an',
'gr-qc',
'hep-ex',
'hep-lat',
'hep-ph',
'hep-th',
'math-ph',
'math.AC',
'math.AG',
'math.AP',
'math.AT',
'math.CA',
'math.CO',
'math.CT',
'math.CV',
'math.DG',
'math.DS',
'math.FA',
'math.GM',
'math.GN',
'math.GR',
'math.GT',
'math.HO',
'math.IT',
'math.KT',
'math.LO',
'math.MG',
'math.MP',
'math.NA',
'math.NT',
'math.OA',
'math.OC',
'math.PR',
'math.QA',
'math.RA',
'math.RT',
'math.SG',
'math.SP',
'math.ST',
'mtrl-th',
'nlin.AO',
'nlin.CD',
'nlin.CG',
'nlin.PS',
'nlin.SI',
'nucl-ex',
'nucl-th',
'patt-sol',
'physics.acc-ph',
'physics.ao-ph',
'physics.app-ph',
'physics.atm-clus',
'physics.atom-ph',
'physics.bio-ph',
'physics.chem-ph',
'physics.class-ph',
'physics.comp-ph',
'physics.data-an',
'physics.ed-ph',
'physics.flu-dyn',
'physics.gen-ph',
'physics.geo-ph',
'physics.hist-ph',
'physics.ins-det',
'physics.med-ph',
'physics.optics',
'physics.plasm-ph',
'physics.pop-ph',
'physics.soc-ph',
'physics.space-ph',
'q-alg',
'q-bio',
'q-bio.BM',
'q-bio.CB',
'q-bio.GN',
'q-bio.MN',
'q-bio.NC',
'q-bio.OT',
'q-bio.PE',
'q-bio.QM',
'q-bio.SC',
'q-bio.TO',
'q-fin.CP',
'q-fin.EC',
'q-fin.GN',
'q-fin.MF',
'q-fin.PM',
'q-fin.PR',
'q-fin.RM',
'q-fin.ST',
'q-fin.TR',
'quant-ph',
'solv-int',
'stat.AP',
'stat.CO',
'stat.ME',
'stat.ML',
'stat.OT',
'stat.TH',
'supr-con'}
print(len(unique_categories))
172
从以上结果发现，共有176种论⽂文种类，⽐比我们直接从 https://arxiv.org/help/api/user-manual 的 5.3
⼩小节的 Subject Classifications 的部分或 https://arxiv.org/category_taxonomy中得到的类别少，这说
明存在⼀一些官⽹网上没有的类别，这是⼀一个⼩小细节。不不过对于我们的计算机⽅方向的论⽂文没有影响，依然是
以下的40个类别，我们从原数据中提取的和从官⽹网的到的种类是可以⼀一⼀一对应的。
'''
'cs.AI': 'Artificial Intelligence',
'cs.AR': 'Hardware Architecture',
'cs.CC': 'Computational Complexity',
'cs.CE': 'Computational Engineering, Finance, and Science',
'cs.CG': 'Computational Geometry',
'cs.CL': 'Computation and Language',
'cs.CR': 'Cryptography and Security',
'cs.CV': 'Computer Vision and Pattern Recognition',
'cs.CY': 'Computers and Society',
'cs.DB': 'Databases',
'cs.DC': 'Distributed, Parallel, and Cluster Computing',
'cs.DL': 'Digital Libraries',
'cs.DM': 'Discrete Mathematics',
'cs.DS': 'Data Structures and Algorithms',
'cs.ET': 'Emerging Technologies',
'cs.FL': 'Formal Languages and Automata Theory',
'cs.GL': 'General Literature',
'cs.GR': 'Graphics',
'cs.GT': 'Computer Science and Game Theory',
'cs.HC': 'Human-Computer Interaction',
'cs.IR': 'Information Retrieval',
'cs.IT': 'Information Theory',
'cs.LG': 'Machine Learning',
'cs.LO': 'Logic in Computer Science',
'cs.MA': 'Multiagent Systems',
'cs.MM': 'Multimedia',
'cs.MS': 'Mathematical Software',
'cs.NA': 'Numerical Analysis',
'cs.NE': 'Neural and Evolutionary Computing',
'cs.NI': 'Networking and Internet Architecture',
'cs.OH': 'Other Computer Science',
'cs.OS': 'Operating Systems',
'cs.PF': 'Performance',
'cs.PL': 'Programming Languages',
'cs.RO': 'Robotics',
'cs.SC': 'Symbolic Computation',
'cs.SD': 'Sound',
'cs.SE': 'Software Engineering',
'cs.SI': 'Social and Information Networks',
'cs.SY': 'Systems and Control',
'''
"\n'cs.AI': 'Artificial Intelligence',\n'cs.AR': 'Hardware Architecture',\n'cs.CC': 'Computational Complexity',\n'cs.CE': 'Computational Engineering, Finance, and Science',\n'cs.CG': 'Computational Geometry',\n'cs.CL': 'Computation and Language',\n'cs.CR': 'Cryptography and Security',\n'cs.CV': 'Computer Vision and Pattern Recognition',\n'cs.CY': 'Computers and Society',\n'cs.DB': 'Databases',\n'cs.DC': 'Distributed, Parallel, and Cluster Computing',\n'cs.DL': 'Digital Libraries',\n'cs.DM': 'Discrete Mathematics',\n'cs.DS': 'Data Structures and Algorithms',\n'cs.ET': 'Emerging Technologies',\n'cs.FL': 'Formal Languages and Automata Theory',\n'cs.GL': 'General Literature',\n'cs.GR': 'Graphics',\n'cs.GT': 'Computer Science and Game Theory',\n'cs.HC': 'Human-Computer Interaction',\n'cs.IR': 'Information Retrieval',\n'cs.IT': 'Information Theory',\n'cs.LG': 'Machine Learning',\n'cs.LO': 'Logic in Computer Science',\n'cs.MA': 'Multiagent Systems',\n'cs.MM': 'Multimedia',\n'cs.MS': 'Mathematical Software',\n'cs.NA': 'Numerical Analysis',\n'cs.NE': 'Neural and Evolutionary Computing',\n'cs.NI': 'Networking and Internet Architecture',\n'cs.OH': 'Other Computer Science',\n'cs.OS': 'Operating Systems',\n'cs.PF': 'Performance',\n'cs.PL': 'Programming Languages',\n'cs.RO': 'Robotics',\n'cs.SC': 'Symbolic Computation',\n'cs.SD': 'Sound',\n'cs.SE': 'Software Engineering',\n'cs.SI': 'Social and Information Networks',\n'cs.SY': 'Systems and Control',\n"
我们的任务要求对于2019年年以后的paper进⾏行行分析，所以⾸首先对于时间特征进⾏行行预处理理，从⽽而得到2019
年年以后的所有种类的论⽂文：
data['year'] = pd.to_datetime(data['update_date']).dt.year
#将update_date从例例如2019-02-20的str变为datetime格式，并提取处year
del data['update_date'] #删除这一列得特征
data = data[data['year']>=2019]#找出 year 中2019年年以后的数据，并将其他数据删除
# data.groupby(['categories','year']) #以 categories 进⾏行行排序，如果同⼀一个categories
#相同则使⽤用 year 特征进⾏行行排序
data.reset_index(drop = True, inplace = True)#重新编号，想保留原来的index，使用参数 drop=True,修改原有数据 采用inplace = True
data # 查看结果
| id | submitter | authors | title | comments | journal-ref | doi | report-no | categories | license | abstract | versions | authors_parsed | year | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 0704.0297 | Sung-Chul Yoon | Sung-Chul Yoon, Philipp Podsiadlowski and Step... | Remnant evolution after a carbon-oxygen white ... | 15 pages, 15 figures, 3 tables, submitted to M... | None | 10.1111/j.1365-2966.2007.12161.x | None | astro-ph | None | We systematically explore the evolution of t... | [{'version': 'v1', 'created': 'Tue, 3 Apr 2007... | [[Yoon, Sung-Chul, ], [Podsiadlowski, Philipp,... | 2019 |
| 1 | 0704.0342 | Patrice Ntumba Pungu | B. Dugmore and PP. Ntumba | Cofibrations in the Category of Frolicher Spac... | 27 pages | None | None | None | math.AT | None | Cofibrations are defined in the category of ... | [{'version': 'v1', 'created': 'Tue, 3 Apr 2007... | [[Dugmore, B., ], [Ntumba, PP., ]] | 2019 |
| 2 | 0704.0360 | Zaqarashvili | T.V. Zaqarashvili and K Murawski | Torsional oscillations of longitudinally inhom... | 6 pages, 3 figures, accepted in A&A | None | 10.1051/0004-6361:20077246 | None | astro-ph | None | We explore the effect of an inhomogeneous ma... | [{'version': 'v1', 'created': 'Tue, 3 Apr 2007... | [[Zaqarashvili, T. V., ], [Murawski, K, ]] | 2019 |
| 3 | 0704.0525 | Sezgin Ayg\"un | Sezgin Aygun, Ismail Tarhan, Husnu Baysal | On the Energy-Momentum Problem in Static Einst... | This submission has been withdrawn by arXiv ad... | Chin.Phys.Lett.24:355-358,2007 | 10.1088/0256-307X/24/2/015 | None | gr-qc | None | This paper has been removed by arXiv adminis... | [{'version': 'v1', 'created': 'Wed, 4 Apr 2007... | [[Aygun, Sezgin, ], [Tarhan, Ismail, ], [Baysa... | 2019 |
| 4 | 0704.0535 | Antonio Pipino | Antonio Pipino (1,3), Thomas H. Puzia (2,4), a... | The Formation of Globular Cluster Systems in M... | 32 pages (referee format), 9 figures, ApJ acce... | Astrophys.J.665:295-305,2007 | 10.1086/519546 | None | astro-ph | None | The most massive elliptical galaxies show a ... | [{'version': 'v1', 'created': 'Wed, 4 Apr 2007... | [[Pipino, Antonio, ], [Puzia, Thomas H., ], [M... | 2019 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |
| 170613 | quant-ph/9904032 | Mikhail Lukin | V. A. Sautenkov, M. D. Lukin, C. J. Bednar, G.... | Enhancement of Magneto-Optic Effects via Large... | None | None | 10.1103/PhysRevA.62.023810 | None | quant-ph | None | We utilize the generation of large atomic co... | [{'version': 'v1', 'created': 'Thu, 8 Apr 1999... | [[Sautenkov, V. A., ], [Lukin, M. D., ], [Bedn... | 2019 |
| 170614 | solv-int/9511005 | Wen-Xiu Ma | Wen-Xiu Ma, Benno Fuchssteiner | Explicit and Exact Solutions to a Kolmogorov-P... | 14pages, Latex, to appear in Intern. J. Nonlin... | None | 10.1016/0020-7462(95)00064-X | None | solv-int nlin.SI | None | Some explicit traveling wave solutions to a ... | [{'version': 'v1', 'created': 'Tue, 14 Nov 199... | [[Ma, Wen-Xiu, ], [Fuchssteiner, Benno, ]] | 2019 |
| 170615 | solv-int/9809008 | Victor Enolskii | J C Eilbeck, V Z Enol'skii, V B Kuznetsov, D V... | Linear r-Matrix Algebra for a Hierarchy of One... | plain LaTeX, 28 pages | None | None | None | solv-int nlin.SI | None | We consider a hierarchy of many-particle sys... | [{'version': 'v1', 'created': 'Wed, 2 Sep 1998... | [[Eilbeck, J C, ], [Enol'skii, V Z, ], [Kuznet... | 2019 |
| 170616 | solv-int/9909010 | Pierre van Moerbeke | M. Adler, T. Shiota and P. van Moerbeke | Pfaff tau-functions | 42 pages | None | None | None | solv-int adap-org hep-th nlin.AO nlin.SI | None | Consider the evolution $$ \frac{\pl m_\iy}{\... | [{'version': 'v1', 'created': 'Wed, 15 Sep 199... | [[Adler, M., ], [Shiota, T., ], [van Moerbeke,... | 2019 |
| 170617 | solv-int/9909014 | David Fairlie | D.B. Fairlie and A.N. Leznov | The General Solution of the Complex Monge-Amp\... | 13 pages, latex, no figures | None | 10.1088/0305-4470/33/25/307 | None | solv-int nlin.SI | None | A general solution to the Complex Monge-Amp\... | [{'version': 'v1', 'created': 'Thu, 16 Sep 199... | [[Fairlie, D. B., ], [Leznov, A. N., ]] | 2019 |
170618 rows × 14 columns
# 这里我们就已经得到了了所有2019年年以后的论⽂文，下⾯面我们挑选出计算机领域内的所有⽂文章：
#爬取所有的类别
website_url = requests.get('https://arxiv.org/category_taxonomy').text # 获取网页的文本数据
soup = BeautifulSoup(website_url, 'lxml')#爬取数据，这⾥里里使⽤用lxml的解析器器，加速
root = soup.find('div', {
'id':'category_taxonomy_list'}) #找出 BeautifulSoup 对应的标签⼊⼝
tags = root.find_all(['h2', 'h3', 'h4', 'p'], recursive = True)#读取 tags
#初始化 str和list变量
level_1_name = ''
level_2_name = ''
level_2_code = ''
level_1_names = []
level_2_names = []
level_2_codes = []
level_3_names = []
level_3_codes = []
level_3_notes = []
#进⾏
for t in tags:
if t.name =='h2':
level_1_name = t.text
level_2_code = t.text
level_2_name = t.text
elif t.name =='h3':
raw = t.text
level_2_code = re.sub(r"(.*)\((.*)\)",r"\2",raw)#正则表达式：模式字符串串：(.*)\((.*)\)；被替换字符串串"\2"；被处理理字符串串：raw
level_2_name = re.sub(r"(.*)\((.*)\)",r"\1",raw)
elif t.name == "h4":
raw = t.text
level_3_code = re.sub(r"(.*) \((.*)\)",r"\1",raw)
level_3_name = re.sub(r"(.*) \((.*)\)",r"\2",raw)
elif t.name == "p":
notes = t.text
level_1_names.append(level_1_name)
level_2_names.append(level_2_name)
level_2_codes.append(level_2_code)
level_3_names.append(level_3_name)
level_3_codes.append(level_3_code)
level_3_notes.append(notes)
#根据以上信息⽣生成dataframe格式的数据
df_taxonomy = pd.DataFrame({
'group_name' : level_1_names,
'archive_name' : level_2_names,
'archive_id' : level_2_codes,
'category_name' : level_3_names,
'categories' : level_3_codes,
'category_description': level_3_notes
})
#按照 "group_name" 进⾏行行分组，在组内使⽤用 "archive_name" 进⾏行行排序
df_taxonomy.groupby(["group_name","archive_name"])
df_taxonomy
| group_name | archive_name | archive_id | category_name | categories | category_description | |
|---|---|---|---|---|---|---|
| 0 | Computer Science | Computer Science | Computer Science | Artificial Intelligence | cs.AI | Covers all areas of AI except Vision, Robotics... |
| 1 | Computer Science | Computer Science | Computer Science | Hardware Architecture | cs.AR | Covers systems organization and hardware archi... |
| 2 | Computer Science | Computer Science | Computer Science | Computational Complexity | cs.CC | Covers models of computation, complexity class... |
| 3 | Computer Science | Computer Science | Computer Science | Computational Engineering, Finance, and Science | cs.CE | Covers applications of computer science to the... |
| 4 | Computer Science | Computer Science | Computer Science | Computational Geometry | cs.CG | Roughly includes material in ACM Subject Class... |
| ... | ... | ... | ... | ... | ... | ... |
| 150 | Statistics | Statistics | Statistics | Computation | stat.CO | Algorithms, Simulation, Visualization |
| 151 | Statistics | Statistics | Statistics | Methodology | stat.ME | Design, Surveys, Model Selection, Multiple Tes... |
| 152 | Statistics | Statistics | Statistics | Machine Learning | stat.ML | Covers machine learning papers (supervised, un... |
| 153 | Statistics | Statistics | Statistics | Other Statistics | stat.OT | Work in statistics that does not fit into the ... |
| 154 | Statistics | Statistics | Statistics | Statistics Theory | stat.TH | stat.TH is an alias for math.ST. Asymptotics, ... |
155 rows × 6 columns
这⾥里里主要说明⼀一下上⾯面代码中的正则操作，这⾥里里我们使⽤用re.sub来⽤用于替换字符串串中的匹配项
'''
pattern : 正则中的模式字符串串。
repl : 替换的字符串串，也可为⼀一个函数。
string : 要被查找替换的原始字符串串。
count : 模式匹配后替换的最⼤大次数，默认 0 表示替换所有的匹配。
flags : 编译时⽤用的匹配模式，数字形式。
其中pattern、repl、string为必选参数
'''
# re.sub(pattern, repl, string, count=0, flags=0)
'\npattern : 正则中的模式字符串串。\nrepl : 替换的字符串串，也可为⼀一个函数。\nstring : 要被查找替换的原始字符串串。\ncount : 模式匹配后替换的最⼤大次数，默认 0 表示替换所有的匹配。\nflags : 编译时⽤用的匹配模式，数字形式。\n其中pattern、repl、string为必选参数\n'
# 实例如下
import re
phone = "2004-959-559 # 这是⼀一个电话号码"
# 删除注释
num = re.sub(r'#.*$', "", phone)
print ("电话号码 : ", num)
# 移除⾮非数字的内容
num = re.sub(r'\D', "", phone)
print ("电话号码 : ", num)
电话号码 : 2004-959-559
电话号码 : 2004959559
详细了了解可以参考：https://www.runoob.com/python3/python3-reg-expressions.html
#对于我门的代码来讲
# re.sub(r"(.*)\((.*)\)",r"\2",raw)
#raw = Astrophysics(astro-ph)
#output = astro-ph
对应的参数
-正则中的模式字符串串 pattern 的格式为 “任意字符” + “(” + “任意字符” + “)”。
- 替换的字符串串 repl 为第2个分组的内容。
- 要被查找替换的原始字符串串 string 为原始的爬取的数据。
推荐⼤大家⼀一个在线正则表达式测试的⽹网站：https://tool.oschina.net/regex/
1.4.3 数据分析及可视化
接下来我们首先看⼀一下所有大类的paper数量分布：
_df = data.merge(df_taxonomy, on="categories",
how="left").drop_duplicates(["id","group_name"]).groupby("group_name").agg({
"id"
:'count'}).sort_values(by="id",ascending=False).reset_index()
_df
| group_name | id | |
|---|---|---|
| 0 | Physics | 38379 |
| 1 | Mathematics | 24495 |
| 2 | Computer Science | 18087 |
| 3 | Statistics | 1802 |
| 4 | Electrical Engineering and Systems Science | 1371 |
| 5 | Quantitative Biology | 886 |
| 6 | Quantitative Finance | 352 |
| 7 | Economics | 173 |
我们使⽤用merge函数，以两个dataframe共同的属性 “categories” 进行合并，并以 “group_name” 作为
类别进行统计，统计结果放⼊入 “id” 列列中并排序。
# 下⾯面我们使⽤用饼图进⾏行行上图结果的可视化：
fig = plt.figure(figsize = (15, 12))
explode = (0, 0, 0, 0.2, 0.3, 0.3, 0.2, 0.1)
plt.pie(_df['id'], labels=_df['group_name'], autopct='%1.2f%%', startangle=180, explode= explode)
plt.tight_layout()
plt.show()
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-lpzBTTdQ-1610444557338)(output_29_0.png)]
下⾯面统计在计算机各个⼦子领域2019年年后的paper数量量：
group_name = 'Computer Science'
cats = data.merge(df_taxonomy, on="categories").query("group_name ==@group_name")
cats.groupby(["year","category_name"]).count().reset_index().pivot(index="category_name",
columns="year",values="id")
# 我们同样使⽤用 merge 函数，对于两个dataframe 共同的特征 categories 进⾏行行合并并且进⾏行行查询。然后
#我们再对于数据进⾏行行统计和排序从⽽而得到以下的结果：
| year | 2019 |
|---|---|
| category_name | |
| Artificial Intelligence | 558 |
| Computation and Language | 2153 |
| Computational Complexity | 131 |
| Computational Engineering, Finance, and Science | 108 |
| Computational Geometry | 199 |
| Computer Science and Game Theory | 281 |
| Computer Vision and Pattern Recognition | 5559 |
| Computers and Society | 346 |
| Cryptography and Security | 1067 |
| Data Structures and Algorithms | 711 |
| Databases | 282 |
| Digital Libraries | 125 |
| Discrete Mathematics | 84 |
| Distributed, Parallel, and Cluster Computing | 715 |
| Emerging Technologies | 101 |
| Formal Languages and Automata Theory | 152 |
| General Literature | 5 |
| Graphics | 116 |
| Hardware Architecture | 95 |
| Human-Computer Interaction | 420 |
| Information Retrieval | 245 |
| Logic in Computer Science | 470 |
| Machine Learning | 177 |
| Mathematical Software | 27 |
| Multiagent Systems | 85 |
| Multimedia | 76 |
| Networking and Internet Architecture | 864 |
| Neural and Evolutionary Computing | 235 |
| Numerical Analysis | 40 |
| Operating Systems | 36 |
| Other Computer Science | 67 |
| Performance | 45 |
| Programming Languages | 268 |
| Robotics | 917 |
| Social and Information Networks | 202 |
| Software Engineering | 659 |
| Sound | 7 |
| Symbolic Computation | 44 |
| Systems and Control | 415 |
我们可以从结果看出，Computer Vision and Pattern Recognition（计算机视觉与模式识别）类是CS中
paper数量量最多的⼦子类，遥遥领先于其他的CS⼦子类，并且paper的数量量还在逐年年增加；另外，
Computation and Language（计算与语⾔言）、Cryptography and Security（密码学与安全）以及
Robotics（机器器⼈人学）的2019年年paper数量量均超过1000或接近1000，这与我们的认知是⼀一致的。
data.drop_duplicates?