版权声明：本文为博主原创文章，未经允许也可以任意转载。http://huyouxiao.com https://blog.csdn.net/fudaxing/article/details/88750133
在网上找书，无意间发现了一个网站，上面有很多电子书。试着打开了几本，感觉很不错，就下了下来。可惜我的人性太贪婪了，想把他们都下下来，哪怕放到磁盘上生灰。至少可以满足对知识的焦虑感。
怎么办？
在刚毕业那会，我会孜孜不倦的一个个打开，然后另存为。过程中会被其他人发现。在学python之前，我会用vim和shell去做这个事情。大致思路是:
- 把网页文件另存为一个txt文档，然后用 vim 命令编辑一顿骚操作搞到所有的url
- 然后再用 vim 一顿骚操作搞成一行一行的命令，会用到wget. 命令的形式: wget http://theWebSiite.com/thePdfFileUrl
- 然后给文件加上执行权限，运行就可以了。
最后的运行文件内容如下：
wget http://www.r-5.org/files/books/ethology/arms/Alexandr_Nikonov_Vladimir_Gagurin-Dont_Fear-RU.pdf
wget http://www.r-5.org/files/books/ethology/arms/Warrax-Arms_FAQ-RU.pdf
wget http://www.r-5.org/files/books/economics/distortion/A_N_Litvinenko_E_U_Kovaleva-Legalisation_of_Incomes-RU.pdf
wget http://www.r-5.org/files/books/economics/distortion/Alexey_Gladkiy-Fraud_and_Provocation_in_Business-RU.pdf
wget http://www.r-5.org/files/books/economics/distortion/Franz_Lautenschlager-Die_Groben_Fouls-RU.pdf
wget http://www.r-5.org/files/books/economics/distortion/Iliya_Danilin-Caching-Out-RU.pdf
wget http://www.r-5.org/files/books/economics/distortion/Marat_Musin-Rospil-RU.pdf
wget http://www.r-5.org/files/books/economics/distortion/N_Kiselev-Preventive_Protection_from_Corporate_Takeover-RU.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Al_Ries_Jack_Trout-The_22_Immutable_Laws_of_Marketing-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Barry_Maher-No_Lie_Truth_Is_The_Ultimate_Sales_Tool-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Brian_Tracy-Advanced_Selling_Strategies-RU.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Ford_Harding-Rain_Making-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Jim_Camp-Start_With_No-RU.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Sergey_Filippov-SalesUp-RU.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Steve_Schiffman-Cold_Calling_Techniques-RU.pdf
wget http://www.r-5.org/files/books/ethology/corporate/cold-calls/Tim_Connor-91_Mistakes_Smart_Salespeople_Make-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Carl_Menger-Principles_of_Economics-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Eugen_von_Bohm_Bawerk-Karl_Marx_And_The_Close_Of_His_System-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Frederic_Bastiat-The_Bastiat_Collection-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Friedrich_A_Hayek-Prices_and_Production-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Gene_Callahan-Economics_for_Real_People-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/George_Charles_Roche_III-Frederic_Bastiat_A_Man_Alone-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Henry_Hazlitt-Economics_in_One_Lesson-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Jeremie_Rostan-Study_Guide_to_Mengers_Principles-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Ludwig_von_Mises-Human_Action_A_Treatise_on_Economics-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Mark_Thornton-Frederic_Bastiats_Views_on_the_Nature_of_Money-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Mark_Thornton-The_Quotable_Mises-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Murray_N_Rothbard-Man_Economy_and_State-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Robert_Murphy-Study_Guide_to_Human_Action-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Robert_Murphy-Study_Guide_to_Man_Economy_and_State-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/Walter_Block-Defending_the_Undefendable-EN.pdf
wget http://www.r-5.org/files/books/economics/a-t/William_Graham_Sumner-The_Forgotten_Man-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Antony_Sutton-The_Federal_Reserve_Conspiracy-RU.pdf
wget http://www.r-5.org/files/books/economics/cycles/Henry_Hazlitt-What_You_Should_Know_About_Inflation-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Jesus_Huerta_De_Soto-Money_Bank_Credit_and_Economic_Cycles-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Jorg_Guido_Hulsmann-The_Ethics_of_Money_Production-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Ludwig_von_Mises-The_Causes_of_the_Economic_Crises-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Maxon-The_Problem_of_Printing_Machine-RU.pdf
wget http://www.r-5.org/files/books/economics/cycles/Murray_N_Rothbard-Economic_Depressions-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Murray_N_Rothbard-The_Case_Against_the_Fed-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Murray_N_Rothbard-What_Has_Government_Done_to_Our_Money-EN.pdf
wget http://www.r-5.org/files/books/economics/cycles/Patrick_Carmack-The_Money_Changers-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Elgin_Groseclose-Money_and_Man-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Friedrich_A_Hayek-Road_to_Serfdom-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Friedrich_A_Hayek-Road_to_Serfdom-RU.pdf
wget http://www.r-5.org/files/books/economics/goverments/Garet_Garrett-The_Blue_Wound-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/John_Denson-A_Century_of_War-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Ludwig_von_Mises-Bureaucracy-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Ludwig_von_Mises-Economic_Policy-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Murray_N_Rothbard-Anatomy_of_the_State-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Peter_L_Bernstein-A_Primer_on_Money_Banking_and_Gold-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Randolph_Bourne-The_State-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Robert_LeFevre-This_Bread_is_Mine-EN.pdf
wget http://www.r-5.org/files/books/economics/goverments/Ron_Paul-Pillars_of_Prosperity-EN.pdf
wget http://www.r-5.org/files/books/economics/historical/H_C_Engelbrecht_and_F_C_Hanighen-The_Merchants_of_Death-EN.pdf
wget http://www.r-5.org/files/books/economics/historical/Richard_Hildreth-The_History_of_Banks-EN.pdf
wget http://www.r-5.org/files/books/economics/structure/Morris_Silver-Economic_Structures_of_Antiquity-EN.pdf
wget http://www.r-5.org/files/books/economics/structure/Valentin_Katasonov-Stalin_Economy-RU.pdf
wget http://www.r-5.org/files/books/economics/structure/Valentin_Katasonov-The_History_of_Capitalism-RU.pdf
wget http://www.r-5.org/files/books/ethology/human-mind/Alexander_Nikolskiy-Logical_Fallacies-RU.pdf
wget http://www.r-5.org/files/books/ethology/human-mind/Boris_Sergeev-One_Brain_is_Good_but_Two_are_Better-RU.pdf
wget http://www.r-5.org/files/books/ethology/human-mind/Georgy_Chelpanov-The_Book_of_Logic-RU.pdf
wget http://www.r-5.org/files/books/ethology/human-mind/Humphrey_B_Neill-The_Art_of_Contrary_Thinking-EN.pdf
wget http://www.r-5.org/files/books/ethology/human-mind/Richard_Bornat-Proof_and_Disproof_in_Formal_Logic-EN.pdf
wget http://www.r-5.org/files/books/ethology/human-mind/Thomas_Gilovich-How_We_Know_What_Isnt_So-EN.pdf
wget http://www.r-5.org/files/books/ethology/human-mind/Victor_Gorbatov-The_Art_of_Logical_Thinking-RU.pdf
wget http://www.r-5.org/files/books/economics/statistics/D_Cox_Michael_Cox-The_Mathematics_of_Banking_and_Finance-EN.pdf
wget http://www.r-5.org/files/books/economics/statistics/Nigel_Da_Costa_Lewis-Operational_Risk-EN.pdf
wget http://www.r-5.org/files/books/economics/statistics/Philip_Hans_Franses_Richard_Paap-Marketing_Research-EN.pdf
wget http://www.r-5.org/files/books/economics/statistics/Philippe_Jorion-Financial_Risk_Manager_Handbook-EN.pdf
wget http://www.r-5.org/files/books/ethology/business/Ark_ShahCh-Guide_to_Success-RU.pdf
wget http://www.r-5.org/files/books/ethology/business/Boris_Berezovsky-Earning_Wealth-RU.pdf
wget http://www.r-5.org/files/books/ethology/business/Brian_Tracy-21_Secrets_of_Self-Made_Millionaires-EN.pdf
wget http://www.r-5.org/files/books/ethology/business/Brian_Tracy-21_Secrets_of_Self-Made_Millionaires-RU.pdf
wget http://www.r-5.org/files/books/ethology/business/Brian_Tracy-Maximum_Achievement-RU.pdf
wget http://www.r-5.org/files/books/ethology/business/Gary_Chapman-The_Five_Love_Languages-RU.pdf
wget http://www.r-5.org/files/books/ethology/business/George_S_Clason-The_Richest_Man_in_Babylon-RU.pdf
wget http://www.r-5.org/files/books/ethology/business/Phil_Porter-Eat_or_Be_Eaten-RU.pdf
wget http://www.r-5.org/files/books/ethology/business/Robert_T_Kiyosaki-Dont_Go_to_School-RU.pdf
wget http://www.r-5.org/files/books/trading/randomness/Nassim_Nicholas_Taleb-Fooled_by_Randomness-RU.pdf
wget http://www.r-5.org/files/books/trading/randomness/Nassim_Nicholas_Taleb-The_Black_Swan-EN.pdf
wget http://www.r-5.org/files/books/ethology/experience/Dan_Ariely-Predictably_Irrational-EN.pdf
wget http://www.r-5.org/files/books/ethology/experience/Louis_V_Gerstner-Who_Says_Elephants_Cant_Dance-EN.pdf
wget http://www.r-5.org/files/books/ethology/experience/Merrill_R_Chapman-In_Search_of_Stupidity-EN.pdf
wget http://www.r-5.org/files/books/ethology/inequality/genesis/Igor_Efimov-Inequality-RU.pdf
wget http://www.r-5.org/files/books/ethology/inequality/food-sc/Joseph_Mercola-Fat_for_Fuel-RU.pdf
wget http://www.r-5.org/files/books/ethology/inequality/food-sc/Michael_D_Fayer-Absolutely_Small-RU.pdf
wget http://www.r-5.org/files/books/ethology/inequality/food-sc/Ray_Kurzweil_Terry_Grossman-Transcend-RU.pdf
wget http://www.r-5.org/files/books/ethology/inequality/v-style/Arshavir_Ter-Hovannessian-Raw-Eating-RU.pdf
wget http://www.r-5.org/files/books/ethology/inequality/v-style/Colin_Campbell-The_China_Study-RU.pdf
wget http://www.r-5.org/files/books/ethology/enterprise/Grant_Gordon_Nigel_Nicholson-Family_Wars-EN.pdf
wget http://www.r-5.org/files/books/ethology/enterprise/Ian_Ruskin_Brown-Practical_Pricing_For_Results-EN.pdf
wget http://www.r-5.org/files/books/ethology/enterprise/Jae_Shim_Joel_Siegel-Budgeting_Basics_and_Beyond-EN.pdf
wget http://www.r-5.org/files/books/ethology/enterprise/Patrick_Gaughan-Mergers_and_Acquisitions-EN.pdf
wget http://www.r-5.org/files/books/ethology/contracts/Henry_Henkin-Drafting_Engineering_Contracts-EN.pdf
wget http://www.r-5.org/files/books/ethology/contracts/M_Horgan_F_Roulston-The_Foundations_of_Engineering_Contracts-EN.pdf
wget http://www.r-5.org/files/books/ethology/contracts/Max_Abrahamson-Engineering_Law_and_the_ICE_Contracts-EN.pdf
wget http://www.r-5.org/files/books/ethology/contracts/Robert_Ribeiro-Engineering_Contracts-EN.pdf
wget http://www.r-5.org/files/books/ethology/brands/W_Park_D_MacInnis_J_Priester-Brand_Attachment-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/behaviour/Keith_Ferrazzi_Tahl_Raz-Never_Eat_Alone-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/behaviour/Nancy_Patrick-Social_Skills_A_Practical_Guide-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/behaviour/Peggy_Klaus-The_Hard_Truth_About_Soft_Skills-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/behaviour/Richard_Templar-The_Rules_of_Life-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/behaviour/Richard_Templar-The_Rules_of_Work-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/leadership/Gerald_Swanson-The_Hyperinflation_Survival_Guide-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/leadership/John_Doorbar-Business_English-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/leadership/Loretta_Malandro-Say_It_Right_The_First_Time-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/interviews/Julie-Ann_Amos-Be_Prepared_Getting_Ready_for_Job_Interview-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/interviews/Julie-Ann_Amos-Be_Prepared_Pass_That_Job_Interview-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/peopleware/Edgar_H_Schein-DEC_Is_Dead_Long_Live_DEC-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/motivation/F_Herzberg_B_Mausner_B_Bloch_Snyderman-The_Motivation_to_Work-RU.pdf
wget http://www.r-5.org/files/books/ethology/corporate/structure/Brian_Dive-The_Accountable_Leader-EN.pdf
wget http://www.r-5.org/files/books/ethology/corporate/structure/Brian_Dive-The_Healthy_Organization-EN.pdf
wget http://www.r-5.org/files/books/trading/historical/Kenneth_Fisher-100_Minds_That_Made_the_Market-EN.pdf
wget http://www.r-5.org/files/books/trading/speculation/Dickson_G_Watts-Speculation_as_a_Fine_Art-EN.pdf
wget http://www.r-5.org/files/books/trading/speculation/Edwin_LeFevre-Reminiscences_of_a_Stock_Operator-EN.pdf
wget http://www.r-5.org/files/books/trading/speculation/Jesse_Livermore-How_To_Trade_In_Stocks_(1940_original)-EN.pdf
wget http://www.r-5.org/files/books/trading/speculation/Richard_Smitten-The_Worlds_Greatest_Stock_Trader-EN.pdf
wget http://www.r-5.org/files/books/trading/speculation/Richard_Smitten-The_Worlds_Greatest_Stock_Trader-RU.pdf
wget http://www.r-5.org/files/books/trading/speculation/Richard_Smitten-Trade_Like_Jesse_Livermore-EN.pdf
wget http://www.r-5.org/files/books/trading/investment/Nicolas_Darvas-How_I_Made_$2_Million_in_the_Stock_Market-EN.pdf
wget http://www.r-5.org/files/books/trading/investment/Steve_Nison-Japanese_Candlestick_Charting_Techniques-EN.pdf
wget http://www.r-5.org/files/books/trading/investment/Thomas_Dorsey-Point_and_Figure_Charting-EN.pdf
wget http://www.r-5.org/files/books/trading/charts/classical/The_Arms_Index-EN.pdf
wget http://www.r-5.org/files/books/trading/charts/gap-analysis/David_Windover-The_Triangle_Trading_Method-EN.pdf
wget http://www.r-5.org/files/books/trading/charts/gap-analysis/Michael_Barclay_Terrence_Hendershott-Trading_After_Hours-EN.pdf
wget http://www.r-5.org/files/books/trading/charts/gap-analysis/Tom_Leksey-Locked-in_Range_Analysis-RU.pdf
wget http://www.r-5.org/files/books/trading/charts/market-profile/CBOT-Market_Profile-EN.pdf
wget http://www.r-5.org/files/books/trading/charts/market-profile/James_Dalton-Markets_in_Profile-EN.pdf
wget http://www.r-5.org/files/books/trading/charts/market-profile/Peter_Steidlmayer_Kevin_Koy-Markets_and_Market_Logic-EN.pdf
wget http://www.r-5.org/files/books/trading/charts/market-profile/Peter_Steidlmayer_Steven_Hawkins-Steidlmayer_on_Markets-EN.pdf
wget http://www.r-5.org/files/books/trading/schoolbooks/Douglas_E_Zalesky-The_25-point_Mantra-EN.pdf
wget http://www.r-5.org/files/books/trading/schoolbooks/Michael_C_Thomsett-Getting_Started_in_Options-EN.pdf
wget http://www.r-5.org/files/books/trading/d-noise/U_N_Known-Principles_of_the_Market-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/d-noise/Vladimir_Kelasev-Price_Channels-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/d-noise/Vladimir_Kelasev-The_Noise-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/d-noise/Vladimir_Kelasev-The_Structure_of_the_Price_Movement-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/behaviour/Alexander_Bruzgin-A_Trick_of_Senses-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/behaviour/Alexander_Bruzgin-Emotional_Control-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/behaviour/Alexander_Bruzgin-Hesitation-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/behaviour/Alexander_Bruzgin-Your_Own_Method-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/behaviour/Alexander_Ilyinsky-Binomial_Tree-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/behaviour/Yury_Akhmetshin-Fuzzy_Logic-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/gold/Alexey_Sinelnikov-The_Moon_Tears-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/gold/Sergey_Moiseev-Bimetallism-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/gold/Sergey_Moiseev-Monometallism-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/gold/Sergey_Moiseev-The_Rise_and_Fall_of_the_Gold_Era-1-RU.pdf
wget http://www.r-5.org/files/books/trading/c-spc/gold/Sergey_Moiseev-The_Rise_and_Fall_of_the_Gold_Era-2-RU.pdf
wget http://www.r-5.org/files/books/trading/t-mechanics/Sergey_Taskaev-Brownian_Movement-RU.pdf
学了Python,知道python里面有对应的模块可以做这些事情。感觉python的强大在于他有现成的模块可以让你用。写代码更象是组装代码。这大大提高了开发效率。如果用 java。可能会用到json解析包，正则匹配，http请求等等。当然python也会用到，但用得特别高效。
我解决的思路是:
- 先请求指定页面，然后存储为一个文档，文档里面就是html文本内容
- 用CSS选择器找到所有的超链接元素，然后根据链接的前缀和后缀特有的形式，筛选出我要的地址，放到List里面
- 创建一个下载目录
- 遍历上面的下载地址，挨个保存文件。
好吧，代码很low(缺少的模块可以用命令 sudo pip3 install xxx 安装上). 但是结果很炫。要是能把下载的资料都看一遍，那岂不是: 好嗨哟！感觉人生已经到达了巅峰！
#! /usr/bin/python3
import requests, bs4, urllib, os
from pathlib import Path
# http://www.r-5.org/books/economics/
# url = 'http://www.r-5.org/books/economics/'
# url = 'http://www.r-5.org/books/ethology.en/'
# url = 'http://www.r-5.org/books/rx-music.en/'
url = 'http://www.r-5.org/books/computers.en/all/'
res = requests.get(url)
try:
res.raise_for_status()
except Exception as exc:
print('There wasa problem: %s' % (exc))
soup = bs4.BeautifulSoup(res.text, 'lxml')
linkElems = soup.select('a')
sumOpen = len(linkElems)
allLinks = []
for i in range(sumOpen):
href = linkElems[i].get('href')
if href.startswith('/files/books') and href.endswith('.pdf'):
fileUrl = 'http://www.r-5.org' + href
allLinks.append(fileUrl)
print(allLinks)
currentPath = os.getcwd() + '/computers'
os.makedirs(currentPath)
for i in range(len(allLinks)):
print('start downlowd file ' + allLinks[i])
fileUrl = allLinks[i]
fileName = currentPath + fileUrl[fileUrl.rfind('/'):]
print('will save file at path : ' + fileName)
saveFile = Path(fileName)
response = requests.get(fileUrl)
saveFile.write_bytes(response.content)
print("Completed")
这个网站还有其他类比的书，我在代码里面注释了地址，要是想下的话，改改就可以了，改的时候注意把创建目录的名称也改一下，不然会报错。
currentPath = os.getcwd() + '/computers'
最后安利一下微信公众号(瓠悠笑软件部落)，如果关注的人多了，嘿嘿，岂不是可以赚点钱了？
或者换句高尚的装逼话术: 如果有好的创意，要马上发表它，否则它只会烂掉！
