11.6 项目：“I’m Feeling Lucky”Google 查找
项目要求：如果我只要在命令行中输入查找主题，就能让计算机自动打开浏览器，并在新的选项卡中显示前面几项查询结果；【这个谷歌不行，那我就换百度呗（摊手）】
#! python3
# baidu search in cmd（requests百度https不会返回预想的内容，但http可以）
import sys, bs4, requests, webbrowser
print('Baiduing...')
res = requests.get('http://www.baidu.com/s?wd=' + ' '.join(sys.argv[1:]))
if res.status_code != 200:
print('requests fail')
# TODO: Retrieve top search result links.
bd_soup = bs4.BeautifulSoup(res.text, "html.parser")
linkElems = bd_soup.select('.t a')
print(len(linkElems))
# TODO: Open a browser tab for each result.
numOpen = min(5,len(linkElems))
for i in range(numOpen):
webbrowser.open(linkElems[i].get('href'))
11.7 项目：下载所有 XKCD 漫画
项目要求：自动下载http://xkcd.com/中的漫画
下面是方法一：
#! python3
# download pic from xkcd
# <div id="comic">
# <img src="//imgs.xkcd.com/comics/barrel_cropped_(1).jpg"
title="Don't we all." alt="Barrel - Part 1" />
# </div>
# -----
# <li><a rel="prev" href="/1888/" accesskey="p">< Prev</a></li>
# end: <li><a rel="prev" href="#" accesskey="p">< Prev</a></li>
import os, bs4, requests
url = 'http://xkcd.com'
os.makedirs('xkcd', exist_ok = True)
while not url.endswith('#'):
# todo: download the page
print('Downloading page %s...' % url)
res = requests.get(url)
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text)
# todo: find the url of comic page
comicElems = soup.select('#comic img')
if comicElems == []:
print('Could not find comic image!')
else:
comicUrl = 'http:' + comicElems[0].get('src')
# todo: download the jpg
print('Downloading image %s' % comicUrl)
res = requests.get(comicUrl)
res.raise_for_status()
# todo: save jpg to ./xkcd
imageFile = open(os.path.join('xkcd', os.path.basename(comicUrl)),'wb')
for chunk in res.iter_content(100000):
imageFile.write(chunk)
imageFile.close()
# todo: get the prev button's url
prevLink = soup.select('a[rel="prev"]')[0]
url = 'http://xkcd.com' + prevLink.get('href')
print('Done.')
下面是方法二：
#! python3
# download pic from xkcd
# <div id="comic">
# <img src="//imgs.xkcd.com/comics/barrel_cropped_(1).jpg"
title="Don't we all." alt="Barrel - Part 1" />
# </div>
# -----
# <li><a rel="prev" href="/1888/" accesskey="p">< Prev</a></li>
# end: <li><a rel="prev" href="#" accesskey="p">< Prev</a></li>
import os, bs4, requests
urls = 'http://xkcd.com'
os.makedirs('xkcd', exist_ok = True)
for num in range(1005,1010):
# todo: download the page
url = urls + '/%s/' % num
print('Downloading page %s...' % url)
res = requests.get(url)
res.raise_for_status()
soup = bs4.BeautifulSoup(res.text, "html.parser")
# todo: find the url of comic page
comicElems = soup.select('#comic img')
if comicElems == []:
print('Could not find comic image!')
else:
comicUrl = 'http:' + comicElems[0].get('src')
# todo: download the jpg
print('Downloading image %s' % comicUrl)
res = requests.get(comicUrl)
res.raise_for_status()
# todo: save jpg to ./xkcd
image_name = os.path.join('xkcd', os.path.basename(comicUrl))
with open(image_name, 'wb') as imageFile:
for chunk in res.iter_content(100000):
imageFile.write(chunk)
# todo: get the prev button's url
prevLink = soup.select('a[rel="prev"]')[0]
print('Done.')
11.11.3 实践项目：2048
项目要求：2048是一个简单的游戏，通过箭头向上、下、左、右移动滑块，让滑块合并。实际上，你可以通过一遍一遍的重复“上、右、下、左”模式，获得相当高的分数。编写一个程序，打开https://gabrielecirulli.github.io/2048/上的游戏，不断发送上、右、下、左按键，自动玩游戏。
#! python3
# 'autoplau' 2048 on https://gabrielecirulli.github.io/2048/
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser = webdriver.Chrome()
browser.get('https://gabrielecirulli.github.io/2048/')
htmlElem = browser.find_element_by_tag_name('html')
for i in range(100):
htmlElem.send_keys(Keys.UP)
htmlElem.send_keys(Keys.LEFT)
htmlElem.send_keys(Keys.RIGHT)
htmlElem.send_keys(Keys.DOWN)
print('done!')
11.11.4 实践项目：链接验证
项目要求：编写一个程序，对给定的网页 URL，下载该页面所有链接的页面。程序应该标记出所有具有 404“Not Found”状态码的页面，将它们作为坏链接输出。
#! python3
# to check seek out <a href='xxx'> which are 'not Found 404' in certificated url
import bs4, requests
# todo: use beautifulsoup4 to seek out all <a> in url page ; (or use webdriver)
url = 'http://www.scut.edu.cn/new/'
res = requests.get(url)
res.raise_for_status()
res_soup = bs4.BeautifulSoup(res.text)
res_soup_a = res_soup.select('a[href]')
print('There are %d <a href=""> in %s' % (len(res_soup_a), url))
# todo: use requests to test the href in <a>; (or use webdriver.find_element)
for i in range(len(res_soup_a)):
a_href_url = res_soup_a[i].get('href')
if a_href_url.startswith("http:") or a_href_url.startswith('https:'):
a_href_url_fullname = a_href_url
else:
a_href_url_fullname = 'http://www.scut.edu.cn/new/' + a_href_url.replace('/new/','')
try:
a_res = requests.get(a_href_url_fullname, timeout=0.1)
# if status_code == 404, write the url and tag_text in txt
if a_res.status_code == 404:
print('This url(%s) link to 404 not found... \n %s' % (res_soup_a[i].getText(),a_href_url))
except requests.exceptions.Timeout:
print('A url request over time...')
print('Done!')
环境：python3
想做这个系列文章，就是因为当时看这本书时，想看看网上有没更优美的解决，但是略难找到。所以就把自己的项目练习放在了一个txt文件中，现在把练习代码放到这里，有不足之处希望大家能给出指导意见及相互交流、提升。
