- 通过网易提供的API和相应的爬虫技术下载某个歌手全部歌曲的歌词
通过接口来得到歌曲的歌词
'http://music.163.com/api/song/lyric?' + 'id=' + str(song_id) + '&lv=-1&kv=-1&tv=-1'
- 获取歌曲id
https://music.163.com/artist?id=5771
去掉url中的‘#’后才是可以直接爬取的网页。view-source:https://music.163.com/artist?id=5771 通过网页查看源代码，可以找到歌曲的名称个对应的id
- 通过BeautifuSoup4或者lxml来获取网页上的内容
歌曲的名称和id直接在a标签中，获取上层的div，div的class属性div的class属性
| <ul class="f-hide"><li><a href="[/song?id=167827](https://music.163.com/song?id=167827)">素颜</a></li><li><a href="[/song?id=167876](https://music.163.com/song?id=167876)">有何不可</a></li><li><a href="[/song?id=2016442586](https://music.163.com/song?id=2016442586)">合拍</a></li><li><a href="[/song?id=411214279](https://music.163.com/song?id=411214279)">雅俗共赏</a></li><li><a href="[/song?id=167873](https://music.163.com/song?id=167873)">多余的解释</a></li><li><a href="[/song?id=27646687](https://music.163.com/song?id=27646687)">玫瑰花的葬礼</a></li><li><a href="[/song?id=167655](https://music.163.com/song?id=167655)">幻听</a></li><li><a href="[/song?id=167882](https://music.163.com/song?id=167882)">清明雨上</a></li><li><a href="[/song?id=167691](https://music.163.com/song?id=167691)">天龙八部之宿敌</a></li><li><a href="[/song?id=167850](https://music.163.com/song?id=167850)">庐州月</a></li><li><a href="[/song?id=167870](https://music.163.com/song?id=167870)">如果当时</a></li><li><a href="[/song?id=28854182](https://music.163.com/song?id=28854182)">惊鸿一面</a></li><li><a href="[/song?id=2012390001](https://music.163.com/song?id=2012390001)">没想到</a></li><li><a href="[/song?id=167844](https://music.163.com/song?id=167844)">灰色头像</a></li><li><a href="[/song?id=167732](https://music.163.com/song?id=167732)">千百度</a></li><li><a href="[/song?id=167937](https://music.163.com/song?id=167937)">断桥残雪</a></li><li><a href="[/song?id=573384240](https://music.163.com/song?id=573384240)">如约而至</a></li><li><a href="[/song?id=167885](https://music.163.com/song?id=167885)">城府</a></li><li><a href="[/song?id=167744](https://music.163.com/song?id=167744)">半城烟沙</a></li><li><a href="[/song?id=1449406576](https://music.163.com/song?id=1449406576)">温泉</a></li><li><a href="[/song?id=412902950](https://music.163.com/song?id=412902950)">最佳歌手</a></li><li><a href="[/song?id=27646702](https://music.163.com/song?id=27646702)">我想牵着你的手</a></li><li><a href="[/song?id=28802028](https://music.163.com/song?id=28802028)">山水之间</a></li><li><a href="[/song?id=402073807](https://music.163.com/song?id=402073807)">燕归巢</a></li><li><a href="[/song?id=167929](https://music.163.com/song?id=167929)">你若成风</a></li><li><a href="[/song?id=1488737309](https://music.163.com/song?id=1488737309)">如果当时2020</a></li><li><a href="[/song?id=1913262091](https://music.163.com/song?id=1913262091)">留香</a></li><li><a href="[/song?id=1842784921](https://music.163.com/song?id=1842784921)">乌鸦</a></li><li><a href="[/song?id=167888](https://music.163.com/song?id=167888)">认错</a></li><li><a href="[/song?id=1992391270](https://music.163.com/song?id=1992391270)">曼陀山庄</a></li><li><a href="[/song?id=448144319](https://music.163.com/song?id=448144319)">今年勇</a></li><li><a href="[/song?id=554191378](https://music.163.com/song?id=554191378)">我乐意</a></li><li><a href="[/song?id=34040693](https://music.163.com/song?id=34040693)">千古</a></li><li><a href="[/song?id=167705](https://music.163.com/song?id=167705)">想象之中</a></li><li><a href="[/song?id=2016388352](https://music.163.com/song?id=2016388352)">歌曲：合拍</a></li><li><a href="[/song?id=167841](https://music.163.com/song?id=167841)">叹服</a></li><li><a href="[/song?id=415792563](https://music.163.com/song?id=415792563)">摄影艺术</a></li><li><a href="[/song?id=167712](https://music.163.com/song?id=167712)">拆东墙</a></li><li><a href="[/song?id=1978137630](https://music.163.com/song?id=1978137630)">纸上雪</a></li><li><a href="[/song?id=167679](https://music.163.com/song?id=167679)">全球变冷</a></li><li><a href="[/song?id=862099032](https://music.163.com/song?id=862099032)">明智之举</a></li><li><a href="[/song?id=1975753022](https://music.163.com/song?id=1975753022)">诗画中国</a></li><li><a href="[/song?id=167894](https://music.163.com/song?id=167894)">星座书上</a></li><li><a href="[/song?id=1397105439](https://music.163.com/song?id=1397105439)">雨幕</a></li><li><a href="[/song?id=167715](https://music.163.com/song?id=167715)">医生</a></li><li><a href="[/song?id=167717](https://music.163.com/song?id=167717)">微博控</a></li><li><a href="[/song?id=167727](https://music.163.com/song?id=167727)">降温</a></li><li><a href="[/song?id=167709](https://music.163.com/song?id=167709)">河山大好</a></li><li><a href="[/song?id=1842801466](https://music.163.com/song?id=1842801466)">冰柜</a></li><li><a href="[/song?id=569213279](https://music.163.com/song?id=569213279)">大千世界</a></li></ul> |
获取该div下的子孙a，获取a标签的href和文本内容就可以得到song_id和song_name了。
使用的lxml模块，通过xpath定位标签的方法。也可以通过BeautifulSoup4提供的方法，定位标签并获取文本内容。
-
获取全部歌曲的id，获取全部专辑id，获取所有专辑里的歌曲
专辑对应的接口地址是：“https://music.163.com/artist/album?id="+str(singer_id)+"&limit=150&offset=0”
三个参数分别是歌手id、limit是每页显示的专辑数量、offset是偏移量。将limit设置的大一点，使得所有专辑就会在一个网页上显示
查看这个网页的源代码：https://music.163.com/#/artist/album?id=5771&limit=100&offset=0
（view-source:https://music.163.com/artist/album?id=5771&limit=100&offset=0）
可以找到所有专辑和专辑对应的id：
通过xpath或者BeautifulSoup4可以找到对应的专辑名和专辑id。
- 获取专辑里的所有歌曲id
通过爬取对应的标签后获取歌曲的id和歌曲名。
def get_all_song_id(self, album_ids):
with requests.Session() as session:
all_song_ids, all_song_names = [], []
for album_id in album_ids:
one_album_url = "https://music.163.com/album?id="+str(album_id)
response = session.get(one_album_url, headers=headers)
text = response.text
html = etree.HTML(text)
album_song_ids = html.xpath("//ul[@class='f-hide']/li/a/@href")
album_song_names = html.xpath("//ul[@class='f-hide']/li/a/text()")
album_song_ids = [ids.split('=')[-1] for ids in album_song_ids]
all_song_ids.append(album_song_ids)
all_song_names.append(album_song_names)
return all_song_ids, all_song_names
- 得到该歌手的全部歌曲名称和id
通过python的json模块解析结果simplejson具有同样的效果
获取歌手全部歌词的关键主要流程代码,本次将爬取的结果以文件的形式保存：
try:
with open('G:/lyric/xs/'+str(song_name)+".txt", 'w+') as f:
f.write(self.parse_lyric(json_text))
# print(song_name)
# print(self.parse_lyric(json_text))
# print('-' * 30)
except Exception as e:
pass
歌词输出保存地址为G:/lyric/xs/
if __name__ == "__main__":
sing_id_xs = '5771' # 许嵩
c = CrawlerLyric()
c.get_all_song_lyric(sing_id_xs)
#c.get_top50(sing_id_xs)
# c.test0(sing_id_xs)
c.get_top50(sing_id_xs)
https://music.163.com/api/song/lyric?id=402073807&lv=1&kv=1&tv=-1
-
待解决 新专辑歌词无法通过
url_song = 'http://music.163.com/api/song/lyric?' + 'id=' + str(song_id) + '&lv=1&kv=1&tv=-1'
找到
<div id="lyric-content" class="bd bd-open f-brk f-ib" data-song-id="1842784921" data-third-copy="false" , copy-from>
该页面无法找到歌词，导致结果文本保存为空
-
解决 url返回的是json数据格式的结果，修改接口参数&lv=-1&kv=-1&tv=-1
以歌曲id名为文档名输出
'1842784921', '1842801269', '1842801328', '1842802095', '1842801466', '1842802179', '1842801494', '1842801496', '1842801513', '1842802205',
'乌鸦', '假摔', '科幻', '万古', '冰柜', '超市', '隔代', '野人', '三尺', '庞贝',
再次以文本输出结果，w+复写得到不为空的歌词内容。
歌词输出结果
代码部分
import requests
from lxml import etree
import simplejson
import re
import operator
from functools import reduce
ua = 'Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36'
headers = {
'User-agent': ua
}
class CrawlerLyric:
def __init__(self):
self.author_name = ""
def get_url_html(self, url):
with requests.Session() as session:
response = session.get(url, headers=headers)
text = response.text
html = etree.HTML(text)
return html
def get_url_json(self, url):
with requests.Session() as session:
response = session.get(url, headers=headers)
text = response.text
text_json = simplejson.loads(text)
return text_json
def parse_song_id(self, html):
song_ids = html.xpath("//ul[@class='f-hide']//a/@href")
song_names = html.xpath("//ul[@class='f-hide']//a/text()")
self.author_name = html.xpath('//title/text()')
song_ids = [ids[9:len(ids)] for ids in song_ids]
return self.author_name, song_ids, song_names
def parse_lyric(self, text_json):
try:
lyric = text_json.get('lrc').get('lyric')
regex = re.compile(r'\[.*\]')
final_lyric = re.sub(regex, '', lyric).strip()
return final_lyric
except AttributeError as k:
print(k)
pass
def get_album(self, html):
album_ids = html.xpath("//ul[@id='m-song-module']/li/p/a/@href")
album_names = html.xpath("//ul[@id='m-song-module']/li/p/a/text()")
album_ids = [ids.split('=')[-1] for ids in album_ids]
return album_ids, album_names
def get_top50(self, sing_id):
url_singer = 'https://music.163.com/artist?id='+str(sing_id) #
html_50 = self.get_url_html(url_singer)
author_name, song_ids, song_names = self.parse_song_id(html_50)
print(author_name, song_ids, song_names)
for song_id, song_name in zip(song_ids, song_names):
url_song = 'http://music.163.com/api/song/lyric?' + 'id=' + str(song_id) + '&lv=1&kv=1&tv=-1'
json_text = self.get_url_json(url_song)
print(song_name)
print(self.parse_lyric(json_text))
print('-' * 30)
def get_all_song_id(self, album_ids):
with requests.Session() as session:
all_song_ids, all_song_names = [], []
for album_id in album_ids:
one_album_url = "https://music.163.com/album?id="+str(album_id)
response = session.get(one_album_url, headers=headers)
text = response.text
html = etree.HTML(text)
album_song_ids = html.xpath("//ul[@class='f-hide']/li/a/@href")
album_song_names = html.xpath("//ul[@class='f-hide']/li/a/text()")
album_song_ids = [ids.split('=')[-1] for ids in album_song_ids]
all_song_ids.append(album_song_ids)
all_song_names.append(album_song_names)
return all_song_ids, all_song_names
def get_all_song_lyric(self,singer_id):
album_url = "https://music.163.com/artist/album?id="+str(singer_id)+"&limit=150&offset=0"
html_album = self.get_url_html(album_url)
album_ids, album_names = self.get_album(html_album)
all_song_ids, all_song_names = self.get_all_song_id(album_ids)
all_song_ids = reduce(operator.add, all_song_ids)
all_song_names = reduce(operator.add, all_song_names)
print(all_song_ids)
print(all_song_names)
for song_id, song_name in zip(all_song_ids, all_song_names):
url_song = 'http://music.163.com/api/song/lyric?' + 'id=' + str(song_id) + '&lv=-1&kv=-1&tv=-1'
json_text = self.get_url_json(url_song)
print(song_name)
try:
with open('G:/lyric/xs/'+str(song_name)+".txt", 'w+') as f:
f.write(self.parse_lyric(json_text))
# print(song_name)
# print(self.parse_lyric(json_text))
# print('-' * 30)
except Exception as e:
pass
def test0(self,singer_id):
a=['1842784921', '1842801269', '1842801328', '1842802095', '1842801466', '1842802179', '1842801494', '1842801496', '1842801513', '1842802205']
for song_id in a:
url_song = 'http://music.163.com/api/song/lyric?' + 'id=' + str(song_id) + '&lv=-1&kv=-1&tv=-1'
json_text = self.get_url_json(url_song)
with open('G:/lyric/xs/hxzy/' + str(song_id) + ".txt", 'w+') as f:
f.write(self.parse_lyric(json_text))
print(json_text)
if __name__ == "__main__":
sing_id_xs = '5771' # 许嵩
c = CrawlerLyric()
c.get_all_song_lyric(sing_id_xs)
#c.get_top50(sing_id_xs)
# c.test0(sing_id_xs)
已知歌曲id的情况下( url = f"http://music.163.com/api/song/lyric?id={song_id}+&lv=-1&tv=-1")
import json # 使用了json格式存储
import requests
def get_lyric(song_id):
headers = {
"user-agent" : "Mozilla/5.0",
"Referer" : "http://music.163.com",
"Host" : "music.163.com"
}
if not isinstance(song_id, str):
song_id = str(song_id)
url = f"http://music.163.com/api/song/lyric?id={song_id}+&lv=-1&tv=-1"
r = requests.get(url, headers=headers)
r.raise_for_status()
r.encoding = r.apparent_encoding
json_obj = json.loads(r.text)
return json_obj["lrc"]["lyric"]
lyric = get_lyric(1842801496)
print(lyric)
