目标站点 有声小说-有声读物-电台广播在线听书-蜻蜓FM
python爬虫实战搞的是PC端,移动端可能有更加方便的接口,欢迎大家留意讨论反正是练手就随便抓一下pc端的包好了
主要内容
- 1.post请求登录
- 2.hmacMD5算法的简单使用
这个例子的登录非常简单没有任何加密直接post就行真的是一点加密和未知参数都没有
私信小编01即可获取大量Python学习资源
python实现,注意此处是类的一个方法不完整无法直接运行
def login(self,user_id,password): data = {account_type:5,device_id:web,user_id: user_id,password: password } response =self.session.post(self.login_url,data=data)ifresponse.status_code==200: temp = response.json() errorno = temp[errorno] errormsg = temp[errormsg]iferrorno ==0:print(login successful!,登录成功!) data = temp[data]self.qingting_id = data[qingting_id]self.access_token = data[access_token]else:print(Login failed,登录失败)print(errormsg)
录成功后我们把access_token和qingting_id拿到,相当于一个登录后的标志如果账号是会员相当于一个会员标志音频的真实地址请求了这样一个url:
https://audio.qingting.fm/audiostream/redirect/294280/11604885
其中294280是专辑id,11604885是当前音频的id
其中还带了一些参数比如access_token,qingting_id(登录成功的response中有,上图没有登录所有是空的),另外还有一些比如t是时间戳,
device_id=MOBILESITE(不变)
关键就在于sign(尝试过不加sign会返回一个签名错误)
可以通过全局搜索试一下是哪个js生成的这个sign我全局搜索了一下
device_id
在mian.一大堆.js找到了生成sign的函数(需要自己分辨一下是一个device_id: “MOBILESITE”的)
搜索其他关键字应该也是可以顺利找到的
这里的sign是u这个变量它是由c这个变量通过一堆加密处理得到的
我们可以控制台输出一下u和c
所以我们就知道了sign实际是加密了请求的其他参数
一开始我误以为是单纯的MD5所以卡了好久(还进入函数内部看他具体是怎么实现的看的一头雾水)
其实代码已经告诉用的是
createHmac(“md5”, “fpMn12&38f_2e”)
查了下Hmac发现就是一种现成的算法,还有不同的模式MD5是其中一种,需要一个秘钥
这里什么都告诉你了,用Hmac-md5秘钥是fpMn12&38f_2e
找个在线加密的网站试了下,果然和刚才控制台输出的一样
python的话需要import
hmac这个库
importhmacimport time base_url ="https://audio.qingting.fm"bookid ="294280"id ="11590788"access_token =""qingting_id =""timestamp = str(round(time.time()*1000))data =f"/audiostream/redirect/{bookid}/{id}?access_token={access_token}&device_id=MOBILESITE&qingting_id={qingting_id}&t={timestamp}"message = data.encode(utf-8)key ="fpMn12&38f_2e".encode(utf-8)sign = hmac.new(key, message, digestmod=MD5).hexdigest()whole_url = base_url+data+"&sign="+signprint(whole_url)
得到一个音频可以做到了剩下的就是得到一堆了,其实我们得到每个音频的id就可以了我请求的是这个接口
info_api = https://i.qingting.fm/capi/channel/{self.bookid}/programs/{self.version}?curpage={str(page)}&pagesize=30&order=asc
version在声书主页的源代码中,只要改curpage就可以翻页了完整源码
import requestsimport reimport hmacimport timefrom tqdm import tqdmfrom bs4 import BeautifulSoupimport osimport jsonimport sysimport urllib3urllib3.disable_warnings() class QingTing(): def __init__(self,user_id,password,bookurl,ifLogin): self.ifLogin = ifLogin self.user_id = user_id self.password = password self.session = requests.session() self.session.headers.update({user-agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/79.0.3945.130 Safari/537.36}) self.login_url ="https://u2.qingting.fm/u2/api/v4/user/login"self.qingting_id =self.access_token =self.bookurl = bookurl self.bookurl =https://www.qingting.fm/channels/257790self.bookid = self.bookurl.split(/)[-1] self.version =self.qingtinghost =https://audio.qingting.fmself.save_path =self.bookname =def login(self,user_id,password): data = {account_type:5,device_id:web,user_id: user_id,password: password } response = self.session.post(self.login_url,data=data,verify=False)ifresponse.status_code==200: temp = response.json() errorno = temp[errorno] errormsg = temp[errormsg]iferrorno ==0:print(login successful!,登录成功!) data = temp[data] self.qingting_id = data[qingting_id] self.access_token = data[access_token]else:print(Login failed,登录失败)print(errormsg)time.sleep(10) sys.exit(0) def __get_version(self): response = self.session.get(url=self.bookurl,verify=False)ifresponse.status_code==200: soup = BeautifulSoup(response.text,lxml) temp_bookname = soup.select(div.album-info-root > div.top > div.info.right > h1)[0].stringreplaced_pattern =[\\\/:\*\?\"<>|]self.bookname = re.sub(replaced_pattern,,temp_bookname,flags=re.M +re.S)ifnotos.path.exists(self.bookname):os.makedirs(self.bookname) matched = re.search(\"version\":\"(\w+)",response.text,re.S)ifmatched: version = matched.group(1) self.version = versionreturnversion def __get_total_page(self): self.__get_version() page =1info_api = fhttps://i.qingting.fm/capi/channel/{self.bookid}/programs/{self.version}?curpage={str(page)}&pagesize=30&order=ascresponse = self.session.get(info_api,verify=False)ifresponse.status_code==200: temp = response.json() total = temp[data][total] total_page = int(int(total)/30)+1returntotal,total_page def get_book_info(self): total,total_page = self.__get_total_page()print(self.bookname,共{}集.format(total))forpageinrange(1,total_page+1): info_api = fhttps://i.qingting.fm/capi/channel/{self.bookid}/programs/{self.version}?curpage={str(page)}&pagesize=30&order=ascresponse = self.session.get(info_api,verify=False) programs = response.json()[data][programs]forprograminprograms:print(program[id],program[title])yieldprogram def get_src(self,id): bookid = self.bookid access_token = self.access_token qingting_id =self.qingting_id timestamp = str(round(time.time()*1000)) data = f"/audiostream/redirect/{bookid}/{id}?access_token={access_token}&device_id=MOBILESITE&qingting_id={qingting_id}&t={timestamp}"message = data.encode(utf-8) key ="fpMn12&38f_2e".encode(utf-8) sign = hmac.new(key, message, digestmod=MD5).hexdigest() whole_url = self.qingtinghost+data+"&sign="+signreturnwhole_url def downloadFILE(self,url,name): resp = self.session.get(url=url,stream=True,verify=False)ifresp.headers[Content-Type] ==audio/mpeg: content_size = int(int(resp.headers[Content-Length])/1024) withopen(name,"wb") as f:print("Pkg total size is:",content_size,k,start...)fordataintqdm(iterable=resp.iter_content(1024),total=content_size,unit=k,desc=name): f.write(data)print(name ,"download finished!")else: errorno = resp.json()[errorno] errormsg = resp.json()[errormsg]print(没有权限下载,请登录已购此音频的账号。)print(errorno:,errorno,errormsg) def run(self):ifself.ifLogin: self.login(self.user_id,self.password) programs = self.get_book_info() count =0forprograminprograms: count+=1try: id = program[id] title = str(count).zfill(4)++program[title]+.m4aifnotself.bookname ==: title =os.path.join(self.bookname,title) whole_url = self.get_src(id) self.downloadFILE(whole_url,title) except Exception as e:print(e) withopen(log.txt,a,encoding=utf-8) as f: f.write(str(count)+str(e)+\n) def get_config_info(): withopen(config.json,r,encoding=utf-8) as f:config= json.loads(f.read())returnconfigif__name__ =="__main__": pyinstaller -F -i ico.ico QingTingFM.pyconfig= get_config_info()ifconfig["ifLogin"]: bookurl =input(请输入要下载音频的主页链接:(如[url=https://www.qingting.fm/channels/257790]https://www.qingting.fm/channels/257790[/url])) isvalid = re.search(https://www.qingting.fm/channels/\d+,bookurl)ifisvalid: q = QingTing(config["user_id"],config["password"],bookurl,1) q.run()else:print("输入的主页格式错误")else: 不登录 bookurl =input(请输入要下载音频的主页链接:(如[url=https://www.qingting.fm/channels/257790]https://www.qingting.fm/channels/257790[/url])) isvalid = re.search(https://www.qingting.fm/channels/\d+,bookurl)ifisvalid: q = QingTing(config["user_id"],config["password"],bookurl,0) q.run()else:print("输入的主页格式错误")
配置文件部分
{"ifLogin":1,"user_id":"135","password":"pwd"}