协程爬取标题和作者

思想和方法上都很简单

就是要注意我们爬取的xpath路径的问题

此处代码我嫌弃有点慢，就用了多线程下载

代码如下：

```

import requests
from lxml import etree
from concurrent.futures import ThreadPoolExecutor

def base_url(i):
    if (i == 1):
        url = 'https://www.qjnu.edu.cn/channels/9260.html'
    else:
        url='https://www.qjnu.edu.cn/channels/9260_'+str(i)+'.html'
    request = requests.get(url)
    request.encoding = 'utf-8'
    html = etree.HTML(request.text)
    return html

def down_1(html):
    name_list = html.xpath('//div[@class="media"]/h4/a/text()')
    for i in range(len(name_list)):
        with open('学校要问标题.txt', 'a', encoding='UTF-8') as fp:
            fp.write(name_list[i] + '\n')

def down_2(html):
    aut_list = html.xpath('//*[@id="list"]/div/div/span/text()')
    for i in range(len(aut_list)):
        with open('学校要问作者.txt', 'a', encoding='UTF-8') as fp:
            fp.write(aut_list[i][21:] + '\n')


if __name__ == '__main__':
    page = int(input('请输入需要爬取的页数：'))
    with ThreadPoolExecutor(10) as x:
        for i in range(1,page+1):
            html = base_url(i)
            x.submit(down_1,html)
            x.submit(down_2, html)

```

标签：__,fp,协程,url,list,html,标题,down,作者
来源： https://www.cnblogs.com/h-y-26/p/16386673.html