scrapy接selenium关键步骤

下载中间件

from taobao.UA import UserAgents
from scrapy import signals
import random
from scrapy.http import HtmlResponse
from selenium import webdriver
b=webdriver.Firefox(executable_path="/root/Downloads/geckodriver")
class TaobaoSpider1Middleware(object):
def process_request(self, request, spider):

    data=b.get(request.url)
    b.execute_script('window.scrollTo(0, document.body.scrollHeight)')
    return HtmlResponse(url=request.url,request=request,body=b.page_source,encoding='utf-8')

标签：关键步骤,body,url,selenium,request,scrapy,import
来源： https://blog.51cto.com/u_15177056/2725503