小爬爬5:重点回顾&&移动端数据爬取1
作者：互联网
1.
(1)什么是selenium
- 基于浏览器自动化的一个模块
(2)在爬虫中为什么使用selenium及其和爬虫之间的关联
- 可以便捷的获取动态加载的数据
- 实现模拟登陆
(3)列举常见的selenium模块的方法及其作用
- get(url)
- find系列的函数进行标签定位 #记住常用的几个
- send_keys（‘key’） #录入1个数据池
- click（） #点击
- excute_script（‘jsCode’） #执行js代码
- page_source #获取页面的数据
- switch_to.frame('iframeID') #iframe需要切换
- quite() #关闭
- save_screenshot() #保存屏幕的内容
- a = ActionChains(bro) #动作链实例化对象
- a.click_and_hold('tag') #点击且长按这个标签
- tag.move_by_offset(x,y).perform() #偏移某个标签
(4)loop的作用:
可以将多个任务对象注册到loop中
loop就可以通过不间断循环的形式异步的执行任务对象
(5)多任务异步协程是如何实现异步的
- 协程
- 任务对象
- loop
2.单线程多任务异步协程回顾
# Author: studybrother sun
import asyncio
import aiohttp
#在实现该函数的时候，其函数实现内部不可以出现非异步模块的代码
async def request(url):
async with aiohttp.ClientSession() as s:
async with await s.get(url=url) as response:
page_text = await response.text() #解析的搜索界面
return page_text
def callback(task): #回调
print(task.result())
def callback1(task):
print(task.result())
#事件循环对象：
loop = asyncio.get_event_loop()
c = request('https://www.baidu.com')
c1 = request('https://www.sogou.com')
task = asyncio.ensure_future(c)
task.add_done_callback(callback)
task1 = asyncio.ensure_future(c1)
task1.add_done_callback(callback1)
tasks = [task,task1]
loop.run_until_complete(asyncio.wait(tasks))
运行的得到下面的结果:
<html>
<head>
<script>
location.replace(location.href.replace("https://","http://"));
</script>
</head>
<body>
<noscript><meta http-equiv="refresh" content="0;url=http://www.baidu.com/"></noscript>
</body>
</html>
<!DOCTYPE html>
<html lang="cn">
<head>
<script>window._speedMark = new Date();
window.lead_ip = '221.218.208.77';window.now = 1559307496193;</script> <meta charset="utf-8">
<link rel="dns-prefetch" href="//img01.sogoucdn.com"><link rel="dns-prefetch" href="//img02.sogoucdn.com"><link rel="dns-prefetch" href="//img03.sogoucdn.com"><link rel="dns-prefetch" href="//img04.sogoucdn.com"><link rel="dns-prefetch" href="//dlweb.sogoucdn.com">
<title>搜狗搜索引擎 - 上网从搜狗开始</title>
<link rel="shortcut icon" href="/images/logo/new/favicon.ico?v=4" type="image/x-icon">
<meta http-equiv="X-UA-Compatible" content="IE=Edge">
<link rel="search" type="application/opensearchdescription+xml" href="/content-search.xml" title="搜狗搜索">
<meta name="keywords" content="搜狗搜索,网页搜索,微信搜索,视频搜索,图片搜索,音乐搜索,新闻搜索,软件搜索,问答搜索,百科搜索,购物搜索">
<meta name="description" content="搜狗搜索是全球第三代互动式搜索引擎，支持微信公众号和文章搜索、知乎搜索、英文搜索及翻译等，通过自主研发的人工智能算法为用户提供专业、精准、便捷的搜索服务。"> <link rel="stylesheet" type="text/css" href="/web/index/css/base.v.1.4.12.css">
<style>.wrapper .suggestion{border: 1px solid #e8e8e8; width:622px;-moz-box-shadow: 0px 1px 8px rgba(0,0,0,0.1);-webkit-box-shadow: 0px 1px 8px rgba(0,0,0,0.1);box-shadow: 0px 1px 8px rgba(0,0,0,0.1);border-top-left-radius: 0px;border-top-right-radius: 0px;border-bottom-right-radius: 2px;border-bottom-left-radius: 2px; top:43px;} .wrapper .suglist{width: 206px;} .wrapper .suglist .keyword {color: #7a77c8;} .big-scn .suggestion {width: 654px;} .big-scn .suglist{width:236px;} .wrapper .suglist{ padding:4px 0}</style></head>
<body >
<div class="bg-gj-w" id="settings-mask" style="display: none;"></div>
<div class="gjss" id="settings-advanced" style="display: none;top:-240px;">
<div class="hf-box" id="settings-save-layer">
<div class="hf-def">已保存设置</div>
</div>
<div class="gjss-tab">
<a uigs-id="tab_set" href="javascript:void(0);" class="js-settings-tab tab-a cur">搜索设置</a>
<a uigs-id="tab_adv" href="javascript:void(0);" class="js-settings-tab tab-a">高级搜索</a>
<a href="javascript:void(0);" class="close-btn" id="settings-close"></a>
</div>
<div class="gjss-main">
<div class="gjss-sz js-settings-content">
<p class="gjss-err js-settings-mask" style="display: none;">搜索设置暂不可用，请启用浏览器的Cookie功能，然后刷新本页。</p>
<div class="bg-wkq js-settings-mask" id="settings-tips" style="display: none;"></div>
<dl class="js-as-select">
<dt>搜索结果显示条数</dt>
<dd>
<a href="javascript:void(0);" class="xz" id="settings-number" data-value="10">每页显示10条</a>
<ul id="settings-number-list">
<li><a uigs-id="set_10" href="javascript:void(0);" data-value="10">每页显示10条</a></li>
<li><a uigs-id="set-20" href="javascript:void(0);" data-value="20">每页显示20条</a></li>
<li><a uigs-id="set-50" href="javascript:void(0);" data-value="50">每页显示50条</a></li>
<li><a uigs-id="set-100" href="javascript:void(0);" data-value="100">每页显示100条</a></li>
</ul>
</dd>
<input type="hidden" name="pageNum" id="settings-show-number" value="10">
</dl>
<p class="enter" style="padding-top: 20px;">
<a href="javascript:void(0);" id="settings-save" uigs-id="set-save" class="a1">保存</a>
<a href="javascript:void(0);" id="settings-reset" uigs-id="set-reset" class="a2">恢复默认</a>
</p>
</div>
<div class="gjss-sz js-settings-content" style="display: none;">
<form action="/web" target="_blank" id="advanced-search-form">
<input type="hidden" name="query" value="">
<input name="fieldtitle" type="hidden" value=""/>
<input name="fieldcontent" type="hidden" value=""/>
<input name="fieldstripurl" type="hidden" value=""/>
<input name="bstype" type="hidden" value=""/>
<input name="ie" type="hidden" value="utf8"/>
<dl>
<dt>搜索关键词</dt>
<dd class="js-as-radio">
<div class="input-box js-input-box" id="advanced-query-box">
<input name="q" type="text" must="1" size="42" maxlength="100" autocomplete="off" placeholder="例如：搜狗真棒（多个关键词可用空格区分）">
<span class="err-word">* 请输入搜索关键词</span>
</div>
<a uigs-id="adv_split-query" href="javascript:void(0);" data-value="checkbox" class="dk-btn cur">拆分关键词</a>
<a uigs-id="adv_no-split-query" href="javascript:void(0);" data-value="" class="dk-btn">不拆分关键词</a>
<input type="hidden" name="include" value="checkbox">
</dd>
</dl>
<dl>
<dt>在指定站内搜索</dt>
<dd>
<div class="input-box js-input-box"><input name="sitequery" type="text" size="40" autocomplete="off" placeholder="例如：www.sogou.com"></div>
</dd>
</dl>
<dl class="js-as-select" style="padding-top:16px">
<dt>搜索词位于</dt>
<dd>
<a href="javascript:void(0);" class="xz">网页中任何地方</a>
<ul>
<li><a href="javascript:void(0);" data-value="0">网页中任何地方</a></li>
<li><a href="javascript:void(0);" data-value="1">仅在标题中</a></li>
<li><a href="javascript:void(0);" data-value="2">仅在正文中</a></li>
<li><a href="javascript:void(0);" data-value="3">仅在网址中</a></li>
</ul>
</dd>
<input type="hidden" name="located" value="0">
</dl>
<dl class="js-as-select" style="padding-top:16px">
<dt>需要搜索的文件格式</dt>
<dd >
<a href="javascript:void(0);" class="xz">全部网页</a>
<ul>
<li><a href="javascript:void(0);" data-value="">全部网页</a></li>
<li><a href="javascript:void(0);" data-value="doc">Microsoft Word (.doc)</a></li>
<li><a href="javascript:void(0);" data-value="xls">Microsoft Excel (.xls)</a></li>
<li><a href="javascript:void(0);" data-value="ppt">Microsoft Powerpoint (.ppt)</a></li>
<li><a href="javascript:void(0);" data-value="pdf">Adobe Acrobat PDF (.pdf)</a></li>
<li><a href="javascript:void(0);" data-value="rtf">RTF (.rtf)</a></li>
<li><a href="javascript:void(0);" data-value="all">全部文档</a></li>
</ul>
</dd>
<input type="hidden" name="filetype" value="">
</dl>
<dl>
<dt>搜索结果排序方式</dt>
<dd class="js-as-radio">
<a uigs-id="adv_relevance-ranking" href="javascript:void(0);" data-value="off" class="dk-btn cur">按相关性排序</a>
<a uigs-id="adv_time-sort" href="javascript:void(0);" data-value="on" class="dk-btn">按时间排序</a>
<input type="hidden" name="tro" value="off">
</dd>
</dl>
<p class="enter"><input id="adv-search-btn" uigs-id="adv_search-btn" type="submit" class="a1" value="开始搜索"></p>
</form>
</div>
</div>
</div>
<div class="wrapper" id="wrap">
<div class="header">
<div class="top-nav">
<ul>
<li><a onclick="st(this,'40030300','news')" href="http://news.sogou.com" uigs-id="nav_news" id="news">新闻</a></li>
<li class="cur"><span>网页</span></li>
<li><a onclick="st(this,'73141200','weixin')" href="http://weixin.sogou.com/" uigs-id="nav_weixin" id="weixinch">微信</a></li>
<li><a onclick="st(this,'40051200','zhihu')" href="http://zhihu.sogou.com/" uigs-id="nav_zhihu" id="zhihu">知乎</a></li>
<li><a onclick="st(this,'40030500','pic')" href="http://pic.sogou.com" uigs-id="nav_pic" id="pic">图片</a></li>
<li><a onclick="st(this,'40030600','video')" href="https://v.sogou.com/" uigs-id="nav_v" id="video">视频</a></li>
<li><a href="http://mingyi.sogou.com?fr=common_index_nav" uigs-id="nav_mingyi" id="mingyi" onclick="st(this,'','myingyi')">明医</a></li>
<li><a href="http://english.sogou.com?fr=pcweb_index_nav" uigs-id="nav_overseas" id="overseas" onclick="st(this,'','overseas')" >英文</a></li>
<li><a onclick="st(this,'web2ww','wenwen')" href="https://wenwen.sogou.com/?ch=websearch" uigs-id="nav_wenwen" id="index_more_wenwen">问问</a></li>
<li><a href="http://scholar.sogou.com?fr=common_index_nav" uigs-id="nav_scholar" id="scholar" onclick="st(this,'','scholar')">学术</a></li>
<li class="show-more">
<a href="javascript:void(0);" id="more-product">更多<i class="m-arr"></i></a>
<div class="pos-more" id="products-box" style="top: 40px;">
<span class="ico-san"></span>
<a onclick="st(this,'40031000')" href="http://map.sogou.com" uigs-id="nav_map" id="map">地图</a>
<a onclick="st(this,'40031500')" href="http://gouwu.sogou.com/" uigs-id="nav_gouwu" id="index_more_gouwu">购物</a>
<a onclick="st(this,'40051203')" href="http://baike.sogou.com/Home.v" uigs-id="nav_baike" id="index_more_baike">百科</a>
<a onclick="st(this)" href="http://zhishi.sogou.com" uigs-id="nav_zhishi" id="index_more_zhishi">知识</a>
<a onclick="st(this,'40051205')" href="http://as.sogou.com/" uigs-id="nav_app" id="index_more_appli">应用</a>
<a onclick="st(this,'40051205','fanyi')" href="http://fanyi.sogou.com?fr=common_index_nav_pc" uigs-id="nav_fanyi" id="index_more_fanyi">翻译</a>
<a href="http://index.sogou.com" uigs-id="nav_index" id="index_more_index">指数</a>
<a href="http://dangjian.sogou.com" uigs-id="nav_dangjian" id="dangjian" onclick="st(this,'','dangjian')">党建</a>
<span class="all"><a onclick="st(this,'40051206')" href="http://www.sogou.com/docs/more.htm?v=1" uigs-id="nav_all" target="_blank">全部</a></span>
</div>
</li>
</ul>
</div> <div class="user-box">
<div class="local-weather" id="local-weather">
<div class="wea-box" id="cur-weather" style="display: none;"></div>
<div class="pos-more" id="detail-weather" style="top:40px;"></div>
</div>
<span class="line" id="user-box-line" style="display: none;"></span>
<div class="user-enter">
<a href="javascript:void(0);" id="show-card" style="display: none" uigs-id="settings_show-card">显示卡片</a>
<a href="javascript:void(0);" uigs-id="settings_change-skin" id="changeSkinBtn" >换肤</a>
<span class="s-dw">
<a href="javascript:void(0);" id="settings">设置</a>
<div class="pos-more" id="settings-box" style="top:40px;">
<span class="ico-san"></span>
<a href="javascript:void(0);" id="search-settings" uigs-id="settings_config">搜索设置</a>
<a href="javascript:void(0);" id="advanced-search" uigs-id="settings_advanced">高级搜索</a>
<a href="http://help.sogou.com/?w=01091500&v=1" uigs-id="settings_help">帮助</a>
</div>
</span>
<a href="javascript:void(0);" class="enter" id="loginBtn">登录</a> </div>
</div>
</div>
<div class="content" id="content">
<div class="pos-header" id="top-float-bar">
<div class="part-one"></div>
<div class="part-two" id="card-tab-layer">
<div class="c-top" id="top-card-tab"></div>
</div>
</div>
<div class="logo2" id="logo-s"><span></span></div> <div class="logo" id="logo-l"><span></span></div> <div class="search-box" id="search-box">
<form action="/web" name="sf" id="sf">
<span class="sec-input-box">
<input type="text" class="sec-input active" name="query" id="query" maxlength="100" len="80" autocomplete="off" />
</span>
<span class="enter-input"><input type="submit" value="" id="stb"></span>
<input type="hidden" name="_asf" value="www.sogou.com" />
<input type="hidden" name="_ast" />
<input type="hidden" name="w" value="01019900" />
<input type="hidden" name="p" value="40040100" />
<input type="hidden" name="ie" value="utf8" />
<input type="hidden" name="from" value="index-nologin" />
<input type="hidden" name="s_from" value="index" />
<div class="keywords-tips" id="keywordsTips" style="display:none">
<i></i><p>搜狗的查询限制在"<strong>40个汉字</strong>"以内。</p>
</div>
</form>
</div>
</div>
<div class="card-box" id="card-box" style="display: none;">
<div class="card-box2" id="card-box2">
<div class="c-top" id="card-tab-box">
<a href="javascript:void(0);" id="card-settings" uigs-id="settings_settings-btn" class="shezhi"></a>
<div class="pos-more" id="card-options">
<span class="ico-san"></span>
<a href="javascript:void(0);" uigs-id="settings_close-card" id="close-card">关闭卡片</a>
</div>
</div>
<div class="c-main" id="card-content"></div>
</div>
</div>
<div class="loog-more" id="scroll-more" style="display: none;">
<a href="javascript:void(0);" uigs-id="scroll-more">滚动查看更多<br><span class="ico_san"></span></a>
</div> <div class="ft" id="footer" style="display: none;">
<a href="http://fuwu.sogou.com/" target="_blank" uigs-id="footer_tuiguang">企业推广</a><span class="line"></span><a href="http://corp.sogou.com/" target="_blank" uigs-id="footer_about">关于搜狗</a><span class="line"></span><a href="http://ir.sogou.com/" target="_blank" uigs-id="footer_aboutEnglish">About Sogou</a><span class="line"></span><a href="http://www.sogou.com/docs/terms.htm?v=1" target="_blank" uigs-id="footer_disclaimer">免责声明</a><span class="line"></span><a href="http://fankui.help.sogou.com/index.php/web/web/index/type/4" target="_blank" uigs-id="footer_feedback">意见反馈及投诉</a><span class="line"></span><a href="http://corp.sogou.com/private.html" target="_blank" uigs-id="footer_private">隐私政策</a><br>
© 2004-2019 Sogou.com / <span class="g">京网文 (2016) 6432-852号</span> / <a href="http://www.miibeian.gov.cn" target="_blank" class="g">京ICP证050897号</a><br>
<span class="g">(京)-经营性-2016-0019</span> / <a href="http://www.miibeian.gov.cn/" target="_blank" class="g">京ICP备11001839号-1</a> / <a href="http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000025" class="ba" target="_blank">京公网安备11000002000025号</a>
</div>
<div class="ft-v1" id="QRcode-footer" style="padding-bottom:53px; ">
<div class="erwm-box">
<span class="ewm"></span>
<div class="erwx">
<p>搜狗搜索APP</p>
<p class="p2">搜你所想</p>
</div>
</div>
<div class="ft-info">
<a uigs-id="mid_pinyin" href="http://pinyin.sogou.com/" target="_blank"><i class="i1"></i>搜狗输入法</a><span class="line"></span><a uigs-id="mid_liulanqi" href="http://ie.sogou.com/" target="_blank"><i class="i2"></i>浏览器</a><span class="line"></span><a uigs-id="mid_daohang" href="http://123.sogou.com/" target="_blank"><i class="i3"></i>网址导航</a><br> <a href="http://corp.sogou.com/" target="_blank" class="g">关于搜狗</a> - <a href="http://ir.sogou.com/" target="_blank" class="g">About Sogou</a> - <a href="http://fuwu.sogou.com/" target="_blank" class="g">企业推广</a> - <a href="http://www.sogou.com/docs/terms.htm?v=1" target="_blank" class="g">免责声明</a> - <a href="http://fankui.help.sogou.com/index.php/web/web/index/type/4" target="_blank" class="g">意见反馈及投诉</a> - <a href="http://corp.sogou.com/private.html" target="_blank" class="g" uigs-id="footer_private">隐私政策</a><br>
© 2004-2019 Sogou.com / <span class="g">京网文 (2016) 6432-852号</span> / <span class="g">(京)-经营性-2016-0019</span><br>
<a href="http://www.miibeian.gov.cn" target="_blank" class="g">京ICP证050897号</a> / <a href="http://www.miibeian.gov.cn/" target="_blank" class="g">京ICP备11001839号-1</a> / <a href="http://www.beian.gov.cn/portal/registerSystemInfo?recordcode=11000002000025" class="ba" target="_blank">京公网安备11000002000025号</a>
</div>
</div> <div class="kuozhan" id="QRcode-box" style="display: none;">
<a href="javascript:void(0);" id="miniQRcode"></a>
<span id="QRcode"></span>
</div>
<a href="javascript:void(0);" class="back-top" id="back-top"></a> </div>
<script>
var SugPara, uigs_para,
msBrowserName = navigator.userAgent.toLowerCase(),
msIsSe = false,
msIsMSearch = false,
hasDoodle = false,
queryinput = document.getElementById('query');
uigs_para={
"uigs_productid": "webapp",
"type": "webindex_new",
"stype": "nologin",
"scrnwi": screen.width,
"scrnhi": screen.height,
"uigs_pbtag": "A",
"uigs_cookie": "SUID,sct",
"protocol": location.protocol.toLowerCase() == "https:" ? "https" : "http"
};
SugPara = {"enableSug":true,"sugType":"web","domain":"w.sugg.sogou.com","productId":"web","sugFormName":"sf","inputid":"query","submitId":"stb","suggestRid":"01015002","normalRid":"01019900","useParent":0 ,"sugglocation":"index","showVr":true,"showHotwords":true,"suggAbtestObject":{"suggestHistoryStrategy1":"","suggestHistoryStrategy2":"0|1|2|3|4|5|6|7|8","suggHistoryAbtest":""}};
function mk_con() {
try {
window.external.metasearch('make_connection', 'www.google.com.hk');
} catch (e) {}
}
if (/se 2\.x/i.test(msBrowserName)) {
msIsSe = true;
}
if (/metasr/i.test(msBrowserName)) {
msIsMSearch = true;
}
if (queryinput) {
if (msIsSe && msIsMSearch) {
if (queryinput.addEventListener) {
queryinput.addEventListener('keypress', mk_con, false);
queryinput.addEventListener('keydown', mk_con, false)
} else if (queryinput.attachEvent) {
queryinput.attachEvent('onkeypress', mk_con);
queryinput.attachEvent('onkeydown', mk_con);
} else {
queryinput.onkeypress = mk_con;
queryinput.onkeydown = mk_con;
}
}
}
function getDomain(){
var domainName = document.domain;
if(domainName.indexOf("sogou.com")==(domainName.length-9)){
return ".sogou.com";
}else if(domainName.indexOf("soso.com")==(domainName.length-8)){
return ".soso.com";
}else if(domainName.indexOf("sogo.com") != -1){
return ".sogo.com"
}
}
window.m_s_index = function() {
var w = document.sf.query,
c = Math.round((new Date().getTime() + Math.random()) * 1000);
w.focus();
if(new RegExp("kw=([^&]+)").test(location.search)) {
if(w.value.length == 0) {
w.value = decodeURIComponent(RegExp.$1);
}
}
if (document.cookie.indexOf("SUV=") < 0) {
document.cookie = "SUV=" + c + ";path=/;expires=Sun, 29 July 2026 00:00:00 UTC;domain="+getDomain();
}
(new Image).src = '//pb6.sogou.com/v6';
};
function st(self, p, product, anchor) {
var searchBox = document.sf.query,
query = encodeURIComponent(searchBox.value),
productUrl = {
"news": 'http://news.sogou.com/news?ie=utf8&query=',
"web": 'web?ie=utf8&query=',
"weixin": 'http://weixin.sogou.com/weixin?type=2&ie=utf8&query=',
"zhihu": 'http://zhihu.sogou.com/zhihu?ie=utf8&query=',
"pic": 'http://pic.sogou.com/pics?ie=utf8&query=',
"video": 'https://v.sogou.com/v?ie=utf8&query=',
"myingyi": 'https://www.sogou.com/web?m2web=mingyi.sogou.com&ie=utf8&query=',
"overseas": 'http://english.sogou.com?b_o_e=1&ie=utf8&fr=pcweb_index_nav&query=',
"scholar": 'http://scholar.sogou.com?ie=utf8&fr=common_index_nav&query=',
"fanyi": 'http://fanyi.sogou.com/?fr=common_index_nav_pc&ie=utf8&keyword=',
"wenwen":'http://wenwen.sogou.com/s/?ch=websearch&w=',
"dangjian":'http://dangjian.sogou.com/dangjian?query='
},
newHref = productUrl[product] || self.href;
function getConnectSymbol(url) {
return url.indexOf("?") > -1 ? '&' : '?';
}
if(searchBox && searchBox.value !== ''){
if(productUrl[product]) {
newHref = productUrl[product] + query;
} else if(newHref.indexOf("kw=") > 0) {
newHref = newHref.replace(new RegExp("kw=[^&$]*"), "kw=" + query)
} else {
newHref += getConnectSymbol(newHref) + 'kw=' + query;
}
}
if(p){
newHref += getConnectSymbol(newHref) + "p=" + p;
}
if (anchor && anchor.length > 0){
newHref += "#" + anchor;
}
if (searchBox && searchBox.value == '' && (product == 'wenwen' || product == 'dangjian')){//问问首页链接单独处理
newHref = self.href;
}
self.href = newHref;
}
window.cid = function(o, p) {
var w = document.sf.query,
q = encodeURIComponent(w.value);
if (!q) {
o.href += "?cid=" + p
} else {
if (p === "web2ww") {
o.href += "s/?cid=web2ww&w=" + q
} else if (p === "web2bk") {
o.href += "Search.e?sp=S" + q + "&cid=web2bk"
}
}
};
window.m_s_index();
</script>
<script src="//dlweb.sogoucdn.com/common/lib/jquery/jquery-1.11.0.min.js"></script>
<script charset="gbk" type="text/javascript" src="/js/sugg_new.v.104.js"></script>
<script src="/js/pb_v.1.9.6.min.js"></script>
<script src="/js/lib/jquery.mousewheel.min.js"></script>
<script src="/js/lib/juicer-min.js"></script>
<script src="/js/common/widget/login_new.min.v.0.5.js"></script>
<script src="//account.sogou.com/static/api/passport-async.js"></script>
<script src="/web/index/js/base.v.1.1.14.js"></script>
<script src="/web/js/voice.min.v.0.0.6.js"></script>
<script src="/web/js/taspeed.min.v.0.0.1.js"></script>
</body>
</html>
<!--zly-->
View Code
3.移动端数据爬取&&环境配置等
实验:参考下面的blog
https://www.cnblogs.com/bobo-zhang/p/10068994.html
- 移动端数据爬取：
- 抓包工具：(定义:代理服务器)
window:- fiddler，mitproxy(两者都是代理服务器)
mac:青花瓷
- 在手机中安装证书：
- 1让电脑开启一个wifi，然后手机连接wifi（手机和电脑是在同一个网段下）
- 手机浏览器中：ip:8888,点击超链进行证书下载
- 需要将手机的代理开启：将代理ip和端口号设置成fiddler的端口和fidd所在机器的ip
(1)将证书发送给"手机"
(2)在Fiddler中,点击Tools=>Options=>
下一步,"允许"其他设备连接:=>"确定"=>OK
在浏览器中访问:http://localhost:8888/http://localhost:8888/
得到下面的结果
我们可以在上图的最后一行下载"证书"
标签：sogou,http,回顾,newHref,爬取,nbsp,&&,query,com 来源： https://www.cnblogs.com/studybrother/p/10957649.html