python – Url开放编码

我有以下urllib和BeautifulSoup的代码：

getSite = urllib.urlopen(pageName) # open current site   
getSitesoup = BeautifulSoup(getSite.read()) # reading the site content 
print getSitesoup.originalEncoding
for value in getSitesoup.find_all('link'): # extract all <a> tags 
    defLinks.append(value.get('href')) 

结果：

/usr/lib/python2.6/site-packages/bs4/dammit.py:231: UnicodeWarning: Some characters could not be decoded, and were replaced with REPLACEMENT CHARACTER.
  "Some characters could not be decoded, and were "

当我尝试阅读网站时,我得到：

�7�e����0*"I߷�G�H����F������9-������;��E�YÞBs���������㔶?�4i���)�����^W�����`w�Ke��%��*9�.'OQB���V��@�����]���(P��^��q�$�S5���tT*�Z

解决方法:

该页面为UTF-8,但服务器以压缩格式将其发送给您：

>>> print getSite.headers['content-encoding']
gzip

在运行Beautiful Soup之前,您需要解压缩数据.我在数据上使用zlib.decompress()时出错了,但是将数据写入文件并使用gzip.open()从中读取工作正常 – 我不知道为什么.

标签：python,beautifulsoup,python-unicode
来源： https://codeday.me/bug/20190620/1249073.html