如何使用urlfetch获取所有cookie？

根据GAE fetch documentation,cookie不会通过重定向处理：

Cookies are not handled upon redirection. If cookie handling is needed, set follow_redirects to False and handle both cookies and redirects manually.

所以,我正在尝试实施手动解决方案：

page = urlfetch.Fetch(
    url = url,
    payload = form_data,
    method = urlfetch.POST,
    headers = headers,
    follow_redirects = False,
    deadline = 60)
cookies = ''
while page.status_code == 302:
    url = page.headers.get('location')
    if page.headers.get('set-cookie'):
        cookies = page.headers.get('set-cookie')
        headers['cookie'] = cookies
    page = urlfetch.Fetch(
        url = url,
        method = urlfetch.GET,
        headers = headers,
        follow_redirects = False,
        deadline = 60)
if page.status_code == 200 and page.content:
    self.response.out.write(page.content)

但它没有按预期工作.看起来我缺少some cookies：

header_msg
An instance of httplib.HTTPMessage containing the response headers. If there may be multiple headers with the same name (for
example, Set-Cookie headers), call header_msg.get_headers(header_name)
to retrieve the values as a list.

但是我应该如何使用header_msg呢？

解决方法:

如果我理解了这个问题,你想从每个响应中收集(并累积传递)cookie,但是使用follow_redirects = True的URLFetch只返回上一个响应中的cookie.此外,默认行为不实现cookie jar,这将导致后面的请求与先前响应中的Set-Cookies对应的正确Cookie头一起发送.据推测,初始POST是一个登录表单,重定向到期望cookie的页面,这个方案无法满足这些限制.

为此,您的代码已关闭,但cookies = page.headers.get(‘set-cookie’)在每次请求后消除以前收集的Cookie.这应该更好：

page = urlfetch.Fetch(
  url = url,
  headers = headers,
  follow_redirects = False)
cookies = []
while page.status_code == 302:
  url = page.headers.get('location')
  if page.headers.get('set-cookie'):
    cookies.extend(page.header_msg.getheaders('set-cookie'))
  headers['cookie'] = '; '.join(cookies)
  page = urlfetch.Fetch(
    url = url,
    method = urlfetch.GET,
    headers = headers,
    follow_redirects = False)
if page.status_code == 200 and page.content:
  self.response.out.write(page.content)

一些警告：

>如果Location是相对路径,则需要修复url.
>如果任何Set-Cookie标头不仅仅是key = value(例如它已过期),您将需要解析标头值,以便只发送键/值对.有关解析的帮助,请参阅Cookie库.
>如果针对特定密钥看到多个Set-Cookie,则此代码将很乐意发送重复的Cookie.
>如果重定向在单独的域上结束,则会错误地从原始域发送cookie.这可能是一个安全问题.正确的cookie jar实现可以推断域和路径限制,以确定何时接受和发出cookie.您可能想要合并cookielib.CookieJar库.如果您希望请求序列位于同一个域中,则只要检测到交换机就可以中止.

标签：urlfetch,python,cookies,google-app-engine
来源： https://codeday.me/bug/20190826/1726170.html