使用psycopg2和Lambda来更新Redshift(Python)

我试图使用python从Lambda函数更新Redshift.为此,我试图合并两个代码片段.当我单独运行它们时,两个片段都是有效的.

>从PyDev for Eclipse更新Redshift

import psycopg2

conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"
conn = psycopg2.connect(conn_string)

cursor = conn.cursor()

cursor.execute("UPDATE table SET attribute='new'")
conn.commit()
cursor.close()

>接收上传到S3 Bucket的内容(Lambda上可用的预建模板)

from __future__ import print_function

import json
import urllib
import boto3

print('Loading function')

s3 = boto3.client('s3')


def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))

    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')

    try:
        response = s3.get_object(Bucket=bucket, Key=key)
        print("CONTENT TYPE: " + response['ContentType'])
        return response['ContentType']

    except Exception as e:
        print(e)
        print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
        raise e

由于这两个段都有效,我尝试将它们组合起来,以便在将文件上传到s3时更新Redshift：

from __future__ import print_function

import json
import urllib
import boto3
import psycopg2

print('Loading function')

s3 = boto3.client('s3')


def lambda_handler(event, context):
    #print("Received event: " + json.dumps(event, indent=2))

    # Get the object from the event and show its content type
    bucket = event['Records'][0]['s3']['bucket']['name']
    key = urllib.unquote_plus(event['Records'][0]['s3']['object']['key']).decode('utf8')

    conn_string = "dbname='name' port='0000' user='name' password='pwd' host='url'"

    conn = psycopg2.connect(conn_string)

    cursor = conn.cursor()

    cursor.execute("UPDATE table SET attribute='new'")
    conn.commit()
    cursor.close()

    try:
        response = s3.get_object(Bucket=bucket, Key=key)
        print("CONTENT TYPE: " + response['Body'].read())
        return response['Body'].read()
    except Exception as e:
        print(e)
        print('Error getting object {} from bucket {}. Make sure they exist and your bucket is in the same region as this function.'.format(key, bucket))
        raise e

由于我使用的是外部库,因此我需要创建一个部署包.我创建了一个新文件夹(lambda_function1)并将我的.py文件(lambda_function1.py)移动到该文件夹​​.我运行以下命令在该文件夹中安装psycopg2：

pip install psycopg2 -t \lambda_function1

我收到以下反馈：

Collecting psycopg2
  Using cached psycopg2-2.6.1-cp34-none-win_amd64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.6.1 

然后我压缩了目录的内容.并将该zip文件上传到我的lambda函数中.当我将文档上传到功能监视器的存储桶时,我在cloudwatch日志中收到以下错误：

Unable to import module 'lambda_function1': No module named _psycopg 

当我查看图书馆时,唯一名为“_psycopg”的是“_psycopg.pyd”.

是什么导致了这个问题？当我使用3.4时,Lambda使用Python 2.7是否重要？我在Windows机器上压缩文件内容是否重要？有没有人能够成功连接到lambda的Redshift？

解决方法:

为了使其工作,您需要使用静态链接的libpq.so库构建psycopg2.看看这个repo https://github.com/jkehler/awslambda-psycopg2.它已经构建了psycopg2包,并说明了如何自己构建它.

回到你的问题：

是什么导致了这个问题？

psycopg2需要构建一个使用静态链接库编译的Linux.

当我使用3.4时,Lambda使用Python 2.7是否重要？

是的,lambda只支持2.7版本.只需创建虚拟环境并在其中安装所有必需的软件包.

我在Windows机器上压缩文件内容是否重要？

只要您压缩的所有库都可以在Linux上运行,它就不会

有没有人能够成功连接到lambda的Redshift？

是.

标签：python,amazon-web-services,aws-lambda,amazon-redshift,aws-sdk
来源： https://codeday.me/bug/20190923/1815656.html