使用AWS Lambda和api-gateway创建API时出错（python）

Question

TL;DR: getting errors using AWS Lambda and API-Gateway

Background

I wrote a python script to help me track some index funds I am using. It basically scrapes (using requests and bs4) the price from different websites.

I wanted to turn this into an API so I can use it anywhere, so I used Flask and Connexion to make this (followed this https://realpython.com/flask-connexion-rest-api/ ). That's all working perfectly fine locally, but I'm having trouble implementing this on AWS.

I have zipped up my code and the corresponding app packages to AWS Lambda and using that in the API Gateway. However I am getting the following errors:

Endpoint response body before transformations: {"errorMessage": "module initialization error"} Method response body after transformations: {"errorMessage": "module initialization error"}

I've googled for this, followed the steps here ( Amazon API Gateway : response body is not transformed when the API is called via Postman? ), deployed the app etc - not sure why this is happening.

here is a snippet of the code:

import requests
from bs4 import BeautifulSoup
from datetime import date, timedelta
import datetime
import json


def CurrentPortfolioValue(event, context):
    initial_porfolio_price = float (100)
    initial_date = date(2018, 8, 19) #set the initial date to a date where the fortnightly payment has gone
    num_of_shares_vanguard = float(500)
    num_of_shares_AMPNZShares = float(300)
    num_of_shares_AMPAustralisianProperty = float(100)
    fortnightly_contrib = float(100)
    current_value = initial_porfolio_price
    #print(initial_date)

    #sets a user-agent    
    example_headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'} #can make this better by having a list of random header


    #vanguard
    vanguard = requests.get("https://www.bloomberg.com/quote/VAN1579:AU",headers=example_headers) #need to have headers to avoid being blocked
    soup = BeautifulSoup(vanguard.content, 'html.parser')
    vanguard_price = soup.find_all("span", class_ = 'priceText__1853e8a5')[0].get_text()
    #print("Vanguard Price:\n" + vanguard_price+"AUD")

    #AMP Capital NZ Index 
    ampnzshares = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/nz-shares-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup1 = BeautifulSoup(ampnzshares.content, 'html.parser')
    ampnzshares_price = soup1.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital NZ Index Price:\n" + ampnzshares_price+"NZD")

    #AMP Capital Australisian Property Index Fund
    ampAustralisianProperty = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/australasian-property-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup2 = BeautifulSoup(ampAustralisianProperty.content, 'html.parser')
    ampAustralianProperty_price = soup2.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital Australisian Property Index Fund:\n" + ampAustralianProperty_price+"NZD")

    #get day difference between today and initial date
    current_date = date.today()
    days_difference = (current_date - initial_date).days #.days shows just the number of days
    #print(days_difference)

    count = days_difference
    while count > 14: #while difference is less than 14 days i.e forntightly payment hasnt been made
        current_value += initial_porfolio_price + (float(50) * float(vanguard_price)) + (float(50) * float(ampnzshares_price))
        count = count - 14
    print("Current Portfolio value is $" + str(current_value))
    return {
        "statusCode": 200,
        "body": json.dumps(current_value)}

CurrentPortfolioValue() #retrieves current portfolio value

my yaml file looks like ths:

swagger: "2.0"
info:
  description: This is the swagger file that goes with our server code
  version: "1.0.0"
  title: Swagger REST Article
consumes:
  - "application/json"
produces:
  - "application/json"

basePath: "/api"

paths:
  /value: #name of the endpoint
    get:
      operationId: "ShareScraper.CurrentPortfolioValue" #link to the module - this is called when you navigate to /value
      tags:
        - "Portfolio"
      summary: "The portfolio data structure supported by the server application"
      description: "Read the current portfolio value"
      responses:
        200:
          description: "Successful read current portfolio value"
          schema:
            type: "string"
            items:
              properties:
                portfolio_value:
                  type: "string"

Thanks in advance!

[Edit] This is my first time creating an api and using AWS so sorry if I am missing something fairly simple/obvious

Answer 1

Your lambda code doesn't have the event and context argument. Which are required arguments for a lambda python handler. You can modify your code as,

def CurrentPortfolioValue(event, context):

    current_value += initial_porfolio_price + (float(50) * float(vanguard_price)) + (float(50) * float(ampnzshares_price))
    count = count - 14
    print("Current Portfolio value is $" + str(current_value))
    return {
        "statusCode": 200,
        "body": json.dumps(current_value)}

CurrentPortfolioValue() #retrieves current portfolio value

Answer 2

Your Lambda function doesn't appear to be correctly formatted as a Lambda. As per the AWS documentation, here , your Lambda must be formatted as a handler and must accept the event and context arguments.

You will need to edit your code to look more like this in order to get it to run:

from botocore.vendored import requests
from bs4 import BeautifulSoup
import datetime
from datetime import date, timedelta
import json


def lambda_handler(event, context):

    initial_porfolio_price = float (100)
    initial_date = date(2018, 8, 19) #set the initial date to a date where the fortnightly payment has gone
    num_of_shares_vanguard = float(500)
    num_of_shares_AMPNZShares = float(300)
    num_of_shares_AMPAustralisianProperty = float(100)
    fortnightly_contrib = float(100)
    current_value = initial_porfolio_price
    #print(initial_date)

    #sets a user-agent
    example_headers = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; WOW64; rv:50.0) Gecko/20100101 Firefox/50.0'} #can make this better by having a list of random header

    #vanguard
    vanguard = requests.get("https://www.bloomberg.com/quote/VAN1579:AU",headers=example_headers) #need to have headers to avoid being blocked
    soup = BeautifulSoup(vanguard.content, 'html.parser')
    print(soup)
    vanguard_price = soup.find_all("span", class_ = 'priceText__1853e8a5')[0].get_text()
    #print("Vanguard Price:\n" + vanguard_price+"AUD")

    #AMP Capital NZ Index
    ampnzshares = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/nz-shares-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup1 = BeautifulSoup(ampnzshares.content, 'html.parser')
    ampnzshares_price = soup1.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital NZ Index Price:\n" + ampnzshares_price+"NZD")

    #AMP Capital Australisian Property Index Fund
    ampAustralisianProperty = requests.get("https://www.ampcapital.com/nz/en/investments/funds/index-funds/australasian-property-index-fund",headers=example_headers) #need to have headers to avoid being blocked
    soup2 = BeautifulSoup(ampAustralisianProperty.content, 'html.parser')
    ampAustralianProperty_price = soup2.find_all("div", class_ = 'ht-module_title ht-highlight_color')[1].get_text()
    #print("AMP Capital Australisian Property Index Fund:\n" + ampAustralianProperty_price+"NZD")

    #get day difference between today and initial date
    current_date = date.today()
    days_difference = (current_date - initial_date).days #.days shows just the number of days
    #print(days_difference)

    count = days_difference
    while count > 14: #while difference is less than 14 days i.e forntightly payment hasnt been made
        current_value += initial_porfolio_price + (float(50) * float(vanguard_price)) + (float(50) * float(ampnzshares_price))
    count = count - 14
    print("Current Portfolio value is $" + str(current_value))
    return {
       "statusCode": 200,
       "body": json.dumps(current_value)}

You didn't post your import statements but at the very least you'll need to import json.

Note that the call to the function is also removed. I've assumed that this function is running in isolation (ie nothing else running with it) and therefore you do not need to call it within the Lambda itself, as it will get executed in the Lambda environment.

EDIT:

I've updated the code block and run it. It is indeed running in Lambda, however, at line 27 it is trying to find "span", class_ = 'priceText__1853e8a5' which doesn't exist in the response to the vanguard = requests.get... request. Instead, as you can see in the screenshot below, we're receiving something else, and Lambda is giving us the error "errorMessage": "list index out of range" .

To find out why, we have to look in the Log Output. When we look at the output we see that instead of a page with the pricing information that we're trying to scrape, what is being received is actually a Captcha page; Bloomberg recognize that you're scraping the data and they don't want people scraping their data, so to prevent this they want you to complete a Captcha, and because haven't built a way to handle this situation the Lambda is erroring out.