Python AWS Lambda Web scraper

Question

I need to scrape a URL to send content to my Lambda function. I am trying to achieve it by packaging BeautifulSoup with my Lambda function, but I am getting import errors like: cannot import name 'CharsetMetaAttributeValue' etc. I am not sure if bs4 should be used in AWS environment or not.

Any suggestions would be helpful.

Answer 1

I am able to use compiled lxml package with Lambda function for web scraper. Compiled package is available at github this link: https://github.com/JFox/aws-lambda-lxml/tree/master/3.6.4