[英]How to increase memory available for chromium in chrome-aws-lambda?
I collect the data using puppeteer and chrome-aws-lambda.我使用 puppeteer 和 chrome-aws-lambda 收集数据。 I plan to push it to AWS Lambda but while testing locally I get an error:
我计划将其推送到 AWS Lambda 但在本地测试时出现错误:
Error: Protocol error (Runtime.callFunctionOn): Target closed.
when I call for waitForSelector
.当我调用
waitForSelector
时。
I've some posts that mentioned there is a chance that chrome process gets too little memory within the docker.我有一些帖子提到铬工艺有可能在 docker 中变得太少 memory。 The question is: how to get it more memory?
问题是:如何获得更多的memory? I also read that
disable-dev-shm-usage
may help, but it doesn't.我还读到
disable-dev-shm-usage
可能有帮助,但没有。 That's how I do it now (the last line is where error happens):这就是我现在的做法(最后一行是发生错误的地方):
const chromium = require('chrome-aws-lambda');
browser = await chromium.puppeteer.launch({
args: [...chromium.args, `--proxy-server=${proxyUrl}`, '--disable-dev-shm-usage'],
defaultViewport: chromium.defaultViewport,
executablePath: await chromium.executablePath,
headless: chromium.headless,
ignoreHTTPSErrors: true,
});
const page = await browser.newPage();
await page.authenticate({ username, password });
await page.goto(MY_URL, { waitUntil: 'domcontentloaded' })
await page.click(SUBMIT_SELECTOR);
await page.waitForSelector('#myDiv')
.then(() => console.log('got it')).
catch((e)=>console.log('Error happens: '+ e));
UPDATE: more info on local setup:更新:有关本地设置的更多信息:
I run it locally using sam local start-api
.我使用
sam local start-api
在本地运行它。 Here is the content of my template.yaml
(just a slightly updated hello-world
template:这是我的
template.yaml
的内容。yaml(只是一个稍微更新hello-world
模板:
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
samnode
Sample SAM Template for samnode
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 60
Resources:
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: hello-world/
Handler: app.lambdaHandler
Runtime: nodejs14.x
MemorySize: 4096
Layers:
- !Sub 'arn:aws:lambda:${AWS::Region}:764866452798:layer:chrome-aws-lambda:22'
Events:
HelloWorld:
Type: Api # More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Properties:
Path: /hello
Method: get
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
HelloWorldApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
HelloWorldFunction:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt HelloWorldFunction.Arn
HelloWorldFunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt HelloWorldFunctionRole.Arn
MemorySize: 4096
You have already configured 4GB memory for the Lambda and it should be more than enough to load couple of pages.您已经为 Lambda 配置了 4GB memory,它应该足以加载几页。 If you still feel this is the issue, you can increase the memory upto 10240. I suspect the error may not be related to memory.
如果您仍然觉得这是问题,您可以将 memory 增加到 10240。我怀疑该错误可能与 memory 无关。
To verify, you can do the following to see if the Lambda is actually getting the specified memory.为了验证,您可以执行以下操作来查看 Lambda 是否真的得到了指定的 memory。
Run the lambda in Eager mode (This keeps the lambda running on local even if there are no active requests)在 Eager 模式下运行 lambda(即使没有活动请求,这也会使 lambda 在本地运行)
sam local start-api --warm-containers EAGER
Now run the following command to track the memory consumption现在运行以下命令来跟踪 memory 的消耗
docker stats
You can send a request to your local api now and track the memory consumption.您现在可以向您当地的 api 发送请求并跟踪 memory 的消耗。 If you see less than 4GB memory allocated to your lambda function, then update the Docker resources and ensure you allocate appropriate memory to Docker.
If you see less than 4GB memory allocated to your lambda function, then update the Docker resources and ensure you allocate appropriate memory to Docker.
Update Docker Resources (Increase memory)更新 Docker 资源(增加内存)
Try out different versions of chrome-aws-lambda
(may be using a local layer with SAM ).尝试不同版本的
chrome-aws-lambda
(可能使用带有 SAM 的本地层)。 I would also run the same block of code on local using Puppeteer
by disabling the headless mode and verify the selector the code is waiting for is actually available.我还将使用
Puppeteer
在本地运行相同的代码块,方法是禁用无头模式并验证代码等待的选择器实际上是否可用。
puppeteer
dependency.puppeteer
依赖项。puppeteer
instead of chrome-aws-lambda
puppeteer
而不是chrome-aws-lambda
const puppeteer = require('puppeteer');
browser = await puppeteer.launch({headless: false});
node <replace-with-your-file-name.js>
eg if the file name is somejsfile.js then the command would be node somefile.js
node <replace-with-your-file-name.js>
运行文件,例如,如果文件名是 somejsfile.js,那么命令将是node somefile.js
Hope this helps you proceed further.希望这可以帮助您进一步进行。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.