Can I script puppeteer line by line in the node.js command line window?

Question

I have a website I am trying to automate scraping data from. I have a legitimate account and need to make my life easier by programmatically gathering data. The problem is, for every iteration of my puppeteer code my account is logged into afresh, and after a couple of rounds of coding I get an email from the service (MLS) that they noticed suspicious activity on my account. My question is, is it possible to script puppeteer code in realtime, so I can test different lines of code within the same session?

So far I've tried this:

const puppeteer = require('puppeteer');
const fs = require("fs");
const browser = puppeteer.launch({headless: false, ignoreHTTPSErrors: true, args: ['--start-maximized'], defaultViewport: null, slowMo: 150})
const page = browser.newPage();
page.goto('https://idp.sdmls.com/idp/Authn/UserPassword',{waitUntil: 'networkidle2'});

The Chromium windows opens but nothing else. I get this in the command line window:

> const puppeteer = require('puppeteer');
undefined
> const fs = require("fs");
undefined
>
> const browser = puppeteer.launch({headless: false, ignoreHTTPSErrors: true, args: ['--start-maximized'], defaultViewport: null, slowMo: 150})
undefined
> const page = browser.newPage();
TypeError: browser.newPage is not a function
> page.goto('https://idp.sdmls.com/idp/Authn/UserPassword',{waitUntil: 'networkidle2'});
ReferenceError: page is not defined
>

Answer 1

puppeteer.launch is an async function that returns a Promise . That means that you should await for it in your async function, or put the subsequent code into .then() .

Since you want to use Node.js command line, you cannot directly await for it. There are several options:

Option 1: put your code in .then() :

$ node
Welcome to Node.js v12.1.0.
Type ".help" for more information.
> const options = {headless: false, ignoreHTTPSErrors: true, args: ['--start-maximized'], defaultViewport: null, slowMo: 150};
...
> const puppeteer = require('puppeteer');
undefined
> const browserPromise = puppeteer.launch(options);
undefined
> const pagePromise = browserPromise.then(browser => { return browser.newPage(); });
undefined
> pagePromise.then(page => { page.goto('http://stackoverflow.com'); });
Promise { <pending> }

Option 2 (good for debugging, but do not try to use it in your real code ): save promise results into the global object and reuse.

> puppeteer.launch(options).then(browser => { global.browser = browser; });
Promise { <pending> }

Wait for some time until the promise resolves and the global object is assigned, then continue:

> global.browser.newPage().then(page => { global.page = page; });
Promise { <pending> }

Wait again, then go ahead:

> global.page.goto('http://stackoverflow.com')
Promise { <pending> }