Get a javascript variable from a web page without interaction/heedlessly

Question

Good afternoon, We're looking to get a javascript variable from a webpage, that we are usually able to retrieve typing app in the Chrome DevTools.

However, we're looking to realize this headlessly as it has to be performed on numerous apps.

Our ideas:

• Using a Puppeteer instance to go on the page, type the command and return the variable, which works, but it's very ressource consuming.

• Using a GET/POST request to the page trying to inject the JS command, but we didn't succeed.

We're then wondering if there will be an easier solution, as a special API that could extract the variable? The goal would be to automate this process with no human interaction.

Thanks for your help!

Answer 1

You can embed Chrome into your application and instrument it. It will be headless.
We've used this approach in the past to copy content from PowerPoint Online.

We were using .NET to do this and therefore used CEFSharp.

Answer 2

Your question is not so much about a JS API (since the webpage is not yours to edit, you can only request it) as it is about webcrawling / browser automation.

You have to add details to get a definitive answer, but I see two scenarios:

• the website actively checks for evidence of human browsing (for example, it sits behind CloudFlare and has requested this option); or the scripts depend heavily on there being a browser execution environment available. In this case, the simplest option is to automate a browser, because a headless option has to get many things right to fool the server or the scripts. I would use karate , which is easier than, say, selenium and can execute in-browser scripts . It is written in Java, but you can execute it externally and just read its reports.
• the website does not check for such evidence and the scripts do not really require a browser execution environment. Then you can simply download everything requires locally and attempt to jury-rig the JS into executing in any JS environment. According to your post, this fails; but it is impossible to help unless you can describe how it fails. This option can be headless.