Can I access elements from a web page with JavaScript?

Question

I'm making a Discord bot in JavaScript and implementing a feature where when you ask a coding question it gives you a snippet. I'm using Grepper and returning the url with the search results. For example: Hello World in JavaScript Search Results . I would like to access the div containing the snippet. Is this possible? And how would I do it?

Here's my code:

if (message.startsWith('programming')) {
    // Command = programming
    message = message.replace('programming ', ''); // Remove programming from the string
    message = encodeURIComponent(message) // Encode the string for a url
    msg.channel.send(`https://www.codegrepper.com/search.php?answer_removed=1&q=${message}`); // Place formatted string into url and send it to the discord server
    
    // Here the program should access the element containing the snippet instead of sending the url:

}

I'm new to JavaScript so sorry if this is a stupid question.

Answer 1

As far as I know the API you are using returns HTML/Text data, not JSON, Grepper has a lot more APIs if you just look into them, you can instead use this API that returns JSON data. If you need more information you can check this Unofficial List of Grepper APIs

https://www.codegrepper.com/api/get_answers_1.php?v=2&s=${SearchQuery}&u=98467

How Do I Access the div containing the snippet?
To access the div you might need to use python web scraping to scrape the innerHTML of the div but I think it's easier to use the other API.

Or

You can put /api/ in the url like:

https://www.codegrepper.com/api/search.php?answer_removed=1&q=js%20loop

Answer 2

The best way is to use the grepper api.

Install node-fetch

npm i node-fetch , You need this package for making requestes to the api. To import It in the code just type:

const fetch = require('node-fetch');

Write this code

Modify your code like this:

if (message.startsWith('programming')) {
    message = message.replace('programming ', '');
    message = encodeURIComponent(message)
    
    // Making the request
    fetch(`https://www.codegrepper.com/api/search.php?answer_removed=1&q=${message}`)
    .then(res => res.json())
    .then(response => {
        // response is a json object containing all the data You need
        // now You need to parse this data
        const answers = response.answers; // this is an array of objects
        const answers_limit = 3; // set a limit for the answers
        
        // cheking if there is any answer
        if(answers.length == 0) {
            return msg.channel.send("No answers were found!");
        }

        // creating the embed
        const embed = new Discord.MessageEmbed()
            .setTitle("Here the answers to your question!")
            .setDescription("")
        
        // parsing
        for(let i = 0; i < answers_limit; i++) {
            if(answers[i]) {
                embed.description += `**${i+1}° answer**:\n\`\`\`js\n${answers[i].answer}\`\`\`\n`;
            }
        }
        console.log(embed)

        msg.channel.send(embed);
    });
        
}

Answer 3

The easiest way for this is to send a GET request to the underlying API

https://www.codegrepper.com/api/search.php?q=hello%20world%20javascript&search_options=search_titles

This will return the answers in JSON format. Obviously you'd have to adjust the parameters.

How did I find out about this?
Simply look at the network tab of your browser's dev tools while loading the page. You'll see a GET request being sent out to the endpoint, returning mentioned answers as JSON.