Web scraping in dialogflow

Question

I am trying to do web scraping in dialogflow.

const rp = require('request-promise');
const ch = require('cheerio');

.

app.intent('get show name', (conv, {show_name}) => {
    const url = 'https://screenrant.com/search/'+show_name;
    rp(url).then(function(html){
        let total_items = ch('article > div > h3 > a', html).length;
        if(total_items > 0){
            conv.ask(ch('article > div > h3 > a', html)[0].children[0].data);
        }
        else{
            conv.ask("Sorry no news on this!");
        }
    });
  });

This is my code. When running the action, I'm getting 'final_response' must be set.

https://pastebin.com/y9Rc0Vd8

Answer 1

This is an asynchronous command, so you need to return a Promise in your intent handler so the code knows to wait.

app.intent('get show name', (conv, {show_name}) => {
  const url = 'https://screenrant.com/search/'+show_name;
  return rp(url).then(function(html){
    let total_items = ch('article > div > h3 > a', html).length;
    if(total_items > 0){
        conv.ask(ch('article > div > h3 > a', html)[0].children[0].data);
    }
    else{
        conv.ask("Sorry no news on this!");
    }
  });
});

Answer 2

While you are doing web scraping in dialogflow. I will suggest using the in-built inline editor in it.

The same code be rewritten as

var request = require('request');
var cheerio = require('cheerio');
function welcome(agent) {
//agent.add(`Welcome to my agent!`);
var search_term = agent.parameters['term'];
st = urlencode(search_term);
request('https://screenrant.com/search/'+st+'/', function (error, response, html) {
  if (!error && response.statusCode == 200) {
    var $ = cheerio.load(html);
    var responseText = "Here are the news";
    $("article").each(function(){
        responseText += "- " + $(this).find("h3").text();
    })
    agent.add(responseText)
  }
});
}

To make it work, make sure the following:

• Your application is using v2 of dialogflow
• You have enabled billing for your firebase account associated with the dialogflow account (Switch to Pay as you go plan via Firebase console). This is mandatory to send external requests to the server.
• You have enabled "enable webhook call for this intent" option in the desired intent
• You have enabled "enable webhook call for slot filling" option in the desired intent