Node.js puppeteer mysql - Inserting fetched values in database inside a loop using mysql

Question

I'm using node.js and puppeteer to get some data. ... now I want to insert the fetched data in a databases ... using mysql. The below seems to work ... yet what confuses me is that in the console.log('DB insert successful. Record: '+i); is always behind and after some time it stops ... although there are still tables with records available.

That's my app:

  let tableCell01;
  let tableCell01Val;
  let tableCell02;
  let tableCell02Val;

  const tableRows = await page.$$('table.tableFile2 > tbody > tr');

  for (let i=1; i < tableRows.length; i++){

    tableRow = tableRows[i];
    tableCell01 = await tableRow.$('td:nth-child(1) a');
    tableCell01Val = await page.evaluate( tableCell01 => tableCell01.innerText, tableCell01 );
    tableCell02 = await tableRow.$('td:nth-child(2)');
    tableCell02Val = await page.evaluate( tableCell02 => tableCell02.innerText, tableCell02 );

    tableCell02ValA.replace(/(^\s+|\s+$)/g,'');

    console.log('\n');
    console.log('ID: '+tableCell01Val);
    console.log('Company: '+tableCell02Val);
    console.log('Iterator: '+i);

    const insertCompanyList = "INSERT INTO companyList ( company_name, id ) values (?,?)";

    connection.query(insertCompanyList,[tableCell02Val, tableCell01Val],function(err, rows) {
      if (err) {
        console.log(err);
      } else {
        console.log('DB insert successful. Record: '+i);
      }
    });

  }

This I can see in the console:

ID: 3136
Company: Company A
Iterator: 1

ID: 3143
Company: Company B
Iterator: 2
DB insert successful. Record: 1

ID: 4497
Company: Company C
Iterator: 3

ID: 3164
Company: Company D
Iterator: 4

ID: 3219
Company: Company E
Iterator: 5

ID: 3071
Company: Company F
Iterator: 6

ID: 3184
Company: Company G
Iterator: 7
DB insert successful. Record: 2

ID: 3130
Company: Company H
Iterator: 8
DB insert successful. Record: 3
DB insert successful. Record: 4
DB insert successful. Record: 5
DB insert successful. Record: 6
DB insert successful. Record: 7
DB insert successful. Record: 8        

ID: 1844
Company: Company I
Iterator: 1

ID: 3687
Company: Company J
Iterator: 2

ID: 4514
Company: ECompany K
Iterator: 3

ID: 3635
Company: Company L
Iterator: 4

ID: 3884
Company: Company M
Iterator: 5

ID: 3482
Company: Company N
Iterator: 6
DB insert successful. Record: 1

ID: 3482
Company: Company O
Iterator: 7

ID: 1827
Company: Company P
Iterator: 8
DB insert successful. Record: 2

ID: 1827
Company: Company Q
Iterator: 9

ID: 6465
Company: Company R
Iterator: 10

ID: 0731
Company: Company S
Country: B9
Iterator: 11
No pagination!
DB insert successful. Record: 3
DB insert successful. Record: 4
DB insert successful. Record: 5
DB insert successful. Record: 6
DB insert successful. Record: 7
DB insert successful. Record: 8
DB insert successful. Record: 9
DB insert successful. Record: 10
DB insert successful. Record: 11

What am I missing? I guess I need to put the connection query in a async.function?! Like here: Issue inserting values in a loop (for) in the database : same value is inserted - node js / sql .?

Answer 1

Just promisify connection.query so you can await it. The link to the other question you posted is very similar to your problem.

This question is asked over and over, because it is hard to understand, but basically connection.query runs immediately, skips to the next line and then some point later (when the database responds and the event loop has time to process it) the function(err, rows) {} parts runs. So in-between some of your pupeteer awaits (or other async processes), it's processing the function(err,rows){} .

Next suggestion: learn to use util.promisify ! ( https://nodejs.org/dist/latest-v8.x/docs/api/util.html#util_util_promisify_original )

  let tableCell01;
  let tableCell01Val;
  let tableCell02;
  let tableCell02Val;

  const tableRows = await page.$$('table.tableFile2 > tbody > tr');

  for (let i=1; i < tableRows.length; i++){

    tableRow = tableRows[i];
    tableCell01 = await tableRow.$('td:nth-child(1) a');
    tableCell01Val = await page.evaluate( tableCell01 => tableCell01.innerText, tableCell01 );
    tableCell02 = await tableRow.$('td:nth-child(2)');
    tableCell02Val = await page.evaluate( tableCell02 => tableCell02.innerText, tableCell02 );

    tableCell02ValA.replace(/(^\s+|\s+$)/g,'');

    console.log('\n');
    console.log('ID: '+tableCell01Val);
    console.log('Company: '+tableCell02Val);
    console.log('Iterator: '+i);

    const insertCompanyList = "INSERT INTO companyList ( company_name, id ) values (?,?)";

    let rows = await new Promise((resolve,reject)=>{
      connection.query(insertCompanyList,[tableCell02Val, tableCell01Val],function(err, rows) {
        if (err) {
          console.log(err);
          reject(err);
        } else {
          console.log('DB insert successful. Record: '+i);
          resolve(rows);
        }
      });
    });

  }