Google表格脚本-太慢

Question

I have google sheets set up as a database for my raspberry pi to send readings of temp, humidity etc. It is doing that just fine however for some reason google sheets will create duplicate huge arrays of the data in the same sheet.

To solve this problem I have been using the standard removeDuplicates google script code that I found (see below). The problem with this is that sometimes google sheets will copy over 100,000 rows of data to duplicate that a couple of time. This means there is a LOT of duplicates for the script to sort out.

I have read that maybe the issue is with so many duplicates, having to write individually to sort each duplicate could be taking much too long and I should try to get a list ALL of the duplicates in one go and then do ALL of the removing of the duplicates in one go. Now I am rather novice at this so I am not sure how I would achieve this.

function removeDuplicates() {
    // [START apps_script_sheets_sheet]
    var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
    var sheet = spreadsheet.getSheetByName('RawData');
    var data = sheet.getDataRange().getValues();
    // [END apps_script_sheets_sheet]
    // [START apps_script_sheets_new_data]
    var newData = [];
    // [END apps_script_sheets_new_data]
    for (var i in data) {
      var row = data[i];
      var duplicate = false;
      for (var j in newData) {
        if (row.join() == newData[j].join()) {
          duplicate = true;
        }
      }
      // [START apps_script_sheets_duplicate]
      if (!duplicate) {
        newData.push(row);
      }
      // [END apps_script_sheets_duplicate]
    }
    // [START apps_script_sheets_clear]
    sheet.clearContents();
    sheet.getRange(1, 1, newData.length, newData[0].length).setValues(newData);
    // [END apps_script_sheets_clear]
  }
  // [END apps_script_sheets_remove_duplicates]

There only errors is the code does not finish in the 6 minutes of allocated time.

How do I solve the problem?

Answer 1

try to specify number of rows, may be its trying to include remaining empty rows.

function removeDuplicates() {
    // [START apps_script_sheets_sheet]
    var spreadsheet = SpreadsheetApp.getActiveSpreadsheet();
    var sheet = spreadsheet.getSheetByName('RawData');
    var startRow = 1;  // First row of data to process
    var numRows = 4000;
    var data = sheet.getRange(startRow, 1, numRows, 1000)

    // [END apps_script_sheets_sheet]
    // [START apps_script_sheets_new_data]
    var newData = [];
    // [END apps_script_sheets_new_data]
    for (var i in data) {
      var row = data[i];
      var duplicate = false;
      for (var j in newData) {
        if (row.join() == newData[j].join()) {
          duplicate = true;
        }
      }
      // [START apps_script_sheets_duplicate]
      if (!duplicate) {
        newData.push(row);
      }
      // [END apps_script_sheets_duplicate]
    }
    // [START apps_script_sheets_clear]
    sheet.clearContents();
    sheet.getRange(1, 1, newData.length, newData[0].length).setValues(newData);
    // [END apps_script_sheets_clear]
  }
  // [END apps_script_sheets_remove_duplicates]