Javascript 从（多行）字符串中提取唯一数组所需的时间太长

Question

I have a huge data set that gets loaded in pre tag, like the following.

00:00:00 INFO SERVER-SYSTEM - Cmd Line Arg: sysName = SERVER
00:00:01 INFO SERVER-SYSTEM - Cmd Line Arg: resultsDirName = github
00:00:02 INFO SERVER-SYSTEM - Cmd Line Arg: Device4Branch = //github/server_manager01/test1
00:00:02 FAIL SERVER-SYSTEM - Cmd Line Arg: testCase = server_manager01
00:00:03 INFO SERVER-SYSTEM - Cmd Line Arg: timestamp_style = RELATIVE
00:00:04 INFO SERVER-SYSTEM - Cmd Line Arg: token = 36
00:00:04 FAIL SERVER-SYSTEM - Cmd Line Arg: Campaign = True

There will be around 30,000+ lines and I want to store unique words in an array. Following are lines of code to fetch data from the div which hold this pre tag data and stores unique words separated by space into an array.

pre_data = document.getElementById("data_div").innerHTML.split('\n');
var words = [];
var reg = new RegExp("\\S*", "ig");

for (x = 0; x < pre_data.length; x++) {
   words = words .concat(pre_data[x].match(reg));
}

// To remove null values
filtered_data = words .filter(function (el) {
   return el != ''; });

// Set gives unique data
unique_data = Array.from(new Set(filtered_data ));

But this is taking 10+ seconds if there are 30,000+ lines. What could be an effective way to get it faster?

Answer 1

I'm not sure whether this is what you are looking for but this is how I would do it;

• Find the element and get the innerHTML or innerText and split it by newLine which will give you an array
• Reduce the lines and create one array of all words of each line
  • Split the line by white space which will give you an array of words
  • Filter out the empty words w => w
  • Add the words array to the accumulator and return it
• Now that you have all words you can filter out the duplicated

PS: you don't have to create 3 variables as I did, you can chain the array functions and create one variable if you want

 var linesArray = document.getElementById('my-pre').innerHTML.split('\n'); var words = linesArray.reduce((acc, cur) => { return acc.concat(cur.split(' ').filter(w => w)); }, []) var uniqueWords = words.filter((value, index, self) => self.indexOf(value) === index) console.log(uniqueWords)

 <pre id="my-pre"> Hello Stackoverflow I'm having some issues with extracting a unique array out of my multiline string this is a test to see if the the array is unique enough this is a test to see if the the array is unique enough this is a test to see if the the array is unique enough </pre>