如何对大量不一致的数据进行排序？

Question

For my programming class, I have been given 103MB of CSV files to work with. With a mid-sized file having 40 000 lines, I estimate everything to be around 300 000 lines. I am trying to sort the data by line and convert it into an object however one field is incredibly inconsistent.

---

Generally, the entire line is:

[Station Number], [Acronym of the Parameter Description], [Parameter Description], [Date (MM/DD/YYYY)], [Time], etc...

The parameter description, however is what is inconsistent. In it, there are different combinations of words and the formatting isn't even the same. Sometimes things are abbreviated and sometimes there is 1 space in between and sometimes there is 10.

Here are some examples of the parameter description field:

(chemical), (filtered/unfiltered) TOTAL

(chemical) TOTAL, (filtered/unfiltered)

CONDUCTIVITY, 25C

STREAM CONDITION

(chemical), DISSOLVED (inorganic/organic)

Also... sometimes after the chemical there is "UNFILTERED REACTIVE" and sometimes there is "UNFIL.REA"

---

Please help as I have no idea how to go about organizing the parameter description field. And these are just some that I have found in 6 000 lines. And I can hardly look over 300 000 lines to see what each line has.

Also, if it helps, this is Ontario Water Stream Quality Data and I am coding in Java (pseudocode is ok, though.)

Answer 1

If Parameter Description is the only field that contains commas, it should be possible to split line using comma as delimiter, and extract the rest of the fields, starting from the beginning and the end respectively. (The first two and the last x resulting strings will correspond to a field each.)

The remaining strings would be the Parameter Description field, wich can be put together again. Don't forget to restore the commas...

Answer 2

I have a different approach for this

I have also applied the same in on of my project.

this may lead lesser time to sort the whole content in comparision to the current state

and also as per KarlP answer you will need to do some extra operation some extra column that may lead more computational time rather than normal

---

My Way

Create a method which remove all spaces, commas and any other special character and also you may replace the numeric too coz they don't play good role when you are shorting with description like field

so the result will be something like this

// fn_getOnlyText replaces all chars which don't play role in alphabetical sorting as per your case

   fn_getOnlyText("Parameter Description");

then use this new field to sorting only this will be dictionary based sorting so it would be much faster than the original content.