如何处理大量数据？

Question

I have a project for school where I was given 103MB of data (Water Stream Quality Results in Ontario) which equates to 200 000+ lines of data. I need to be able to take in the data and create a visual out of it (eg a map of Ontario with dots representing where X chemical was found in 20XX).

I have tried taking in everything more or less all at once which my computer obviously couldn't handle. I was wondering if there is a strategy or function that can help me with this.

I would preferable like to have the information in a 2D ArrayList after the data has been taken in.

---

Example

Array Name: rawData

[0], [1], [2]

2002, 2003, 2004

station num, station num, station num

chem. found, chem. found, chem. found

etc.

If this is an inefficient way to have the data stored, what would be a better way?

---

Any help would be appreciated.

Side Note: the data has been given to me over 10+ files. Also, the file extension is a .csv so each row is separated by a comma (eg [year, chemical found, concentration percent>, etc.])

Answer 1

You can use ArrayLists. Create a class with the values you need to store in the objects.

For eg :

class Data{
  String Year;
  String found;
  ---one property for each of the value you need--
}

You can then populate the objects of this class and add them to the arrayList.

Answer 2

In addition to Ardash's answer, you will probably need to read in the data in chunks. If you use a BufferedReader, you can read CSV files line by line.