Parsing huge XML file for getting information from huge CSV file

Question

I have huge problem.

I have a task to put in database information about products. All information's except price are in XML file that is big 100mb .

In another file there are information's about price of that product. That is CSV file with size about 200mb

In simple logic

1. Parse XML into array
2. Search price by unique ID of product in another file
3. Add found price to element of array where is that product
4. Put information's in database

But i Cannot do that. Problem is in SIZE .

I manage to parse XML and put it in array, but I cannot handle CSV file. If there is 1000 products, for each product he need to parse CSV file (that CSV file have more than 30.000.000 lines). I cannot parse that CSV one, much less 1000.

I need suggestion how to solve this problem

Answer 1

Looks like a job for a database to me. Load both files into a database and then do a simple join.

Answer 2

Use XMLReader for the XML , SplFileObject for the CSV .

Do not keep the information in memory. Write it directly into a database like MySQL or SQlite. After all the data is imported you can use SQL to aggregate it.