Reading huge txt files from C++?

Question

I'm trying to read a huge txt through c++. It has 70mb. My objective is to substring line by line and generate another smaller txt containing only the information that I need.

I got to the code below to read the file. It works perfectly with smaller files, but not with the 70mb monster.

#include "stdafx.h"
#include <iostream>
#include <fstream>
#include <string>

using namespace std;

int main()
{
  ifstream myReadFile;
  myReadFile.open("C:/Users/Lucas/Documents/apps/COTAHIST_A2010.txt");
  char output[100];
  if (myReadFile.is_open()) {
    while (myReadFile.eof()!=1) {
         myReadFile >> output;
         cout<<output;
         cout<<"\n";
     }


    }
  system("PAUSE");
  return 0;
}

This is the error I get: Unhandled exception at 0x50c819bc (msvcp100d.dll) in SeparadorDeAcoes.exe: 0xC0000005: Access violation reading location 0x3a70fcbc.

If someone can point a solution in C or even in C#, that would be acceptable too!

Thanks =)

Answer 1

your char output[100] buffer is not able to take the content of one of the lines.

Ideally you should use a string destination, and not a char[] buffer.

Edit As has been pointed out, this is bad practice, and leads to reading the last line twice or a stray empty last line. A more correct writing of the loop would be:

string output;
while (getline(myReadFile, output)) {
  cout<<output<<"\n";
}

**Edit - Leaving the bad, evil code here:

A quick rewrite of your inner while loop might be:

string output;
while (myReadFile.good()) {
  getline(myReadFile, output);
  cout<<output<<"\n";
}

Answer 2

I think that your problem is that one of your lines is over 100 characters long. Need to increase the size of the character array.

Answer 3

You are not using std::string , but you include the header file. Decide. Use either std::string or character array.

Also, use std::istream::read and provide the size of the array to the function. You will need to repeat many times as 100 characters is far smaller than 70mb.

Try allocating a much larger array using dynamic memory:

const unsigned int array_size = 1024 * 1024 * 1024;

int main(void)
{
  char * output;
//...
  output = new char [array_size];
// read into output
// ...
// clean up
  delete [] output;
  return EXIT_SUCCESS;
}

If you use std::string , use the constructor that takes a size parameter so you can specify the initial size of the string.