Reading in from a .tsv file

Question

I'm trying to read in information from a tab separated value file with the format:

<string>    <int>    <string>

Example:

Seaking 119 Azumao
Mr. Mime    122 Barrierd
Weedle  13  Beedle

This is currently how I'm doing it:

string americanName;
int pokedexNumber;
string japaneseName;

inFile >> americanName;
inFile >> pokedexNumber
inFile >> japaneseName;

My issue stems from the space in the "Mr. Mime" as the strings can contain spaces.

I would like to know how to read the file in properly.

Answer 1

Standard library uses such things as locales to determine the categories of different symbols and other locale-dependent things depending on your system locale. Standard streams use that to determine what is a space because of various unicode issues.

You can use this fact to control the meaning of ' ' in your case:

#include <iostream>
#include <locale>
#include <algorithm>

struct tsv_ws : std::ctype<char>
{
    mask t[table_size]; // classification table, stores category for each character

    tsv_ws() : ctype(t) // ctype will use our table to check character type
    {
        // copy all default values to our table;
        std::copy_n(classic_table(), table_size, t);
        // here we tell, that ' ' is a punctuation, but not a space :)
        t[' '] = punct; 
    }
};

int main() {
    std::string s;
    std::cin.imbue(std::locale(std::cin.getloc(), new tsv_ws)); // using our locale, will work for any stream
    while (std::cin >> s) {
        std::cout << "read: '" << s << "'\n";
    }
}

Here we make ' ' a punctuation symbol, but not a space symbol, so streams don't consider it a separator anymore. The exact category isn't important, but it mustn't be space .

That's quite powerful technique. For example, you could redefine ',' to be a space to read in CSV format.

Answer 2

You can use std::getline to extract strings with non-tab whitespace.

std::getline(inFile, americanName, '\t'); // read up to first tab
inFile >> pokedexNumber >> std::ws; // read number then second tab
std::getline(inFile, japaneseName); // read up to first newline

Answer 3

Seems like you want to read csv data or in your case tsv data. But let's stick to the common term "csv". This is a standard task and I will give you detailed explanations. In the end all the reading will be done in a one-liner.

I would recommend to use "modern" C++ approach.

After searching for "reading csv data", people are still are linking to How can I read and parse CSV files in C++? , the questions is from 2009 and now over 10 years old. Most answers are also old and very complicated. So, maybe its time for a change.

In modern C++ you have algorithms that iterate over ranges. You will often see something like "someAlgoritm(container.begin(), container.end(), someLambda)". The idea is that we iterate over some similar elements.

In your case we iterate over tokens in your input string, and create substrings. This is called tokenizing.

And for exactly that purpose, we have the std::sregex_token_iterator . And because we have something that has been defined for such purpose, we should use it.

This thing is an iterator. For iterating over a string, hence sregex. The begin part defines, on what range of input we shall operate, then there is a std::regex for what should be matched / or what should not be matched in the input string. The type of matching strategy is given with last parameter.

• 1 --> give me the stuff that I defined in the regex and
• -1 --> give me that what is NOT matched based on the regex.

So, now that we understand the iterator, we can std::copy the tokens from the iterator to our target, a std::vector of std::string . And since we do not know, how may columns we have, we will use the std::back_inserter as a target. This will add all tokens that we get from the std::sregex_token_iterator and append it ot our std::vector<std::string>> . It does'nt matter how many columns we have.

Good. Such a statement could look like

std::copy(                          // We want to copy something
    std::sregex_token_iterator      // The iterator begin, the sregex_token_iterator. Give back first token
    (
        line.begin(),               // Evaluate the input string from the beginning
        line.end(),                 // to the end
        re,                         // Add match a comma
        -1                          // But give me back not the comma but everything else 
    ),
    std::sregex_token_iterator(),   // iterator end for sregex_token_iterator, last token + 1
    std::back_inserter(cp.columns)  // Append everything to the target container
);

Now we can understand, how this copy operation works.

Next step. We want to read from a file. The file conatins also some kind of same data. The same data are rows.

And as for above, we can iterate of similar data. If it is the file input or whatever. For this purpose C++ has the std::istream_iterator . This is a template and as a template parameter it gets the type of data that it should read and, as a constructor parameter it gets a reference to an input stream. It doesnt't matter, if the input stream is a std::cin , or a std::ifstream or a std::istringstream . The behaviour is identical for all kinds of streams.

And since we do not have files an SO, I use (in the below example) a std::istringstream to store the input csv file. But of course you can open a file, by defining a std::ifstream testCsv(filename) . No problem.

And with std::istream_iterator , we iterate over the input and read similar data. In our case one problem is that we want to iterate over special data and not over some build in data type.

To solve this, we define a Proxy class, which does the internal work for us (we do not want to know how, that should be encapsulated in the proxy). In the proxy we overwrite the type cast operator, to get the result to our expected type for the std::istream_iterator .

And the last important step. A std::vector has a range constructor. It has also a lot of other constructors that we can use in the definition of a variable of type std::vector . But for our purposes this constructor fits best.

So we define a variable csv and use its range constructor and give it a begin of a range and an end of a range. And, in our specific example, we use the begin and end iterator of std::istream_iterator .

If we combine all the above, reading the complete CSV file is a one-liner , it is the definition of a variable with calling its constructor.

Please see the resulting code:

#include <iostream>
#include <sstream>
#include <fstream>
#include <string>
#include <vector>
#include <iterator>
#include <regex>
#include <algorithm>

std::istringstream testCsv{ R"(Seaking  119 Azumao
Mr. Mime    122 Barrierd
Weedle  13  Beedle)" };


// Define Alias for easier Reading
using Columns = std::vector<std::string>;
using CSV = std::vector<Columns>;


// Proxy for the input Iterator
struct ColumnProxy {
    // Overload extractor. Read a complete line
    friend std::istream& operator>>(std::istream& is, ColumnProxy& cp) {

        // Read a line
        std::string line; cp.columns.clear();
        if (std::getline(is, line)) {

            // The delimiter
            const std::regex re("\t");

            // Split values and copy into resulting vector
            std::copy(std::sregex_token_iterator(line.begin(), line.end(), re, -1),
                std::sregex_token_iterator(),
                std::back_inserter(cp.columns));
        }
        return is;
    }

    // Type cast operator overload.  Cast the type 'Columns' to std::vector<std::string>
    operator std::vector<std::string>() const { return columns; }
protected:
    // Temporary to hold the read vector
    Columns columns{};
};


int main()
{
    // Define variable CSV with its range constructor. Read complete CSV in this statement, So, one liner
    CSV csv{ std::istream_iterator<ColumnProxy>(testCsv), std::istream_iterator<ColumnProxy>() };

    // Print result. Go through all lines and then copy line elements to std::cout
    std::for_each(csv.begin(), csv.end(), [](Columns & c) {
        std::copy(c.begin(), c.end(), std::ostream_iterator<std::string>(std::cout, " ")); std::cout << "\n";   });
}

I hope the explanation was detailed enough to give you an idea, what you can do with modern C++.

This example does basically not care how many rows and columns are in the source text file. It will eat everything.