Read every word in a string C++

Question

I am trying to read every word a string. I want a string to go in and the first word to come out, then I'll process it, then the second, and so on. But the internet isn't helping me, I know it's probably right under my nose but I can't figure it out!

string lex(string filecontent) {

string t = filecontent;
getline(cin, t);

istringstream iss(t);
string word;

while (iss >> word) {
    return word;
}

}
int main() {
    string data = load_file(); // Returns a string of words
    cout << data;
    cout << lex(data);
    getchar();
}

Right now this works... sort of it prints out a lot of random gibberish and crazy characters, The file I'm reading's output is ok I check this at cout << data and it is what I expect. Any ideas?

Answer 1

Here is the solution I think you are looking for:

int main() {
   string data = load_file(); // Returns a string of words

   istringstream iss(data);

   while(iss)
   {
      string tok;
      iss >> tok;
      cout << "token: " << tok << endl;
      //you can do what ever you want with the token here

   }
}

Answer 2

Have a look at this, it should help you.

main.cpp

#include "stdafx.h"
#include "Utility.h"

int main() {
    using namespace util;

    std::string fileName( "sample.txt" );

    if ( fileName.empty() ) {
        std::cout << "Missing or invalid filename." << std::endl;
        return RETURN_ERROR;
    }

    std::string line;
    std::vector<std::string> results;
    std::fstream fin;

    // Try To Open File For Reading
    fin.open( fileName.c_str(), std::ios_base::in );
    if ( !fin.is_open() ) {
        std::cout << "Can not open file(" << fileName << ") for reading." << std::endl;
        return RETURN_ERROR;
    }

    // Read Line By Line To Get Data Contents Store Into String To Be Parsed
    while ( !fin.eof() ) {
        std::getline( fin, line );
        // Parse Each Line Using Space Character As Delimiter
        results = Utility::splitString( line, " " );

        // Print The Results On Each Iteration Of This While Loop 
        // This Is Where You Would Parse The Data Or Store Results Into
        // Class Objects, Variables Or Structures.
        for ( unsigned u = 0; u < results.size(); u++ ) {
            std::cout << results[u] << " ";
        }
        std::cout << std::endl;
    }

    // Close File Pointer
    fin.close();

    // Now Print The Full Vector Of Results - This Is To Show You That Each
    // New Line Will Be Overwritten And That Only The Last Line Of The File Will
    // Be Stored After The While Loop.
    std::cout << "\n-------------------------------------\n";
    for ( unsigned u = 0; u < results.size(); u++ ) {
        std::cout << results[u] << " ";
    }

    Utility::pressAnyKeyToQuit();
    return RETURN_OK;
} // main

sample.txt

Please help me parse this text file
It spans multiple lines of text
I would like to get each individual word

stdafx.h - Some of these include files may not be needed they are here for I have a larger solution that requires them.

#ifndef STDAFX_H
#define STDAFX_H

#include <Windows.h>

#include <stdio.h>
#include <tchar.h>
#include <conio.h>

#include <string>
#include <sstream>
#include <fstream>
#include <iostream>
#include <iomanip>
#include <vector>
#include <array>
#include <memory>

#include <queue>
#include <functional>

#include <algorithm>

// User Application Specific
// #include "ExceptionHandler.h" - One Of My Class Objects Not Used Here

namespace util {

enum ReturnCode {
    RETURN_OK = 0,
    RETURN_ERROR = 1,
}; // ReturnCode

extern const unsigned INVALID_UNSIGNED;
extern const unsigned INVALID_UNSIGNED_SHORT;

} // namespace util

#endif // STDAFX_H

stdafx.cpp

#include "stdafx.h"

namespace util {

const unsigned INVALID_UNSIGNED = static_cast<const unsigned>( -1 );
const unsigned INVALID_UNSIGNED_SHORT = static_cast<const unsigned short>( -1 );

} // namespace util

Utility.h

#ifndef UTILITY_H
#define UTILITY_H

namespace util {

class Utility {
public:

    static void pressAnyKeyToQuit();

    static std::string  toUpper(const std::string& str);
    static std::string  toLower(const std::string& str);
    static std::string  trim(const std::string& str, const std::string elementsToTrim = " \t\n\r");

    static unsigned     convertToUnsigned(const std::string& str);
    static int          convertToInt(const std::string& str);
    static float        convertToFloat(const std::string& str);

    static std::vector<std::string> splitString(const std::string& strStringToSplit, const std::string& strDelimiter, const bool keepEmpty = true);

private:
    Utility(); // Private - Not A Class Object
    Utility(const Utility& c); // Not Implemented
    Utility& operator=(const Utility& c); // Not Implemented

    template<typename T>
    static bool stringToValue(const std::string& str, T* pValue, unsigned uNumValues);

    template<typename T>
    static T getValue(const std::string& str, std::size_t& remainder);

}; // Utility

#include "Utility.inl"

} // namespace util

#endif // UTILITY_H

Utility.inl

// ----------------------------------------------------------------------------
// stringToValue()
template<typename T>
static bool Utility::stringToValue(const std::string& str, T* pValue, unsigned uNumValues) {
    int numCommas = std::count(str.begin(), str.end(), ',');
    if (numCommas != uNumValues - 1) {
        return false;
    }

    std::size_t remainder;
    pValue[0] = getValue<T>(str, remainder);

    if (uNumValues == 1) {
        if (str.size() != remainder) {
            return false;
        }
    }
    else {
        std::size_t offset = remainder;
        if (str.at(offset) != ',') {
            return false;
        }

        unsigned uLastIdx = uNumValues - 1;
        for (unsigned u = 1; u < uNumValues; ++u) {
            pValue[u] = getValue<T>(str.substr(++offset), remainder);
            offset += remainder;
            if ((u < uLastIdx && str.at(offset) != ',') ||
                (u == uLastIdx && offset != str.size()))
            {
                return false;
            }
        }
    }
    return true;
} // stringToValue

Utility.cpp

#include "stdafx.h"
#include "Utility.h"

namespace util {

// ----------------------------------------------------------------------------
// pressAnyKeyToQuit()
void Utility::pressAnyKeyToQuit() {
    std::cout << "\nPress any key to quit" << std::endl;
    _getch();
} // pressAnyKeyToQuit

// ----------------------------------------------------------------------------
// toUpper()
std::string Utility::toUpper( const std::string& str ) {
    std::string result = str;
    std::transform( str.begin(), str.end(), result.begin(), ::toupper );
    return result;
} // toUpper

// ----------------------------------------------------------------------------
// toLower()
std::string Utility::toLower( const std::string& str ) {
    std::string result = str;
    std::transform( str.begin(), str.end(), result.begin(), ::tolower );
    return result;
} // toLower

// ----------------------------------------------------------------------------
// trim()
// Removes Elements To Trim From Left And Right Side Of The str
std::string Utility::trim( const std::string& str, const std::string elementsToTrim ) {
    std::basic_string<char>::size_type firstIndex = str.find_first_not_of( elementsToTrim );
    if ( firstIndex == std::string::npos ) {
        return std::string(); // Nothing Left
    }

    std::basic_string<char>::size_type lastIndex = str.find_last_not_of( elementsToTrim );
    return str.substr( firstIndex, lastIndex - firstIndex + 1 );
} // trim

// ----------------------------------------------------------------------------
// getValue()
template<>
float Utility::getValue( const std::string& str, std::size_t& remainder ) {
    return std::stof( str, &remainder );
} // getValue <float>

// ----------------------------------------------------------------------------
// getValue()
template<>
int Utility::getValue( const std::string& str, std::size_t& remainder ) {
    return std::stoi( str, &remainder );
} // getValue <int>

// ----------------------------------------------------------------------------
// getValue()
template<>
unsigned Utility::getValue( const std::string& str, std::size_t& remainder ) {
    return std::stoul( str, &remainder );
} // getValue <unsigned>

// ----------------------------------------------------------------------------
// convertToUnsigned()
unsigned Utility::convertToUnsigned( const std::string& str ) {
    unsigned u = 0;
    if ( !stringToValue( str, &u, 1 ) ) {
        std::ostringstream strStream;
        strStream << __FUNCTION__ << " Bad conversion of [" << str << "] to unsigned";
        throw strStream.str();
    }
    return u;
} // convertToUnsigned

// ----------------------------------------------------------------------------
// convertToInt()
int Utility::convertToInt( const std::string& str ) {
    int i = 0;
    if ( !stringToValue( str, &i, 1 ) ) {
        std::ostringstream strStream;
        strStream << __FUNCTION__ << " Bad conversion of [" << str << "] to int";
        throw strStream.str();
    }
    return i;
} // convertToInt

// ----------------------------------------------------------------------------
// convertToFloat()
float Utility::convertToFloat(const std::string& str) {
    float f = 0;
    if (!stringToValue(str, &f, 1)) {
        std::ostringstream strStream;
        strStream << __FUNCTION__ << " Bad conversion of [" << str << "] to float";
        throw strStream.str();
    }
    return f;
} // convertToFloat

// ----------------------------------------------------------------------------
// splitString()
std::vector<std::string> Utility::splitString( const std::string& strStringToSplit, const std::string& strDelimiter, const bool keepEmpty ) {
    std::vector<std::string> vResult;
    if ( strDelimiter.empty() ) {
        vResult.push_back( strStringToSplit );
        return vResult;
    }

    std::string::const_iterator itSubStrStart = strStringToSplit.begin(), itSubStrEnd;
    while ( true ) {
        itSubStrEnd = search( itSubStrStart, strStringToSplit.end(), strDelimiter.begin(), strDelimiter.end() );
        std::string strTemp( itSubStrStart, itSubStrEnd );
        if ( keepEmpty || !strTemp.empty() ) {
            vResult.push_back( strTemp );
        }

        if ( itSubStrEnd == strStringToSplit.end() ) {
            break;
        }

        itSubStrStart = itSubStrEnd + strDelimiter.size();
    }

    return vResult;

} // splitString

} // namspace util

In my small utility library I have a function that will split a string that can use any delimiter that the user defines. It will search for the first occurrence of that character delimiter and it will save everything before it into a string and it will push that string into a vector of strings, and it will continue this for every occurrence of that character until it is finished parsing the full string that is passed to it. It will then return a vector of strings back to the user. This is very helpful when engaged in parsing text files or even just data types with long strings that need to be broken down. Now if there is a case where you are parsing a text file and lets say you need to have more than one word as a single string, this can be done but requires more work on your part. For example a text file might have personal record on a single line.

LastName, FirstName MiddleInitial Age Phone# Address
Cook, John S 33 1-888-323-4545 324 Complex Avenue

And you would want the 324 Complex Avenue to be in a single string also you don't want the comma stored after the last name. Your structure in code to store this info might look like this:

struct PersonalRecord {
    std::string firstName;
    std::string lastName;
    char middleInitial;
    unsigned age;
    std::string phoneNumber;
    std:string address;
};

What you would have to do is after you read this line in from your file on that same iteration of the while loop is you would have to do multiple parsing.

You would first start by using a temporary string and vector of strings and use the utility function splitString with the delimeter being the comma. So this would save 2 strings in the temp vector of strings the first being: Cook and the second being the rest of the line after the comma including the leading space. The reason you have the temp string and temp vector of strings is that you will need to pop values at when needed. So in this case we would have to do the following, but first how do we resolve the case with multiple words to one string? We can change the line of text in the text file to be enclosed with double quotes as such:

textfile

Cook, John S 33 1-888-323-4545 "324 Complex Avenue"
Evens, Sue A 24 1-888-323-6996 "128 Mission Rd"
Adams, Chris B 49 1-777-293-8234 "2304 Helms Drive"

Then parse it with this logic flow or algorithm.

main.cpp

#including "stdafx.h"
#including "Utility.h"

int main() {
    using namespace util;

    std::string strFilename( "personalRecord.txt" );
    std::ifstream file; 

    std::string strLine;
    std::vector<std::string> vTemp;
    std::vector<std::string> vResult;

    std::vector<PersonalRecord> vData;

    // Open File For Reading
    file.open( strFilename.c_str() );
    // Check For Error Of Opening File
    if ( !file.is_open() ) {
        std::cout << "Error opening file (" << strFilename << ")" << std::endl;
        return RETURN_ERROR;
    }

    // Continue Until End Of File
    while( !file.eof() ) {
        // Get Single Full Line Save To String
        std::getline( file, strLine );

        // Check For Comma
        vTemp = Utility::splitString( strLine, ",");
        // Save First String For Laster
        std::string lastName = vTemp[0];

        // Split String Using A Double Quote Delimiter Delimiter
        vTemp = Utility::splitString( vTemp[1], "\"" );

        // Check To See If vTemp Has More Than One String
        if ( vTemp.size() > 1 ) {
            // We Need To Use Pop Back To Account For Last Double Quote
            vTemp.pop_back(); // Remove Last Double Quote
            std::string temp = vTemp.back(); 
            vTemp.pop_back(); // Remove Wanted String From vTemp.

            // At This Point We Need To Parse vTemp Again Using Space Delimiter
            vResult = Utility::splitString( vTemp[0], " " );

            // Need To Account For Leading Space In Vector
            vResult[0].erase();
            // Need To Account For Last Space In Vector
            vResult.pop_back();

            // Now We Can Push Our Last String Back Into vResult
            vResult.push_back( temp );

            // Replace The First String " " With Our LastName
            vResult[0] = lastName;

        } else if ( vTemp.size() == 1 ) {
            // Just Parse vTemp Using Space Delimiter
            vResult = Utility::splitString( vTemp[0], " " );
        }


        // Print Out Results For Validity
        for ( unsigned u = 0; u < vResult.size(); u++) {
            std::cout << vResult.at(u) << " ";
        }
        std::cout << std::endl;

        // Here Is Where You Would Populate Your Variables, Structures Or Classes On Each Pass Of The While Loop.
        // With This Structure There Should Only Be 8 Entries Into Our vResult
        PersonalRecord temp;
        temp.lastName      = vResult[0];
        temp.firstName     = vResult[1];
        temp.middleInitial = vResult[2][0];
        temp.age           = Utility::convertToUnsigned( vResult[3] );
        temp.phoneNumber   = vResult[4];
        temp.address       = vResult[5];

        vData.push_back( temp );
    } // while

    // Close File
    file.close();

    std::cout << std::endl << std::endl;

    // Print Using Structure For Validity
    std::cout << "---------------------------------------\n";
    for ( unsigned u = 0; u < vData.size(); u++ ) {
        std::cout << vData[u].lastName << " " 
                  << vData[u].firstName << " "
                  << vData[u].middleInitial << " "
                  << vData[u].age << " "
                  << vData[u].phoneNumber << " "
                  << vData[u].address << std::endl;
    }

    Utility::pressAnyKeyToQuit();
    return RETURN_OK;
} // main

So both consideration and are has to be taken when parsing text or strings. You have to account for every single character including your carriage returns, spaces etc. So the format that the text file is written in has to be considered. Yes the splitString() will also parse tabs, you would just have to use "\\t" for tabs, etc. Just remember that it will make a split at every occurrence. So if you have a sentence that has a colon ":" in it, but then you decide to use the colon as your delimiter between values, it will split that sentence as well. Now you could have different rules for each line of text from the file and if you know what line you are on you can parse each line accordingly. This is why most people prefer to write their code to read and parse binary, because it is much easier to program, then writing a text parser.

I chose to use the PersonalRecord structure to show you how you can extract strings from a line of text and to convert them to basic types such as int, float or double by using some of my other functions in my Utility class. All methods in this class are declared as static and the constructor is private, so the class name acts as a wrapper or a namespace so to speak. You can not create an instance of a Utility util; // invalid object Utility util; // invalid object . Just include the header file and use the class name with the scope resolution operator :: to access any of the functions and make sure you are using the namespace util .