Readlines function for an xlsx file works inproper

Question

The goal is sentiment classification. The steps are to open 3 xlsx files, read them, process with gensim.doc2vec methods and classify with SGDClassificator. Just try to repeat this code on doc2vec . Python 2.7

with open('C:/doc2v/trainpos.xlsx','r') as infile:
    pos_reviews = infile.readlines()
with open('C:/doc2v/trainneg.xlsx','r') as infile:
    neg_reviews = infile.readlines()
with open('C:/doc2v/unsup.xlsx','r') as infile:
    unsup_reviews = infile.readlines()

But it turned out that the resulting lists are not what they are expected to be:

print 'length of pos_reviews is %s' % len(pos_reviews)
>>> length of pos_reviews is 1

The files contain 18, 1221 and 2203 raws correspondingly. I thought that the lists will have the same number of elements.

The next step is to concatenate all the sentences.

y = np.concatenate((np.ones(len(pos_reviews)), np.zeros(len(neg_reviews))))
x_train, x_test, y_train, y_test = train_test_split(np.concatenate((pos_reviews, neg_reviews)), y, test_size=0.2)

This leads to the situation when x-train, x-test are lists of sentences as they should be while

y_train = [0.]
y_test = [1.]

After this division every sentence gets a label:

def labelizeReviews(reviews, label_type):
labelized = []
for i,v in enumerate(reviews):
    label = '%s_%s'%(label_type,i)
    labelized.append(LabeledSentence(v, [label]))
return labelized
x_train = labelizeReviews(x_train, 'TRAIN')
x_test = labelizeReviews(x_test, 'TEST')
unsup_reviews = labelizeReviews(unsup_reviews, 'UNSUP')

As written in the numpy documentation , the arrays should be equal in size. But when I reduce the bigger files to 18 lines, nothing changes. As I searched on the forum noone has a similar error. I've broken my head what went wrong and how to fix it. Thanks for help!

Answer 1

Generally you can't read Microsoft Excel files as a text files using methods like readlines or read . You should convert files to another format before (good solution is .csv which can be readed by csv module) or use a special python modules like pyexcel and openpyxl to read .xlsx files directly.