Reading txt file as a list instead of a string in python

Question

I have a file that is made of lines of the following format:-

[123, something, some other thing, "text that i want", more details]

eg:-

[1393349463, u'Tue Feb 25 17:31:03 +0000 2014', 438365537261735936, u'A Falcon character poster for Captain America: The Winter Soldier has swooped in', [], [u'totalfilm'], [u'//1bJdCJ2'], [u'http://pbs.twimg.com/media/BhViUNICQAAoBue.jpg'], 369, 362]

Now i want to read this as list directly into python instead of a string first and then spliting the string by , and joining it back and all because the text section can have a ',' and I dont want to split that.

I am looking for something like this:

with open("input.txt") as fp:
   for line in fp:
       corpus.append(line[3]) #read only text

Answer 1

Your input is obviously generated by calling just print ing out Python lists (or calling str or repr on them).

This particular example can be handled by using literal_eval :

with open("input.txt") as fp:
    for line in fp:
        obj = ast.literal_eval(line)
        corpus.append(obj[3])

However, that won't work for all Python list displays in general. And when it doesn't work… well, there's not much you can do in general. But you can just literal_eval until you get an error, and then, for each error, laboriously work out how to pre-process things to work around it.

The right thing to do is generate output that's actually parseable, like JSON, and then you can just parse it trivially.