splitting long string at indices in parallel in python

Question

I have many files that each have several million rows; each row is a dumped data entry and is several hundred characters long. The rows come in groups and the first two characters tell me the type of row it is, and I use that to parse it. This structure prohibits me from loading the rows to a dataframe, for example, or anything else that does not go through the rows one at a time.

For each row, I currently create a dictionary vals = {}, and then sequentially run through about fifty keys along the lines of

vals{'name'} = row[2:24]

vals{'state'} = row[24:26]

Instead of doing fifty assignments sequentially, can I do this simultaneously or in parallel in some simple manner?

Is

vals{'name'},vals{'state'} = row[2:24],row[24:26]

faster if I do this simultaneous assignment for many entries? I could also reformulate this as a list comprehension. Would that be faster than running through sequentially?

Answer 1

To answer your question, no, doing multiple assignment will not speed up your program. This is because the multiple assignment syntax is just a different way of writing multiple assignments on different lines.

For example

vals{'name'},vals{'state'} = row[2:24],row[24:26]

is equivalent to

vals{'name'}= row[2:24]
vals{'state'} = row[2:24]

If you want to optimize your code, your should start by profiling it to determine the parts that are taking the largest amount of time. I would also check to ensure that you are not doing multiple reads from the same file, as these are very slow compared to reading from memory. If possible, you should read the entire file into memory first, and then process it.