Python-将类似列表的字符串转换为列表列表的最快，最有效的方法

Question

I'm writting a python program which has to run a c++ program (compiled) in order to manage a very heavy load opperation.

The way I call this executable is subprocess.check_output()

Doing so, it returns a very long string like the following:

executable_output_1 = [0.011, 0.544, 2.314], [7.895, 6.477, 2.573]

executable_output_2 = [4.255, 6.235, 7.566], [9.522, 7.321, 1.234]

type(executable_output) >>> <type 'str'>

In this example I wrote a quite short string, but in the real output it's really long.

I'd like to do some operations with that data back in python, so I need a list of lists (as I'll call that executable multiple times).

How could I convert that string into a list of lists?

Desired output:

executables_outputs_list = [[[0.011, 0.544, 2.314], [7.895, 6.477, 2.573]], [[4.255, 6.235, 7.566], [9.522, 7.321, 1.234]]]

type(executable_output_list) >>> <type 'list'>

type(executable_output_list[0]) >>> <type 'list'>

type(executable_output_list[0][0][0]) >>> <type 'float'>

Answer 1

Use ast.literal_eval :

>>> from ast import literal_eval
>>> executable_output = '[0.011, 0.544, 2.314], [7.895, 6.477, 2.573]'
>>> literal_eval(executable_output)
([0.011, 0.544, 2.314], [7.895, 6.477, 2.573])

This is a tuple . To convert to a list :

>>> list(literal_eval(executable_output))
[[0.011, 0.544, 2.314], [7.895, 6.477, 2.573]]

Answer 2

You can make use of python's json package which should be faster than the ast.literal_eval .

# setup
from ast import literal_eval
import json

executable_output_1 = "[0.011, 0.544, 2.314], [7.895, 6.477, 2.573]"
executable_output_2 = "[4.255, 6.235, 7.566], [9.522, 7.321, 1.234]"

outputs = (executable_output_1, executable_output_2)

The json approach has the following timing: 100000 loops, best of 3: 14.5 µs per loop :

def extract(output_strings):
    template = '{{"values": [{}]}}'
    parse_func = lambda x: json.loads(template.format(x))
    return [parse_func(x)["values"] for x in output_strings]

extract(outputs)

>>> [[[0.011, 0.544, 2.314], [7.895, 6.477, 2.573]],
     [[4.255, 6.235, 7.566], [9.522, 7.321, 1.234]]]

The ast approach is 3x slower for the dummy data with 10000 loops, best of 3: 51.6 µs per loop :

def extract_ast(output_strings):
    return [list(literal_eval(x)) for x in output_strings]

extract_ast(outputs)

>>> [[[0.011, 0.544, 2.314], [7.895, 6.477, 2.573]],
     [[4.255, 6.235, 7.566], [9.522, 7.321, 1.234]]]

The json approach improves with increasing amount of data to be parsed. With the following setup, the json approach yields 1000 loops, best of 3: 291 µs per loop compared to the ast with 100 loops, best of 3: 3.95 ms per loop which is 13x faster.

executable_output_1 = ",".join(["[0.011, 0.544, 2.314], [7.895, 6.477, 2.573]"] * 100)
executable_output_2 = ",".join(["[4.255, 6.235, 7.566], [9.522, 7.321, 1.234]"] * 100)

outputs = (executable_output_1, executable_output_2)