熊猫：将字符串元组列表更快地转换为数据框？

Question

From a text field I have the following input series, containing geographic coordinate tuples as a string:

import pandas as pd

coords = pd.Series([
   '(29.65271977700047, -82.33086252299967)',
   '(29.652914019000434, -82.42682220199964)',
   '(29.65301114200048, -82.36455186899968)',
   '(29.642610841000476, -82.29853169599966)',
])

I would like to parse the numbers in these tuples and end up with the following result DataFrame:

         lat        lon
0  29.652720 -82.330863
1  29.652914 -82.426822
2  29.653011 -82.364552
3  29.642611 -82.298532

This is what I have come up with:

str_coords = coords.str[1:-1].str.split(', ')
latlon = str_coords.apply(pd.Series).astype(float)
latlon.columns = ['lat', 'lon']

My problem : The call to .apply(pd.Series) takes "forever" on the real list, which has around 1.2 million entries. Is there a faster way?

Answer 1

Another way to access the first and second element of the list also through the str :

In [174]: coords = pd.Series([
   .....:    '(29.65271977700047, -82.33086252299967)',
   .....:    '(29.652914019000434, -82.42682220199964)',
   .....:    '(29.65301114200048, -82.36455186899968)',
   .....:    '(29.642610841000476, -82.29853169599966)'])

In [175]: str_coords = coords.str[1:-1].str.split(', ')

In [176]: coords_df = pd.DataFrame({'lat': str_coords.str[0], 'lon': str_coords.str[1]})

In [177]: coords_df.astype(float).head()
Out[177]:
         lat        lon
0  29.652720 -82.330863
1  29.652914 -82.426822
2  29.653011 -82.364552
3  29.642611 -82.298532
4  29.652720 -82.330863

Some timings indicate that both my solution as that of @ajcr is much faster than the apply(pd.Series) approach (and the difference between both is negligible):

In [197]: coords = pd.Series([
   .....:    '(29.65271977700047, -82.33086252299967)',
   .....:    '(29.652914019000434, -82.42682220199964)',
   .....:    '(29.65301114200048, -82.36455186899968)',
   .....:    '(29.642610841000476, -82.29853169599966)'])

In [198]: coords = pd.concat([coords]*1000, ignore_index=True)


In [199]: %%timeit
   .....: str_coords = coords.str[1:-1].str.split(', ')
   .....: df_coords = pd.DataFrame({'lat': str_coords.str[0], 'lon': str_coords.str[1]}, dtype=float)
   .....:
100 loops, best of 3: 14.1 ms per loop

In [200]: %%timeit
   .....: str_coords = coords.str[1:-1].str.split(', ')
   .....: df_coords = str_coords.apply(pd.Series).astype(float)
   .....:
1 loops, best of 3: 821 ms per loop

In [201]: %%timeit
   .....: df_coords = coords.str.extract(r'\((?P<lat>[\d\.]+),\s+(?P<lon>[^()\s,]+)\)')
   .....: df_coords.astype(float)
   .....:
100 loops, best of 3: 16.2 ms per loop

Answer 2

Another way could be to use the vectorised string method extract :

>>> coords.str.extract(r'\((?P<lat>[\-\d\.]+),\s+(?P<lon>[\-\d\.]+)\)')
                  lat                 lon
0   29.65271977700047  -82.33086252299967
1  29.652914019000434  -82.42682220199964
2   29.65301114200048  -82.36455186899968
3  29.642610841000476  -82.29853169599966

You can pass named regex capture groups to extract - it will create a DataFrame with the group names as column names.

You can then cast this DataFrame df to a float datatype:

>>> df.astype(float)
         lat        lon
0  29.652720 -82.330863
1  29.652914 -82.426822
2  29.653011 -82.364552
3  29.642611 -82.298532