How to read in a .txt file when the column names are inline with the data with Pandas?

Question

I'm trying to read in a.txt file into a dataframe using Pandas. The problem I'm getting is that the column names are in line with the data for each row. This makes it a little hard for me to get only the data since I'm not sure what the delimiter should be. My data looks like this: (full file here )

f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=    2020958 P[GPa]=       1877.100    24.300 E[Ha]=       -59.56300000  1.39000000

f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=    4041916 P[GPa]=       4249.300    18.400 E[Ha]=       160.64900000  1.07400000

f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=    8083831 P[GPa]=       9208.000    31.500 E[Ha]=       513.26500000  1.80900000

What I've tried doing is:

Al = pd.read_csv('Al_EOS_09-18-20.txt', skiprows=18, delimiter='=', names=['f', 'N', 'rho[g/cc]', 'V[A^3]', 'T[K]', 'P[GPa]', 'E[Ha]'])

What this returns is a dataframe with the correct columns, but the values under each column contain the value and the name of the next column. So under Al['f'] I get 'Al N' instead of just 'Al'.

Any help would be GREATLY appreciated!

Answer 1

pd.read_fwf

Fixed Width

import pandas as pd
import io
import requests
url = "http://militzer.berkeley.edu/FPEOS/files/Al_EOS_09-18-20.txt"
s = requests.get(url).content.decode('utf-8')

df = pd.read_fwf(
    io.StringIO(s), skiprows=18, header=None,
    usecols=[1, 3, 5, 7, 9, 11, 12, 14, 15],
    names='f N rho[g/cc] V[A^3] T[K] P[GPa]0 P[GPa]1 E[Ha]0 E[Ha]1'.split()
)

df

      f  N  rho[g/cc]      V[A^3]       T[K]    P[GPa]0  P[GPa]1     E[Ha]0  E[Ha]1
0    Al  1   0.269861  166.025618    2020958     1877.1     24.3    -59.563   1.390
1    Al  1   0.269861  166.025618    4041916     4249.3     18.4    160.649   1.074
2    Al  1   0.269861  166.025618    8083831     9208.0     31.5    513.265   1.809
3    Al  1   0.269861  166.025618   16167663    18629.3     36.3   1055.213   2.077
4    Al  1   0.269861  166.025618   32335325    37424.5     33.7   2129.898   1.924
..   .. ..        ...         ...        ...        ...      ...        ...     ...
235  Al  1   2.383356    1.383547   16167663  2126283.1   5177.3    938.925   2.532
236  Al  1   2.383356    1.383547   32335325  4411441.4   4979.6   2046.679   2.373
237  Al  1   2.383356    1.383547   64670651  8944752.2   4660.1   4212.630   2.231
238  Al  1   2.383356    1.383547  129341301  7991672.2   4802.6   8525.231   2.314
239  Al  1   2.383356    1.383547  215568835    17138.4   7894.0  14252.907   8.470

[240 rows x 9 columns]

Answer 2

Since you know the names of you columns you in theory also know your separators. You can do some regex.

import pandas as pd
from io import StringIO

s = """f= Al N= 1 rho[g/cc]= 0.269861 V[A^3]= 166.02561792 T[K]= 2020958 P[GPa]= 1877.100 24.300 E[Ha]= -59.56300000 1.39000000
f= Al N= 1 rho[g/cc]= 0.269861 V[A^3]= 166.02561792 T[K]= 4041916 P[GPa]= 4249.300 18.400 E[Ha]= 160.64900000 1.07400000
f= Al N= 1 rho[g/cc]= 0.269861 V[A^3]= 166.02561792 T[K]= 8083831 P[GPa]= 9208.000 31.500 E[Ha]= 513.26500000 1.80900000"""


sep = 'f= |N= |rho\[g/cc]= |V\[A\^3]= |T\[K]= |P\[GPa]= |E\[Ha]= '

df = pd.read_csv(StringIO(s), sep=sep, 
                 names=['f', 'N', 'rho[g/cc]', 'V[A^3]', 'T[K]', 'P[GPa]', 'E[Ha]'],
                 engine='python').reset_index(drop=True)


     f  N  rho[g/cc]      V[A^3]     T[K]            P[GPa]  \
0  Al   1   0.269861  166.025618  2020958  1877.100 24.300    
1  Al   1   0.269861  166.025618  4041916  4249.300 18.400    
2  Al   1   0.269861  166.025618  8083831  9208.000 31.500    

                     E[Ha]  
0  -59.56300000 1.39000000  
1  160.64900000 1.07400000  
2  513.26500000 1.80900000  

Answer 3

An option is to create dummys that will store the unwanted values and then drop them when you are done extracting the data:

names = [1, 'f', 2, 'N', 3, 'rho[g/cc]', 4, 'V[A^3]', 5, 'T[K]', 6, 'P[GPa]0', 'P[GPa]1', 7, 'E[Ha]0', 'E[Ha]1']
df = pd.read_csv('test.txt',
                 sep='\s+',
                 index_col=False,
                 names=names)
df.drop(range(1, 8), axis=1, inplace=True)

df

    f  N  rho[g/cc]      V[A^3]     T[K]  P[GPa]0  P[GPa]1   E[Ha]0  E[Ha]1
0  Al  1   0.269861  166.025618  2020958   1877.1     24.3  -59.563   1.390
1  Al  1   0.269861  166.025618  4041916   4249.3     18.4  160.649   1.074
2  Al  1   0.269861  166.025618  8083831   9208.0     31.5  513.265   1.809

Answer 4

Instead of using pandas I would have used a regex and then constructed a dataframe

import re

text = '''f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=    2020958 P[GPa]=       1877.100    24.300 E[Ha]=       -59.56300000  1.39000000
f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=    4041916 P[GPa]=       4249.300    18.400 E[Ha]=       160.64900000  1.07400000
f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=    8083831 P[GPa]=       9208.000    31.500 E[Ha]=       513.26500000  1.80900000
f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=   16167663 P[GPa]=      18629.300    36.300 E[Ha]=      1055.21300000  2.07700000
f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=   32335325 P[GPa]=      37424.500    33.700 E[Ha]=      2129.89800000  1.92400000
f= Al N= 1 rho[g/cc]=   0.269861 V[A^3]=     166.02561792 T[K]=   64670651 P[GPa]=      75127.600    29.100 E[Ha]=      4284.24400000  1.66400000
'''

m = re.findall('f=\s*(.+?)\s*N=\s*(.+?)\s*rho\[g\/cc\]=\s*(.+?)\s*V\[A\^3\]=\s*(.+?)\s*T\[K\]=\s*(.+?)\s*P\[GPa\]=\s*(.+?)\s*E\[Ha\]=\s*(.*)', text)


df6 = pd.DataFrame()

df6[['f','N','rho[g/cc]', 'V[A^3]','T[K]', 'P[GPa]','E[Ha]']] = pd.DataFrame(m)
df6

If you don't know about regex, they are pretty easy to learn and you can test them on sites such as https://regex101.com/ . In any case, I will explain the one that I used.

\s* says zero or more occurrence of whitespace characters.+? requires at least one character () are used to extract the regex.

Keep in mind that you could have searched for a specific type of character. For example, if you need one or more digits use d+, where d stands for digit and + means one or more.

Hope I could help, even if the answer is a little bit different from the logic you were using.

Answer 5

You could write a custom parser and generate a list of dictionaries to pass to the dataframe constructor.

NOTE: this leaves the columns with 2 floats as strings. You'd have to parse them separately.

import pandas as pd
import io
import requests
url = "http://militzer.berkeley.edu/FPEOS/files/Al_EOS_09-18-20.txt"
s = requests.get(url).content.decode('utf-8')

def parse_line(line):
    line = [x.strip() for x in line.split('=')]
    line[0] = (None, line[0])
    line[-1] = (line[-1], None)
    line[1:-1] = [x.rsplit(maxsplit=1) for x in line[1:-1]]
    (_, *values), (*keys, _) = zip(*line)
    return dict(zip(keys, values))

df = pd.DataFrame(map(parse_line, s.splitlines()[18:]))

df

      f  N  rho[g/cc]        V[A^3]       T[K]                  P[GPa]                       E[Ha]
0    Al  1   0.269861  166.02561792    2020958      1877.100    24.300    -59.56300000  1.39000000
1    Al  1   0.269861  166.02561792    4041916      4249.300    18.400    160.64900000  1.07400000
2    Al  1   0.269861  166.02561792    8083831      9208.000    31.500    513.26500000  1.80900000
3    Al  1   0.269861  166.02561792   16167663     18629.300    36.300   1055.21300000  2.07700000
4    Al  1   0.269861  166.02561792   32335325     37424.500    33.700   2129.89800000  1.92400000
..   .. ..        ...           ...        ...                     ...                         ...
249  Al  1  32.383356    1.38354684   16167663   2126283.100  5177.300    938.92500000  2.53200000
250  Al  1  32.383356    1.38354684   32335325   4411441.400  4979.600   2046.67900000  2.37300000
251  Al  1  32.383356    1.38354684   64670651   8944752.200  4660.100   4212.63000000  2.23100000
252  Al  1  32.383356    1.38354684  129341301  17991672.200  4802.600   8525.23100000  2.31400000
253  Al  1  32.383356    1.38354684  215568835  30017138.400 17894.000  14252.90700000  8.47000000

[254 rows x 7 columns]