从非结构化文本创建 pandas DataFrame

Question

Sorry for the noob question but here it goes. I'm trying to analyse some Facebook messages. So far I downloaded an html file, turned it into a neat list with BeautifulSoup and now I'm trying to create a dataframe out of it.

I'm looking at this resource: https://datatofish.com/list-to-dataframe/ but it's not working out.

Here's the format of what I have now:

list = ['2019-01-07 12:51 PM', 'name1', 'hi how are you', 'im at home', 'wanna come over?', '2019-01-07 01:02 PM', 'name2', 'hell yeah', '🐟', 'ill bring beer', '2019-01-07 01:06 PM', 'name1', 'awesome', 'and so on']

I tried a couple of different things but I'm starting to think I bit more than I can chew. I'm learning at the moment.

Here's the output I'm hoping to get:

index date          time       name            message
0      2019-01-07   12:51 PM   name1           hi how are you
1      2019-01-07   12:51 PM   name1           im at home
2      2019-01-07   12:51 PM   name1           wanna come over?
3      2019-01-07   12:56 PM   name2           hell yeah

I tried iterating over the list and filling columns as it went along and hit a date, name or message.

As I said, I'm learning, so rather than solutions it would be amazing if you could point me in the right direction to research into. I'd be very grateful. Thanks!

Edit: I tried a couple of existing message parsers but they all stopped being supported in 2018 for some reason. They also all give me parse error messages.

Answer 1

It is a bit ugly, but it works. I'll gladly upvote a more elgant solution !

l = iter(['2019-01-07 12:51 PM', 'name1', 'hi how are you', 'im at home', 'wanna come over?', '2019-01-07 01:02 PM', 'name2', 'hell yeah', '🐟', 'ill bring beer', '2019-01-07 01:06 PM', 'name1', 'awesome', 'and so on'])
df = pd.DataFrame()

# get first element in list
x = next(l)

# if element is the last, catch the IterationError and stop
try:
    while 1:
        # try to convert element to datetime
        datetime = pd.to_datetime(x, format="%Y-%m-%d %H:%M %p")
        # if successful get next element as name
        x = next(l)
        name = x

        # get next elements as messages while they do not match datetime format
        x = next(l)
        while 1:
            try:
                # if datetime conversion is successful break while 
                pd.to_datetime(x, format="%Y-%m-%d %H:%M %p");
                break
            except ValueError:
                # else add message to dataframe
                df = df.append([{"datetime":datetime,"name":name,"msg":x}])
                x = next(l)
except StopIteration:
    pass

df["date"] = df["datetime"].dt.date
df["time"] = df["datetime"].dt.time
print(df)

             datetime               msg   name        date      time
0 2019-01-07 12:51:00    hi how are you  name1  2019-01-07  12:51:00
0 2019-01-07 12:51:00        im at home  name1  2019-01-07  12:51:00
0 2019-01-07 12:51:00  wanna come over?  name1  2019-01-07  12:51:00
0 2019-01-07 01:02:00         hell yeah  name2  2019-01-07  01:02:00
0 2019-01-07 01:02:00                 🐟  name2  2019-01-07  01:02:00
0 2019-01-07 01:02:00    ill bring beer  name2  2019-01-07  01:02:00
0 2019-01-07 01:06:00           awesome  name1  2019-01-07  01:06:00
0 2019-01-07 01:06:00         and so on  name1  2019-01-07  01:06:00

Answer 2

Using regular expressions and list comprehensions, the list content is extracted and transformed into a Pandas dataframe:

import pandas as pd

import re

datetime_regex = re.compile(r"\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}\sPM")
name_regex = re.compile(r"name\d+")

cols = ["date",
        "time",
        "name",
        "message"
        ]

l = ['2019-01-07 12:51 PM',
     'name1',
     'hi how are you',
     'im at home',
     'wanna come over?',
     '2019-01-07 01:02 PM',
     'name2',
     'hell yeah',
     '🐟',
     'ill bring beer',
     '2019-01-07 01:06 PM',
     'name1',
     'awesome',
     'and so on'
     ]
tmp = ''.join(l)

datetimes = re.findall(datetime_regex, tmp)
dates = [datetime[:11] for datetime in datetimes]
times = [datetime[11:] for datetime in datetimes]
names = re.findall(name_regex, tmp)

messages = [line
            for line in l
            if not line.startswith(('2019', 'name1', 'name2'))
            ]

data = [[[dates[0], times[0], names[0], msg]
        for msg in messages[:3]],
        [[dates[1], times[1], names[1], messages[3]]],
        [[dates[2], times[2], names[2], msg]
        for msg in messages[4:]]
        ]

flatten = [item for sublist in data for item in sublist]

df = pd.DataFrame(flatten, columns=cols)
print(df)

Which returns:

          date      time   name           message
0  2019-01-07   12:51 PM  name1    hi how are you
1  2019-01-07   12:51 PM  name1        im at home
2  2019-01-07   12:51 PM  name1  wanna come over?
3  2019-01-07   01:02 PM  name2         hell yeah
4  2019-01-07   01:06 PM  name1                 🐟
5  2019-01-07   01:06 PM  name1    ill bring beer
6  2019-01-07   01:06 PM  name1           awesome
7  2019-01-07   01:06 PM  name1         and so on