标记熊猫数据框中的文本

Question

I have a Pandas DataFrame with scripts collected from an external source. The column text_content contains the script contents. The longest script consists of 85.617 characters.

A sample to give you an idea:

The scripts contain table names and other useful information. Currently, the dataframe is written to a SQLite database table, which can then be searched using ad-hoc SQL statements (and distributed to a larger crowd).

A common use case is that we'll have a list of table names, and would like to know the scripts in which they appear. If we need to do this in SQL, it would require us to execute wildcard searches using the LIKE operator, which kinda sucks performance-wise.

Thus, I wanted to extract the words from the script while it's still in a DataFrame, resulting in a two columns table, with each row consisting of:

• a link to the original script row
• a word that was found in the script

Each script would result in a number of rows (depending on the amount of matches).

So far, I wrote this to extract the words from the script:

DataFrame(df[df.text_type == 'DISCRIPT']
    .dropna(subset=['text_content'])
    .apply(lambda x: re.findall('([a-zA-Z]\w+)', x['text_content']), axis=1)
    .tolist())

The result:

So far, so good (?).

There are two more steps I need to go through, but I'm a little stuck here.

1. Remove a list of common words (eg SQL reserved words).
2. Reshape the DataFrame so each row is a match, but with a link to the script in the original DataFrame.

I can use T to transpose the DataFrame, use replace() in combination with a predefined list of keywords (replacing them with an NA value) and finally use dropna() to shorten the list to just the keywords. However, I'm not sure if this is the best approach.

I'd very much appreciate your comments and suggestions!

Answer 1

IIUC you can try add index=df.index to df2 constructor, then reshape by stack and filter by isin :

print df
                            text_content text_name text_type
1614  CHECK FOR LOCK STATUS CACHETABLEDB      TEXT  DISCRIPT
1615  CHECK FOR LOCK STATUS CACHETABLEDB      TEXT  DISCRIPT

df2 = pd.DataFrame(df[df.text_type == 'DISCRIPT']
    .dropna(subset=['text_content'])
    .apply(lambda x: re.findall('([a-zA-Z]\w+)', x['text_content']), axis=1)
    .tolist(), index=df.index)
print df2
          0    1     2       3             4
1614  CHECK  FOR  LOCK  STATUS  CACHETABLEDB
1615  CHECK  FOR  LOCK  STATUS  CACHETABLEDB

#reshape all rows to column
df2 = df2.stack().reset_index(level=0)
df2.columns = ['id', 'words']

L = ['CACHETABLEDB','STATUS']
#remove reserved words
df2 = df2.loc[~df2.words.isin(L)].reset_index(drop=True)
print df2
     id  words
0  1614  CHECK
1  1614    FOR
2  1614   LOCK
3  1615  CHECK
4  1615    FOR
5  1615   LOCK