将数据从特定列移动到新行中的不同列

Question

I have a csv containing details of documents located on Google Drive. I am trying to make it easier to read and deal with as this example has over 400 columns.

Each row in the csv represents a file on Google drive. There are multiple columns to denote who owns the file and who it is shared with.

Every time a file has been shared, the details of the person it has been shared with are appended as a new column to that row.

I have loaded the data into Pandas data frame and I'm struggling to move the contents of certain columns to a new row.

Below is an example

Input:
owner  |  id  |  title | permissions.0.name | permissions.0.email | permsissions.1.name | permissions.1.email
value     1      doc1    Tommy                tommy@office.com      Timmy                 timmy@office.com
value     2      doc2    Tommy                tommy@office.com
value     3      doc3    Timmy                timmy@office.com
value     4      doc4    Tammy                tammy@office.com      Tommy                 tommy@office.com

Output:
owner  |  id  |  title | permissions.0.name | permissions.0.email 
value     1      doc1    Tommy                tommy@office.com      
value     2      doc2    Tommy                tommy@office.com
value     3      doc3    Timmy                timmy@office.com
value     4      doc4    Tammy                tammy@office.com      
value     5      doc1    Timmy                timmy@office.com
value     6      doc4    Tommy                tommy@office.com

I began by creating a list of and finding out the maximum number in the column headings (it is 46 in the full data). Then loop through from 1 to 46 building the column name to look at and moving the contents from that column to a different column on a new row. But I had no idea how to move the contents...

import pandas

df = pandas.read_csv(input.csv)

cols = list(df) #list of column names

maxcol =[]

for c in cols:
    if '.' in c:
        n = c.split('.')[1]
        maxcol.append(int(n))

maxval = max(maxcol)

for i in range(1 to maxval):
    colname = 'permissions.' + str(i) + '.name'
    # move contents from this column to permissions.0.name in new row somehow

There are many more columns (over 400) and do not appear in an organised structure. For example columns are created when required. So we have columns like this:

permissions.5.email | permissions.1.withPhoto | permissions.6.name | permission.6.email

Answer 1

You can use this code. I think there are easier ways to do this, but it seems to be giving the required result:

cols = list(df)
# Get the variable column names
per_cols = [c for c in cols if '.' in c]
# Get the constant column names, i.e. title
main_cols = [c for c in cols if '.' not in c]
# Get the set of numbers present in columns
col_no = set([c.split('.')[1] for c in per_cols])

result = pd.DataFrame()
for col in col_no:
    # Get the list of columns with the current number
    part = [c for c in per_cols if c.split('.')[1] == col]
    # Add the constant columns to get a complete df
    part =  main_cols + part
    temp = df[part]
    # Change the name of columns to '0' for unified result
    temp.columns = [c.replace(col, '0') for c in temp.columns]
    # Drop the NaN rows (preferrably use a subset you are certain wouldn't be null)
    temp = temp.dropna()
    # Append the chunck of df to result
    # If the chunck has a new column, it will be added to result df
    result = result.append(temp)