How do I use the pandas.melt function to unpivot a few columns while keeping the rest intact

Question

I am working with a database with 66 columns and I wish to unpivot only 3 columns using python pandas.melt function.

df = pd.melt(df,value_vars=["RFR 1","RFR 2","RFR 3"],var_name="RFR Index",value_name="RFR Mode")

I'm finding all the other columns are dropped unless I set them as id_vars . How do I keep them all without listing all of them? (since there are so many of them)

Answer 1

IIUC, you can use pandas.Index.difference to get all columns of your dataframe that are not in your specified list.

A bit of a nonsensical example, but:

df = pd.DataFrame(data=np.random.randn(5,10),
                  columns=['a','b','c','d','e','f','g','h','i','j'])

val_vars = ['e','f','g']
other_vars = df.columns.difference(val_vars)

df.melt(id_vars=other_vars, value_vars=val_vars)

An alternative approach not using pandas-specific functionality would be to use sets:

other_vars = set(df.columns) - set(val_vars)

Answer 2

Just create list that doesn't include the columns that are in the value_vars

value_vars = ["RFR 1","RFR 2","RFR 3"]
id_vars = [x for x in df.columns if x not in value_vars]
df = pd.melt(df,value_vars=value_vars,var_name="RFR Index",value_name="RFR Mode", id_vars=id_vars)