Pandas create new column with values from other columns, selected based on column value

Question

I have a dataframe that looks like a bit like this example. For some reasons, the raw data has the value replicated across.

  Node Node 1 Value Node 2 Value Node 3 Value
0    1            A            B            C
1    2            A            B            C
2    3            A            B            C

I want to transform it to look like this:

  Node Value
0    1     A
1    2     B
2    3     C

This iterrows code works as intended but it is very slow for my data (48 nodes with ~20,000 values).

I feel like there must be a faster way, perhaps with apply but I can't figure it out.

import pandas as pd

df = pd.DataFrame({"Node": ["1", "2", "3"],
                   "Node 1 Value": ["A","A","A"],
                   "Node 2 Value": ["B","B","B"],
                   "Node 3 Value": ["C","C","C"]})

print(df)

for index, row in df.iterrows():
    df.loc[index, 'Value'] = row["Node {} Value".format(row['Node'])]

print(df[['Node','Value']])

Answer 1

Use DataFrame.lookup and then DataFrame.assign :

a = df.lookup(df.index, "Node " + df.Node.astype(str) + " Value")

df = df[['Node']].assign(Value = a)
print (df)
   Node Value
0     1     A
1     2     B
2     3     C

EDIT: If some values missing you can extract this values by numpy.setdiff1d for dictionary with default value, eg np.nan and add to DataFrame before lookup :

print (df)
   Node Node 1 Value Node 2 Value Node 3 Value
0     1            A            B            C
1     2            A            B            C
3     5            A            B            C

s = "Node " + df.Node.astype(str) + " Value"
new = dict.fromkeys(np.setdiff1d(s, df.columns), np.nan)
print (new)
{'Node 5 Value': nan}

print (df.assign(**new))
   Node Node 1 Value Node 2 Value Node 3 Value  Node 5 Value
0     1            A            B            C           NaN
1     2            A            B            C           NaN
3     5            A            B            C           NaN

a = df.assign(**new).lookup(df.index, s)
print (a)
['A' 'B' nan]

df = df[['Node']].assign(Value = a)
print (df)
   Node Value
0     1     A
1     2     B
3     5   NaN

Another idea with definition of lookup :

def f(row, col):
    try:
        return df.at[row, col]
    except:
        return np.nan

s = "Node " + df.Node.astype(str) + " Value"
a = [f(row, col) for row, col in zip(df.index, s)]

df = df[['Node']].assign(Value = a)
print (df)
   Node Value
0     1     A
1     2     B
3     5   NaN

And solution with DataFrame.melt :

s = "Node " + df.Node.astype(str) + " Value"
b = (df.assign(Node = s)
        .reset_index()
        .melt(['index','Node'], value_name='Value')
        .query('Node == variable').set_index('index')['Value'])


df = df[['Node']].join(b)
print (df)
   Node Value
0     1     A
1     2     B
3     5   NaN