How to subtract two columns of lists from each other in pandas?

Question

I have data in a tab-separated value text file that look like this:

FileName    Onsets          Offsets
FileName1   [9, 270, 763]   [188, 727, 1252]
FileName2   [52, 634, 1166, 1775, 2104] [472, 1034, 1575, 1970, 2457]
FileName3   [180, 560, 1332, 1532]  [356, 1286, 1488, 2018]

These are data from audio files. Each row contains a series of onset and offset times for each of the sounds I'm researching.

In the first row of data, 9 is the onset time of the first sound, and 188 is the offset time of the first sound. That means it lasted for 179 ms.

I need the durations for each sound, and the gaps of silence between each sound.

Currently I read the data as follows:

import pandas as pd
import numpy as np

data = pd.read_csv('/path/file.txt', delimiter='\t')
    
FileName = data[["FileName"]].to_numpy()  
Onsets = data[["Onsets"]].to_numpy()  
Offsets = data[["Offsets"]].to_numpy() 

That gives me three numpy arrays. For the onsets and offsets, each row is actually an array of the numbers in the original data file.

What code can I use to extract those numbers so that I can then subtract the onset times from the offset times to determine the durations?

Answer 1

• The first issue is, you have columns of strings that must be converted to lists, using ast.literal_eval
• In order to perform array subtracting, convert the values in 'Onsets' and 'Offsets' to numpy.arrays
• To calculate the gaps of silence:
  • The first gap of silence between [9, 270, 763] and [188, 727, 1252] begins at 188 and end at 270 .
  • To perform the array calculation, subtract the first two elements of Offsets from the last two elements of Onsets
    • 270 - 188 and 763 - 727
    • x[0][1:] is all but the first element of Onsets
    • x[1][:-1] is all but the last element of Offsets

import pandas as pd
import numpy as np
from ast import literal_eval

# load data and use literal_eval to converts strings to lists
data = pd.read_csv('/path/file.txt', delimiter='\t', converters={'Onsets': literal_eval, 'Offsets': literal_eval})

# convert rows of lists to numpy arrays
data[['Onsets', 'Offsets']] = data[['Onsets', 'Offsets']].applymap(np.array)

# subtract the values in the arrays
data['duration'] = data.Offsets.sub(data.Onsets)  # data.Offsets - data.Onsets can also be used

# calculate the gaps of silence
data['gaps'] = data[['Onsets', 'Offsets']].apply(lambda x: x[0][1:] - x[1][:-1], axis=1)

# display(data)
    FileName                       Onsets                        Offsets                   duration                  gaps
0  FileName1                [9, 270, 763]               [188, 727, 1252]            [179, 457, 489]              [82, 36]
1  FileName2  [52, 634, 1166, 1775, 2104]  [472, 1034, 1575, 1970, 2457]  [420, 400, 409, 195, 353]  [162, 132, 200, 134]
2  FileName3       [180, 560, 1332, 1532]        [356, 1286, 1488, 2018]       [176, 726, 156, 486]         [204, 46, 44]