Getting correct frequencies using a fast Fourier transform

Question

I am trying to take understand the frequencies of a dataset and am having issues in getting the fast Fourier transform to work. The main problem is that I cannot figure out how to get the correct frequencies on the x-axis.

Background

I have a dataset with many columns but the columns of interest are TOF(time of flight) and dE/dx. I have attached the CSV file containing the data. Here is how I access it.

import pandas as pd
file = 'np_15us.csv'
dataset = pd.read_csv(file,skiprows=8)
df = dataset[:-1] #necesssary because last row of the dataset is null for some reason
x = df['TOF'] #TOF is in micro-seconds
y= df['dE/dx']

Now, when you plot x vs. y it's roughly a sinusoid. I can eyeball the frequency to be roughly 116kHz. I want to get the exact frequencies by using a fast Fourier transform because I expect other datasets to be imperfect sinusoids.

Problem

When I try to take the fft of the data set using this code:

import numpy as np
x_new = np.arange(0,14, dt)
y_new = func_1(x_new)
fs = len(y_new)

fig = plt.figure(2)
plt.subplot(2,1,1)
plt.plot(x_new, y_new)
plt.xlabel('time (usec)')
plt.ylabel('E (V/mm)')
plt.subplot(2,1,2)
fft = np.fft.fft(y)/len(y)
fft = fft[range(int(len(y)/2))]
tpCount = len(fft)
values = np.arange(int(tpCount))
timePeriod = tpCount/samp
frequencies = (values/(2*timePeriod))*10**6 #followed some tutorial to get here
plt.plot(frequencies[:100], abs(fft)[:100]) #zooming in to one of the peaks 
fig.tight_layout()
plt.show()

I get,

This has a frequency of roughly 260kHz which is an overestimate. I can run the same script using np.fft.fftfreq to get

func_1 = interpolate.interp1d(x, y)
samp = 100
dt = 1/samp
x_new = np.arange(0,14, dt)
y_new = func_1(x_new)
fs = len(y_new)
fig = plt.figure(2)
plt.subplot(2,1,1)
plt.plot(x_new, y_new)
plt.xlabel('time (usec)')
plt.ylabel('E (V/mm)')
plt.subplot(2,1,2)
fft = np.fft.fft(y)/len(y)
freqs = np.fft.fftfreq(len(fft),dt)
fft_shift = np.fft.fftshift(fft)
freqs = np.fft.fftshift(freqs)
plt.plot(freqs[int(len(freqs)/2):int(len(freqs))-300], abs(fft_shift[int(len(fft_shift)/2):int(len(freqs))-300]) #now I don't understand the frequnecies
fig.tight_layout()
plt.show()

Whichever way I use, I get incorrect frequencies. So, I am definitely doing something wrong. I don't really understand how the frequencies are calculated using np.fft.fftfreq.

I don't exactly know the sampling frequency of the dataset which is why I am interpolating to get more control over that. I am not sure if I am supposed to be doing that. I would like to use np.fft.fftfreq since the code for that seems cleaner.

Thank you for your help. Please let me know if you have any questions.

Please note - in this case dE/dx in the dataset should actually be dV/dx like shown in the first plot. The CSV file just has is named incorrectly.

link to CSV file - https://drive.google.com/file/d/1LNcue82K2y4ZgKr8cPIgp7VCC2vFk_J9/view?usp=sharing

Answer 1

If your data is very close to a sinusoid like this, and you generally have at least one period I think you will get a much better estimate in the time domain. FFTs have energy in bins and it just picking the top bin will not give you the best result.

I would measure the distance between zero crossings of the dE/dx value. (Do a linear interpolation to get a more exact zero crossing location). Without even doing the interpolation I got a value of 117.9kHz.