尝试计算 dataframe 列子集中的 NaN 时出现 Pandas 类型错误

Question

I'm writing a script to perform LLoD analysis for qPCR assays for my lab. I import the relevant columns from the.csv of data from the instrument using pandas.read_csv() with the usecols parameter, make a list of the unique values of RNA quantity/concentration column, and then I need to determine the detection rate / hit rate at each given concentration. If the target is detected, the result will be a number; if not, it'll be listed as "TND" or "Undetermined" or some other non-numeric string (depends on the instrument). So I wrote a function that (should) take a quantity and the dataframe of results and return the probability of detection for that quantity. However, on running the script, I get the following error:

Traceback (most recent call last):
  File "C:\Python\llod_custom.py", line 34, in <module>
    prop[idx] = hitrate(val, data)
  File "C:\Python\llod_custom.py", line 29, in hitrate
    df = pd.to_numeric(list[:,1], errors='coerce').isna()
  File "C:\Users\wmacturk\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\core\frame.py", line 3024, in __getitem__
    indexer = self.columns.get_loc(key)
  File "C:\Users\wmacturk\AppData\Local\Programs\Python\Python39\lib\site-packages\pandas\core\indexes\base.py", line 3080, in get_loc
    return self._engine.get_loc(casted_key)
  File "pandas\_libs\index.pyx", line 70, in pandas._libs.index.IndexEngine.get_loc
  File "pandas\_libs\index.pyx", line 75, in pandas._libs.index.IndexEngine.get_loc
TypeError: '(slice(None, None, None), 1)' is an invalid key

The idea in the line that's throwing the error ( df = pd.to_numeric(list[:,1], errors='coerce').isna() ) is to change any non-numeric values in the column to NaN, then get a boolean array telling me whether a given row's entry is NaN, so I can count the number of numeric entries with df.sum() later. I'm sure it's something that should be obvious to anyone who's worked with pandas / dataframes, but I haven't used dataframes in python before, so I'm at a loss. I'm also much more familiar with C and JavaScript, so something like python that isn't as rigid can actually be a bit confusing since it's so flexible. Any help would be greatly appreciated.

NB the conc column will consist of 5 to 10 different values, each repeated 5-10 times (ie 5-10 replicates at each of the 5-10 concentrations); the detect column will contain either a number or a character string in each row -- numbers mean success, strings mean failure... For my purposes the value of the numbers is irrelevant, I only need to know if the target was detected or not for a given replicate. My script (up to this point) follows:

import os
import pandas as pd
import numpy as np
import statsmodels as sm
from scipy.stats import norm
from tkinter import filedialog
from tkinter import *

# initialize tkinter
root = Tk()
root.withdraw()


# prompt for data file and column headers, then read those columns into a dataframe
print("In the directory prompt, select the .csv file containing data for analysis")
path = filedialog.askopenfilename()

conc = input("Enter the column header for concentration/number of copies: ")
detect = input("Enter the column header for target detection: ")
tnd = input("Enter the value listed when a target is not detected (e.g. \"TND\", \"Undetected\", etc.): ")

data = pd.read_csv(path, usecols=[conc, detect])

# create list of unique values for quantity of RNA, initialize vectors of same length
# to store probabilies and probit scores for each
qtys = data[conc].unique()
prop = probit = [0] * len(qtys)

# Function to get the hitrate/probability of detection for a given quantity
def hitrate(qty, dataFrame):
    list = dataFrame[dataFrame.iloc[:,0] == qty]
    df = pd.to_numeric(list[:,1], errors='coerce').isna()
    return (len(df) - (len(df)-df.sum()))/len(df)

# iterate over quantities to calculate the corresponding probability of Detection
# and its associate probit score
for idx, val in enumerate(qtys):
    prop[idx] = hitrate(val, data)
    probit[idx] = norm.ppf(hitrate(val, data))

# create an array of the quantities with their associated probabilities & Probit scores
hitTable = vstack([qtys,prop,probit])

sample dataframe can be created with:

d = {'qty':[1,1,1,1,1, 10,10,10,10,10, 20,20,20,20,20, 50,50,50,50,50, 100,100,100,100,100], 'result':['TND','TND','TND',5,'TND', 'TND',5,'TND',5,'TND', 5,'TND',5,'TND',5, 5,6,5,5,'TND', 5,5,5,5,5]}
exData = pd.DataFrame(data=d)

Then just use exData as the dataframe data in the original code

EDIT: I've fixed the problem by tweaking Loic RW's answer slightly. The function hitrate should be

def hitrate(qty, df):
    t_s = df[df.qty == qty].result
    t_s = t_s.apply(pd.to_numeric, args=('coerce',)).isna()
    return (len(t_s)-t_s.sum())/len(t_s)

Answer 1

Does the following achieve what you want? I made some assumptions on the structure of your data.

def hitrate(qty, df):
    target_subset = df[df.qty == qty].target
    target_subset = target_subset.apply(pd.to_numeric, args=('coerce',)).isna()
    return 1-((target_subset.sum())/len(target_subset))

If i run the following:

data = pd.DataFrame({'qty': [1,2,2,2,3],
                     'target': [.5, .8, 'TND', 'Undetermined', .99]})
hitrate(2, data)

I get: 0.33333333333333337