How to use regex re.compile Match() or findall() in list comprehension

Question

I am trying to use regex in list comprehension without needing to use the pandas extract() functions.

I want to use regex because my code might need to change where I need to use more complex pattern matching. A kind user here suggested I use the str accessor functions but again it mainly works because the current pattern is simple enough.

As of now, I need to return pandas rows that either contain nan or whose values under ODFS_FILE_CREATE_DATETIME are not 10 string numbers ie: does not match the current format: 2020012514 . To this intent I tried to bypass the str method and use regex. However this doesn't do anything. It puts everything into my list of tuples even though I told it to only put values that only contain nan or where the bool(regex.search()) is not true:

def process_csv_formatting(csv):
odfscsv_df = pd.read_csv(csv, header=None,names=['ODFS_LOG_FILENAME', 'ODFS_FILE_CREATE_DATETIME', 'LOT', 'TESTER', 'WAFER_SCRIBE'], dtype={'ODFS_FILE_CREATE_DATETIME': str})
odfscsv_df['CSV_FILENAME'] = csv.name
odfscdate_re = re.compile(r"\d{10}")
errortup = [(odfsname, "Bad_ODFS_FILE_CREATE_DATETIME= " + str(cdatetime), csv.name) for odfsname,cdatetime in zip(odfscsv_df['ODFS_LOG_FILENAME'], odfscsv_df['ODFS_FILE_CREATE_DATETIME']) if not odfscdate_re.search(str(cdatetime))]
emptypdf = pd.DataFrame(columns=['ODFS_LOG_FILENAME', 'ODFS_FILE_CREATE_DATETIME', 'LOT', 'TESTER', 'WAFER_SCRIBE'])

#print([tuple(x) for x in odfscsv_df[odfscsv_df.isna().any(1) | odfscdate_re.search(str(odfscsv_df['ODFS_FILE_CREATE_DATETIME'])) ].values])
m1 = odfscsv_df.isna().any(1)

m1 = odfscsv_df.isna().any(1)
s = odfscsv_df['ODFS_FILE_CREATE_DATETIME']
m2 = ~s.astype(str).str.isnumeric()
m2 = bool(odfscdate_re.search(str(s)))
m4 = not m2
print(m4)
m3 = s.astype(str).str.len().ne(10)

#print([tuple(x) for x in odfscsv_df[m1 | m2 | m3].values])
print([tuple(x) for x in odfscsv_df[m1 | ~bool(odfscdate_re.search(str(s)))].values])

if len(errortup) != 0:
    #print(errortup)  #put this in log file statement somehow
    #print(errortup[0][2])
    return emptypdf
else:

    return odfscsv_df

Answer 1

If you want to use re module. You need to use it with map . For 10-digit strings, use this pattern r"^\d{10}$"

import re

odfscdate_re = re.compile(r"^\d{10}$")

m1 = odfscsv_df.isna().any(1)
m2 = odfscsv_df['ODFS_FILE_CREATE_DATETIME'].map(lambda x: 
                                                 odfscdate_re.search(str(x)) == None)
[tuple(x) for x in odfscsv_df[m1 | m2].values]

Note : depend on your requirement, I think you may also use match instead of search .