Python csv file reader not reading whole file

Question

Morning all, Firstly I've read the post which is similar to this question,but it doesn't solve my problem.

I have 3 csv files(broker-4393 rows;skips-27761 rows;tippers-19118 rows). each csv file is read by the same function:

Long story short:

broker csv file (containing 4393 rows) produces a list of 1359 rows. MISSING

skip csv file (containing 27761 rows) produces a list of 27761 rows. FINE

tipper csv file (containing 19118 rows) produces a list of 19118 rows. FINE

Anyone managed to find a fix?

[see cost below]

import os, re, csv

# -------------------------------------- General Functions -------------------------------------- #
# function: find journey summary file
def FileFinder(fl, begin):
    regex = begin
    pattern = re.compile(regex)
    for f in fl:
        if re.findall(pattern, f):   #empty seq = False
            global found;
            found = re.findall(pattern, f)

# function: read from 'Enquiry-...'
def ReadEnquiry():
    with open(d + found[0], "r") as fR:
        r = csv.reader(fR)

        # capture data from csv file into 'clist'
        for row in r:
            global rlist;
            rlist.append(row)
    fR.close()
# ----------------------------------------------------------------------------------------------- #
# --------------------------------------- Broker Functions -------------------------------------- #
# function: Find and Read from BrokerExport.
def BrokerExp():
    FileFinder(filelist, 'BrokerExport.*')
    ReadEnquiry()
    CreateBrokerList(rlist, 48, 17, 74, brokerlist)

# function: create a list of broker data.  Format: Account Number,Date,Price(ex-VAT),Profit
def CreateBrokerList(rlist, col1, col2, col3, expList):
    for row in rlist:
        if row[41] == '':         # exclude jobs that were cancelled.
            expList.append([row[col1], row[col2], row[col3]])
# ----------------------------------------------------------------------------------------------- #
# ---------------------------------------- Skip Functions --------------------------------------- #
# function: Find and Read from SkipsExport.
def SkipExp():
    FileFinder(filelist, 'SkipsExport.*')
    ReadEnquiry()
    CreateSkipList(rlist, 2, 42, 46, skiplist)

# function: create a list of skip data.  Format: Account Number,Date,Price(ex-VAT),Profit
def CreateSkipList(rlist, col1, col2, col3, expList):
    for row in rlist:
        expList.append([row[col1], row[col2], row[col3]])
# ----------------------------------------------------------------------------------------------- #
# ---------------------------------------- Skip Functions --------------------------------------- #
# function: Find and Read from TipperExport.
def TipperExp():
    FileFinder(filelist,'TipperExport.*')
    ReadEnquiry()
    CreateSkipList(rlist,3,4,34,tipperlist)

# function: create a list of tipper data.  Format: Account Number,Date,Price(ex-VAT),Profit
def CreateTipperList(rlist, col1, col2, col3, expList):
    for row in rlist:
        expList.append([row[col1], row[col2], row[col3]])
# ----------------------------------------------------------------------------------------------- #

# --- General Variables --- #
rlist = [];                               # 'rlist' list read from csv.
found = ''                                # string to hold filename found through 'FileFinder()'
d = 'U:/rmarshall/To Do/'                 # directory to use
headings = ['Company Name', 'Rep', \
        'Month 1 Calls', 'Month 1 Inv Tots', 'Month 1 No. of Invs', \
        'Month 2 Calls', 'Month 2 Inv Tots', 'Month 2 No. of Invs', \
        'Month 3 Calls', 'Month 3 Inv Tots', 'Month 3 No. of Invs', \
        'Month 4 Calls', 'Month 4 Inv Tots', 'Month 4 No. of Invs', \
        'Month 5 Calls', 'Month 5 Inv Tots', 'Month 5 No. of Invs', \
        'Month 6 Calls', 'Month 6 Inv Tots', 'Month 6 No. of Invs', \
        'Month 7 Calls', 'Month 7 Inv Tots', 'Month 7 No. of Invs', \
        'Month 8 Calls', 'Month 8 Inv Tots', 'Month 8 No. of Invs', \
        'Month 9 Calls', 'Month 9 Inv Tots', 'Month 9 No. of Invs', \
        'Month 10 Calls', 'Month 10 Inv Tots', 'Month 10 No. of Invs', \
        'Month 11 Calls', 'Month 11 Inv Tots', 'Month 11 No. of Invs', \
        'Month 12 Calls', 'Month 12 Inv Tots', 'Month 12 No. of Invs']
cp=[headings]; da=[headings]; mb=[headings]; apd=[headings]; bobs=[headings]    # separate Rep lists
filelist=os.listdir(d)                                                          # place directory filenames into a list
dlist=[]; brokerlist=[]; skiplist=[]; tipperlist=[]; book1=[]                   # lists used throughout code
brklist=[]; skplist=[]; tprlist=[]                                              # a list of names
# ------------------------- #

# --- main --- #
Enquiry_Main()          # call 'Enquiry_Main()' to run all work to create 'cp,da,mb,apd,bob' list data.
rlist=[]; dlist=[]      # reset lists
print('1')
BrokerExp()             # call 'BrokerExp()' to run all work to create 'brokerlist' data.
rlist=[]                # reset list
print('2')
SkipExp()               # call 'SkipExp()' to run all work to create 'skiprlist' data.
rlist=[]                # reset list
print('3')
TipperExp()             # call 'TipperExp()' to run all work to create 'tipperlist' data.
rlist=[]                # reset list

a=0
for row in brokerlist:a+=1
print(a)

a=0
for row in skiplist:a+=1
print(a)

a=0
for row in tipperlist:a+=1
print(a)

Answer 1

You are using a lot of globals everywhere, it over complicates your program. Functions can create and return filtered lists, so you can reuse them and pass as an input to another function which will also filter on some other parameter.

This function creates local list expList and returns nothing. The same is with CreateSkipList, CreateTipperList functions.

def CreateBrokerList(rlist, col1, col2, col3, expList):
      for row in rlist:
        if row[41] == '':         # exclude jobs that were cancelled.
            expList.append([row[col1], row[col2], row[col3]])

Example of list returned in proper way:

def ReadEnquiry(file_to_read, rlist):
  with open(file_to_read, "r") as fR:
      r = csv.reader(fR)
      for row in r:
          rlist.append(row)
  return rlist

Usage example:

rlist = []
read_list = ReadEnquiry(d + found[0], rlist)
# pass read_list to other function as parameter
brokerlist = []
CreateBrokerList(read_list, 48, 17, 74, brokerlist)

Answer 2

The answer is that there is no problem with csv reader. After looking at the code I saw the following line:

if row[41] == '':         # exclude jobs that were cancelled.

The list produced 1359 results instead of 4393 because the difference was excluded by the above line of code.