how to read several files into a python code and loop over them?

Question

I am coding in python and have written a code which basically reads ina file takes some information from it does some calculations and then outputs the answers to a new file; I have now tried to generalise so that it loops over many files (as I have a catalogue of files I need to go through) by reading there file names in from a file in which they are listed. However I get this error message IOError: [Errno socket error] [Errno -2] Name or service not known

I assume its because I am now trying to run my code for several files systematically so here is the bit of code in which I do it:

f = open('//disk2/ps1/cech/CFHTLenS/cluster_catalogues/field_list')
fields = f.readlines()
f.close()
for W in fields:

    file = open('//disk2/ps1/cech/CFHTLenS/cluster_catalogues/clusters_%s_info.cat' %W)
    data = np.loadtxt(file)
    file.close()


    sig_cl = data[:,3].copy()

    m200 = 10**(0.124 * sig_cl + 12.493)
    np.savetxt('//disk2/ps1/bertbert/z_ref_%s.cat'%W,m200)

Answer 1

I'm not sure why you're getting a socket error from file handling routines, but the most obvious issue is that W will have an end of line character at the end which is likely to cause problems.

Try:

W = W.strip()

as the first line of your for loop.

This will remove all leading and trailing spaces from W before you add it to the name of the file to open.

Answer 2

For looping over a arbitrary sets of files the module fileinput could help you. It sets up a nice command line interface where you can specify a set of files and easily loop over them.