This is the part of the code that I want to package into function. I tired this way
def mm(*files, writer=None):
for file in files:
if fnmatch.fnmatch(file, 'ticr_calculated_2*'):
gunzip(file, file.replace(".gz", ""))
json_fi = glob.glob("*.json")
table_list=[]
for filename in json_fi:
with open(filename, encoding='utf-8', mode='r') as i:
data = pd.read_json(i, lines=True)
table_list.append(data)
for table in table_list:
writer = append_to_parquet_table(table, filepath, writer)
if writer:
writer.close()
return writer
Latter I will have 'tick_calculated_3*' and similar file names.
I invoked my function
files = [fi for fi in files if fi.endswith(".gz")]
mm(files,writer)
Files are list type and it MUST stay that way.
I got error
File "/usr/local/Cellar/python@3.8/3.8.3/Frameworks/Python.framework/Versions/3.8/lib/python3.8/posixpath.py", line 54, in normcase
return os.fspath(s)
TypeError: expected str, bytes or os.PathLike object, not list
You can use * with your parameter which will catch any passed arguments in a tuple:
def f(*filenames):
# filenames is a tuple with all your arguments and you can iteratore over
print(type(filenames)) # prints <class 'tuple'>
for arg in args:
pass # do something here
If you would also want a positional argument (writer in your case), it must be placed as first parameter:
def f(writer, *filenames):
print(writer)
print(len(filenames)) # will be zero if only a writer is passed
print(type(filenames))
In your case it's sufficient to use:
def mm(filenames): # you can pass anything that is iterable (list, tuples, strings, ...)
for file in filenames:
The technical post webpages of this site follow the CC BY-SA 4.0 protocol. If you need to reprint, please indicate the site URL or the original address.Any question please contact:yoyou2525@163.com.