[英]Python code running slow
Python code running extremely slow. Python代码运行极其缓慢。 Starts fast, then turns into a crawl. 快速启动,然后变为爬网。 ANything I can do to speed things up? 我可以做些什么来加快速度吗? I'm pulling up a text file, reading the contents of the file, filtering the contents of the text file, and writing that to a csv for later use by someone else for json. 我正在拉一个文本文件,读取文件的内容,过滤文本文件的内容,并将其写入csv,以供其他人稍后用于json。
I just started doing this, as I am sure you can tell. 我相信您可以告诉我,我才刚刚开始这样做。 Any help would be greatly appreciated. 任何帮助将不胜感激。
import glob
from pathlib import Path
import datetime
import re
import csv
get_this = []
thislist = []
def timeteller():
now = datetime.datetime.now()
month = str('{:02d}'.format(now.month))
day1 = now.day -1
day = str('{:02d}'.format(day1))
year =str(now.year)
time =year+month+day
return time
def these_files(x, y):
configfiles = Path('O:/Unit Management/Reports/G4S/').glob('{}*/{}*Transaction.txt'.format(x, y))
for files in configfiles:
thislist.append(files)
return thislist
def hasNumbers(inputString):
numberfinal = []
numberfinal = re.findall("\d+", inputString)
if numberfinal == []:
numberfinal = '1'
return numberfinal
def get_odometers(thesepath):
for thispath in thesepath:
with open(thispath,"r") as f:
searchlines = f.readlines()
for i, line in enumerate(searchlines):
if "Odometers" in line:
get_this.append(line)
elif "Lifetime" in line:
get_this.append(line)
return get_this
def make_pretty(checkthis):
the_numbers = {}
the_numbers['Serial'] = banumber
for i, line in enumerate(checkthis):
the_numbers['Serial'] = banumber
if '(BNR) Odometers Accept' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['BNR'] = numberschecked[0]
elif '(BNR 2) Odometers Accept' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['BNR 2'] = numberschecked[0]
elif '(BCR) Odometers Accept' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['BCR'] = numberschecked[0]
elif '(BCR) Odometers Hopper1' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper1'] = numberschecked[0]
elif '(BCR) Odometers Hopper2' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper2'] = numberschecked[0]
elif '(BCR) Odometers Hopper3' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper3'] = numberschecked[0]
elif '(BCR) Odometers Hopper4' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper4'] = numberschecked[0]
elif '(BCR) Odometers Hopper5' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper5'] = numberschecked[0]
elif '(BCR) Odometers Hopper6' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper6'] = numberschecked[0]
elif '(BCR) Odometers Hopper7' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper7'] = numberschecked[0]
elif '(BCR) Odometers Hopper8' in line:
for l in checkthis[i:i+2]:
numbers = l[0:20]
numberschecked = hasNumbers(numbers)
the_numbers['Hopper8'] = numberschecked[0]
return the_numbers
intnow = int(timeteller())
intnow -= 1
now = str(intnow)
thelist = []
thispath = open('banumberlist1.txt')
finallist = []
for files in thispath:
getem = files
banumber = getem[0:8]
print(banumber)
combined = '{}_{}'.format(banumber,now)
thepaths = these_files(banumber, combined)
needtomakepretty = get_odometers(thepaths)
goeslast = make_pretty(needtomakepretty)
finallist.append(goeslast)
f = open ('odom01.txt', 'w')
for ba in finallist:
sba = str(ba)
f.write(sba)
If you want to know which part of your code is slow, you will need to use a profiler. 如果您想知道代码的哪一部分很慢,则需要使用探查器。
You could use cProfile
which is part of the standard library: 您可以使用cProfile
作为标准库的一部分:
python -m cProfile -o profile.txt -s cumtime myscript.py
This will write the profiling results to profile.txt
, sorting the profile by cumulative time. 这会将概要分析结果写入profile.txt
, profile.txt
累积时间对概要文件进行排序。 This will give you an overview of where your program is spending its time. 这将为您提供程序花费时间的概述。
For a more detailed view, you could use the line_profiler module. 要获得更详细的视图,可以使用line_profiler模块。 This can even show you how much time every line of code uses. 这甚至可以显示每行代码使用多少时间。
After installing line_profiler
, add the following to the beginning of your script: 安装line_profiler
,将以下内容添加到脚本的开头:
import line_profiler
import atexit
profile = line_profiler.LineProfiler()
atexit.register(profile.print_stats)
Now add the @profile
decorators to all your functions, like so: 现在,将@profile
装饰器添加到所有函数中,如下所示:
@profile
def timeteller():
now = datetime.datetime.now()
month = str('{:02d}'.format(now.month))
day1 = now.day -1
day = str('{:02d}'.format(day1))
year =str(now.year)
time =year+month+day
return time
You should put the stuff at the end of your script into a function (eg main
) and add @profile
to that as well. 您应该将脚本末尾的内容放入函数(例如main
)中,并在其中添加@profile
。
Now run your script, and you will see a profile. 现在运行您的脚本,您将看到一个配置文件。
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.