[英]Collect dict frames in a python object with appropriate and fast data structure
I'm using an API which provides 128 signals (and signal quality values) per second as a dict in a dict. 我正在使用一个API,该API每秒提供128个信号(和信号质量值)作为dict中的一个dict。 I want to collect them into windows of 1 (or more) seconds.
我想将它们收集到1(或更多)秒的窗口中。
My native approach was to use the same dict structure and append the values to lists (Like this: Appending values to dictionary in Python ). 我的本机方法是使用相同的dict结构并将值追加到列表中(例如: 在Python中将值追加到字典中 )。
See the example: 参见示例:
# the API provides data like this
def getFrame(i):
return {"X":
{"value" : i,
"quality": i*2},
"AF3":
{"value" : i*3,
"quality": i*4}
}
# object to collect single frames
class WindowDto(object):
def __init__(self, windowSize, header):
self.header = header
self.windowSize = windowSize
self.data = {}
for key in self.header:
self.data[key] = {"value": [], "quality": []}
def addFrame(self, frame):
for key, val in frame.iteritems():
field = self.data[key]
field["value"].append(val["value"])
field["quality"].append(val["quality"])
# the keys of a frame are known
header = ["X", "AF3"]
# the max frame count is also known, but not used yet
maxFrameCount = 8
dto = WindowDto(maxFrameCount, header)
for i in range(8):
dto.addFrame(getFrame(i))
print(dto.data)
# {'X': {'quality': [0, 2, 4, 6, 8, 10, 12, 14], 'value': [0, 1, 2, 3, 4, 5, 6, 7]},
# 'AF3': {'quality': [0, 4, 8, 12, 16, 20, 24, 28], 'value': [0, 3, 6, 9, 12, 15, 18, 21]}}
I assume the addFrame()
method is a potential bottleneck, as I have to iterate over the whole structure. 我假设
addFrame()
方法是一个潜在的瓶颈,因为我必须遍历整个结构。 I am looking for a smarter way to perform the add
operation. 我正在寻找一种执行
add
操作的更智能方法。 Any suggestions how to do this? 有什么建议怎么做?
I am also free to change the way the values are stored (eg in an array?) to improve performance. 我还可以自由更改值的存储方式(例如在数组中?)以提高性能。 I know the header values and the max size of a window object.
我知道标题值和窗口对象的最大大小。 The values are later processed as lists (eg
sum(data["X"]["values"])
), but the access is wrapped by the WindowDto
. 这些值稍后将作为列表进行处理(例如
sum(data["X"]["values"])
),但是访问由WindowDto
包裹。
Did you do any profiling,maybe what you have is already good enough for your server?! 您是否进行了任何性能分析,也许您的服务器已经足够好了!
import cProfile
cProfile.run('for i in range(128):dto.addFrame(getFrame(i))')
899 function calls in 0.001 seconds
cProfile.run('for i in range(128**2):dto.addFrame(getFrame(i))')
114691 function calls in 0.046 seconds
声明:本站的技术帖子网页,遵循CC BY-SA 4.0协议,如果您需要转载,请注明本站网址或者原文地址。任何问题请咨询:yoyou2525@163.com.