How to compute the performance of a python program on different machines

Question

I would like to know what are the different performance characteristics that can be used to find the performance of a python code on 2 different systems. Also is it possible to extend about its performance on a different machine? Is this kind of stuff possible?

Lets assume that one of the two systems is computation on GPU and other is on a CPU I want to extend the python code's performance on a CPU enabled different system.

Can this be also derived analytically?

Answer 1

In my experiences making assumptions based on hands on performance analysis has been sufficient for identifying initial instance sizes/requirements, and then using real time telemetry and instrumentation to closely monitor those solutions.

There are a couple ways, I've used, to commute performance (the terms are gibberish i've made up):

• Informal Characterization of Bottlenecks This has involved informally understanding where the bottlenecks of your application are likely to be, to give a very rough idea of capacity/machine requirements. If you're performing CPU bound calculations with little to no network, then could bypassing starting with a network optimized instance. Also if you're materializing processing to filesystem, and memory overhead is pretty small or bounded then you don't need a high memory instance.

• External Performance Experiments This involves creating performance test harnesses to establish base line experiments allowing you to change computer variables to determine what sort of effect they have on your program performance. I like to setup queue based systems with throughput tests ie @10k requests / second what is the queue saturation, what is the service time. It involves adding logging/telemetry to code to log those numbers. Also setup a backlog to understand how fast a single instance can process a backlog.

For HTTP there are many tools to generate load.

Hopefully there is an automated tool to support your input format but if not you may have to write your own.

• Performance Profiling

I consider this using "low level" tools to scientifically (opposed to the informal analysis) determine where your code is spending its time. Usually involves using python profiler to determine which routines you're spending time in, and then try to optimize them. http://www.brendangregg.com/linuxperf.html

For this step if the performance test harness has acceptable performance then this can be ignored :p

• Real time telemetry After acceptable performance and instance size has been determined, real time telemetry is critical to see how program is perform in real-time-ish to real life workloads.
  I've found Throughput, processing counts, errors, etc to all be critical to maintaining high performance systems: http://www.brendangregg.com/usemethod.html