How to bundle Python packages (case study: Microsoft Visual Studio 2017)

Question

I am building Python from source, the advantage of this being it will be optimized for my hardware (PGO). But I also want to compile 3rd party packages like NumPy or SciPy to take advantage of intel MKL (I have a Xeon CPU) and optimize them instead of just installing them via PIP which would download pre-compiled, "generic", non-optimized code.

Because I would potentially compile "a lot" of packages by myself I am looking for suggestions as to how to automatize this process. The "traditional" or "standard" way of doing this would be something like this:

1) compile a package

2) include that package to python (maybe using PIP?)

3) repeat the previous 2 steps for each package.

I intend to use Microsoft Visual Studio (under Windows 10 x64), so my question is this:

Is there a way to automatize this process using Visual Studio? if so, Can anyone give me a hint as to what to look for?

I am new to Visual Studio and I am not asking for a complete "How To" but I would appreciate someone telling me if this can be done (or feasible, depending on what packages need to be compiled), otherwise I would revert back to manually make that process by compiling and adding them to Python one by one.

Answer 1

If you're looking to optimize, you can't really automate much as each library will need its own customization. otherwise pip is probably the way to go. Realistically however, you'll only probably need to optimize a few libraries such as the scientific stack and possibly numba for better jit compilation.

TLDR; install numpy from source using the blas/lapack (math libs) optimized for your system and use pip for the rest.