unit testing embedded python

Question

I have a third party software which is able to run some python scripts using something like:

software.exe -script pythonscript.py

My company is heavily dependent on this software as well as on the scripts we develop for it. Currently we have some QA that checks the output of the scripts, but we really want to start unit testing the scripts to make it easier to find bugs and make the test system more complete.

My problem is how is it possible to run "embedded" unit tests? We use pydev+eclipse and I tried to use it's remote debbuging to make it work with the unit tests, but I cannot really make it work. How can I make the server connection "feed" the unit test?

The other idea would be to parse the stdout of the software, but that would not really be a unit test... And the added complexity it seems to bring makes this approach less interesting.

I would expect that something like this has already been done somewhere else and I tried googling for it, but maybe I am just not using the correct keywords. Could anyone give me a starting point?

Thank you

Answer 1

A bit more info would be helpful. Are you using a testing framework (eg unittest or nose), or if not, how are the tests structured? What is software.exe ?

In python, unit tests are really nothing more than a collection of functions which raise an exception on failure, so they can be called from a script like any other function. In theory, therefore, you can simply create a test runner (if you're not already using one such as nose), and run it as software.exe -script runtests.py . In pydev, you can set up software.exe as a customised python interpreter.

If the problem is that software.exe hides stdout, then simply write the results to a log file instead. You could also create a environment which mocks that provided by software.exe and run the tests using python.exe .

Answer 2

If unit tests are for your code and not for the functionality provided by software.exe then you could run the tests using a standalone python mocking software.exe parts where necessary. As an intermediate step you could try to run unittest -based scripts using `software.exe'

Answer 3

Well, generally speaking, testing software shall be done by a Continuous Integration suite (And Jenkins is your friend).

Now, I think you'll have to test your scripts pythonscript.py by setting a test() function inside the python script that will emulate the possible environments you'll give to the entry point of your script. And you'll be able to use unittest to execute the test functions of all your scripts. You can also embed tests in doctests, but I personally don't like that.

And then, in your software.exe , you'll be able to execute tests by emulating all the environment combinations. But as you don't say much about software.exe I won't be able to help you more... (what language ? is software.exe already unit tested ?)