How to automatically test a QnA Maker Bot (Bot Framework V4, C#)?

Question

I am building bots based on the QnA Maker and the Microsoft Bot Framework V4 on C#. I have several Knowledge Bases and i want to verify that the bot responds with the right Answers to my questions. Is there a way to automatically test this type of bot?

And is there a possibility to implement this automated testing process into Azure Pipelines?

Answer 1

I can't help you with C# specifically, but I am doing this type of testing with my nodejs bot so this may be of some help. I replied to another of you questions here where I've given my specific nodejs code for testing. This answer will focus more on the approach to automate testing and use Azure Pipelines to execute it.

I'm using mocha as my test framework, but if there is something similar in C#, this method should work. This is where my knowledge is incomplete; I don't know what the C# equivalent of npm test is.

Anyway, I'm hopeful that my approach in nodejs is transferrable for you in C#. I want to mention up from that there are really two different approaches you can take to implementing your test.

1. Create a test bot adapter and simulate your entire bot, testing specifically recognizing the utterance, sending it to QnA Maker, processing the response as necessary, and posting the response to user (and validating that response content).
2. Make a REST call directly to QnA maker and validate the result

If you are concerned about the bot processing the activities correctly, invoking the QnA Maker connecter at the right time, parsing the response correctly, AND having the correct answer, go with option 1. If you simply want to make sure your QnA Service is functioning and returning the expected results, you can go with method 2. Personally I use method 1, but I mock the webservice response (using nock from the npm library) so I'm not testing QnA Maker at all, only the bot. Having live tests in your pipeline (if your tests determine if a release is created or not) can be problematic.

If you check the answer on your other question , that should give you the approach needed for option 1. For option 2, I would suggest using an Azure Function and just write your own test to make the REST call to QnA Maker and validate the result.

Either of these tests can be done in Azure Pipelines. For nodejs/option 1, I just run the npm test command which is configured to execute all of my test files. If any tests fail, the pipeline fails and (per my configuration) no release is created. I am unsure what the equivalent is for C#. For option 2, honestly I wouldn't even worry about putting it in the pipeline because it's independent from your bot code. Just set up your testing schedule in Azure Functions (if that's what you use) or whichever execution trigger you want, and have some actionable output. Eg I have a live integration tester in Azure Functions which is a similar concept, and it sends out service down and service restored messages based on results and changes in status.