单元测试在“全部运行”时失败，但在单次运行时通过

Question

I have this test class and when I run these tests one at a time they pass but when I try to run all tests from this class, first test is always passed and others usually fail (sometimes randomly one of them pass) and when I run all tests from my project all of these tests fail.

I use NUnit and Moq frameworks.

Here is the code:

using System.Security;
using DebtDiary.Core;
using DebtDiary.DataProvider;
using Moq;
using NUnit.Framework;

namespace DebtDiary.Tests.ViewModels
{
    [TestFixture]
    public class LoginPageViewModelTests
    {
        [Test]
        public void TestLoginCommandCallsLoginUserInClientDataStoreWhenDataIsValid()
        {
            Mock<IApplicationViewModel> stubApplicationVM = new Mock<IApplicationViewModel>();
            Mock<IDiaryPageViewModel> stubDiaryPageVM = new Mock<IDiaryPageViewModel>();
            Mock<IDialogFacade> stubDialogFacadeVM = new Mock<IDialogFacade>();
            Mock<IClientDataStore> mockClientDataStore = new Mock<IClientDataStore>();
            Mock<IDataAccess> stubDataAccess = new Mock<IDataAccess>();
            var loginPageVM = new LoginPageViewModel(stubApplicationVM.Object, stubDiaryPageVM.Object, stubDialogFacadeVM.Object, mockClientDataStore.Object, stubDataAccess.Object);
            loginPageVM.Username = "test";
            Mock<IHavePassword> stubPassword = new Mock<IHavePassword>();
            SecureString ss = new SecureString();
            ss.AppendChar('t');
            stubPassword.Setup(x => x.Password).Returns(ss);
            User user = new User();
            stubDataAccess.Setup(x => x.UserExist(It.IsAny<string>(), It.IsAny<string>())).Returns(true);
            stubDataAccess.Setup(x => x.TryGetUser(It.IsAny<string>(), It.IsAny<string>(), out user)).Returns(true);

            loginPageVM.LoginCommand.Execute(stubPassword.Object);

            mockClientDataStore.Verify(x => x.LoginUser(It.IsAny<User>()), Times.Once());
        }

        [Test]
        public void TestLoginCommandUpdatesDebtorsListInDiaryPageViewModelWhenDataIsValid()
        {
            Mock<IApplicationViewModel> stubApplicationVM = new Mock<IApplicationViewModel>();
            Mock<IDiaryPageViewModel> mockDiaryPageVM = new Mock<IDiaryPageViewModel>();
            Mock<IDialogFacade> stubDialogFacadeVM = new Mock<IDialogFacade>();
            Mock<IClientDataStore> stubClientDataStore = new Mock<IClientDataStore>();
            Mock<IDataAccess> stubDataAccess = new Mock<IDataAccess>();
            var loginPageVM = new LoginPageViewModel(stubApplicationVM.Object, mockDiaryPageVM.Object, stubDialogFacadeVM.Object, stubClientDataStore.Object, stubDataAccess.Object);
            loginPageVM.Username = "test";
            Mock<IHavePassword> stubPassword = new Mock<IHavePassword>();
            SecureString ss = new SecureString();
            ss.AppendChar('t');
            stubPassword.Setup(x => x.Password).Returns(ss);
            User user = new User();
            stubDataAccess.Setup(x => x.UserExist(It.IsAny<string>(), It.IsAny<string>())).Returns(true);
            stubDataAccess.Setup(x => x.TryGetUser(It.IsAny<string>(), It.IsAny<string>(), out user)).Returns(true);

            loginPageVM.LoginCommand.Execute(stubPassword.Object);

            mockDiaryPageVM.Verify(x => x.UpdateDebtorsList(), Times.Once());
        }

        [Test]
        public void TestLoginCommandUpdatesUsersDataInDiaryPageViewModelWhenDataIsValid()
        {
            Mock<IApplicationViewModel> stubApplicationVM = new Mock<IApplicationViewModel>();
            Mock<IDiaryPageViewModel> mockDiaryPageVM = new Mock<IDiaryPageViewModel>();
            Mock<IDialogFacade> stubDialogFacadeVM = new Mock<IDialogFacade>();
            Mock<IClientDataStore> stubClientDataStore = new Mock<IClientDataStore>();
            Mock<IDataAccess> stubDataAccess = new Mock<IDataAccess>();
            var loginPageVM = new LoginPageViewModel(stubApplicationVM.Object, mockDiaryPageVM.Object, stubDialogFacadeVM.Object, stubClientDataStore.Object, stubDataAccess.Object);
            loginPageVM.Username = "test";
            Mock<IHavePassword> stubPassword = new Mock<IHavePassword>();
            SecureString ss = new SecureString();
            ss.AppendChar('t');
            stubPassword.Setup(x => x.Password).Returns(ss);
            User user = new User();
            stubDataAccess.Setup(x => x.UserExist(It.IsAny<string>(), It.IsAny<string>())).Returns(true);
            stubDataAccess.Setup(x => x.TryGetUser(It.IsAny<string>(), It.IsAny<string>(), out user)).Returns(true);

            loginPageVM.LoginCommand.Execute(stubPassword.Object);

            mockDiaryPageVM.Verify(x => x.UpdateUsersData(), Times.Once());
        }

        [Test]
        public void TestLoginCommandResetsCurrentSubpageInApplicationViewModelWhenDataIsValid()
        {
            Mock<IApplicationViewModel> mockApplicationVM = new Mock<IApplicationViewModel>();
            Mock<IDiaryPageViewModel> stubDiaryPageVM = new Mock<IDiaryPageViewModel>();
            Mock<IDialogFacade> stubDialogFacadeVM = new Mock<IDialogFacade>();
            Mock<IClientDataStore> stubClientDataStore = new Mock<IClientDataStore>();
            Mock<IDataAccess> stubDataAccess = new Mock<IDataAccess>();
            var loginPageVM = new LoginPageViewModel(mockApplicationVM.Object, stubDiaryPageVM.Object, stubDialogFacadeVM.Object, stubClientDataStore.Object, stubDataAccess.Object);
            loginPageVM.Username = "test";
            Mock<IHavePassword> stubPassword = new Mock<IHavePassword>();
            SecureString ss = new SecureString();
            ss.AppendChar('t');
            stubPassword.Setup(x => x.Password).Returns(ss);
            User user = new User();
            stubDataAccess.Setup(x => x.UserExist(It.IsAny<string>(), It.IsAny<string>())).Returns(true);
            stubDataAccess.Setup(x => x.TryGetUser(It.IsAny<string>(), It.IsAny<string>(), out user)).Returns(true);

            loginPageVM.LoginCommand.Execute(stubPassword.Object);

            mockApplicationVM.Verify(x => x.ResetCurrentSubpage(), Times.Once());
        }
    }
}

Do you know what can be the reason? As you can see I moved all repeatable code to these methods to avoid dependencies and it doesnt work.

Answer 1

Since, as you mentioned, each test is creating it's own data and there aren't any shared objects, there a few likely reasons causing these failures:

Option A

Your system under test (ie LoginPageViewModel) is reusing its dependencies in a thread-safe way.

If that is the case , then LoginPageViewModel is being instantiated with the mock interfaces created on the first test being run on a given thread. It would then re-use those dependencies for any other test run on that thread. This would result in the first test on that thread to pass (so when you run each test individually, they all pass). However, any subsequent tests run on that thread will fail because even though your passing in new mock interfaces for each test, the mock interfaces from the first test on that thread are being re-used.

Solutions to this problem would be to:

1. Remove any locking or Singelton implementations in your code and instead use an IoC container to define the lifestyle of your objects in your application. Then you can be in control of the lifestyle of the objects being used in your tests.
2. Use the RequiresThread attribute in NUnit to ensure that each test is run on it's own thread, removing the instance re-use.

Option B (this seems to be the case after looking further at your code)

Your system under test is calling some async code without awaiting the result. This will cause the async code to continue to run while the control is returned to the caller (in this case, the unit test). This would then potentially cause race conditions if the assertions in the test are executed before the async code finishes. And these race conditions would be more likely to occur when the code is executing slower due to some load, such as running multiple tests at once.

This seems to be the case in the code that you're testing. The constructor for LoginPageViewModel constructs a RelayParameterizedCommand , passing in an async delegate. However, your unit tests then call RelayParameterizedCommand to execute that delegate that was passed in, without awaiting the result.

The solutions for that would be:

1. Update the delegate passed into RelayParameterizedCommand from Action<object> to Func<object, Task> . Then you can, either
   1. Make the Execute method on RelayParameterizedCommand async. Then make your unit tests aysnc and call the method in your system under test with an await. Or,
   2. Keep the Execute method on RelayParameterizedCommand synchronous, but still get the result from the Task of the delegate: _action(parameter).GetAwaiter().GetResult();