JUnit test method comparing DateTimes fails only when run from suites

Question

A single JUnit test being run under JUnit 4.11 fail the majority of the time while being run via either to module test suite ( 40 runs: 2 failures, 38 passes ), or the class test suite ( 40 runs: 6 failures, 34 passes ), but running the test method by itself did not produce a single failure ( 50 runs: 0 failures, 50 passes ).

To summarize what is happening, the equals(Object MyObject) implementation returns true if the org.joda.time.DateTime corresponding to the key Stamp.START or the key Stamp.STOP is the same for the current instance as the one in instance passed to the method. Here's the code:

import org.joda.time.DateTime;
...
private final Map<Stamp, DateTime> timeStampMap;
...
@Override
public boolean equals(Object obj) {
    if (this == obj) { return true; }
    if (obj == null || getClass() != obj.getClass()) { return false; }
    final MyObject other = (MyObject) obj;
    return (Objects.equals(this.timeStampMap.get((Stamp.START)),
                           other.timeStampMap.get(Stamp.START))
            && Objects.equals(this.timeStampMap.get(Stamp.STOP),
                              this.timeStampMap.get(Stamp.STOP)));
}
...
public enum Stamp {
    START,
    STOP
}

And the test itself:

@Test
@Config(configuration = TestConfig.NO_CONFIG)
public void equalityTest() {
    MyObject a = new MyObject(BigDecimal.TEN);
    MyObject b = a;

    assertThat(a.hashCode(), is(b.hashCode()));
    assertTrue(a.equals(b));

    b = new MyObject(BigDecimal.TEN);

    // This line produces the failure
    assertThat(a, is(not(b)));
}

Why would this test only fail when run under either test suite, but not when run on it's own?

Answer 1

Since you are using Joda time, an alternative approach might be to fix the current time to something of your choosing using DateTimeUtils.setCurrentMillisFixed(val) .

For example:

@Test
@Config(configuration = TestConfig.NO_CONFIG)
public void equalityTest() {
    DateTimeUtils.setCurrentMillisFixed(someValue);

    MyObject a = new MyObject(BigDecimal.TEN);
    MyObject b = a;

    assertThat(a.hashCode(), is(b.hashCode()));
    assertTrue(a.equals(b));

    DateTimeUtils.setCurrentMillisFixed(someValue + someOffset);

    b = new MyObject(BigDecimal.TEN);

    // This line produces the failure
    assertThat(a, is(not(b)));
}

Answer 2

I suggest making the code more testable. Instead of having the code get the date directly, you can pass in an interface named Clock :

public interface Clock {
  DateTime now();
}

Then you could add Clock to the constructor:

MyObject(BigDecimal bigDecimal, Clock clock) {
  timeStampMap.put(Stamp.START, clock.now());
}

For production code, you can make a helper constructor:

MyObject(BigDecimal bigDecimal) {
  this(bigDecimal, new SystemClock());
}

...where SystemClock looked like this:

public class SystemClock implements Clock {

  @Override
  public DateTime now() {
    return new DateTime();
  }
}

Your tests could either mock Clock or you could create a fake clock implementation.

Answer 3

Over the process of trying to produce an MCVE and author the question, I discovered something interesting:

When the test is run at the method level, note the timestamp difference of 1 millisecond. The difference is never less than that:

[START: 2015-02-26T11:53:20.581-06:00, STOP: 2015-02-26T11:53:20.641-06:00, DURATION: 0.060]    
[START: 2015-02-26T11:53:20.582-06:00, STOP: 2015-02-26T11:53:20.642-06:00, DURATION: 0.060]

But when I run the test ends up being run as part of the suites, this happens nearly every single time:

[START: 2015-02-26T12:25:31.183-06:00, STOP: 2015-02-26T12:25:31.243-06:00, DURATION: 0.060]
[START: 2015-02-26T12:25:31.183-06:00, STOP: 2015-02-26T12:25:31.243-06:00, DURATION: 0.060]

Zero difference. Weird right?

My best guess is that the JVM is proverbially all warmed up and has some momentum built by the time it reaches this particular test when running the test suites. So much so, that the instantiations occur so quickly as to be nearly simultaneous. The tiny amount of time that passes between the time that MyObject a is instantiated and b is assigned until b is reassigned as a new MyObject is so minute as to produce a MyObject with an identical pair of DateTime s.

Turns out, there are a few usable solutions:

The Solution I Went With:

This is really similar to Duncan's . Call DateTimeUtils.setCurrentMillisOffset(val) before reassigning MyObject b and then reset immediately afterward, since I only need the offset long enough to force a difference in the DateTimes between MyObjects a and b :

@Test
@Config(configuration = TestConfig.NO_CONFIG)
public void equalityTest() {
    MyObject a = new MyObject(BigDecimal.TEN);
    MyObject b = a;

    assertThat(a.hashCode(), is(b.hashCode()));
    assertTrue(a.equals(b));

    // Force an offset
    DateTimeUtils.setCurrentMillisOffset(1000);
    b = new MyObject(BigDecimal.TEN);
    // Clears the offset
    DateTimeUtils.setCurrentMillisSystem();

    assertThat(a, is(not(b)));
}

Namshubwriter's Solution (link to answer):

Easily the best solution in cases where this issue will likely be seen throughout a project and/or in actual use.

Duncan's Solution (link to answer):

Set the current time to return a fixed time by calling DateTimeUtils.setCurrentMillisFixed(val) at the beginning of the unit test, then adding an offset to that time by calling DateTimeUtils.setCurrentMillisFixed(val + someOffset) to force the difference before reassigning MyObject b . Click the link to jump right to his solution with the code.

It is worth pointing out that you'll need to call DateTimeUtils.setCurrentMillisSystem() at some point to reset the time, otherwise other tests dependent on the time could be affected.

Original Solution:

I think it is worth mentioning here that, it is my understanding this is the only solution that does not depend on the program having certain security privileges on the parent system.

Place a call to Thread.sleep() ensure that there is a time separation between the DateTime timestamps of the two MyObjects :

@Test
@Config(configuration = TestConfig.NO_CONFIG)
public void equalityTest() {
    MyObject a = new MyObject(BigDecimal.TEN);
    MyObject b = a;

    assertThat(a.hashCode(), is(b.hashCode()));
    assertTrue(a.equals(b));

    try {
        Thread.sleep(0, 1);
    } catch (Exception e) {
        e.printStackTrace();
    }
    b = new MyObject(BigDecimal.TEN);

    // Line that was failing
    assertThat(a, is(not(b)));
}