Tensorflow Java API-第1次预测时间与第2次或更长时间

Question

I'm experiencing that if I load a model with the java api and does prediction the first one takes significantly longer than any other later. There's some lazy evaluation like thing, I mean the model weights not loaded till the first prediction? What's the reason behind this?

        try (Session s = load.session()) {
            Tensor result = null;
            startTime = System.nanoTime();
            result = s.runner().feed("input", data).fetch("prediction").run().get(0);
            endTime = System.nanoTime();
            System.out.println(String.format("First prediction performance: %.4f ms", ((double)endTime - startTime)/1000000));

            startTime = System.nanoTime();
            result = s.runner().feed("input", data).fetch("prediction").run().get(0);
            endTime = System.nanoTime();
            System.out.println(String.format("2nd prediction performance: %.4f ms", ((double)endTime - startTime)/1000000));

            System.out.println(result.toString());

        }

1. First prediction performance: 10.6066 ms
2. 2nd prediction performance: 0.4776 ms

Answer 1

Short Answer

Yes, this is normal behavior and not a cause of alarm.

Longer Answer

Tensorflow uses a graph (as I'm sure you've come to realize) which defines the order and flow of operations. It does not define how to optimally execute them while you define them. That is all sorted out upon the first run. So yes, some lazy loading or more precisely lazy computation.

Most likely you'll be using TensorFlow to process thousands or millions of entries, so the fact that the first 1 takes 10ms longer than normal should not be a problem. If you are using TensorFlow as a service then you'll probably want to keep a session open for a long time, similar to how you'd keep an SQL connection open for multiple queries and not reconnect over tcp/ip or each request.