在应用程序启动时在Android中加载（反序列化）_quickly_ 2MB数据

Question

I need to load around 2MB of data quickly on startup of my Android application. I really need all this data in memory, so something like SQLite etc. is not an alternative.

The data consists of about 3000 int[][] arrays. The array dimension is around [7][7] on average.

I first implemented some prototype on my desktop, and ported it to android. On the desktop, I simply used Java's (de)serialization. Deserialization of that data takes about 90ms on my desktop computer.

However on Android 2.2.1 the same process takes about 15seconds(!) on my HTC Magic. It's so slow that if I don't to the deserialization in a seperate thred, my app will be killed. All in all, this is unacceptably slow.

What am I doing wrong? Should I

• switch to something like protocol buffers ? Would that really speed up the process of deserialization of several magnitudes - after all, it's not complex objects that I am deserializing, just int[][] arrays?!
• design my own custom binary file format? I've never done that before, and no clue where to start
• do something else?

Answer 1

Why not bypass the built-in deserialization, and use direct binary I/O? When speed is your primary concern, not necessarily ease of programming, you can't beat it.

For output the pseudo-code would look like this:

write number of arrays
for each array
  write n,m array sizes
  for each element of array
    write array element

For input, the pseudo-code would be:

read number of arrays
for each array
  read n,m array sizes
  allocate the array
  for each element of array
    read array element

When you read/write numbers in binary, you bypass all the conversion between binary and characters. The speed should be limited only by the data transfer rate of the file storage media.

Answer 2

after trying out several things, as Mike Dunlavey suggested, direct binary I/O seemed fastest. I almost verbatim used his sketched out version. For completeness however, and if someone else wants to try, I'll post my full code here; even though it's very basic and without any kind of sanity check. This is for reading such a binary stream; writing is absolutely analogous.

import java.io.*;

public static int[][][] readBinaryInt(String filename) throws IOException {
    DataInputStream in = new DataInputStream(
            new BufferedInputStream(new FileInputStream(filename)));
int dimOfData = in.readInt();
int[][][] patternijk = new int[dimofData][][];
for(int i=0;i<dimofData;i++) {
    int dimStrokes = in.readInt(); 
    int[][] patternjk = new int[dimStrokes][];      
    for(int j=0;j<dimStrokes;j++) {
        int dimPoints = in.readInt();
        int[] patternk = new int[dimPoints];
        for(int k=0;k<dimPoints;k++) {
                patternk[k] = in.readInt();
            }
            patternjk[j] = patternk;
    }
    patternijk[i] = patternjk;
    }
    in.close();
return patternijk;  
}

Answer 3

I had the same kind of issues on a project some months ago. I think you should split your file in various parts, and only load relevant parts following a choice from the user for example. Hope it will be helpful!

Answer 4

I dont know your data but if you optimize your loop, it will effect the deserialize time unbelievably.

if you look at example below

 computeRecursively(30); computeRecursivelyWithLoop(30); // 270 milisecond computeIteratively(30); // 1 milisecond computeRecursivelyFasterUsingBigInteger(30); // about twice s fast as before version computeRecursivelyFasterUsingBigIntegerAllocations(50000); // only 1.3 Second !!! 

public class Fibo {
    public static void main(String[] args) {
        // try the methods
    }

    public static long computeRecursively(int n) {

        if (n > 1) {
            System.out.println(computeRecursively(n - 2)
                    + computeRecursively(n - 1));
            return computeRecursively(n - 2) + computeRecursively(n - 1);
        }
        return n;
    }

    public static long computeRecursivelyWithLoop(int n) {
        if (n > 1) {
            long result = 1;
            do {
                result += computeRecursivelyWithLoop(n - 2);
                n--;
            } while (n > 1);
            System.out.println(result);
            return result;
        }
        return n;
    }

    public static long computeIteratively(int n) {
        if (n > 1) {
            long a = 0, b = 1;
            do {
                long tmp = b;
                b += a;
                a = tmp;
                System.out.println(a);
            } while (--n > 1);
            System.out.println(b);
            return b;
        }
        return n;
    }

    public static BigInteger computeRecursivelyFasterUsingBigInteger(int n) {
        if (n > 1) {
            int m = (n / 2) + (n & 1); // not obvious at first – wouldn’t it be
                                        // great to have a better comment here?
            BigInteger fM = computeRecursivelyFasterUsingBigInteger(m);
            BigInteger fM_1 = computeRecursivelyFasterUsingBigInteger(m - 1);
            if ((n & 1) == 1) {
                // F(m)^2 + F(m-1)^2
                System.out.println(fM.pow(2).add(fM_1.pow(2)));
                return fM.pow(2).add(fM_1.pow(2)); // three BigInteger objects
                                                    // created
            } else {
                // (2*F(m-1) + F(m)) * F(m)
                System.out.println( fM_1.shiftLeft(1).add(fM).multiply(fM));
                return fM_1.shiftLeft(1).add(fM).multiply(fM); // three
                                                                // BigInteger
                                                                // objects
                                                                // created
            }
        }
        return (n == 0) ? BigInteger.ZERO : BigInteger.ONE; // no BigInteger
                                                            // object created
    }

    public static long computeRecursivelyFasterUsingBigIntegerAllocations(int n) {
        long allocations = 0;
        if (n > 1) {
            int m = (n / 2) + (n & 1);
            allocations += computeRecursivelyFasterUsingBigIntegerAllocations(m);
            allocations += computeRecursivelyFasterUsingBigIntegerAllocations(m - 1);
            // 3 more BigInteger objects allocated
            allocations += 3;
            System.out.println(allocations);
        }
        return allocations; // approximate number of BigInteger objects
                            // allocated when
                            // computeRecursivelyFasterUsingBigInteger(n) is
                            // called
    }
}