Java creating new objects performance

Question

I have the below class.

class MyObject implements Serializable {
    private String key;
    private String val;
    private int num;

    MyObject(String a, String b, int c) {
        this.key = a;
        this.val = b;
        this.num = c;
    }
}

I need to create a list of Objects, the following method is called repeatedly (say 10K times or more)

public void addToIndex(String a, String b, int c) {
    MyObject ob = new MyObject(a,b,c);
    list.add(ob); // List<MyObject>
}

I used a profiler to see the memory footprint, and it increases so much due to creation of object everytime. Is there a better way of doing this? I am writing the list then to disk.

EDIT: This is how I write once the list is fully populated. Is there a way to append once the memory goes beyond a value (size of list).

ObjectOutputStream oos = new ObjectOutputStream(
                        new DeflaterOutputStream(new FileOutputStream(
                                list)));
                oos.writeObject(list);
                oos.close();

Answer 1

I used a profiler to see the memory footprint, and it increases so much due to creation of object everytime. Is there a better way of doing this?

Java Serialization doesn't use that much memory in your situation. What it does so is create a lot of garbage, far more than you might imagine. It also has a very verbose output which can be improved using compression as you do.

A simple way to improve this situation is to use Externalizable instead of Serializable. This can reduce the garbage produced dramatically and make it more compact. It can also be much faster with lower over head.

BTW You can get even better performance if you use custom serialization for the list itself.

public class Main {
    public static void main(String[] args) throws IOException, ClassNotFoundException {
        List<MyObject> list = new ArrayList<>();
        for (int i = 0; i < 10000; i++) {
            list.add(new MyObject("key-" + i, "value-" + i, i));
        }

        for (int i = 0; i < 10; i++) {
            timeJavaSerialization(list);
            timeCustomSerialization(list);
            timeCustomSerialization2(list);
        }
    }

    private static void timeJavaSerialization(List<MyObject> list) throws IOException, ClassNotFoundException {
        File file = File.createTempFile("java-serialization", "dz");
        long start = System.nanoTime();
        ObjectOutputStream oos = new ObjectOutputStream(
                new DeflaterOutputStream(new FileOutputStream(file)));
        oos.writeObject(list);
        oos.close();
        ObjectInputStream ois = new ObjectInputStream(
                new InflaterInputStream(new FileInputStream(file)));
        Object o = ois.readObject();
        ois.close();
        long time = System.nanoTime() - start;
        long size = file.length();
        System.out.printf("Java serialization uses %,d bytes and too %.3f seconds.%n",
                size, time / 1e9);
    }

    private static void timeCustomSerialization(List<MyObject> list) throws IOException {
        File file = File.createTempFile("custom-serialization", "dz");
        long start = System.nanoTime();
        MyObject.writeList(file, list);
        Object o = MyObject.readList(file);
        long time = System.nanoTime() - start;
        long size = file.length();
        System.out.printf("Faster Custom serialization uses %,d bytes and too %.3f seconds.%n",
                size, time / 1e9);
    }

    private static void timeCustomSerialization2(List<MyObject> list) throws IOException {
        File file = File.createTempFile("custom2-serialization", "dz");
        long start = System.nanoTime();
        {
            DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(
                    new DeflaterOutputStream(new FileOutputStream(file))));
            dos.writeInt(list.size());
            for (MyObject mo : list) {
                dos.writeUTF(mo.key);
            }
            for (MyObject mo : list) {
                dos.writeUTF(mo.val);
            }
            for (MyObject mo : list) {
                dos.writeInt(mo.num);
            }
            dos.close();
        }
        {
            DataInputStream dis = new DataInputStream(new BufferedInputStream(
                    new InflaterInputStream(new FileInputStream(file))));
            int len = dis.readInt();
            String[] keys = new String[len];
            String[] vals = new String[len];
            List<MyObject> list2 = new ArrayList<>(len);
            for (int i = 0; i < len; i++) {
                keys[i] = dis.readUTF();
            }
            for (int i = 0; i < len; i++) {
                vals[i] = dis.readUTF();
            }
            for (int i = 0; i < len; i++) {
                list2.add(new MyObject(keys[i], vals[i], dis.readInt()));
            }
            dis.close();
        }
        long time = System.nanoTime() - start;
        long size = file.length();
        System.out.printf("Compact Custom serialization uses %,d bytes and too %.3f seconds.%n",
                size, time / 1e9);
    }


    static class MyObject implements Serializable {
        private String key;
        private String val;
        private int num;

        MyObject(String a, String b, int c) {
            this.key = a;
            this.val = b;
            this.num = c;
        }

        MyObject(DataInput in) throws IOException {
            key = in.readUTF();
            val = in.readUTF();
            num = in.readInt();
        }

        public void writeTo(DataOutput out) throws IOException {
            out.writeUTF(key);
            out.writeUTF(val);
            out.writeInt(num);
        }

        public static void writeList(File file, List<MyObject> list) throws IOException {
            DataOutputStream dos = new DataOutputStream(new BufferedOutputStream(
                    new DeflaterOutputStream(new FileOutputStream(file))));
            dos.writeInt(list.size());
            for (MyObject mo : list) {
                mo.writeTo(dos);
            }
            dos.close();
        }

        public static List<MyObject> readList(File file) throws IOException {
            DataInputStream dis = new DataInputStream(new BufferedInputStream(
                    new InflaterInputStream(new FileInputStream(file))));
            int len = dis.readInt();
            List<MyObject> list = new ArrayList<>(len);
            for (int i = 0; i < len; i++) {
                list.add(new MyObject(dis));
            }
            dis.close();
            return list;
        }
    }
}

prints finally

Java serialization uses 61,168 bytes and too 0.061 seconds.
Faster Custom serialization uses 62,519 bytes and too 0.024 seconds.
Compact Custom serialization uses 68,225 bytes and too 0.020 seconds.

As you can see my attempts to make the file more compact instead made it faster, which is a good example of why you should test performance improvements.

Answer 2

Consider using fast-serialization. It is source-level compatible to JDK-serialization, and creates less bloat. Additionally it beats most of handcrafted "Externalizable" serialization, as its not only the JDK-serialization implementation itself, but also inefficient In/Output stream implementations of stock JDK which hurt performance.

http://code.google.com/p/fast-serialization/