如何将大尺寸的.txt文件数据拆分为小部分并插入数据库中？

Question

Below is my code to read and split the text file content.

  try {

        br = new BufferedReader(new FileReader("F:\\Test.txt"));
        final char[] cbuf = new char[2048];
        final int length = br.read(cbuf);

        cbuf[length] = '@';
        String packet = new String(cbuf, 0, length + 1);
        final String[] splitedPacket=packet.split("@");
        for(int i=0;i<splitedPacket.length;i++)
        {
            if(splitedPacket[i].contains("POS"))
            {
                System.out.println(splitedPacket[i]+"@");
                preparedstatement=connection.prepareStatement("insert into inserttextfile([file]) values(?)");
                preparedstatement.setString(1, splitedPacket[i]+"@");
                preparedstatement.executeUpdate();
            }
        }
    } catch (IOException e) {
        e.printStackTrace();
    } finally {
        try {
            if (br != null)br.close();
        } catch (IOException ex) {
            ex.printStackTrace();
        }
    }

It was working fine in my small size .txt file, but if I use more than 30 MB text file I get array index out of bound exception.

So how to split and read this kind of .txt file and insert into database? (particularly expecting for split the large size file(ex. like 5 MB from 30 MB) and insert into database) Please guide me in this issue.

Answer 1

Your doing it the hard way by not using readLine

    try {
        FileInputStream fis = new FileInputStream("F:\\Test.txt");
        reader = new BufferedReader(new InputStreamReader(fis));          
        String line = reader.readLine();
        while(line != null){

            //process your line here, it's just a String...   

            line = reader.readLine();
        }           

    } catch (FileNotFoundException ex) {
        ...
    } catch (IOException ex) {

Answer 2

 try (Scanner read = new Scanner(new File("/tmp/datafile.txt"));) {
        read.useDelimiter("@");
        while (read.hasNext()) {
            String splitedPacket = read.next();
            System.out.println(splitedPacket);
            // Perform DB Operation
        }
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    }

1. Dont read the entire file.
2. Use Scanner to read and hold only one read @ any give time - to avoid OOM
3. If Possible Perform Db Operation as bulk . If you are running in one transaction, read 1000 records and process as separate bulk insert.