Why do I get "BadPaddingException: Message is larger than modulus" when decrypting data stored as a bit array from text file?

Question

I'm using the RSA algorithm to encrypt some numbers, then store those numbers in a bit array within a text file. The numbers are encrypted and added to the text file just fine, but when I try and decrypt them using the method below, I get "BadPaddingException: Message is larger than modulus" this error in my console. I'm pretty new to encryption and this is really throwing me off.

 Cipher cipher;
    KeyPair pair;

    public byte[] encryptData(String data) throws BadPaddingException, IllegalBlockSizeException {
        try{
            KeyPairGenerator keyPairGen = KeyPairGenerator.getInstance("RSA");
            pair = keyPairGen.generateKeyPair();
            PublicKey publicKey = pair.getPublic();
            cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
            cipher.init(Cipher.ENCRYPT_MODE, publicKey);
            cipher.update(data.getBytes());
            cipher.doFinal();
        }
        catch (NoSuchAlgorithmException | NoSuchPaddingException | InvalidKeyException | BadPaddingException | IllegalBlockSizeException ex){
            ex.printStackTrace();
        }
        return cipher.doFinal();
    }

writing data into a file

 String nums = request.getParameter("numbers");
        String password = (String) session.getAttribute("password");
        String filename =  (password +".txt");



        File dir = new File("/volume");
        dir.mkdir();
        File myfile = new File(dir, filename);
        System.out.println("filename: " + filename);
        FileOutputStream output;

        try {
            if (myfile.isFile()){
                 output = new FileOutputStream(myfile, true);
            }
            else{
                 output = new FileOutputStream(myfile);
            }
            byte[] encryptednums = encryptData(nums);
            output.write(encryptednums);
            output.close();
        } catch (BadPaddingException | IllegalBlockSizeException  | IOException e) {
            e.printStackTrace();
        }

Reading the file

    public byte[] bytesFileReader(File filename){
        try {
            return Files.readAllBytes(Paths.get(String.valueOf(filename)));
        } catch (IOException e) {
            e.printStackTrace();
        }
        return null;
    }

Decryption

  HttpSession session = request.getSession();
        KeyPair pair = (KeyPair) session.getAttribute("keypair");
        Cipher cipher = (Cipher) session.getAttribute("cipher");
        String password = (String) session.getAttribute("password");

        String filename = (password +".txt");
        File dir = new File("/volume");
        dir.mkdir();
        File myfile = new File(dir, filename);

        byte[] encryptedBytes = bytesFileReader(myfile);

        try {
            cipher.init(Cipher.DECRYPT_MODE, pair.getPrivate());
            byte[] decipheredText = cipher.doFinal(encryptedBytes);
            System.out.println(new String(decipheredText, StandardCharsets.UTF_8));

        } catch (InvalidKeyException | BadPaddingException | IllegalBlockSizeException e) {
            e.printStackTrace();
        }

    }

Probably worth noting that this is all taking place across two java servlets.

Answer 1

There is one main issue and some minor issues in your code. Let's start will a minor one: when converting a string to a byte array and vice versa all conversions have to use a fixed encoding (see your encryptData method):

cipher.update(data.getBytes());

The main issue is the plaintext size limitation when doing RSA encryption. You can calculate the maximum plaintext size with this formula

1) RSA keysize divided by 8 - e.g. 2048 / 8 = 256 bytes
2) PKCS1 padding takes 11 bytes
3) maximum of plaintext bytes: 256 - 11 = 245 bytes

In my example code I'm using a RSA keypair with size of ( UNSECURE ) 512 bits = 64 bytes, minus 11 bytes for padding there are 53 bytes that can get encrypted. The first round will run like expected, the next encryption with 54 bytes of data runs into the

IllegalBlockSizeException: Data must not be longer than 53 bytes

As your code has only the methods but does not show what data is written I can only assume that you try to decrypt all written data in one run - there is a high chance for failure.

The reason why my code runs into the error (and yours not) is the update- and double (!) final-call (beware I left out the catching that you maybe need):

change:
cipher.update(data.getBytes());
cipher.doFinal();
return cipher.doFinal();
to:
return cipher.doFinal(data.getBytes(StandardCharsets.UTF_8));

output:

BadPaddingException: Message is larger than modulus
encrypting 12345678901234567890123456789012345678901234567890123
encryptedData length: 64
decryptedData: 12345678901234567890123456789012345678901234567890123
encrypting 123456789012345678901234567890123456789012345678901234
Exception in thread "main" javax.crypto.IllegalBlockSizeException: Data must not be longer than 53 bytes

Security warning: this code uses an UNSECURE 512 bit long RSA key (use minimum 2048 bit) and encrypts with RSA padding PKCS1 that is vulnerable - please use OEAP padding! .

code:

import javax.crypto.BadPaddingException;
import javax.crypto.Cipher;
import javax.crypto.IllegalBlockSizeException;
import javax.crypto.NoSuchPaddingException;
import java.nio.charset.StandardCharsets;
import java.security.*;

public class Main {

    static KeyPair keyPair;

    public static void main(String[] args) throws NoSuchAlgorithmException, BadPaddingException, IllegalBlockSizeException, InvalidKeyException, NoSuchPaddingException {
        System.out.println("BadPaddingException: Message is larger than modulus");
        // https://stackoverflow.com/questions/64722370/why-do-i-get-badpaddingexception-message-is-larger-than-modulus-when-decrypti
        keyPair = generateKeyPair(512); // ### do not use RSA keys with 512 bits length, minimum 2048 bits
        // 512 bit = 64 byte - 11 byte for padding = maximal 53 bytes data to encrypt
        System.out.println("\nencrypting 53 chars");
        String data53Chars = "12345678901234567890123456789012345678901234567890123";
        //String data53Chars = "12345678901234567890";
        System.out.println("encrypting " + data53Chars);
        byte[] encryptedData = encryptData(data53Chars);
        System.out.println("encryptedData length: " + encryptedData.length);
        String decryptedData = decryptData(encryptedData);
        System.out.println("decryptedData: " + decryptedData);
        // now 54 bytes
        System.out.println("\nencrypting 54 chars");
        String data54Chars = "123456789012345678901234567890123456789012345678901234";
        System.out.println("encrypting " + data54Chars);
        encryptedData = encryptData(data54Chars);
        System.out.println("encryptedData length: " + encryptedData.length);
        decryptedData = decryptData(encryptedData);
        System.out.println("decryptedData: " + decryptedData);
    }

    public static byte[] encryptData(String data) throws BadPaddingException, IllegalBlockSizeException, NoSuchPaddingException, NoSuchAlgorithmException, InvalidKeyException {
        PublicKey publicKey = keyPair.getPublic();
        Cipher cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
        cipher.init(Cipher.ENCRYPT_MODE, publicKey);
        return cipher.doFinal(data.getBytes(StandardCharsets.UTF_8));
    }

    public static String decryptData(byte[] encryptedBytes) throws InvalidKeyException, BadPaddingException, IllegalBlockSizeException, NoSuchPaddingException, NoSuchAlgorithmException {
        byte[] decipheredText = new byte[0];
        Cipher cipher = Cipher.getInstance("RSA/ECB/PKCS1Padding");
        cipher.init(Cipher.DECRYPT_MODE, keyPair.getPrivate());
        decipheredText = cipher.doFinal(encryptedBytes);
        return new String(decipheredText, StandardCharsets.UTF_8);
    }

    static KeyPair generateKeyPair(int keyLength) throws NoSuchAlgorithmException {
        final String KEY_ALGORITHM = "RSA";
        KeyPairGenerator keygen = KeyPairGenerator.getInstance(KEY_ALGORITHM);
        keygen.initialize(keyLength, new SecureRandom());
        KeyPair keyPair = keygen.generateKeyPair();
        return keyPair;
    }
}