signature verification process in java security

Question

I want to send a original_message between two person. let say Alice and Bob, and I want to know if these steps are correct for verification of the signature or not

1. Alice hash the original_message with her PrivateKey -> h(m)
2. Alice cipher the hashed message -> c(h(m))
3. Alice sign the message with her PrivateKey -> s(c(h(m)))

Alice send the final signed message with her (PublicKey) and (the original_message) to Bob. in Bob side:

1. Bob hash the original_message -> h(m)
2. Bob decipher the signed message with Alice public key -> d(s(c(h(m))))
3. Bob check deciphered message with hashed message if they are equal or not ? if ( h(m) == d(s(c(h(m)))) )

I know that I am doing some mistake. does any one knows what is the good order in both side ?

Here I used java.security for doing this but when I check the hashes at final step it gives me false !

in Alice part :

public byte[] Sign(byte[] aMessage) {

        try {
            // get an instance of a cipher with RSA with ENCRYPT_MODE
            // Init the signature with the private key
            // Compute signature
            Cipher cipher = Cipher.getInstance("RSA");
            cipher.init(Cipher.ENCRYPT_MODE, thePrivateKey);

            Signature instance = Signature.getInstance("MD5withRSA");
            instance.initSign(thePrivateKey);

            // get an instance of the java.security.MessageDigest with MD5
            // process the digest
            MessageDigest md5_digest = MessageDigest.getInstance("MD5");
            md5_digest.update(aMessage);
            byte[] digest = md5_digest.digest();

            // return the encrypted digest
            byte[] cipherText = cipher.doFinal(digest);

            instance.update(cipherText);            
            byte[] signedMSG = instance.sign();

            return signedMSG;

        } catch (Exception e) {
            System.out.println("Signature error");
            e.printStackTrace();
            return null;
        }

    }

in Bob part :

public boolean CheckSignature(byte[] aMessage, byte[] aSignature,
            PublicKey aPK) {
        try {
            // get an instance of a cipher with RSA with ENCRYPT_MODE
            // Init the signature with the private key
            // decrypt the signature
            Cipher cipher = Cipher.getInstance("RSA");
            cipher.init(Cipher.DECRYPT_MODE, aPK);
            byte[] decrypted_digest =  cipher.doFinal(aSignature);

            // get an instance of the java.security.MessageDigest with MD5
            MessageDigest md5_digest = MessageDigest.getInstance("MD5");

            // process the digest
            md5_digest.update(aMessage);
            byte[] digest = md5_digest.digest();

            // check if digest1 == digest2
            if (decrypted_digest == digest) {
                return true;
            }else {
                return false;
            }

        } catch (Exception e) {
            System.out.println("Verify signature error");
            e.printStackTrace();
            return false;
        }
    }

Answer 1

Finally I found the answer. The mistake was, doing a sign() in Alice part. Because when you are doing the hashing and encryption it already became a signature, and when you sign() it another time in the Bob's part became impossible to recover the hashed signature .

my code is also a good example of "One-way hashing in java security public key encryption"

this is the modification in Alice part and everything work after.

public byte[] Sign(byte[] aMessage) {

        try {
            // get an instance of a cipher with RSA with ENCRYPT_MODE
            // Init the signature with the private key
            // Compute signature
            Cipher cipher = Cipher.getInstance("RSA");
            cipher.init(Cipher.ENCRYPT_MODE, thePrivateKey);

            // get an instance of the java.security.MessageDigest with MD5
            // process the digest
            MessageDigest md5_digest = MessageDigest.getInstance("MD5");
            byte[] digest = md5_digest.digest(aMessage);

            // return the encrypted digest
            byte[] cipherText = cipher.doFinal(digest);

            return cipherText;

        } catch (Exception e) {
            System.out.println("Signature error");
            e.printStackTrace();
            return null;
        }

    }

Answer 2

Digital Signature is the Encrypted (with private key) cipher of the Hash of the message.

Signature s = c(h(m))

Now s is attached to the message m. Signed message to be transmitted from Alice to Bob is m+s

After receiving m+s on Bob's side, Bob would decrypt the signature with Alice's public key which would be present in the certificate. So here he does d(s) = d(c(h(m)) = h(m)

Also the Bob received the message so he would compute the hash of the message m which is h(m)

Now he would compare the outputs from both the above steps to see if they match. This ensures the message has not been tampered with by anyone in between.

This is the general idea of how Digital signatures work. Hope this helps.

Wikipedia has a graphical representation of the same process here: http://upload.wikimedia.org/wikipedia/commons/2/2b/Digital_Signature_diagram.svg