Grails：验证域对象时出现“ methodInvocation”错误代码

Question

Since two weeks, a wild issue appears ! It comes only in production (not in dev, integration, preproduction modes) and after hours....

Symptoms : once "something" happens, users cannot sign up anymore. After days and days of investigations, I have no clue how to reproduce neither where come this issue. For now, we have the ugly workaround to restart tomcat instances, but this is VERY ugly.

For information, differences between two deliveries doesn't touch around the sign up process.

Here what I think is relevant :

Domain :

class UserAccount implements Serializable {
    static transients = ['tokenLogin']
    String email
    String pwd
    Date creationDate = new Date()
    String tokenLogin


    //[skip] dozen of other attributes

    static constraints = {
        email(matches:EmailRegexp,blank:false)
        pwd(blank:false)
        tokenLogin(editable:false)
    }

    public void setEmail(String email) {
        this.email = email
        authId = authenticationChannel.getUsername(email)
        // We must recalculate token login
        this.tokenLogin = generateTokenLogin(email, creationDate, pwd)
    }

    public void setPwd(String pwd) {
        this.pwd = pwd
        // We must recalculate token login
        this.tokenLogin = generateTokenLogin(email, creationDate, pwd)
    }

    public static String generateTokenLogin(String email, Date creationDate, String pwd) {
        // work well
        // even tested when return null or with NPE in it and does not produce the same log
        generated = // work with MessageDigest, MD5, salt, etc. (not disclosed, because we are currently under security audit)
        return generated;
    }
}

Controller :

def saveAccount = {
    def user = new UserAccount(email: params.email?.trim(), pwd: params.pwd) // More parameters, but here what is relevant
    user.validate()
    if(user.hasErrors()) log.info("${user.errors}") // added since the issue appears
    // Other verification (unicity, clear password verification, etc.)
    // Password encryption
    // user.save()
}

Output log :

2013-06-03 15:37:32,165 [TP-Processor46] INFO  controller.UserAccountController  - org.springframework.validation.BeanPropertyBindingResult: 1 errors
Field error in object 'domain.UserAccount' on field 'pwd': rejected value [foobar]; codes [methodInvocation.domain.UserAccount.pwd,methodInvocation.pwd,methodInvocation.java.lang.String,methodInvocation]; arguments [org.springframework.context.support.DefaultMessageSourceResolvable: codes [domain.UserAccount.pwd,pwd]; arguments []; default message [pwd]]; default message [Property 'pwd' threw exception; nested exception is java.lang.NullPointerException]

Questions :

• How this output might be appears? We saw this ouput either for pwd or email field, never for others.
• What is the signification of "methodInvocation" code? (I have seen in spring-context that MethodInvocationException is thrown when BeanWrapperImpl.setPropertyValue catch an exception)
• How to get more information? (Now, we run a patched spring-context.jar displaying the StackTrace in BeanWrapperImpl, but we cannot reproduce until now)

Informations :

• Grails 1.3.7 => spring 3.0.5-GA
• Mysql jdbc
• Tomcat 6

In start.sh (tomcat options)

JAVA_OPTS="-Dapp=prod -Xms1024m -Xmx1024m -XX:MaxPermSize=256m \
-Dfile.encoding=UTF-8 -XX:+UseParNewGC -XX:+UseConcMarkSweepGC \
-XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled \
-XX:+HeapDumpOnOutOfMemoryError \
-Xloggc:/path/to/gc.log"

Edit : We change and compile spring-context.jar to be able to see a StackTrace. and we get this one :

org.springframework.beans.PropertyBatchUpdateException; nested PropertyAccessException details (1) are:
PropertyAccessException 1:
org.springframework.beans.MethodInvocationException: Property 'pwd' threw exception; nested exception is java.lang.NullPointerException
        at org.springframework.beans.BeanWrapperImpl.setPropertyValue(BeanWrapperImpl.java:1127)
        at org.springframework.beans.BeanWrapperImpl.setPropertyValue(BeanWrapperImpl.java:900)
        at org.springframework.beans.AbstractPropertyAccessor.setPropertyValues(AbstractPropertyAccessor.java:76)
        at org.springframework.validation.DataBinder.applyPropertyValues(DataBinder.java:673)
        at org.springframework.validation.DataBinder.doBind(DataBinder.java:569)
        at org.springframework.web.bind.WebDataBinder.doBind(WebDataBinder.java:191)
[snip]
Caused by: java.lang.NullPointerException
        at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.checkCall(PojoMetaClassSite.java:54)
        at org.codehaus.groovy.runtime.callsite.PojoMetaClassSite.call(PojoMetaClassSite.java:42)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:124)
        at domain.UserAccount.generateTokenLogin(UserAccount.groovy:369)
        at domain.UserAccount$generateTokenLogin.callStatic(Unknown Source)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallStatic(CallSiteArray.java:48)
        at domain.UserAccount$generateTokenLogin.callStatic(Unknown Source)
        at domain.UserAccount.setPwd(UserAccount.groovy:184)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.springframework.beans.BeanWrapperImpl.setPropertyValue(BeanWrapperImpl.java:1114)

The root cause is that exists an creationDate that might be null on some weird conditions. I guess the solution is not to call generateTokenLogin() in setters (and only in getters).

I cannot explain why we did not have this issue before (no recent upgrade, etc.)

Answer 1

It sounds like once this happens it isn't fixed without a restart. I would look at the permgen size for your application. If it works in every environment expect production it might be a memory issues.

Easy way to test is add 200megs or so to start.

Answer 2

just a wild guess - when you override a setter, you shouldn't set the value of that field inside this setter, because it's a kind of recursion, or at least unpredicted behavior (like this) could be expected, maybe try using raw field access operator:

public void setPwd(String pwd) {
    this.@pwd = pwd
    // We must recalculate token login
    this.tokenLogin = generateTokenLogin(email, pwd)
}

the same goes for email field/setter

Answer 3

Is your pre-prod server exactly the same as your prod server ? Which are the minors modifications you make, are you sur there is no impact ?

It could be the war generated which contains errors, did you try a rollback on your modifications ?

Mayby you could take your prod war and put it into your pre-prod env to reproduce the issue.

Is there a recent update from a plugin, grails version, or whatever in your dependencies ?

Not answering your question, but peraps I can give you investigations ideas.

Snite