Java Mission Control - Access Denied Connecting to Remote

Question

I cannot connect to a remote VM using Java Mission Control . I can connect using VisualVM with relative ease. The reason I want to use Mission Control is due to a long-standing bug with VisualVM having to be restarted whenever the remote VM is restarted. Therefore, most of the leg-work involved in remote JMX connections is already in-place.

I have already enhanced the configuration for Mission Control as instructed here: https://technology.first8.nl/using-mission-controle-for-remote-profiling/

Java Version: 1.7.0_79-b15

JVM parameters:

-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=redacted 
-XX:+HeapDumpOnOutOfMemoryError 
-XX:HeapDumpPath=/foo/bar/service
-XX:+UnlockCommercialFeatures
-XX:+FlightRecorder
-Dcom.sun.management.jmxremote.port=8401
-Dcom.sun.management.jmxremote.rmi.port=8402
-Dcom.sun.management.jmxremote.access.file=/foo/bar/service/jmxremote.access
-Djava.security.auth.login.config=ldap.config
-Djava.rmi.server.hostname=< redacted public IP address >
-Dcom.sun.management.jmxremote.login.config=< redacted JMX config name >
-Dcom.sun.management.jmxremote.local=false
-Djavax.net.ssl.keyStore=keystore.jks
-Djavax.net.ssl.keyStorePassword=< redacted password >
-Dcom.sun.management.jmxremote.registry.ssl=false
-Djava.net.preferIPv4Stack=true
-Djava.util.logging.config.file=/foo/bar/service/logging.properties

I am using authentication and SSL because this is being used in a production environment. The JMX server and RMI ports are different because for some reason I could not get them working on the same port.

Custom JMX Remote Access jmxremote.access :

monitorRole   readonly
controlRole   readwrite \
              create javax.management.monitor.*,javax.management.timer.*,com.sun.management.*,com.oracle.jrockit.* \
              unregister

Whenever I attempt to connect to either Flight Control or the Console I get the following message:

Could not connect to Foo Bar Service : access denied ("javax.management.MBeanPermission" "javax.management.MBeanServerDelegate#-[JMImplementation:type=MBeanServerDelegate]" "addNotificationListener")
Unable to resolve the connection credentials for Foo Bar Service. Problem was: access denied ("javax.management.MBeanPermission" "javax.management.MBeanServerDelegate#-[JMImplementation:type=MBeanServerDelegate]" "addNotificationListener")

This makes no sense to me because authentication and authorization are working properly with VisualVM, in fact when connecting with Mission Control I see this in server logs:

[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:initialize:481]:              [LdapLoginModule] search-first mode; SSL disabled
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:login:508]:           [LdapLoginModule] user provider: ldap://localhost/ou=redacted-ou,dc=redacted-dc-1,dc=redacted-dc-2
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:findUserDN:868]:              [LdapLoginModule] searching for entry belonging to user: redacted-user
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:findUserDN:895]:              [LdapLoginModule] found entry: uid=redacted-user,ou=redacted-ou,dc=redacted-dc-1,dc=redacted-dc-2
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:attemptAuthentication:807]:           [LdapLoginModule] attempting to authenticate user: redacted-user
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:login:570]:           [LdapLoginModule] authentication succeeded
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:commit:621]:          [LdapLoginModule] added LdapPrincipal "uid=redacted-user,ou=redacted-ou,dc=redacted-dc-1,dc=redacted-dc-2" to Subject
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:commit:631]:          [LdapLoginModule] added UserPrincipal "redacted-user" to Subject
[16:46:47] [RMI TCP Connection(2044)-some.redacted.ip.address/INFO] [STDOUT]: [com.sun.security.auth.module.LdapLoginModule:commit:642]:          [LdapLoginModule] added UserPrincipal "controlRole" to Subject

I figure it's safe to disable LDAP server SSL because it is not exposed outside of the VPS (feedback welcome). As you can see I take as confirmation the messages "authentication succeeded" and "added UserPrincipal "controlRole" to Subject" that it's working, but Mission Control disagrees. There doesn't appear to be any javax.management.* specific log messages indicating what went wrong.

Answer 1

The error message says it all - grant the permission to add notification listeners. IIRC, JMC will listen on the notifications to properly update the MBean tree when MBeans are added or removed.

Answer 2

I resolved this according to Hirt's answer, but it was non-trivial. I amended the default Java security policy with the following:

//
// permissions for the user/principal "controlRole", for all codebases:
//
grant principal com.sun.security.auth.UserPrincipal "controlRole" {

    //
    // jconsole:
    //  - most of these permissions are needed to let JConsole query the 
    //    MBean server and display information about Derby's mbeans as well
    //    as some default platform MBeans/MXBeans.
    //  - if you don't use JConsole, but query the MBean server from your
    //    JMX client app, some of these permissions may be needed.
    permission javax.management.MBeanPermission 
        "sun.management.*#-[java.*:*]", 
        "getMBeanInfo,isInstanceOf,queryNames";
    permission javax.management.MBeanPermission 
        "sun.management.*#*[java.*:*]", "getAttribute,invoke";
    permission javax.management.MBeanPermission 
        "sun.management.*#-[com.sun.management*:*]", 
        "getMBeanInfo,isInstanceOf,queryNames";
    permission javax.management.MBeanPermission 
        "com.sun.management.*#-[java.*:*]", 
        "getMBeanInfo,isInstanceOf,queryNames";
    permission javax.management.MBeanPermission 
        "com.sun.management.*#*[java.*:*]", "getAttribute,invoke";
    permission javax.management.MBeanPermission "java.*#-[java.*:*]", 
        "getMBeanInfo,isInstanceOf,queryNames";
    permission javax.management.MBeanPermission "javax.management.MBeanServerDelegate#[JMImplementation:type=MBeanServerDelegate]", 
        "getMBeanInfo,isInstanceOf,queryNames,addNotificationListener";
    permission java.net.SocketPermission "*", "resolve";
    permission java.util.PropertyPermission "java.class.path", "read";
    permission java.util.PropertyPermission "java.library.path", "read";
    permission java.lang.management.ManagementPermission "monitor";
    // end jconsole
};

It was key to use the com.sun.security.auth.UserPrincipal class here due to how I'm using LDAP to authenticate.