Cannot access Spotfire server - no Web interface (only SSH works)

Question

Recently one of our Spotfire environments got down out of a sudden (not sure what happened). When I try to open the server's web interface (ie access "Notes & Services") I get that:

When I SSHed to it and ran ps -ef | grep tomcat ps -ef | grep tomcat it listed only one process:

user 27247 26393  0 11:33 pts/1    00:00:00 grep --color=auto tomcat

When I did the same in a working Spotfire server environment it showed 2 processes (including the one above):

root      1623     1  1 Nov02 ?        03:15:21 /home/ec2-user/tss/jdk/jre/bin/java -Djava.util.logging.config.file=/home/ec2-user/tss/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -Xms512M -Xmx4096M -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dcom.sun.management.jmxremote -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dignore.endorsed.dirs= -classpath /home/ec2-user/tss/tomcat/bin/bootstrap.jar:/home/ec2-user/tss/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/home/ec2-user/tss/tomcat -Dcatalina.home=/home/ec2-user/tss/tomcat -Djava.io.tmpdir=/home/ec2-user/tss/tomcat/temp org.apache.catalina.startup.Bootstrap start
user 17726 17677  0 13:38 pts/2    00:00:00 grep --color=auto tomcat

So what I did is I copied the command (from the working ENV) that was missing and executed it in the broken ENV as root:

sudo /home/user/tss/jdk/jre/bin/java -Djava.util.logging.config.file=/home/user/tss/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -server -XX:+AlwaysPreTouch -XX:+UseG1GC -XX:+ScavengeBeforeFullGC -XX:+DisableExplicitGC -Xms512M -Xmx4096M -Djdk.tls.ephemeralDHKeySize=2048 -Djava.protocol.handler.pkgs=org.apache.catalina.webresources -Dorg.apache.catalina.security.SecurityListener.UMASK=0027 -Dcom.sun.management.jmxremote -Dorg.apache.catalina.session.StandardSession.ACTIVITY_CHECK=true -DLog4jContextSelector=org.apache.logging.log4j.core.async.AsyncLoggerContextSelector -Dignore.endorsed.dirs= -classpath /home/user/tss/tomcat/bin/bootstrap.jar:/home/user/tss/tomcat/bin/tomcat-juli.jar -Dcatalina.base=/home/user/tss/tomcat -Dcatalina.home=/home/user/tss/tomcat -Djava.io.tmpdir=/home/user/tss/tomcat/temp org.apache.catalina.startup.Bootstrap start

After that, I tried to access the web interface I saw this error:

When I checked the console output after I saw the above error on the Web interface I saw that:

14-Nov-2019 13:46:39.443 INFO [main] org.apache.catalina.startup.Catalina.start Server startup in 52682 ms
14-Nov-2019 13:47:49.187 SEVERE [http-nio-80-exec-6] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [default] in context with path [/spotfire] threw exception [Filter execution threw an exception] with root cause
 java.lang.IllegalArgumentException: Could not resolve placeholder 'SPOTFIRE_AUTH_CONFIG' in value "file:${SPOTFIRE_AUTH_CONFIG}/spotfire-auth.properties"
        at org.springframework.util.PropertyPlaceholderHelper.parseStringValue(PropertyPlaceholderHelper.java:172)
        at org.springframework.util.PropertyPlaceholderHelper.replacePlaceholders(PropertyPlaceholderHelper.java:124)
        at org.springframework.core.env.AbstractPropertyResolver.doResolvePlaceholders(AbstractPropertyResolver.java:237)
        at org.springframework.core.env.AbstractPropertyResolver.resolveRequiredPlaceholders(AbstractPropertyResolver.java:211)
        at org.springframework.core.env.AbstractEnvironment.resolveRequiredPlaceholders(AbstractEnvironment.java:571)
        at org.springframework.context.annotation.ConfigurationClassParser.processPropertySource(ConfigurationClassParser.java:451)
        at org.springframework.context.annotation.ConfigurationClassParser.doProcessConfigurationClass(ConfigurationClassParser.java:272)
        at org.springframework.context.annotation.ConfigurationClassParser.processConfigurationClass(ConfigurationClassParser.java:245)
        at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:202)
        at org.springframework.context.annotation.ConfigurationClassParser.parse(ConfigurationClassParser.java:170)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.processConfigBeanDefinitions(ConfigurationClassPostProcessor.java:316)
        at org.springframework.context.annotation.ConfigurationClassPostProcessor.postProcessBeanDefinitionRegistry(ConfigurationClassPostProcessor.java:233)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanDefinitionRegistryPostProcessors(PostProcessorRegistrationDelegate.java:271)
        at org.springframework.context.support.PostProcessorRegistrationDelegate.invokeBeanFactoryPostProcessors(PostProcessorRegistrationDelegate.java:91)
        at org.springframework.context.support.AbstractApplicationContext.invokeBeanFactoryPostProcessors(AbstractApplicationContext.java:692)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:530)
        at org.springframework.context.annotation.AnnotationConfigApplicationContext.<init>(AnnotationConfigApplicationContext.java:99)
        at com.ihsenergy.spotifre.SpringContext.<clinit>(SpringContext.java:14)
        at com.ihsenergy.spotifre.IHSMarkitCustomAuthenticator.authenticate(IHSMarkitCustomAuthenticator.java:19)
        at com.spotfire.server.security.ExternalAuthenticator.authenticateUsingCustomAuthenticator(ExternalAuthenticator.java:213)
        at com.spotfire.server.security.ExternalAuthenticator.authenticate(ExternalAuthenticator.java:195)
        at com.spotfire.server.security.ExternalAuthenticator.authenticate(ExternalAuthenticator.java:187)
        at com.spotfire.server.security.AuthenticationManager.doAuthenticate(AuthenticationManager.java:401)
        at com.spotfire.server.security.AuthenticationManager.authenticateNonAnonymously(AuthenticationManager.java:344)
        at com.spotfire.server.security.AuthenticationManager.authenticateFrontend(AuthenticationManager.java:214)
        at com.spotfire.server.security.AuthenticationManager.authenticate(AuthenticationManager.java:198)
        at com.spotfire.server.security.SecurityFilter$AuthenticationFilter.attemptAuthentication(SecurityFilter.java:445)
        at com.spotfire.server.security.SecurityFilter$AuthenticationFilter.doFilter(SecurityFilter.java:242)
        at com.spotfire.server.security.SecurityFilter$InternalFilter.doFilter(SecurityFilter.java:980)
        at com.spotfire.server.security.SecurityFilter$InternalFilterChain.doFilter(SecurityFilter.java:1009)
        at com.spotfire.server.security.SecurityFilter$UserDataConstraintFilter.doFilter(SecurityFilter.java:853)
        at com.spotfire.server.security.SecurityFilter$InternalFilter.doFilter(SecurityFilter.java:980)
        at com.spotfire.server.security.SecurityFilter$InternalFilterChain.doFilter(SecurityFilter.java:1009)
        at com.spotfire.server.security.SecurityFilter$ConnectorFilter.doFilter(SecurityFilter.java:813)
        at com.spotfire.server.security.SecurityFilter$InternalFilter.doFilter(SecurityFilter.java:980)
        at com.spotfire.server.security.SecurityFilter$InternalFilterChain.doFilter(SecurityFilter.java:1009)
        at com.spotfire.server.security.SecurityFilter.doFilter(SecurityFilter.java:160)
        at com.spotfire.server.security.AbstractFilter.doFilter(AbstractFilter.java:124)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.spotfire.server.security.CustomAuthFilterWrapper.doFilter(CustomAuthFilterWrapper.java:89)
        at com.spotfire.server.security.AbstractFilter.doFilter(AbstractFilter.java:124)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.spotfire.server.security.CsrfFilter.doFilter(CsrfFilter.java:74)
        at com.spotfire.server.security.AbstractFilter.doFilter(AbstractFilter.java:124)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.spotfire.server.security.HttpMethodsFilter.doFilter(HttpMethodsFilter.java:179)
        at com.spotfire.server.security.AbstractFilter.doFilter(AbstractFilter.java:124)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.spotfire.server.security.headers.HeadersFilter.doFilter(HeadersFilter.java:213)
        at com.spotfire.server.security.AbstractFilter.doFilter(AbstractFilter.java:124)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.spotfire.server.security.AccessLogFilter.doFilter(AccessLogFilter.java:73)
        at com.spotfire.server.security.AbstractFilter.doFilter(AbstractFilter.java:124)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.spotfire.server.security.RequestContextFilter.doFilter(RequestContextFilter.java:120)
        at com.spotfire.server.security.RequestContextFilter.doFilter(RequestContextFilter.java:85)
        at com.spotfire.server.security.AbstractFilter.doFilter(AbstractFilter.java:124)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:791)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1417)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)

I am not sure what what has caused that error, perhaps this is what is preventing the broken environment from running normally.

Answer 1

It turned out that somebody has started the tomcat server without root. That's why our environment variable "SPOTFIRE_AUTH_CONFIG" was not "recognised". As soon as I executed:

/etc/init.d/tss-10.0.1 stop|start

Everything worked fine.