MobileFirst Foundation 8 on Bluemix Java Buildpacks: “No space left on device”

Question

I deployed MobileFirst Foundation version 8.0.0.00-20160809-2334 on Bluemix Java buildpacks via scripts .

Starting last week, the configuration for adapters has disappeared, the mobile apps do not work anymore and any configuration action I try to perform results in one of the following errors:

mfp///10.254.0.6 FAILURE FWLSE0321: State change failure. Caused by: java.io.IOException: No space left on device

Or:

FAILURE java.io.IOException: No space left on device

dashDB reports only 4% memory used and we only used 375MB out of 1GB disk space, so it's unclear what could be going wrong.

Updated as requested in comments, Below a stack trace of the error when trying to upload an adapter:

APP/0[INFO ] Adapter initialized!Jan 24, 2017 9:35:06 AM
APP/0[ERROR ] RuntimeMBeanCallable.call() exceptionJan 24, 2017 9:35:09 AM
APP/0java.io.IOException: No space left on deviceJan 24, 2017 9:35:09 AM
APP/0[err] java.lang.RuntimeException: java.io.IOException: No space left on deviceJan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.app.internal.webresources.ApplicationWebResourceDescriptorHandler.deploy(ApplicationWebResourceDescriptorHandler.java:34)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.app.internal.webresources.ApplicationWebResourceDescriptorHandler.sign(ApplicationWebResourceDescriptorHandler.java:99)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.app.internal.webresources.ApplicationWebResourceDescriptorHandler.deploy(ApplicationWebResourceDescriptorHandler.java:64)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.core.internal.deployment.DeploymentManagerImpl.changeDeploymentState(DeploymentManagerImpl.java:380)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.core.internal.deployment.DeploymentManagerImpl.redeployIfNeeded(DeploymentManagerImpl.java:434)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.mgmt.internal.connectivity.RuntimeMXBeanImpl.changeDeploymentState(RuntimeMXBeanImpl.java:93)Jan 24, 2017 9:35:09 AM
APP/0[err] at java.lang.reflect.Method.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.misc.Trampoline.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at java.lang.reflect.Method.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.misc.MethodUtil.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.MBeanSupport.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.MXBeanProxy$InvokeHandler.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.ws.kernel.boot.jmx.internal.PlatformMBeanServer.invoke(PlatformMBeanServer.java:373)Jan 24, 2017 9:35:09 AM
APP/0[err] at javax.management.MBeanServerInvocationHandler.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.actions.ArtifactDeploymentTransaction.prepareMBean(ArtifactDeploymentTransaction.java:822)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.ws.kernel.boot.jmx.service.MBeanServerForwarderDelegate.invoke(MBeanServerForwarderDelegate.java:197)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.jmx.mbeanserver.MXBeanProxy.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.sun.proxy.$Proxy361.changeDeploymentState(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.actions.util.RuntimeMBeanWorkerThreadCaller.callSynchronously(RuntimeMBeanWorkerThreadCaller.java:206)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.actions.BaseTransaction.prepare(BaseTransaction.java:561)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.actions.util.RuntimeMBeanWorkerThreadCaller$RuntimeMBeanCallable.call(RuntimeMBeanWorkerThreadCaller.java:76)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.actions.util.RuntimeMBeanPoolCaller.callRuntimeMBeans(RuntimeMBeanPoolCaller.java:91)Jan 24, 2017 9:35:09 AM
APP/0[err] at java.lang.reflect.Method.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.CreateInvocationParametersHandler.handleRequest(CreateInvocationParametersHandler.java:54)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.InvokeMethodHandler.handleRequest(InvokeMethodHandler.java:63)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.actions.BaseTransaction.run(BaseTransaction.java:327)Jan 24, 2017 9:35:09 AM
APP/0[err] at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.services.v2.AdapterService.addAdapter(AdapterService.java:351)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.actions.BaseTransaction.internalRun(BaseTransaction.java:454)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:33)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandler.handleRequest(AbstractHandler.java:34)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleResourceMethod(FindResourceMethodHandler.java:151)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceLocator(FindResourceMethodHandler.java:230)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceLocator(FindResourceMethodHandler.java:230)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:65)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:115)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:115)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleRequest(FindResourceMethodHandler.java:115)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindResourceMethodHandler.handleSubResourceLocator(FindResourceMethodHandler.java:230)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.HeadMethodHandler.handleRequest(HeadMethodHandler.java:53)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.OptionsMethodHandler.handleRequest(OptionsMethodHandler.java:46)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.FindRootResourceHandler.handleRequest(FindRootResourceHandler.java:95)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.handlers.SearchResultHandler.handleRequest(SearchResultHandler.java:33)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.log.ResourceInvocation.handleRequest(ResourceInvocation.java:92)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:26)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.RequestHandlersChain.handle(RequestHandlersChain.java:22)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.doChain(AbstractHandlersChain.java:67)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.handlers.AbstractHandlersChain.run(AbstractHandlersChain.java:52)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.log.Requests.handleRequest(Requests.java:76)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.RequestProcessor.handleRequestWithoutFaultBarrier(RequestProcessor.java:207)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.RequestProcessor.handleRequest(RequestProcessor.java:154)Jan 24, 2017 9:35:09 AM
APP/0[err] at org.apache.wink.server.internal.servlet.RestServlet.service(RestServlet.java:133)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.services.RestServlet.service(RestServlet.java:43)Jan 24, 2017 9:35:09 AM
APP/0[err] at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.ws.webcontainer.servlet.ServletWrapper.service(ServletWrapper.java:1285)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.admin.services.RestFilter.doFilter(RestFilter.java:61)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:207)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.worklight.admin.common.filter.HSTSFilter.doFilter(HSTSFilter.java:110)Jan 24, 2017 9:35:09 AM
APP/0[err] at [internal classes]Jan 24, 2017 9:35:09 AM
APP/0[err] at [internal classes]Jan 24, 2017 9:35:09 AM
APP/0[err] at [internal classes]Jan 24, 2017 9:35:09 AM
APP/0[err] Caused by: Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:207)Jan 24, 2017 9:35:09 AM
APP/0[err] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] java.io.IOException: No space left on deviceJan 24, 2017 9:35:09 AM
APP/0[err] at java.io.FileOutputStream.write(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] at java.lang.Thread.run(Unknown Source)Jan 24, 2017 9:35:09 AM
APP/0[err] ... 128 moreJan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.app.internal.webresources.ApplicationWebResourceDescriptorHandler.getBase64EncodedFile(ApplicationWebResourceDescriptorHandler.java:151)Jan 24, 2017 9:35:09 AM
APP/0[err] at java.io.FileOutputStream.writeBytes(Native Method)Jan 24, 2017 9:35:09 AM
APP/0[err] at com.ibm.mfp.server.app.internal.webresources.ApplicationWebResourceDescriptorHandler.sign(ApplicationWebResourceDescriptorHandler.java:91)Jan 24, 2017 9:35:09 AM
APP/0[INFO ] Adapter destroyed!Jan 24, 2017 9:35:09 AM
APP/0[ERROR ] CNFSRVE115: configuration with id 'ADAPTER_CONTENT' for schema 'mfp_default_schema' with version '1.0' not foundJan 24, 2017 9:35:09 AM
APP/0[ERROR ] Unable to delete configuration with id ADAPTER_CONTENT due to exception FWLSE3208E: An invalid status code "404" was returned. The response content is "{"reason":"configuration_not_found","details":"CNFSRVE115: configuration with id 'ADAPTER_CONTENT' for schema 'mfp_default_schema' with version '1.0' not found"}".

Answer 1

There was a defect in the initial versions of the MFP server kit. The issue was that the BLOB size was 1M, so any adapter with 1M would fail to upload. Ideally you should shift to the latest iFix where the issue is fixed. However, if you want to patch the existing DB then follow the procedure

1. Find the Schema name you are using : For Ex If the Schema name is MFP_447EE21A
2. Log on to the Dash-DB.
3. Select the Default schema as Schema name you are using

4. Go to the Run SQL and execute the below SQL statements:
   (Update the below SQL commands with the Schema name.)

     ALTER TABLE MFP_447EE21A.CS_ATTACHMENTS DROP COLUMN CONTENT; ALTER TABLE MFP_447EE21A.CS_ATTACHMENTS ADD COLUMN CONTENT BLOB(1 G); call ADMIN_CMD('REORG TABLE MFP_447EE21A.CS_ATTACHMENTS'); 

5. Make sure the above SQL statements are run successfully.

6. Restart the MFP App instance.