Error while trying to write on parquet file in datastage 11.7 (File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem)

Question

we have recently upgraded the DataStage from 9.1 to 11.7 on Server AIX 7.1. and i'm trying to use the new connector "File Connector" to write on parquet file. i created simple job takes from teradata as a source and write on the parquet file as a target. Image of the job

but facing below error:

> File_Connector_20,0: java.lang.NoClassDefFoundError: org.apache.hadoop.fs.FileSystem
    at java.lang.J9VMInternals.prepareClassImpl (J9VMInternals.java)
    at java.lang.J9VMInternals.prepare (J9VMInternals.java: 304)
    at java.lang.Class.getConstructor (Class.java: 594)
    at com.ibm.iis.jis.utilities.dochandler.impl.OutputBuilder.<init> (OutputBuilder.java: 80)
    at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 340)
    at com.ibm.iis.jis.utilities.dochandler.impl.Registrar.getBuilder (Registrar.java: 302)
    at com.ibm.iis.cc.filesystem.FileSystem.getBuilder (FileSystem.java: 2586)
    at com.ibm.iis.cc.filesystem.FileSystem.writeFile (FileSystem.java: 1063)
    at com.ibm.iis.cc.filesystem.FileSystem.process (FileSystem.java: 935)
    at com.ibm.is.cc.javastage.connector.CC_JavaAdapter.run (CC_JavaAdapter.java: 444)

i followed the steps in below link: https://www.ibm.com/support/knowledgecenter/SSZJPZ_11.7.0/com.ibm.swg.im.iis.conn.s3.usage.doc/topics/amaze_file_formats.html

1- i uploaded the jar files into "/ds9/IBM/InformationServer/Server/DSComponents/jars"

2- added them to CLASSPATH in agent.sh then restarted the datastage.

3- i have set The environment variable CC_USE_LATEST_FILECC_JARS to the value parquet-1.9.0.jar:orc-2.1.jar.

i tried also to add the CLASSPATH as an environment variable in the job but not worked.

noting that i'm using Local in File System.

so any hint is appreciated as i'm searching a lot time ago. Thanks in advance,

Answer 1

Which File System mode you are using? If you are using Native HDFS as File System mode, then you would need to configure CLASSPATH to include some third party jars.

Perhaps these links should provide you with some guidance.

• https://www.ibm.com/support/pages/node/301847
• https://www.ibm.com/support/pages/steps-required-configure-file-connector-use-parquet-or-orc-file-format

Note : Based on the hadoop distribution and version you are using, the version of the jars could be different.

If the above information does not help in resolving the issue, then you may have to reach out to IBM Support to get this addressed.

Answer 2

TO use File Connector, there is no need to add CLASSPATH in agent.sh unless you want to import HDFS files from IMAM.

If your requirement is reading Parquet files, then set $CC_USE_LATEST_FILECC_JARS=parquet-1.9.0.jar $FILECC_PARQUET_AVRO_COMPAT_MODE=TRUE

If you are still seeing issue, then run job with $CC_MSG_LEVEL=2 and open IBM support case along with job design, FULL job log and Version.xml file from Engine tier.