使用apache ignite在hdfs上写一个文件

Question

I want to insert data in hdfs with help of ignite write through cache. I am using following example config file to run ignite node.

ignite.sh /app/apache-ignite-fabric-1.9.0-bin/examples/config/filesystem/example-igfs.xml

This is my core-site.xml file

<configuration>
<configuration>
<property>
    <name>fs.defaultFS</name>
    <value>hdfs://hmaster:9000/</value>
</property>
<property>
   <name>fs.file.impl</name>
   <!-- value>org.apache.hadoop.fs.LocalFileSystem</value  -->
<value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
   <description>The FileSystem for file: uris.</description>
</property>

<property>
   <name>fs.hdfs.impl</name>
   <value>org.apache.hadoop.hdfs.DistributedFileSystem</value>
   <description>The FileSystem for hdfs: uris.</description>
</property>

 <property>
      <name>fs.igfs.impl</name>
      <value>org.apache.ignite.hadoop.fs.v1.IgniteHadoopFileSystem</value>
  </property>
</configuration>
</configuration>

when I do hadoop fs -cat igfs:/// it shows igfs file system. If I run any hadoop job by below command it inserts data in igfs. But I need to insert data in hdfs file system . How to insert data in hdfs?

hadoop --config /app/apache-ignite-fabric-1.9.0-bin/examples/config/filesystem  jar /app/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount igfs:///workDir/myFile1 /outputWC

Answer 1

You should configure IGFS with a secondary file system to have write-through caching in Ignite.

This doc page says something about that: https://apacheignite-fs.readme.io/docs/secondary-file-system The config (default-config.xml) of Ignite Hadoop edition has the following code, that commented out by default:

  <property name="secondaryFileSystem"> <bean class="org.apache.ignite.hadoop.fs.IgniteHadoopIgfsSecondaryFileSystem"> <property name="fileSystemFactory"> <bean class="org.apache.ignite.hadoop.fs.CachingHadoopFileSystemFactory"> <property name="uri" value="hdfs://your_hdfs_host:9000/"/> </bean> </property> </bean> </property> 

You need to uncomment it and provide appropriate secondary file system URI. Please note known bug that a trailing slash should be present in the end of the 2ndary file system URI, hdfs://your_hdfs_host:9000/ . By default DUAL_ASYNC mode will be used. To set DUAL_SYNC mode set "defaultMode" property of "fileSystemConfiguration" bean.

General comments.

1. There should not be nested <configuration> tags in Hadoop configuration files.
2. You likely don't need to redefine 'fs.file.impl' and 'fs.hdfs.impl', please use $IGNITE_HOME/config/hadoop/core-site.ignite.xml as a core-site.xml file template.
3. hadoop fs -cat ... will not work for directories, please use hadoop fs -ls ...