如何從 Java 代碼運行 Hadoop HDFS 命令

Question

我是 Hadoop 的新手！ 如何從 Java 代碼運行一些 hdfs 命令？ 我一直在使用 java 代碼和 hdfs 命令直接從 cloudera vm 的終端成功測試 mapreduce，但現在我想學習如何使用 java 代碼來完成它。 我一直在尋找任何可以學習的材料，但我還沒有找到。 謝謝

Answer 1

我想這可能對你有幫助

我用它很好地執行shell命令。這是java示例

public class JavaRunShell {
    public static void main(String[] args){  
        try {  
            String shpath="  your command";
            Process ps = Runtime.getRuntime().exec(shpath);  
            ps.waitFor();  

            }  
        catch (Exception e) {  
            e.printStackTrace();  
            }  
    } 
}

Answer 2

您可以在 Java 代碼中使用FileSystem API與 HDFS 交互。

Answer 3

正如 Jagrut 所提到的，您可以在 Java 代碼中使用 FileSystem API 與 hdfs 命令進行交互。 下面是我試圖檢查特定目錄是否存在於 hdfs 中的示例代碼。 如果存在，則刪除該 hdfs 目錄。

    Configuration conf = new Configuration();
    Job job = new Job(conf,"HDFS Connect");

    FileSystem fs = FileSystem.get(conf);
    Path outputPath = new Path("/user/cloudera/hdfsPath");
    if(fs.exists(outputPath))
        fs.delete(outputPath);

您還可以參考給定的博客以進一步參考 -

https://dzone.com/articles/working-with-the-hadoop-file-system-api , https://hadoop.apache.org/docs/r2.8.2/api/org/apache/hadoop/fs/ FileSystem.html https://blog.knoldus.com/2017/04/16/working-with-hadoop-filesystem-api/

Answer 4

您可以在 java 代碼中使用 FileSystem API 來執行 Hdfs 命令。 https://hadoop.apache.org/docs/r2.8.2/api/org/apache/hadoop/fs/FileSystem.html

請找到以下示例代碼。

package com.hadoop.FilesystemClasses;
import java.io.IOException;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.log4j.Logger;
import com.hadoop.Constants.Constants;
public class HdfsFileSystemTasks {

public static Logger logger = Logger.getLogger(HdfsFileSystemTasks.class
        .getName());

public FileSystem configureFilesystem(String coreSitePath,
        String hdfsSitePath) {
    FileSystem fileSystem = null;

    try {
        Configuration conf = new Configuration();
        Path hdfsCoreSitePath = new Path(coreSitePath);
        Path hdfsHDFSSitePath = new Path(hdfsSitePath);
        conf.addResource(hdfsCoreSitePath);
        conf.addResource(hdfsHDFSSitePath);

        fileSystem = FileSystem.get(conf);
        return fileSystem;
    } catch (Exception ex) { 
        ex.printStackTrace();
        return fileSystem;
    }
}

    public String writeToHDFS(FileSystem fileSystem, String sourcePath,
        String destinationPath) {
    try {
        Path inputPath = new Path(sourcePath);
        Path outputPath = new Path(destinationPath);
        fileSystem.copyFromLocalFile(inputPath, outputPath);
        return Constants.SUCCESS;
    } catch (IOException ex) {
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}

 public String readFileFromHdfs(FileSystem fileSystem, String hdfsStorePath,
        String localSystemPath) {
    try {
        Path hdfsPath = new Path(hdfsStorePath);
        Path localPath = new Path(localSystemPath);
        fileSystem.copyToLocalFile(hdfsPath, localPath);
        return Constants.SUCCESS;
    } catch (IOException ex) {
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}

public String deleteHdfsDirectory(FileSystem fileSystem,
        String hdfsStorePath) {
    try {
        Path hdfsPath = new Path(hdfsStorePath);

        if (fileSystem.exists(hdfsPath)) {
            fileSystem.delete(hdfsPath);
            logger.info("Directory{} Deleted Successfully "
                    + hdfsPath);
        } else {
            logger.info("Input Directory{} does not Exists " + hdfsPath);
        }

        return Constants.SUCCESS;
    } catch (Exception ex) {
        System.out
                .println("Some exception occurred while reading file from hdfs");
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}

public String deleteLocalDirectory(FileSystem fileSystem,
        String localStorePath) {
    try {
        Path localPath = new Path(localStorePath);

        if (fileSystem.exists(localPath)) {
            fileSystem.delete(localPath);
            logger.info("Input Directory{} Deleted Successfully "
                    + localPath);
        } else {
            logger.info("Input Directory{} does not Exists " + localPath);
        }
        return Constants.SUCCESS;
    } catch (Exception ex) {
        System.out
                .println("Some exception occurred while reading file from hdfs");
        ex.printStackTrace();
        return Constants.FAILURE;
    }
}


public void closeFileSystem(FileSystem fileSystem) {
    try {
        fileSystem.close();
    } catch (Exception ex) {
        ex.printStackTrace();
        System.out.println("Unable to close Hadoop filesystem : " + ex);
    }
  }
}

package com.hadoop.FileSystemTasks;

import com.hadoop.Constants.HDFSParameters;
import com.hadoop.Constants.HdfsFilesConstants;
import com.hadoop.Constants.LocalFilesConstants;
import com.hadoop.FilesystemClasses.HdfsFileSystemTasks;
import org.apache.hadoop.fs.FileSystem;
import org.apache.log4j.Logger;
public class ExecuteFileSystemTasks {

public static Logger logger = Logger.getLogger(ExecuteFileSystemTasks.class
        .getName());

public static void main(String[] args) {

    HdfsFileSystemTasks hdfsFileSystemTasks = new HdfsFileSystemTasks();
    FileSystem fileSystem = hdfsFileSystemTasks.configureFilesystem(
            HDFSParameters.CORE_SITE_XML_PATH,
            HDFSParameters.HDFS_SITE_XML_PATH);

    logger.info("File System Object {} " + fileSystem);

    String fileWriteStatus = hdfsFileSystemTasks.writeToHDFS(fileSystem,
            LocalFilesConstants.SALES_DATA_LOCAL_PATH,
            HdfsFilesConstants.HDFS_SOURCE_DATA_PATH);

    logger.info("File Write Status{} " + fileWriteStatus);

    String filereadStatus = hdfsFileSystemTasks.readFileFromHdfs(
            fileSystem, HdfsFilesConstants.HDFS_DESTINATION_DATA_PATH
                    + "/MR_Job_Res2/part-r-00000",
            LocalFilesConstants.MR_RESULTS_LOCALL_PATH);
    logger.info("File Read Status{} " + filereadStatus);

    String deleteDirStatus = hdfsFileSystemTasks.deleteHdfsDirectory(
            fileSystem, HdfsFilesConstants.HDFS_DESTINATION_DATA_PATH
                    + "/MR_Job_Res2");

    hdfsFileSystemTasks.closeFileSystem(fileSystem);

 }

}

Answer 5

@HbnKing 我試過運行你的代碼，但一直出錯。 這是我得到的錯誤

 java.io.IOException: Cannot run program "your": CreateProcess error=2, The system cannot 
 find the file specified
       at java.lang.ProcessBuilder.start(Unknown Source)
       at java.lang.Runtime.exec(Unknown Source)
       at java.lang.Runtime.exec(Unknown Source)
       at java.lang.Runtime.exec(Unknowenter code heren Source)
       at jrs.main(jrs.java:5)