使用FTP协议列出包含大约2,000,000个文件的目录

Question

我在FTP服务器上有一个包含约2,000,000个文件的目录。 该FTP服务器基于Linux。 我想列出该目录中的文件（文件名的最后修改日期），但是Filezilla和Core FTP Pro都不能这样做，并且列表操作将失败。 我尝试使用FTPWebRequest类在C＃中编写自己的应用程序并运行ListDirectory方法，但是它也无法列出目录中的文件，并且ftpwebrequest超时。
有什么方法可以使用任何协议（例如FTP，SMB，...）甚至通过执行bash脚本来列出目录中的这么多文件的名称？

在c＃中列出目录的功能是：

public static List<string> ListFtpDirectory(string address, string userName, string password)
    {
        List<string> filesName = new List<string>();
        try
        {
            FtpWebRequest request = (FtpWebRequest) WebRequest.Create(address);
            request.Method = WebRequestMethods.Ftp.ListDirectory;
            request.Credentials = new NetworkCredential(userName, password);
            request.UsePassive = true;
            using (FtpWebResponse response = (FtpWebResponse) request.GetResponse())
            using (StreamReader reader = new StreamReader(response.GetResponseStream()))
            {
                filesName =
                    reader.ReadToEnd().Split(new string[] {"\r\n"}, StringSplitOptions.RemoveEmptyEntries).ToList();
            }
        }
        catch (Exception ex)
        {
            Console.WriteLine(ex.Message);
        }
        return filesName;
    }

任何帮助表示赞赏。

Answer 1

我不太熟练使用C＃，但是您没有提供所收到错误的确切文本。

以下是应该在您的环境中运行的Powershell脚本：

$Server = "ftp://ftp.example.com/"
$User = "anonymous@example.com"
$Pass = "anonymous@anonymous.com"

Function Get-FtpDirectory($Directory) {

    # Credentials
    $FTPRequest = [System.Net.FtpWebRequest]::Create("$($Server)$($Directory)")
    $FTPRequest.Credentials = New-Object System.Net.NetworkCredential($User,$Pass)
    $FTPRequest.Method = [System.Net.WebRequestMethods+FTP]::ListDirectoryDetails

    # Don't want Binary, Keep Alive unecessary.
    $FTPRequest.UseBinary = $False
    $FTPRequest.KeepAlive = $False

    $FTPResponse = $FTPRequest.GetResponse()
    $ResponseStream = $FTPResponse.GetResponseStream()

    # Create a nice Array of the detailed directory listing
    $StreamReader = New-Object System.IO.Streamreader $ResponseStream
    $DirListing = (($StreamReader.ReadToEnd()) -split [Environment]::NewLine)
    $StreamReader.Close()

    # Remove first two elements ( . and .. ) and last element (\n)
    # and take into account empty directories
    If ($DirListing.Length -gt 3) {
        $DirListing = $DirListing[2..($DirListing.Length-2)]
    }
    else {
        $DirListing = @{}    $DirListing = $DirListing[2..($DirListing.Length-2)] 
    }

    # Close the FTP connection so only one is open at a time
    $FTPResponse.Close()

    # This array will hold the final result
    $FileTree = @()

    # Loop through the listings
    foreach ($CurLine in $DirListing) {

        # Split line into space separated array
        $LineTok = ($CurLine -split '\ +')

        # Get the filename (can even contain spaces)
        $CurFile = $LineTok[8..($LineTok.Length-1)]

        # Figure out if it's a directory. Super hax.
        $DirBool = $LineTok[0].StartsWith("d")

        # Determine what to do next (file or dir?)
        If ($DirBool) {
            # Recursively traverse sub-directories
            $FileTree += ,(Get-FtpDirectory "$($Directory)$($CurFile)/")
        } Else {
            # Add the output to the file tree
            $FileTree += ,"$($Directory)$($CurFile)"
        }
    }

    Return $FileTree

}