Need to compare a flat directory to a text file list of path+files, and copy to new location with path structure from text file

Question

I am trying to write a script that saves the file structure of a large share to a text file, where each line is a folder or a path+file, and rebuild this structure from a lot of files that have been extracted and saved in a flat folder. So I want to search through the text file, when I find a match that is in the flat folder, I want to save this file from the flat folder in a NEW location but preserve the folder structure of where the file was (in the text file). I don't really care if the file is in the structure multiple times (duplicated within other folders in the folder structure).

For example: I have the following items in a text file ( $outfile ) from their original structure:

H:\
H:\a.doc
H:\b\c\d.pdf
H:\b\c\e\f\g.xls

Then I have

a.doc
d.pdf
g.xls

all in a flat folder with no folders ( $extractpath ), and I want to copy them to something like

I:\a.doc
I:\b\c.pdf
I:\b\c\e\f\g.xls

So I would need to

1. search $outfile for c.doc ,
2. then extract the path H:\\b\\ from H:\\b\\c.doc , and
3. copy c.doc to I:\\b\\

Given that $outfile is somewhat large for a one per line text file (27m), I was hoping there was an elegant way to do the directory struct write once, then read the file into memory and check each filename in $extractpath , extract the $filepath from $outfile , and copy this file to $filepath .

Here's what I have so far (not working yet):

# Path to log file date/timestamped
$logfile = "D:\Shares\Extracted\_extracted_2_$(Get-Date -format 'yyyyMMdd_hhmm').log"

# Path of root directory
$rootfolder = "H:\"

# Text file with Structure of the root directory, recursively
$outfile = "D:\Shares\Extracted\__struct.txt"

# Path of the files that were previously extracted with extract.ps1
$extractpath = "D:\Shares\Extracted"

# Path to move files to
$newpath = "D:\Shares\New"

if (!(Test-Path $outfile -PathType Leaf)) {
  Get-ChildItem $rootfolder | select -Expand fullname > $outfile
  Get-ChildItem $rootfolder -recurse | select -Expand fullname > $outfile
} else {
  Write-Output "File $outfile exists.. Reading from it.."
}

$files = get-childitem $extractpath
foreach ($file in $files) {
  $filepath  = $null
  $filepaths = (Get-Content -Path $outfile | Where-Object { $_ -like '*$filename'}) 
  foreach ($filepath in $filepaths){ 
    if ($filepath) {
      Copy-Item $file -Destination "$newpath\$filename"
    } else {
      Write-Output "Not found: $filename..."
    }
  }
}

EDIT: Thanks to Ansgar Wiechers below, I am not up to 15 posts so I cannot up vote you yet. This is what I have created and finally worked perfectly on my test data (my testpath() function is currently broken so ignore the scope issues I was having there)..

$outfile   = "d:\Shares\Extracted\__struct.txt"
$outfolder = "d:\Shares\Extracted\__folder.txt"
$extract   = "d:\Shares\Extracted"
$oldpath   = "H:\"
$newpath   = "d:\newH\"  #Make sure this has trailing backslash
$multfile  = "d:\newH\__multiple.log"
$logfile   = "d:\newH\__ext2_$(get-date -format `"yyyyMMdd_hhmm`").log"

$tpok=$false

function log($string, $color)
{
  if ($Color -eq $null) {$color = "white"}
  write-host $string -foregroundcolor $color
  $string | out-file -Filepath $logfile -append

}

function multlog($string, $color)
{
  if ($Color -eq $null) {$color = "white"}
  write-host $string -foregroundcolor $color
  $string | out-file -Filepath $logfile -append   # put in both log files     but output to screen only once
  $string | out-file -Filepath $multfile -append

}


function testpath($tpok)
{
$outpath = Split-Path -Path $outfile
$logpath = Split-Path -Path $logfile
$multpath = Split-Path -Path $multfile
if (!(Test-Path $outpath)) { log "bad Output path:  $outfile" red  return     $false }
if (!(Test-Path $multpath)) { log "bad Multiples path:  $multpath" red return $false }
if (!(Test-Path $extract)) { log "bad Extract path:     $extract" red return     $false }
if (!(Test-Path $oldpath)) { log "bad Oldpath path:     $oldpath" red return $false }
if (!(Test-Path $newpath)) { log "bad Newpath:      $newpath" red return $false }
if (!(Test-Path $logpath)) { log "bad Logfile:      $logfile" red return $false}
log "Paths test good.." yellow
$tpok = $true
return $true
}

function buildstructfiles()
{
  $testoutfile=(Test-Path $outfile -PathType Leaf)
  if (!($testoutfile))  {
    log "Writing $outfile.."  yellow
    log "Files in the root only.." yellow
    get-childitem $oldpath | where {!($_.psiscontainer)} | select -expand fullname > $outfile |
    log "Recursive directory export.." yellow
    get-childitem $oldpath -recurse | where {!($_.psiscontainer)} | select -expand fullname >> $outfile
    log "Done." yellow
  } else {
    log "File $outfile exists.. Reading from it.." yellow  
  }
  $testoutfolder=(Test-Path $outfolder -PathType Leaf)
  if (!($testoutfolder))  {  
    log "Writing $outfolder.."  yellow
    log "Folders in the root only.." yellow
    get-childitem $oldpath | where {($_.psiscontainer)} | select -expand fullname > $outfolder
    log "Folders recursively.." yellow
    get-childitem $oldpath -recurse | where {($_.psiscontainer)} | select -expand fullname >> $outfolder
  } else {
    log "File $outfolder exists.. Reading from it.." yellow  
  }
}

function Main()
{

  $destination = @{}
    if (!(testpath)) { Exit }
  buildstructfiles

  # Read full path+filenames from $outfile and create new destinations for each file in array $destination
  log "$(get-date -format `"yyyyMMdd_HHmm`") Reading $outfile check and writing $newpath destinations.. " yellow
  Get-Content $outfile | Where-Object {
    $_ -notmatch '\\$' # remove folders (lines with trailing backslash)
  } | ForEach-Object {
    $filename = Split-Path -Leaf $_
    log "Filename:  $filename"
    $chgoldpath = Split-Path -Path $_            # take path of current     object
    $chg = $chgoldpath -replace [regex]::Escape($oldpath), $newpath        #replace $oldpath with $newpath
    if ($destination.Contains($filename)) {
      # it has already been found.  Log this in a separate file as well for later comparison
      multlog "$filename has already been found in $($destination[$filename]), now in $chgoldpath" red
    }
    $destination[$filename]=$chg              #save this into an array
    log "New Destination: $($destination[$filename])"
    log "----"
  }

  #  create directory structure from saved txt file $outfolder
  log "$(get-date -format `"yyyyMMdd_HHmm`") Create folders in new structure     $newpath" red
  Get-Content $outfolder | ForEach-Object {
    log "object: $_"
    $d = ($_ -replace [regex]::Escape($oldpath), $newpath)
    log "New Directory to be created: $d" yellow
    if (!(Test-Path($d))) { 
      new-item $d -itemtype directory
    } else {
      log "Directory already exists! $d" gray
    }
  }

#  Or use robocopy to replicate folder structure  
#  log "$(get-date -format `"yyyyMMdd_HHmm`") Starting folder structure     Robocopy.." yellow
#  robocopy $oldPath $newPath /e /xf /LOG+:$logFile

  # Copy the files from $extractpath to new $destination
  Get-ChildItem $extract | ForEach-Object {
    $dname = Split-Path -Leaf $_
    $dfolder = Split-Path -Path $_
    if ($destination.Contains($dname)) {
      log "$($dname): Destination $($destination[$dname])" darkcyan
      Copy-Item $_ -Destination ($destination[$dname])
    } else {
      log "$($dname) does not exist in $oldpath, not restored." red
    }
  }
}

Main 

Answer 1

Assuming that all your filenames are unique (since they are in a single folder) you could read the destination file into a hashtable mapping each filename to its destination path:

$outfile   = 'D:\Shares\Extracted\__struct.txt'
$rootdrive = 'H:'
$newpath   = 'D:\Shares\New'

$destination = @{}
Get-Content $outfile | Where-Object {
  $_ -notmatch '\\$' # remove folders (lines with trailing backslash)
} | ForEach-Object {
  $filename = Split-Path -Leaf $_
  $destination[$filename] = $_ -replace "^$rootdrive", $newpath
}

If you need to (re-)create missing folders (your original question didn't mention that), you can do it in the same step:

$destination = @{}
Get-Content $outfile | Where-Object {
  $_ -notmatch '\\$' # remove folders (lines with trailing backslash)
} | ForEach-Object {
  $path     = $_ -replace "^$rootdrive", $newpath
  $filename = Split-Path -Leaf $path
  $folder   = Split-Path -Parent $path

  $destination[$filename] = $path

  if (-not (Test-Path -PathType Container -LiteralPath $folder)) {
    New-Item -Type Directory $folder | Out-Null
  }
}

Then you could copy the files from $extractpath like this:

$extractpath = 'D:\Shares\Extracted'

Get-ChildItem $extractpath | Copy-Item -Destination {$destination[$_.Name]}