从多个文件（结构输出）中提取数据并打印到一个文件

Question

Please, I need help with extracting values from 400 files. So far I have never been doing something similar and I don't know where to start from. Since I am not a programmer, I don' know which software program would be good to use: R, SAS, Python, command prompt, bash, awk. I have some experience with data manipulation/management using SAS and R (mostly “regular” files with rows and columns) while running some applications with command prompt, bash.

1. I run Structure (Software for population genetics) on Cloud Computing.
2. Output were 400 files/runs. Their names are : job_01_01-output_f; job_01_02-output_f …… job_40_10-output_f
3. These outputs don't have any extensions (like .txt), but I normally open them using Textpad, Notepad++
4. In each of these 400 files/outputs there is a line: Estimated Ln Prob of Data = -5570597.3
5. I would like to extract numeric value -5570597.3 from all these files/outputs and save it into .csv, .txt like column (one under another - the same order like files)
6. Also, this line is not always in the same line within all the files because it depends on number of “parameters”.
7. So I guess something like “take value that comes after “Estimated Ln Prob of Data =” would be option.
8. For example, one file/outout has around 60000 lines. The size from these files goes from 800kb to 5mb.
9. I will try to upload the file/output for example.

Best regards

[LINK - an example of Structure/file output][1]

https://www.dropbox.com/sh/idvoigkky7ldgb7/AAD5foVSKc5Ty6ijc08ge230a?dl=0

Answer 1

Using grep with PCREs for positive look-behind and data from the Dropbox link:

$ grep -Pohm 1 "(?<=^Estimated Ln Prob of Data   = ).*" job_*

Output:

-5570597.3
-2834943326.2

Used switches:

-P, --perl-regexp
          Interpret PATTERNS as Perl-compatible regular expressions (PCREs).

-h, --no-filename
          Suppress the prefixing of file names on output.

-o, --only-matching
          Print only the matched (non-empty) parts of a matching line

-m NUM, --max-count=NUM
          Stop reading a file after NUM matching lines.

Another using awk:

$ for f in job* ; do awk '/^Estimated Ln Prob of Data/{print $NF;exit}' $f ; done

and GNU awk:

$ awk '/^Estimated Ln Prob of Data/{print $NF;nextfile}' job_*

Answer 2

A simple implementation in Python. Let me know if it works for you.

import glob
import os.path as os
import re
import uuid


def extract_data(source: str,
                 export: str = None,
                 nested: bool = False,
                 delimit: str = ",",
                 extract: str = "Estimated Ln Prob of Data") -> None:
  """
  Extracts values of `Estimated Ln Prob of Data` from source and exports
  it in a text file.
  
  Args:
    source: Directory which has `job_01_01-output_f` files.
    export: Path of the output file.
    nested: Boolean, if you want to use nested files as well.
    extract: Keyword whose respective value needs to be extracted.
  """
  regex = r"^\b{}\b.+$".format(extract)
  nest = "**" if nested else "*"
  values = []

  for file in glob.glob(f"{source}/{nest}", recursive=True):
    raw = os.basename(file)
    if raw.startswith("job_") and raw.endswith("-output_f"):
      with open(file, "r") as _file:
        matches = re.finditer(regex, _file.read(), re.MULTILINE)
        entry = f"{raw}{delimit}{list(matches)[0].group().rsplit('= ')[-1]}\n"
        values.append(entry)

  export = export if export else os.join(source, f"{str(uuid.uuid4())}.txt")
  with open(export, "w") as _file:
    _file.writelines(values)


# Where "/home/SOME_USER/Downloads" is the path where you have these 400 files.
extract_data("/home/SOME_USER/Downloads")

Answer 3

batch for your literal question:

(for /f "tokens=2 delims==" %%a in ('findstr /c:"Estimated Ln Prob of Data" "job_??_??-output_f"') do echo %%a)>result.csv

In case you need the filenames too:

(for /f "tokens=1,3 delims=:=" %%a in ('findstr /c:"Estimated Ln Prob of Data" "job_??_??-output_f"') do echo %%a,%%b)>result.csv

Answer 4

Firstly, I'm offering this answer to give additional options, I think the best answer is James Brown 's grep solution as learning to be proficient with grep will be a particularly useful skill. Stephan 's solution is also handy if you think you might get stuck in a Windows environment, especially a if you're in a minimal one that won't necessarily have PowerShell.

Here's an option in PowerShell:

Get-Content "job_01_01-output_f" | ForEach-Object { if ($_ -match "Estimated Ln Prob of Data * = * ([-.\d]+)") { $Matches[1]} }

And another option using sed :

sed -ne "s/Estimated Ln Prob of Data *= *\([-.0-9]\+\)/\1/gp" "job_01_01-output_f"