Python-从一个文本文件复制到另一个

Question

I'm writing a Python script, and want to copy specific lines of a Flexible I/O (FIO) data text file to my output text file.

I've tried to copy data from my data text file using this tutorial , but sadly, I'm not getting my ideal result. After I execute my code and I concatenate into output.txt, output.txt returns nothing and not the text lines of cpu or lat (usec) from data.txt.

Here is my code to obtain data.txt:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import subprocess
import os

cmd = subprocess.Popen(["sudo", "fio", "--filename=/dev/sdg", "--name=test1", "--ioengine=libaio",
    "--bs=4k", "--rw=read", "--iodepth=32", "--runtime=10"],
    stdout = subprocess.PIPE, stderr = subprocess.PIPE, preexec_fn = os.setpgrp)

f = open("logfile.txt", "w+")
g = open("logs.txt", "w+")

f.write(cmd.stdout.read().decode())

with g as f:
    for line in f:
        # finds the leading text in the text file that tarts with "", and also removes all
        # leading and trailing spaces from a strin.
        if line.strip().startswith("lat (usec)") or line.strip().startswith("cpu"):
            # adds items to end of list
            g.write(line + "\n")

Here is my data text file (data.txt):

test1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32
fio-2.16
Starting 1 process

test1: (groupid=0, jobs=1): err= 0: pid=9401: Tue Jul  2 14:28:30 2019
  read : io=4066.2MB, bw=416418KB/s, iops=104104, runt= 10001msec
    slat (usec): min=1, max=2650, avg= 7.75, stdev=41.93
    clat (usec): min=19, max=2790, avg=298.85, stdev=180.63
     lat (usec): min=101, max=2792, avg=306.59, stdev=181.00
    clat percentiles (usec):
     |  1.00th=[  105],  5.00th=[  112], 10.00th=[  118], 20.00th=[  120],
     | 30.00th=[  129], 40.00th=[  137], 50.00th=[  153], 60.00th=[  454],
     | 70.00th=[  474], 80.00th=[  490], 90.00th=[  502], 95.00th=[  516],
     | 99.00th=[  548], 99.50th=[  580], 99.90th=[  756], 99.95th=[  932],
     | 99.99th=[ 1592]
    lat (usec) : 20=0.01%, 100=0.01%, 250=51.56%, 500=36.57%, 750=11.76%
    lat (usec) : 1000=0.06%
    lat (msec) : 2=0.04%, 4=0.01%
  cpu          : usr=18.48%, sys=38.28%, ctx=16300, majf=0, minf=40
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%
     issued    : total=r=1041149/w=0/d=0, short=r=0/w=0/d=0, drop=r=0/w=0/d=0
     latency   : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
   READ: io=4066.2MB, aggrb=416417KB/s, minb=416417KB/s, maxb=416417KB/s, mint=10001msec, maxt=10001msec

Disk stats (read/write):
  sdg: ios=16094/0, merge=0/0, ticks=14376/0, in_queue=14368, util=99.01%

Now I expect lines 8, 15, and 18 from the data text file into the output text file, but it's returning nothing. This is what the output text file would look like (output.txt):

     lat (usec): min=101, max=2792, avg=306.59, stdev=181.00
    lat (usec) : 250=5.09%, 500=67.38%, 750=23.29%, 1000=3.87%
  cpu          : usr=18.48%, sys=38.28%, ctx=16300, majf=0, minf=40

Is there a similar way of extracting data from my data text file into my result output file using the tutorial above?

Thank you!

Answer 1

something like this?

with open("/tmp/log.txt") as f:
        to_keep = []
        for line in f:
                if line.strip().startswith("lat (usec)") or line.strip().startswith("cpu"):
                        to_keep.append(line)
        print(to_keep)

btw: not really the cleanest way of doing it, but I guess the simplest one (in python)

even simpler would be using simple linux commands, something like:

   cat log.txt | grep "lat (usec)"

Answer 2

I figured out my problem, thank you @AlessioM and @pkfm for your help!

Here is my answer:

#!/usr/bin/python
# -*- coding: utf-8 -*-

import subprocess
import os

def fioCall():
    cmd = subprocess.Popen(["sudo", "fio", "--filename=/dev/sdg", "--name=test1", "--ioengine=libaio",
    "--bs=4k", "--rw=read", "--iodepth=32", "--runtime=10"],
    stdout = subprocess.PIPE, stderr = subprocess.PIPE, preexec_fn = os.setpgrp)

    f = open("data.txt", "w+")
    f.write(cmd.stdout.read().decode())
    f.close()

def copyFiles():
    with open("data.txt") as f:
        with open("output.txt", "w") as f1:
            for line in f:
                if line.strip().startswith("lat (usec)") or line.strip().startswith("cpu"):
                    f1.write(line)

def main():
    fioCall()
    copyFiles()

if __name__ == "__main__":
    main()

and here is the output I obtained (output.txt):

     lat (usec): min=161, max=10624, avg=458.78, stdev=148.10
    lat (usec) : 250=5.09%, 500=67.38%, 750=23.29%, 1000=3.87%
  cpu          : usr=17.24%, sys=63.28%, ctx=81571, majf=0, minf=40