使用 python pandas 解析日志文件

Question

問題一

我正在嘗試從模擬中讀取和解析日志文件。 理想情況下，我想用熊貓來做到這一點，但我遇到了問題。 有問題的文件稱為log ，下面包含一個示例。 現在我嘗試做

     import pandas as pd
     import numpy as np
     import csv
    
     # This method works:
     results = []
     with open('log') as inputfile:
             for row in csv.reader(inputfile):
                 results.append(row)
    
   
      # This method doesn't work:
      pd.read_csv('log') 

錯誤是

pandas.parser.CParserError: Error tokenizing data. C error: Expected 1 fields in line 77, saw 4

為什么會這樣？

問題二

解析日志文件的最有效方法是什么？ 請記住，該文件實際上會非常大。 基本上，我想將每個時間步（每個時間步由Time= x指定）的殘差解析為一個 numpy 數組以進行繪圖。 例如

smoothSolver:  Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2

我想將0.813857557031附加到數組res_gamma_init

執行此操作的最有效方法是什么？

/*---------------------------------------------------------------------------*\
| =========                 |                                                 |
| \\      /  F ield         | OpenFOAM: The Open Source CFD Toolbox           |
|  \\    /   O peration     | Version:  2.3.x                                 |
|   \\  /    A nd           | Web:      www.OpenFOAM.org                      |
|    \\/     M anipulation  |                                                 |
\*---------------------------------------------------------------------------*/
Build  : 2.3.x-590d57f32fed
Exec   : simpleFoam
Date   : Oct 30 2015
Time   : 17:24:08
Host   : "jack"
PID    : 4034
Case   : /home/jack/OpenFOAM/jack-2.3.x/run/sim
nProcs : 1
sigFpe : Enabling floating point exception trapping (FOAM_SIGFPE).
fileModificationChecking : Monitoring run-time modified files using timeStampMaster
allowSystemOperations : Allowing user-supplied system call operations

// * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * * //
Create time

Create mesh for time = 0

Reading field p

Reading field U

Reading/calculating face flux field phi

Selecting incompressible transport model Newtonian
Selecting RAS turbulence model kOmegaSST
Using gammaReThetat-correlations by Langtry and Menter (2009)
gammaReThetatSSTCoeffs
{
    ca1             2;
    ce1             1;
    ca2             0.06;
    ce2             50;
    cThetat         0.03;
    sigmaf          1;
    sigmaThetat     2;
    s1              2;
    dUds            false;
    alphaK1         0.85034;
    alphaK2         1;
    alphaOmega1     0.5;
    alphaOmega2     0.85616;
    gamma1          0.5532;
    gamma2          0.4403;
    beta1           0.075;
    beta2           0.0828;
    betaStar        0.09;
    a1              0.31;
    c1              10;
    kInf            0;
    omegaInf        0;
}

No finite volume options present


SIMPLE: convergence criteria
    field p  tolerance 1e-06
    field U  tolerance 1e-06
    field "(k|omega|ReThetaTilda|nut|gamma)"     tolerance 1e-06


Starting time loop


forceCoeffs forceCoeffs:
    Not including porosity effects

Time = 1

smoothSolver:  Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2
DICPCG:  Solving for p, Initial residual = 1, Final residual = 0.00976261323556, No Iterations 326
time step continuity errors : sum local = 8.88353600919, global = -0.154492041821, cumulative = -0.154492041821
smoothSolver:  Solving for omega, Initial residual = 1, Final residual = 0.0110458877993, No Iterations 2
smoothSolver:  Solving for k, Initial residual = 0.999999999804, Final residual = 0.022014322402, No Iterations 2
bounding k, min: 0 max: 0.415754458684 average: 0.410526729282
smoothSolver:  Solving for ReThetatTilda, Initial residual = 1, Final residual = 0.0259070897862, No Iterations 2
smoothSolver:  Solving for gamma, Initial residual = 1, Final residual = 1.51565684699e-05, No Iterations 2
ExecutionTime = 1.41 s  ClockTime = 1 s

forceCoeffs forceCoeffs output:
    Cm    = 0
    Cd    = -19811.1501384
    Cl    = 4.01861806726e-15
    Cl(f) = 2.00930903363e-15
    Cl(r) = 2.00930903363e-15

Time = 2

smoothSolver:  Solving for Ux, Initial residual = 0.510606164486, Final residual = 0.0151751146151, No Iterations 2
smoothSolver:  Solving for Uy, Initial residual = 0.570639175137, Final residual = 0.0164524779792, No Iterations 2
DICPCG:  Solving for p, Initial residual = 0.160667758287, Final residual = 0.00156013775625, No Iterations 143
time step continuity errors : sum local = 88.8245306208, global = 11.6476210117, cumulative = 11.4931289699
smoothSolver:  Solving for omega, Initial residual = 0.531389189498, Final residual = 0.00899532326598, No Iterations 2
smoothSolver:  Solving for k, Initial residual = 0.562103650426, Final residual = 0.0138022250836, No Iterations 2
bounding k, min: 0 max: 0.425403200237 average: 0.411310758596
smoothSolver:  Solving for ReThetatTilda, Initial residual = 0.301017401945, Final residual = 0.00693946297375, No Iterations 2
smoothSolver:  Solving for gamma, Initial residual = 0.813857557031, Final residual = 6.74558898691e-06, No Iterations 2
ExecutionTime = 2.13 s  ClockTime = 2 s

forceCoeffs forceCoeffs output:
    Cm    = 0
    Cd    = -20283.0256988
    Cl    = 3.99364280461e-15
    Cl(f) = 1.9968214023e-15
    Cl(r) = 1.9968214023e-15

Time = 3

Answer 1

回答你的第二個問題。 這可能會有所幫助，盡管正則表達式可能更好。

line1 = 'smoothSolver:  Solving for Ux, Initial residual = 0.999999999388, Final residual = 0.00692443749034, No Iterations 2'
line2 = 'smoothSolver:  Solving for Uy, Initial residual = 0.999999994742, Final residual = 0.0027182321684, No Iterations 2'
test = [line1, line2]
import re
for line in test:
    if line.startswith('smoothSolver'):
        print re.findall(r'[residual = ][\d][.][\d]+', line)

控制台中的結果是：

[' 0.999999999388', ' 0.00692443749034']
[' 0.999999994742', ' 0.0027182321684']