For command token doesn't work when there are empty fields in the record

Question

I am trying to extract the values from the third field of a file which has data records.

The fields are separated by vertical bar characters:

9001||10454145||60|60
9001|234467|10454145||60|60
9001|234457|10454145||60|60

Command is -

for /f "tokens=3 delims=|" %%A IN ('Findstr /i "9001" .\itemloc\%%~nf.dat') do (
 echo  %%A >> log.txt
)

But the output I am getting is

60
10454145
10454145

The empty fields are messing up my output. Any suggestions how to make the for token work with empty fields in the record?

Answer 1

@ECHO OFF
SETLOCAL ENABLEDELAYEDEXPANSION
rem The following settings for the directories and filenames are names
rem that I use for testing and deliberately includes spaces to make sure
rem that the process works using such names. These will need to be changed to suit your situation.

SET "sourcedir=u:\your files"
SET "destdir=u:\your results"
SET "filename1=%sourcedir%\q75199035.txt"
SET "outfile=%destdir%\outfile.txt"

(
FOR /f "usebackqtokens=1*delims=" %%e IN ("%filename1%") DO (
 SET "line=%%e"
 FOR /f "tokens=3 delims=|" %%y IN ("!line:||=|(missing)|!") DO ECHO %%y
)
)>"%outfile%"

TYPE "%outfile%"

GOTO :EOF

Always verify against a test directory before applying to real data.

Note that if the filename does not contain separators like spaces, then both usebackq and the quotes around %filename1% can be omitted.

The magic is that for each line, || is replaced by |(missing)|.

This simple solution has its faults - for instance if there is ||| in the source data, or the usual suspects (some punctuation symbols like ! ) but should be quite happy with alphameric source text.

Another way would be to use a third-party utility like sed to pre-process the source data.

The fundamental reason for this phenomenon is that for/f parses the line as [delimiters]token1[delimiters]token2... , where [delimiters] is any sequence of any of the delimiter characters.