如何通过批处理删除文本文件的重复部分？

Question

So currently I have to run a CLI command to generate data of all the playlists in a folder, and the output text file is something like below.

********************
PLAYLIST: 1
********************

<--- BEGIN FORUMS PASTE --->

Some unnecessary Data
Some unnecessary Data
Some unnecessary Data
......

<---- END FORUMS PASTE ---->

QUICK SUMMARY:

********************
PLAYLIST: 2
********************

<--- BEGIN FORUMS PASTE --->

Some unnecessary Data
Some unnecessary Data
Some unnecessary Data
.....

<---- END FORUMS PASTE ---->

QUICK SUMMARY:

********************
PLAYLIST: 3
********************

<--- BEGIN FORUMS PASTE --->

Some unnecessary Data
Some unnecessary Data
Some unnecessary Data
.....

<---- END FORUMS PASTE ---->

QUICK SUMMARY:
Required Data

From the little knowledge that I have, batch file can't remove it on the original file, but can instead create a new file with the processed data and that is fine for me.

Now to what I am trying to achieve is that there are recurring sections for all the playlists in the output text file. These are the sections beginning with <--- BEGIN FORUMS PASTE ---> and ending with <---- END FORUMS PASTE ----> . So I am trying to remove every section that begins and ends with them, basically leaving out all the data that is not enclosed within that section.

I am not sure how to go about it, but I feel like the findstr command will come into use here or maybe a VBscript.

Answer 1

The task can be done with the following batch file:

@echo off
setlocal EnableExtensions DisableDelayedExpansion

rem Remove all environment variables defined by default for faster lines
rem processing with exception of the environment variable ComSpec, Path,
rem PATHEXT and SystemRoot. The last one is really used below.
for /F "delims==" %%I in ('set ^| %SystemRoot%\System32\findstr.exe /B /I /L /V "Comspec Path PATHEXT SystemRoot"') do set "%%I="

set "SourceFile=PlayList.txt"
if not exist "%SourceFile%" (
    echo ERROR: File not found: "%SourceFile%"
    echo/
    pause
    goto EndBatch
)

set "EmptyLine="
set "IgnoreLines="
set "FileModified="
set "TempFile=%SourceFile%.tmp"

(for /F delims^=^ eol^= %%I in ('%SystemRoot%\System32\findstr.exe /N "^" "%SourceFile%" 2^>nul') do (
    set "Line=%%I"
    setlocal EnableDelayedExpansion
    if not defined IgnoreLines (
        if "!Line:<--- BEGIN FORUMS PASTE --->=!" == "!Line!" (
            if "!Line:*:=!" == "" (
                if not "!EmptyLine!" == "2" echo(
                endlocal
                set "EmptyLine=1"
            ) else (
                echo(!Line:*:=!
                endlocal
                set "EmptyLine="
            )
        ) else (
            endlocal
            set "IgnoreLines=1"
            set "FileModified=1"
        )
    ) else (
        if "!Line:<---- END FORUMS PASTE ---->=!" == "!Line!" (
            endlocal
        ) else (
            endlocal
            set "IgnoreLines="
            if defined EmptyLine set "EmptyLine=2"
        )
    )
))>"%TempFile%"

if defined FileModified move /Y "%TempFile%" "%SourceFile%" >nul
if exist "%TempFile%" del "%TempFile%"

:EndBatch
rem Restore the initial execution environment with the initial variables list.
endlocal

Please read my answer on How to read and print contents of text file line by line? It explains the reason why using FINDSTR and why using such a difficult code to process the lines of a text file with FOR of Windows command processor cmd.exe not designed for text file modifications at all.

The batch file is designed to ignore also the first empty line below a line containing <---- END FORUMS PASTE ----> if there is an empty line above a line containing <--- BEGIN FORUMS PASTE ---> to avoid having finally two empty lines in the file on a removed block.

Example: The source file contains the lines:

********************
PLAYLIST: 1
********************
    <--- BEGIN FORUMS PASTE --->

Some unnecessary Data
Some unnecessary Data
Some unnecessary Data
......

    <---- END FORUMS PASTE ---->

QUICK SUMMARY:

********************
PLAYLIST: 2
********************
BEGINNING LIST 2 <--- BEGIN FORUMS PASTE --->

Some unnecessary Data
Some unnecessary Data
Some unnecessary Data
.....

<---- END FORUMS PASTE ----> END LIST 2
QUICK SUMMARY:

********************
PLAYLIST: 3
********************

<--- BEGIN FORUMS PASTE --->

Some unnecessary Data
Some unnecessary Data
Some unnecessary Data
.....

<---- END FORUMS PASTE ---->

QUICK SUMMARY:
Required Data

This source file is modified by the batch file to:

********************
PLAYLIST: 1
********************

QUICK SUMMARY:

********************
PLAYLIST: 2
********************
QUICK SUMMARY:

********************
PLAYLIST: 3
********************

QUICK SUMMARY:
Required Data

Leading or trailing spaces/tabs or other characters around <--- BEGIN FORUMS PASTE ---> and <---- END FORUMS PASTE ----> do not matter for the identification of the beginning and the end of a block to remove from source file.

For understanding the used commands and how they work, open a command prompt window, execute there the following commands, and read entirely all help pages displayed for each command very carefully.

• del /?
• echo /?
• endlocal /?
• findstr /?
• for /?
• goto /?
• if /?
• move /?
• pause /?
• rem /?
• set /?
• setlocal /?