Prepare a string in a block of parenthesis variable expansion is disabled to return it to a calling scope that enables expansion (no call allowed)

Question

i have a request for theorical knowledge purpose only in Windows Batch scripting.

Today my question is about preparing the value of a local string loc_str INSIDE A BLOCK OF PARENTHESIS and WITHOUT ENABLING DELAYED EXPANSION, in order to return it to a local scope that enables variable expasion (or eventually to prepare it for an incoming local for loop with in("%loc_str%") if nasty characters in loc_str have been properly escaped before).

We assume that loc_str contains at least one ^ and one ! and do not contain double quotes. Lets consider the following code:

@echo off
setlocal disabledelayedexpansion
set "flag_dde_prev=%flag_dde%" & set "flag_dde=!"
(
    set "loc_str=Hello^^^ planet!!!! ^^Earth^"

    if not defined flag_dde_prev (
        call set "out_str=%%loc_str:^=^^%%"
        set out_str

        call set "out_str=%%out_str:^=^^%%"
        set out_str
    )
)
endlocal & set "str=%out_str%"
set str

As you know the local flags flag_{dde, dde_prev} are used to test the type of the calling and current scopes. Typically they're both defined at the beginning of a block setlocal..endlocal . The flag_dde is equal to ! if the current scope disables variable expansion, or is undefined if the current scope enables it. The local value of flag_dde_prev is the inherited value of flag_dde in the calling scope.

What we must do here is to escape all ^ and ! to prepare the return of out_str with %..% to the calling scope when this latter enable variable expansion (ie when flag_dde_prev is defined). Two substitutions ^=^^ then !=^! with a simple set would be enough, but being inside a block (..) forces us to use the call set statement. Unfortunatly the caret ^ is not replaced in the same way they are in a scope that enables variable expansion.

Precisely, the first substitution ^=^^ doubles even sequences of carets, but does not double odd ones (it doubles them minus one). Then, the second substitution !=^! replace each ! with ^^! .

To sump up,

loc_str=Hello^^^ planet!!!! ^^Earth^               ;init
out_str=Hello^^^^^ planet!!!! ^^^^Earth^           ;1st substitution ^=^^
out_str=Hello^^^^^ planet^^!^^!^^!^^! ^^^^Earth^   :2nd substitution !=^!

^=^^ seems to be reversible with ^^=^ if done BEFORE !=^! only. If ^^=^ is done after !=^!then sequences of ^ not preceeding ! are modified only. If ^^=^ is done after !=^!i didn't find a way to replace the sequences of ^^ before each " . It behaves like a sequence ^^ before ! is the atomic unreplacable one, longer caret sequences can be replaced but i couldn't obtain ^! whatever i tried.

The same problem is quite easy to solve when variable expansion is enabled, even with nasty strings containing quotes by replacing "" by " at first (jeb already talked about this in another thread). For example if locs_tr doesn't contain quotes, the substitutions would be the following:

set "out_str=!loc_str:^=^^^^!"              ;1st substitution, multiplicates the number of ^ by 4 as expected
call set "out_str=%%out_str:^!=^^^!%%" !    ;2nd substitution, replaces each ! by ^^!
set "out_str=!out_str:^^=^!"                ;3rd substitution required, replaces each ^^ by ^ to divide by 2 the total number of ^

So my question is simple: Is there a way using substitution IN THE EXACT SAME CONTEXT (ie no call ) to obtain the desired prepared output value for out_str , so it can be returned safely to a scope that enables delayed expansion ?

The prepared output value should double the number of ^ and add ^ before each ! ie:

out_str=Hello^^^^^^ planet^!^!^!^! ^^^^Earth^^

Note that's my question is for theorical knowledge purpose ONLY. Indeed calling a label to do the two substitutions with a simple set is the reasonable way to proceed.

Answer 1

Your assumptions are wrong!

What we must do here is to escape all ^ and ! to prepare the return ...
Two substitutions ^=^^ then !=^! with a simple set would be enough ...

The problem is here the fact that caret folding is different if there is an exclamation mark in the line or not!

setlocal EnableDelayedExpansion
echo "one caret^"
echo "no caret^ but a bang^!"
echo "caret ^^ and a bang^!"
echo one caret ^^
echo no caret^^ but a bang^^!
echo caret ^^^^ and a bang^^!

call and carets and exclamation marks are really funny.

Because call always double all carets before any other rule, but you don't see it with quotes, because in the next parser phase the doubled carets are folded to single carets again.

Parse flow for a single line like

Starting with:
   call echo my caret^^^^ "caret^^" and a bang !

1. Caret escaping outside quotes
   call echo my caret^^ "caret^^" and a bang !

2. delayed expansion caret escaping (outside and **inside** quotes), the bang itself is lost here
   call echo my caret^ "caret^" and a bang

3. The `CALL` caret doubling
   call echo my caret^^ "caret^^" and a bang

4. Caret escaping outside quotes
   call echo my caret^ "caret^^" and a bang

The problem itself was solved by dbenham and me a few years ago.
See SO: Macro to preserve variables when leaving setlocal scope
or SO: Make an environment variable survive ENDLOCAL .

In your case it's much simpler, because you don't support quotes nor line feeds in the variable.
And you know, that you leave the current scope and will return into a scope with delayed expansion enabled, this simplifies the problem, too.

@echo off

setlocal EnableDelayedExpansion

setlocal disabledelayedexpansion
(
    set "loc_str=Hello^^^ planet!!!! ^^Earth^"
    set "loc_str"

    setlocal EnableDelayedExpansion
    set "out_str=!loc_str:^=^^^^!"              # 1st substitution, multiply all carets by four
    set "out_str"
    call set "out_str=%%out_str:^!=^^^!%%" !  # 2nd substitution, replace all exclamation marks to two carets with exclam
    set "out_str"
    set "out_str=!out_str:^^=^!"              # 3rd substitution reduce all carets by factor of two
    set "out_str"
    for /F "delims=" %%V in ("!out_str!") do (
        endlocal 
        endlocal
        set "str=%%V" !  # The trailing exclamation mark is intentionally, but will not part of the result
    )
    
)

echo str = '!str!'

Answer 2

Ok jeb i just executed your macro. The code below can be copied and run. If i understood properly, the general principle is the following (correct me if i didn't).

@echo off
setlocal enableextensions disabledelayedexpansion
set "_(echo=echo: & echo"
set "_(set=echo: & set"

:: Initialize the DDE flags of the current/previous scopes.
set "flag_dde_prev=" & set "flag_dde=!"

:: Base string ; one of your nasty strings defined in another post ;p
set "str_base=caret ^ bang ! and some (other ) <>&| %% \"

setlocal enabledelayedexpansion
set "flag_dde_prev=!flag_dde!" & set "flag_dde=!"

  setlocal enabledelayedexpansion
  set "flag_dde_prev=!flag_dde!" & set "flag_dde=!"

  :: Nasty local string to be returned ; still one of yours !
  set "loc_str_nasty=!str_base! and also quoted stuff "!str_base!""
  set loc_str & echo:

  if defined flag_dde_prev (
    set "out_str=!loc_str_nasty!"
  ) else (
    rem STEP1: Each " is replaced by ""q so everything is consider outside quotes
    set "out_str=!loc_str_nasty:"=""q!"
    set out_str

    rem STEP2: Each ^ is replaced with ^^
    set "out_str=!out_str:^=^^!"
    set out_str

    rem STEP3: Each ! is replaced by ""e! 
    rem WARN A simple `set` must be avoided because of special characters.
    call set "out_str=%%out_str:^!=""e^!%%"
    set out_str

    rem STEP4/5: Initial " and ^! were marked differently with letters q and e so we can restore them by replacing ""q and ""e! by " and ! respectively
    set "out_str=!out_str:""e=^!"
    set out_str
    set "out_str=!out_str:""q="!"
    set out_str
  )
    
  rem Return the value of out_str to the calling DDE/EDE scope by using a safe `for`.
  for /F "delims=" %%@ in ("!out_str!") do endlocal & set "str_pro=%%@"
    
  :: Echo safely the protected string str_pro in the current DDE/EDE scope
  %_(set% str_pro

  if defined flag_dde (
    for /F "tokens=1* delims==" %%v in ('set str_pro') do echo %%str_pro%%=%%w
  ) else (
    %_(echo% ^^^!str_pro^^^!=!str_pro!
  )

endlocal

Thanks for your remarks, indeed i talked too fast there is no problem at all with ""q and ""e in the input value of loc_str. You're very right about avoiding special characters with rem , i totally forgot. I did typing mistake also by assigning flag_dde_prev, it must be done with !..! inside an EDE scope of course. I struggle with the code editor here, i'm not used to such stuffs for now (yes i know i sound like a cavern man lol).

I edited my previous post, the protection block (..) is skipped when returning to a DDE scope (i forgot also to assign out_str when flag_dde_prev when defined to !). The returned protected string to DDE contains nasty characters so can't be echoed with %..% , so i used a for to echo it.

Is there a way to deal with " and &|<> to protect a such nasty loc_str to echo it in the calling DDE scope with a brutal echo %..% or call set %..% , or at the contrary is an undirect echoing can't be avoided ?

By undirect i mean without a slow echo only ie with a for like above, or with a very fast set str_pro only by deleting the leading string "str_pro=" by adding a dynamic leading string of backspace in the out_str value.

OUTPUT Run twice, first when the calling scope is DDE, second when it's EDE (written as EDE in the code above).