上下文无关的语法来识别行尾空格

Question

I am trying to write a context-free grammar to do something very simple—to parse a string into a list of alternating sections of (1) end-of-line whitespace and (2) everything else. For example:

This.first.line...\n..and.this....second.line\n.\n..and.final.line

(showing " " as "." and newlines as "\\n" for readability) is parsed as

"This.first.line", "...\n..", "and.this....second.line", "\n.\n..", "and.final.line"

I wrote this grammar:

string = raw_start | newline_start
raw_start = raw_section [newline_start]
newline_start = newline_section [raw_start]
raw_section = {any_character_except_newline}
newline_section = {whitespace_except_newline} new_line {any_whitespace_character}

But this is not correct because the {any_character_except_newline} will consume the spaces leading up to newlines, when I want those included with the new_line_section .

Is it possible to say "Consume spaces, unless they are right before a newline" without losing the context-free property of the grammar?

Answer 1

Sure, context-free is not a problem. Both "end-of-line whitespace" and "everything else" are regular languages.

For reference, here are the regular expressions (formally regular, not "recognizable with some 'regex' package"). We suppose that A is the alphabet, and define:

 = { ∀x | x ∈  ∧ x ≠ NL ∧ x ≠ SPACE }
   = { ∀x | x ∈  ∧ x ≠ NL }
 = { xωy | x,y ∈  ∧ ω ∈ * } ⋃ 
 = { ωNLγ | ω,γ ∈ {SPACE, NL}* }

That can easily be transformed into a CFG. (It's possible that the text ends with whitespace which doesn't include a newline. The following ignores that possibility, but it could easily be added):

S → Spaces
S → S Other
S → S EOL_WS
Spaces → ε
Spaces → Spaces [ ]
Other → [^ \n] Line [^ \n]
Other → [^ \n]
Line → ε
Line → Line [^\n]
EOL_WS → Spaces NL_Spaces
NL_Spaces → NL_Space
NL_Spaces → NL_Spaces NL_Space
NL_Space → [/n] Spaces
 

As written, the above is ambiguous because it does not insist that Other and EOL_WS be maximally long. That's easy to fix but tedious, and since the OP only asks for a CFG and not an unambiguous or LR(1) CFG, I'll leave it at that.

Answer 2

This is a translation of rici's great answer into the EBNF format I used in my question:

string = raw_start | newline_start
raw_start = raw_section [newline_start]
newline_start = newline_section [raw_start]
raw_section = any_nonwhite_character [{any_character_except_newline} any_nonwhite_character]
newline_section = {whitespace_except_newline} new_line {any_whitespace_character}

The key was changing the definition of raw_section to require that it end with a nonwhite character. This simple grammar will not match empty strings or strings that end with a space, but that is easy to fix.