Parsing text from a table of contents using regex

Question

Following is the text I'd like to parse, stored in variable named "toc"

                                  Table of Contents
I.   INTRODUCTION          ....................................                                  1
II.  FACTUAL ASPECTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .     2
     A.   The Clean Air Act . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  3
     B.   EPA's Gasoline Rule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .    3
          1.     Establishment of Baselines . . . . . . . . . . . . . . . . . . . . . . . .      3
          2.     Reformulated Gasoline . . . . . . . . . . . . . . . . . . . . . . . . . .       4
          3.     Conventional Gasoline (or "Anti-Dumping Rules") . . . . . . . .                 4
     C.   The May 1994 Proposal . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .      5
III. MAIN ARGUMENTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .      5
     A.   General          ....................................                                  5
     B.   The General Agreement on Tariffs and Trade . . . . . . . . . . . . . . . .             6
          1.     Article I - General Most-Favoured-Nation Treatment . . . . . . .                6
          2.     Article III - National Treatment on Internal Taxation
                 and Regulation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .    7
                 a)      Article III:4 . . . . . . . . . . . . . . . . . . . . . . . . . . . .   7
                 b)      Article III:1 . . . . . . . . . . . . . . . . . . . . . . . . . . . .  14
          3.     Article XX - General Exceptions . . . . . . . . . . . . . . . . . . . .        15
          4.     Article XX(b) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  15
                 a)      "Protection of Human, Animal and Plant Life
                         or Health" . . . . . . . . . . . . . . . . . . . . . . . . . . . . .   15
                 b)      "Necessary" . . . . . . . . . . . . . . . . . . . . . . . . . . . .    15
          5.     Article XX(d) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  21
          6.     Article XX(g) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .  22
                 a)      "Related to the conservation of exhaustible natural
                         resources..." . . . . . . . . . . . . . . . . . . . . . . . . . . . .  22
                 b)      "... made effective in conjunction with restrictions
                         on domestic production or consumption" . . . . . . . . . .             23
          7.     Preamble to Article XX . . . . . . . . . . . . . . . . . . . . . . . . . .     23
          8.     Article XXIII - Nullification and Impairment . . . . . . . . . . . .           25

I want the result like this:

['I.INTRODUCTION ...... 1', 'A. The Clean Air Act ....3', 'B. EPA\'s Gasoline Rule ... 3', (AND_SO_ON) ]

INPUT:

re.search(r"((?<=(\n))\s+(?P<name>[A-Z \.]*?)(\n))", toc_s).group() 

OUTPUT:

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-64-4aa240f6e378> in <module>()
----> 1 re.search(r"((?<=(\n))\s+(?P<name>[A-Z \.]*?)(\n))", toc_s).group()

AttributeError: 'NoneType' object has no attribute 'group'

What was my problem?

Answer 1

Let's assume this entire TOC content resides in a multiline string text . You can use re.findall or re.finditer with the re.MULTILINE switch enabled;

for match in re.finditer('(.*?)[\W]+(\d+)(?=\n|$)', text, flags=re.M):
     chapter, page = map(str.strip, match.groups())
     ... # do something with these

Or,

contents = re.findall('(.*?)[\W]+(\d+)(?=\n|$)', text, flags=re.M)

Which returns something along these lines -

[('I.   INTRODUCTION', '1'),
 ('II.  FACTUAL ASPECTS', '2'),
 ('     A.   The Clean Air Act', '3'),
 ("     B.   EPA's Gasoline Rule", '3'),
 ('          1.     Establishment of Baselines', '3'),
 ('          2.     Reformulated Gasoline', '4'),
 ...
]

A list of 2-tuples. Each tuple has a) the chapter, and b) the corresponding page number. If a line is not matched by the pattern, it is, of course, ignored.

Details

The pattern is very specific, and requires some trial-and-error.

(         # first capture group - the chapter name
  .*?     # non-greedy match 
)
[\W]+     # match characters that are not alphanumeric
(         # second capture group - the page number
  \d+     # one or more digits
)
(?=       # lookahead for a newline or EOL (multiline)
  \n      # literal newline
  |       # regex OR
  $       # EOL
)