How pandoc lua filters can be used to add the title as the level-1 header?

Question

I'd like to use pandoc lua filter when I convert multiple markdown files from markdown to pdf. I'd like the titles of individual markdown files to be used as chapters (first-level headers).

I learned the existing examples, and I think this one is close to what I need - basically I need to add pandoc.Header(1, doc.meta.title) to all my markdown files, however I'm struggling to write the lua filter and make it work.

I think this question is doing a similar action pandoc filter in lua and walk_block

The pandoc command:

pandoc -N --lua-filter add-title.lua blog/*.md --pdf-engine=xelatex --toc -s -o my_book.pdf

The add-title.lua (this is just wrong, no exceptions but nothing happens to the output):

function add_header (header)
return {
  {Header = pandoc.Header(1, meta.title)}}
end

Input files:

1.md

---
title: Topic1
---

## Sample Header from file 1.md

text text text

2.md

---
title: Topic2
---

## Sample Header from file 2.md

text text text

Expected output equivalent to this markdown (but my final format is pdf)

---
title: Title from pandoc latex variable
---
# Topic1

## Sample Header from file 1.md

text text text

# Topic2

## Sample Header from file 2.md

text text text

Answer 1

I think the key problem is that the lua filters only run once the full set of documents have been parsed into a single AST. So the individual files are effectively concatenated prior to parsing to create a single document with a single set of metadata. The individual title settings in the yaml metadata blocks are being overridden before the filter has a chance to run. Assuming that you need to get the heading from each separate metadata block (and can't just put the header in directly) this means that you cannot let pandoc join the files. You will need to read and parse each file separately. Fortunately this is pretty easy with filters.

The first step is to make a single reference file that contains links to all of the other files.

---
title: Combined title
---

![First file](1.md){.markdown}

![Second file](2.md){.markdown}

Note that the links are specified using images with a special class .markdown . You could use some other method, but images are convenient because they support attributes and are easy to recognise.

Now we just need a filter that will replace these images with the parsed elements from the linked markdown file. We can do this by opening the files from lua, and parsing them as complete documents with pandoc.read (see https://www.pandoc.org/lua-filters.html#module-pandoc ). Once we have the documents we can read the title from the metadata and insert the new header. Note that we apply the filter to a Para element rather than the Image itself. This is because pandoc separates Block elements from Inline elements, and the return value of the filter must be of the same type. An Image filter cannot return the list of blocks parsed from the file but a Para can.

So here is the resulting code.

function Para(elem)
    if #elem.content == 1 and elem.content[1].t == "Image" then
      local img = elem.content[1]    
      if img.classes:find('markdown',1) then
        local f = io.open(img.src, 'r')
        local doc = pandoc.read(f:read('*a'))
        f:close()

        -- now we need to create a header from the metadata
        local title=pandoc.utils.stringify(doc.meta.title) or "Title has not been set"
         local newHeader=pandoc.Header(1, {pandoc.Str(title)})
        table.insert(doc.blocks, 1, newHeader) 
        return doc.blocks
      end
    end
end

If you run it on the combined file with

pandoc -f markdown -t markdown -i combined.md  -s --lua-filter addtitle.lua

you will get

---
title: Combined title
---

Topic 1
=======

Sample Header from file 1.md
----------------------------

text text text

Topic 2
=======

Sample Header from file 2.md
----------------------------

text text text

as required.

Note that any other yaml metadata in the included files is lost. You could capture anything else by taking it from the individual meta object and placing it into the global one.