Sphinx documentation and links to Markdown

Question

I'm trying to use Sphinx to build some documentation from Markdown source. My conf.py is as follows...

conf.py

from recommonmark.parser import CommonMarkParser
project = 'DS'
copyright = '2018, DS'
author = 'DS, Work'

version = ''
release = ''

extensions = []

templates_path = ['_templates']

source_suffix = ['.rst', '.md']

master_doc = 'index'

language = None
exclude_patterns = []
pygments_style = 'sphinx'
html_theme = 'classic'
html_static_path = ['_static']
source_parsers = {
   '.md': CommonMarkParser,
}
htmlhelp_basename = 'DSDocumentationdoc'
latex_elements = {
}
latex_documents = [
    (master_doc, 'DSDocumentation.tex', 'DS Documentation',
     'DS, Work', 'manual'),
]
man_pages = [
    (master_doc, 'dsdocumentation', 'DS Documentation',
     [author], 1)
]
texinfo_documents = [
    (master_doc, 'DSDocumentation', 'DS Documentation',
     author, 'DSDocumentation', 'One line description ofproject.',
 'Miscellaneous'),
]

index.rst

Welcome to DS Documentation!
======================================

The following documentation is produced and maintained by the Data Science team.

Contents:

.. toctree::
   :maxdepth: 2
   :glob:

   README.md
   documentation.md
   getting_started/*
   how-tos/*
   statistics_data_visualisation.md

The documents build and html output is generated, however README.md has links to other markdown documents in the two sub-directories such as the following...

... [this document](./getting_started/setting_your_machine_up.md)...

...which in the translated README.html document the target has not been converted to the translated HTML target as its been recognised as reference external ...

...<a class="reference external" href="./getting_started/setting_your_machine_up.md">this document</a>...

...I was half-expecting/hoping it would output as reference internal and convert the file extension approrpiately...

...<a class="reference internal" href="./getting_started/setting_your_machine_up.html">this document</a>...

...so that links worked in the same vein as the Table of Contents does in the sidebar.

Any suggestions as to whether this can be achieved would be appreciated.

Cheers.

EDIT

Trying out the solution suggested by @waylan I have added the following to by conf.py to enable_auto_doc_ref ...

def setup(app):
    app.add_config_value('recommonmark_config', {
        'enable_auto_doc_ref': True,
    }, True)
    app.add_transform(AutoStructify)

...and on running make html I get the following error.....

❱ cat /tmp/sphinx-err-57rejer3.log 
# Sphinx version: 1.8.0
# Python version: 3.6.6 (CPython)
# Docutils version: 0.14 
# Jinja2 version: 2.10
# Last messages:
#   building [mo]: targets for 0 po files that are out of date
#   
#   building [html]: targets for 16 source files that are out of date
#   
#   updating environment:
#   
#   16 added, 0 changed, 0 removed
#   
#   reading sources... [  6%] README
#   
# Loaded extensions:
#   sphinx.ext.mathjax (1.8.0) from /home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/ext/math
jax.py                                                                                                                                
#   alabaster (0.7.11) from /home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/alabaster/__init__.py
Traceback (most recent call last):
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/cmd/build.py", line 304, in build_ma
in                                                                                                                                    
    app.build(args.force_all, filenames)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/application.py", line 341, in build
    self.builder.build_update()
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 347, in 
build_update                                                                                                                          
    len(to_build))
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 360, in 
build                                                                                                                                 
    updated_docnames = set(self.read())
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 468, in 
read                                                                                                                                  
    self._read_serial(docnames)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 490, in 
_read_serial                                                                                                                          
    self.read_doc(docname)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/builders/__init__.py", line 534, in 
read_doc                                                                                                                              
    doctree = read_doc(self.app, self.env, self.env.doc2path(docname))
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/io.py", line 318, in read_doc
    pub.publish()
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/docutils/core.py", line 218, in publish
    self.apply_transforms()
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/docutils/core.py", line 199, in apply_trans
forms                                                                                                                                 
    self.document.transformer.apply_transforms()
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/sphinx/transforms/__init__.py", line 90, in
 apply_transforms                                                                                                                     
    Transformer.apply_transforms(self)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/docutils/transforms/__init__.py", line 171,
 in apply_transforms                                                                                                                  
    transform.apply(**kwargs)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 325, in ap
ply                                                                                                                                   
    self.traverse(self.document)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 297, in tr
averse                                                                                                                                
    self.traverse(child)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 297, in tr
averse                                                                                                                                
    self.traverse(child)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 297, in tr
averse                                                                                                                                
    self.traverse(child)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 287, in tr
averse                                                                                                                                
    newnode = self.find_replace(c)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 267, in fi
nd_replace                                                                                                                            
    newnode = self.auto_doc_ref(node)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/transform.py", line 175, in au
to_doc_ref                                                                                                                            
    return self.state_machine.run_role('doc', content=content)
  File "/home/neil.shephard@samba.sheffield.thefloow.com/.local/lib/python3.6/site-packages/recommonmark/states.py", line 134, in run_r
ole                                                                                                                                   
    content=content)
TypeError: 'NoneType' object is not callable

I've looked through the last two calls and I think this might be down to content not being set, which may be something to do with my index.rst but I'm really out of my depth here.

Answer 1

The recommonmark documentation suggests enabling AutoStructify by adding the following to your config.py file:

from recommonmark.transform import AutoStructify

github_doc_root = 'https://github.com/rtfd/recommonmark/tree/master/doc/'
def setup(app):
    app.add_config_value('recommonmark_config', {
            'url_resolver': lambda url: github_doc_root + url,
            'auto_toc_tree_section': 'Contents',
            }, True)
    app.add_transform(AutoStructify)

This will give you the following features:

• enable_auto_toc_tree : whether enable Auto Toc Tree feature.
• auto_toc_tree_section : when enabled, Auto Toc Tree will only be enabled on section that matches the title.
• enable_auto_doc_ref : whether enable Auto Doc Ref feature.
• enable_math : whether enable Math Formula
• enable_inline_math : whether enable Inline Math
• enable_eval_rst : whether Embed reStructuredText is enabled.
• url_resolver : a function that maps a existing relative position in the document to a http link

Of note is the Auto Doc Ref feature:

It is common to refer to another document page in one document. We usually use reference to do that. AutoStructify will translate these reference block into a structured document reference. For example

 [API Reference](api_ref.md) 

will be translated to the AST of following reStructuredText code

 :doc:`API Reference </api_ref>` 

And it will be rendered as API Reference

Why is this necessary? Because, unlike Rst, Markdown does not have any knowledge of anything outside of the given document and has no support for Rst style directives. Therefore, there is no mechanism to transform a URL.

Instead, AutoStructify waits until after the recommonmark bridge converts the Markdown to Sphinx's underlying document structure (docutils document object), then it runs a series of transformers on it to provide limited Rst like functionality. Even with AutoStructify, you will never get full feature support when using Markdown. That would require Markdown to have native support for directives, which is not likely to ever happen.