Converting Silo's docs from FrameMaker to What? reStructuredText, Markdown, ...

[markcmiller86]

I am investigating converting Silo's FrameMaker API reference manual to something better that supports docs-like-code workflow. The two I am most familiar with are reStructuredText and Markdown. I've used both (though not so much for API reference documentation) and prefer Markdown because its simpler. But, am open to suggestions at this point too.

Some of the key considerations...

• conversion process (which will likely require a Python script) is simple.
  • I've taken a quick look and think best approach is to export XML from FrameMaker, read that into an OrderedDict in Python and then iterate and recurse over that to output a separate .md file for each of the main chapters (API sections) in the manual.
  • How will table of contents be handled? Maybe manually
• Formatting needs are met
  • I haven't yet seen anything in the manual that markdown doesn't approximate well
• Math equations can be handled (they can...via Latex even with Jekyll+Kramdown)
• Creating PDF
  • Adobe Acrobat Pro has a feature to create a PDF from a URL (page tree).
  • Create this and add to each release.
• Hosting on air-gapped networks
  • If tooling dependencies to produce once there is problematic, then render to HTML somewhere else and copy over
  • Host from a file:// URL
  • Also include the PDF file
• Images and Diagrams
  • We have a few of these. What is the source format for these? Currently, this is FrameMaker drawings. Can we convert to PowerPoint drawings?
• Stylized typing for some words (e.g. typeface, bold, italics)
  • Might have to make a manual pass over converted docs to fix any issues resulting from conversion.
• Links and cross-refs within the documentation
  • python converter script needs to handle the common cases
  • Sometimes would be nice to just link to actual code instead of cut-n-pasting copies of code. Might be able to render in an <iframe> but there is no include directive in Markdown I don't think like there is in reStructuredText.
• Pagination...we want each API func on its own page (kinda sorta)
  • I think we just need to be sure each API func has an anchor
  • markdown generates anchors automatically for level 1 through 6 headings
  • Might want to put API function short descriptions on their own line then instead of in the heading
• Column spacing in tables (the one below doesn't look very good)
  • I think you just add a number   to table header to force spacing.
• Local preview
  • Chrome browse has a markdown viewer extension that works ok
  • May need full jekyll tooling (for sure if you wanna see math equations rendered)
• Would like to stick to one sentence per line but when I do that (below), it doesn't seem to be meging adjacent lines to a single paragraph.
• Basic GitHub pages vs. Jekyll based vs. ReadTheDocs (which now supports markdown)
  • Basic GitHub pages is desireble because its simple...the doc tree is small and simple too
  • ReadTheDocs handles things like equations but has issues with horiz. scrolling of tables

Below, I've manually converted two functions in the API reference manual to Markdown. They look pretty good to me. You can compare to the current FrameMaker results for DBErrfuncname and DBShowErrors.

DBErrfuncname — Get name of error-generating function

Synopsis:

char const *DBErrfuncname (void)

Fortran Equivalent:

None

Returns:

A char const * containing the name of the function that generated the last error. This call cannot fail.

Description:

The DBErrfuncname function is used to find the name of the function that generated the last Silo error. It is implemented as a macro.

DBShowErrors — Set the error reporting mode.

Synopsis:

void DBShowErrors (int level, void (*func)(char*))

Fortran Equivalent:

integer function dbshowerrors(level)

Arguments:

• level: Error reporting level. One of DB_ALL, DB_ABORT, DB_TOP, or DB_NONE.
  Adding some extra text here to see how it handles this in a bulleted list.
• func: Function pointer to an error-handling function.

Returns:

Returns nothing (void). It cannot fail.

Description:

The DBShowErrors function sets the level of error reporting done by Silo when it encounters an error.
The following table describes the action taken upon error for different values of level.
Ordinarily, error reporting from the HDF5 library is disabled.
However, DBShowErrors also influences the behavior of error reporting from the HDF5 library.

Error level value	Error action
DB_ALL	Show all errors, beginning with the (possibly internal) routine that first detected the error and continuing up the call stack to the application.
DB_ALL_AND_DRVR	Same as DB_ALL execpt also show error messages generated by the underlying driver library (PDB or HDF5).
DB_ABORT	Same as DB_ALL except abort is called after the error message is printed.
DB_TOP	(Default) Only the top-level Silo functions issue error messages.
DB_NONE	The library does not handle error messages. The application is responsible for checking the return values of the Silo functions and handling the error.

[markcmiller86]

@cyrush if you have a moment and any thoughts you'd be willing to share regarding either anything I am missing in thinking about this or in what you think best approaches are, that would be great.

[cyrush]

@markcmiller86 would sphinx do the job?

[markcmiller86]

@markcmiller86 would sphinx do the job?

It definitely would. But, I find it slower to write with and harder to use than Markdown. I don't tend to write in reStructuredText but for the VisIt project. Spellcheckers, linkcheckers (and I worry other quality checks I haven't considered) seem to be less prevalent and more combersome for Sphinx. Preview requires local tooling too whereas its somewhat easier for Markdown

[markcmiller86]

@markcmiller86 would sphinx do the job?

@cyrush perhaps I did not initially undersand your question here...were you suggesting to just go ahead and use sphinx with markdown?

[markcmiller86]

I didn't know sphinx supported that but it apparently does.

[markcmiller86]

Capturing part of a conversion script here...

#  \r \n   D   B   E   r   r   n   o  \r  \n   —  **  ** 
# DBShowErrors
# —Set the error reporting mode.
# Synopsis:
# void DBShowErrors (int level, void (*func)(char*))
# Fortran Equivalent:
# integer function dbshowerrors(level)
# Arguments:
# level
# Error reporting level. One of DB_ALL, DB_ABORT, DB_TOP, or DB_NONE.
# func
# Function pointer to an error-handling function.
# Returns:
# DBShowErrors returns nothing (void). It cannot fail.
# Description:
# The DBShowErrors function sets the level of error reporting done by Silo when it encounters an error. The following table describes the action taken upon error for different values of level.
# Ordinarily, error reporting from the HDF5 library is disabled. However, DBShowErrors also influences the behavior of error reporting from the HDF5 library.
# Error level value       DB_ALL  DB_ALL_AND_DRVR DB_ABORT        DB_TOP  DB_NONE
# Error action    Show all errors, beginning with the (possibly internal) routine that first detected the error and continuing up the call stack to the application.      Same as DB_ALL execpt also show error messages generated by the underlying driver library (PDB or HDF5).        Same as DB_ALL except abort is called after the error message is printed.       (Default) Only the top-level Silo functions issue error messages.       The library does not handle error messages. The application is responsible for checking the return values of the Silo functions and handling the error.

import re

# 
# Get all documentation lines from text file
#
def GetFileLines(fname):
    with open(fname) as txtfile:
        return txtfile.readlines()

#
# Begin a new API section (new md file)
#
def StartNewSection(i, s, lines):
    mdfile = open("Chapter2-Section%d.md"%s,"wt")
    s += 1
    hs = re.search(r'^[0-9]*\s*API Section\s*(.*)',lines[i])
    if len(hs.groups()) > 0:
        mdfile.write("## %s"%hs.groups()[0])
    else:
        mdfile.write("## Unknown")
    mdfile.write("\n\n")
    i += 1
    while i < len(lines) and \
        not re.search(r'^_visit_defvars\s*275$', lines[i]) and \
        not re.search(r'^dbmkptr\s*283$', lines[i]) and \
        not re.search(r'^Silo.Open$', lines[i]) and \
        not re.search(r'^DBGetComponentNames$', lines[i]) and \
        not re.search(r'^DB[A-Za-z0-9]*\s*[0-9]*$', lines[i]):
        as_sentences = re.sub(r'(?!\.\.)([\.!?]\s*(?!\)))',r'\1\n',lines[i])
        mdfile.writelines(as_sentences)
        i += 1
    mdfile.write("\n")
    return i, s, mdfile

"""
       (re.search(r'^DB[A-Za-z0-9_]*$', lines[i]) or \
        re.search(r'^json-c extensions$', lines[i]) or \
        re.search(r'^<DBfile>\.', lines[i]) or \
        re.search(r'^PMPIO_[A-Za-z]*', lines[i]) or \
        re.search(r'^_visit_[a-z_]*$', lines[i]) or \
        re.search(r'^AlphabetizeVariables$', lines[i]) or \
        re.search(r'^ConnectivityIsTimeVarying$', lines[i]) or \
        re.search(r'^MultivarToMultimeshMap_vars$', lines[i]) or \
        re.search(r'^MultivarToMultimeshMap_meshes$', lines[i]) or \
        re.search(r'^dbmkptr$', lines[i]) or \
        re.search(r'^dbrmptr$', lines[i]) or \
        re.search(r'^db.et2dstrlen$', lines[i]) or \
        re.search(r'^dbwrtfl$', lines[i]) or \
"""

def IsMethodHeader(i, lines):
    if i < len(lines)-1 and re.search(r'^—', lines[i+1]):
        return True
    return False

def ProcessSynopsis(mdfile, i, lines):
    i += 1
    mdfile.write("#### C Signature\n")
    mdfile.write("```\n")
    mdfile.write(lines[i])
    i += 1
    while i < len(lines) and not \
        (re.search(r'^Fortran Equivalent:$', lines[i]) or \
         re.search(r'^Returns:$', lines[i]) or \
         re.search(r'^Arguments:$', lines[i])):
        indented_line = '    ' + lines[i].strip()
        mdfile.write(indented_line)
        mdfile.write("\n")
        i += 1
    mdfile.write("```\n")
    if re.search(r'^Fortran Equivalent:$', lines[i]):
        i += 1
        if re.search(r'^None$', lines[i]):
            mdfile.write("#### Fortran Signature: None\n")
            i += 1
        else:
            mdfile.write("#### Fortran Signature\n")
            mdfile.write("```\n")
            while i < len(lines) and not \
                (re.search(r'^Returns:$', lines[i]) or \
                 re.search(r'^Arguments:$', lines[i])):
                indented_line = '    ' + lines[i].strip()
                mdfile.write(indented_line)
                mdfile.write("\n")
                i += 1
            mdfile.write("```\n")
    return i

def ProcessMethod(mdfile, i, lines):
    mdfile.write("### `%s()` - %s\n"%(lines[i][:-1], lines[i+1][1:]))
    i += 2
    while i < len(lines) and not IsMethodHeader(i, lines):
        if re.search(r'^Synopsis:$', lines[i]):
            i = ProcessSynopsis(mdfile, i, lines)
            mdfile.write("\n")    
        else:
            i += 1
    return i

#
# main program
#
lines = GetFileLines("Chapter2-man_pages2.txt")

s = 0
for i in range(len(lines)):
    if re.search(r'^[0-9]* API Section', lines[i]):
        i, s, mdfile = StartNewSection(i, s, lines)
    if IsMethodHeader(i, lines):
        i = ProcessMethod(mdfile, i, lines)

[markcmiller86]

I have started the work in the mcm86-27jan22-convert-docs-fm2md branch. So far, I've got it converting...

• API Section headings and intro text
• Function headings and C/fortran signatures
• Argument list blocks
• Return value block
• Options tables
• Other tables
• Some figures
• Descriptions
• cross-references
• type styling

[markcmiller86]

Pick a ghpages theme..

Hack	Slate	Leap Day	Cayman
.

[markcmiller86]

@brugger1 might you have anything to recommend or suggest here?

[brugger1]

I think Hack is probably the best option. I'm also not a huge fan of the coloring scheme. I think it would look better if the colors were lighter, more like the grays in Slate. The highlighting colors seem fine.

[markcmiller86]

Thanks @brugger1 for look over. I've also tested creating a PDF from the rendered html. This was just using some simplistic workflow.

Couple things I don't like...

• It wouldn't be good for actual printing due to black background color
• It doesn't start new methods on page boundaries

So, the above experiences suggest Sphinx+ReadTheDocs is potentially a better path. The conversion script (to get this far) would be about the same coding/logic, just different ascii formatting.

[brugger1]

Yes, I agree with the 2 down sides you mentioned. I'd see what the Sphinx pdfs look like as well, those sometimes have some issues, although I think for the Silo manual it would probably look good since you probably don't have screen shots or lots of images.

[markcmiller86]

Yeah, took a look at VisIt's manual as PDF downloaded from RTD. Its way better than what I produced, above, for what I've done with Silo's manual here so far but it also suffers from some issues...mostly related I think to unusually large native image sizes. It doesn't know what to do with these and produces many pages with wasted vert. space. It also doesn't appear to do anything to start major headings on new pages.

[markcmiller86]

Ok, got markdown to render on RTD too.

[markcmiller86]

It looks like RTD's PDF version is broken...it does't show the whole tree, just the top page, all the content on index.md. I wonder if this could have to do with the use of markdown instead of reStructuredText.

[markcmiller86]

I have an solution for producing the PDF from the markdown. It results in a PDF file size that is 10x what I would expect (and what I've had in the past) and I don't get bookmarks for each function so I still need to figure those issues out.

• cat all the .md files together to a single .md file.
• sed replace some keywords to add embedded HTML for page breaks ahead of each major section or function
• Open in a markdown-capable browser (Chrome with markdown extension)
• Print to a PDF file