batocera-ES-onArch

Emulationstation from batocera with configs and scripts for integration as Arch-Linux package

This project is maintained by GB609

/create-docs.js

This script parses all source files and generates documentation out of them in md style, based on shdoc.

Overview

The document generation process searches for all files of supported types and tries to run them through shdoc to generate a .md file out of it.
The following types are supported:

Javascript code is not supported by shdoc directly. Instead, an adapter is inserted into the pipe between the script file and shdoc. This adapter, coded in js-to-shdoc.js, performs on-the-fly transformation of a restricted set of JSDoc to shell-style comments.

This script provides the following helpers:

  1. Generate a doclet of .md files from defined source directories. This doclet will be placed in tmp/docs.
  2. Integrate such a doclet into the pages build/branch of the repository
  3. Include additional documents and reports, like coverage into the doclet/pages sub-section.

Generation process

  1. Source directories specified by a supplied config file (default sources/page-template/page_sources.json) will be searched for supported source files. These are processed with shdoc to generate ‘User’ and ‘Developer’ manuals.
    • ‘User’ manuals will only be generated for files without extension (such as those in /usr/bin). Functions marked with @internal will not be included. Instead, the output of executable --help is added.
      It is also possible to manually add more custom files here by config file.
    • ‘Developer’ manuals will contain the output of shdoc for all applicable files, including all @internals.
      They will not contain the --help output.
  2. For each input directory, the config file contains up to 2 relative paths of output directories.
    These are for the user and developer manuals. The user manual is optional, developer output path must be given.
    The process described in 1. uses the specified output paths and creates matching subdirectories under tmp/docs
  3. Generic/static parts of the doclet are copied from sources/page-template/.manuals to tmp/docs.
    These parts are mostly index.md files in several directories used for descriptions and to serve as link tables.
  4. index.md files in tmp/docs are searched recursively and will be enhanced by auto-generated TOCs and links to
    the files generated by shdoc.

Doclet integration

  1. A doclet always represents the documentation for ONE specified version or release.
  2. The integration process expects that a doclet has been generated beforehand.
  3. This process takes a version argument. This is required to determine the correct sub-directory of the pages root.
  4. The doclet from tmp/docs is moved to docs/version/<version>, overwriting any existing docs which might
    have been generated for the same version previously.
  5. Files and folders located under sources/page-template/* are copied to docs/. These are version-independent
    index files like the initial welcome page. To make a new version visible, these index files have to be processed
    the same way in which the version-specific index files are enhanced.
  6. Optionally, when a report shall be integrated into a version, some of the index files from
    sources/page-template/.manuals have to be updated and re-generated.

Usage
Set the environment variable DEBUG to any value for extended output.

: `--integrate-all | [--integrate-docs ] [--integrate-reports]` - Help: `create-docs.js --help` It is possible to run all steps from generation to integration at once, but it will fail when reports are selected to be included, but there is no coverage data in `tmp/results`. ## See also * [JS adapter](/batocera-ES-onArch/version/main/dev/tools/js-to-shdoc.sh.html) * [shdoc](https://github.com/GB609/shdoc) ## Index * [Generic helper utilities](#generic-helper-utilities) * [makeDirs](#makedirs) * [options](#options) * [logGroup](#loggroup) * [prepareShDoc](#prepareshdoc) * [cleanSubPathName](#cleansubpathname) * [Source processing](#source-processing) * [findSourceFiles](#findsourcefiles) * [handleOverrides](#handleoverrides) * [processJsFiles](#processjsfiles) * [processShellScripts](#processshellscripts) * [generateMdFiles](#generatemdfiles) * [runShDoc](#runshdoc) * [class LinkDef](#class-linkdef) * [class MdLinkIndex](#class-mdlinkindex) * [class MdHeaderVars](#class-mdheadervars) * [MdHeaderVars.setValue](#mdheadervarssetvalue) * [Index file update helpers](#index-file-update-helpers) * [getLinksRecursive](#getlinksrecursive) * [updateIndexFiles](#updateindexfiles) ## Generic helper utilities Small functions used several times. ### makeDirs Create multiple directories, including any missing parent. ### options Used to build option arguments for child processes. ### logGroup To get structured logging on github runners. ### prepareShDoc Used internally to download a compatible version of shdoc. ### cleanSubPathName The documentation template directory adds more nesting layers to the original file hierarchy. These should not appear in sub path names of files which are linked automatically. This function does not create valid links, it only attempts to clean the 'tmp/doc/../files' part from any given filename. ## Source processing The functions listed in this section deal with locating and processing source files during `*.md` creation. These functions ultimately work on the input from the sources config file supplied at start (or the default). The following example show the expected structure of the config file. ### Example ```bash [ { "root": "sources/fs-root", // base source directory "manTarget": "user/files", // (optional) where to place user manual styled md files "docTarget": "dev/files", // where to place developer docs "manualAdds": { // Force add files to the user manual which are not included by the automatic search "js": { // must be separated by type because of the adapter required for none-sh files }, "sh": { "opt/batocera-emulationstation/user-paths.lib": "Common Paths", "opt/emulatorlauncher/lib/.operations.lib": "Emulatorlauncher Operations" } } }, // *1-n ] ``` ### findSourceFiles Search the given root path for files with one of the extensions and (optional) executables matching `fileType`. ### handleOverrides Registers configured additions to the user manual. ### processJsFiles Entry function for `.js` processing. 1. Locates files according to one entry of the configuration 2. Add manual overrides. 3. Convert found files to `.md` with `shdoc` and an adapter. This function is basically just a simple configuration proxy for [generateMdFiles](#generateMdFiles). ### processShellScripts Entry function for `.sh`|`.lib` processing. 1. Locates files according to one entry of the configuration 2. Add manual overrides. 3. Convert found files to `.md` with `shdoc`. This function is basically just a simple configuration proxy for [generateMdFiles](#generateMdFiles). ### generateMdFiles This function is responsible for the actual generation of .md-files out of all passed in sources. The actual creation of md files is handled by `shdoc` internally, the source file contents are piped into it. This function generates up to 2 files: 1. One goes into the developer manual. This will show all functions, even those marked with `@internal` (lines containing `@internal` are filtered out). 2. Executables meant to be called by the end user additionally go into a dedicated folder for user manuals. The creation of a user manual file is optional. It only happens when the file - does not have an extension OR - was added manually to a hard-coded list of files for the 'user' manual As the no-extension check is inaccurate in some situations, it is possible to override and force-disable user manual generation. To do so, the special comment `#NOUSERMAN#` must be used in the first 2 lines of the file. The generation of user manual files follows slightly different rules: 1. As a first step, it is attempted to call ` --help` to get what the executable would produce on the command line for help. If that command executes without errors and produces any output, a first section in the document will be generated for that. 2. The file is piped into `shdoc` next. Contrary to the generation of developer docs, `@internal` is not filtered away here. **Usage of `shdoc`** This function calls `[runShDoc](#runShDoc)` and just passes through the optional `shdocAdapter` which can be used to transform files with different comment styles into shell-style comments on the fly. ### runShDoc Wrapper around the real execution of `shdoc` for a source file. ## class LinkDef Used during enhancement of index files. Represents one link to another file. Includes some helpers to handle - title - sorting - output format (md-style links are html links) ## class MdLinkIndex A collection of [LinkDef](#class-linkdef). Used to generate TOCs. Provides methods for structured output, TOCs can be created in a way that replicates a file system tree structure. ## class MdHeaderVars This class handles Jekyll Front Matter in md files. It is capable of reading and writing variables to it. However, it does not by itself re-write the source file, but maintains a in-memory copy/buffer of it. This class is used to process the static template files during index file updates. ### MdHeaderVars.setValue Set header value, update source array. Return true if anything was changed. ## Index file update helpers The functions documented in this section are used when the static `index.md` files from `sources/page-template` are processed during the generation of a doclet or its integration into `docs/version`. ### getLinksRecursive Starting from `root`, recursively walks down the directory tree to files which have to appear in a TOC of an index file located at `root`. This function tries to 'stop early' and not include links to files, which are 'closer' to other index files. This function will be executed once per `index.md` in the doc creation target directory, but only if the respective index file contains a marker line with the code ``. This line will be replaced with the generated TOC. Files to link are searched with the following logic/rules: - If a directory != root contains an `index.[md|html]`, the file path is returned. - If an entry of the current dir starts with '_' it is ignored. These are jekyll resources. - If the entry is a directory containing an index.md file, add it to list of links - Otherwise, recurse into the directory - If the entry is a file ending on 'md|html', but NOT an index, add it. This rule picks up the `shdoc`-generated files in the same directory (or sub-directories) of an index file. ### updateIndexFiles 1. Locate index files 2. For each index file, search none-index md|html files closest to it 3. Generate TOCs 4. Write a few front matter variables for jekyll, if the file has a front matter block. Generated with shdoc from [/create-docs.js](https://github.com/GB609/batocera-ES-onArch/blob/befd01c3618b2e08a4da21972df39f296b9774b0 /scripts/create-docs.js)