RFC: Formatted Comments As Documentation

[Ratstail91]

Inspired by a comment from @hiperiondev, I'm thinking about reworking how my documentation system is set up.

Currently, I'm using https://toylang.com as the primary source of documentation, with liberal application of comments within the code itself. However, I'm considering reworking this so that the website actually derives it's content from formatted comments within the code.

An example would be like so:

/*!
### void Toy_initLexer(Toy_Lexer* lexer, const char* source)

This function initializes a lexer, binding it to the source parameter; the lexer is now ready to be passed to the parser.
*/
TOY_API void Toy_initLexer(Toy_Lexer* lexer, const char* source);

Which, when scanned, would produce markdown like this:

### void Toy_initLexer(Toy_Lexer* lexer, const char* source)

This function initializes a lexer, binding it to the `source` parameter; the lexer is now ready to be passed to the parser.

The exact specification is currently vague, but it would likely plug into the existing github pages system.

Potential

While I initially investigated doxygen for this purpose, I felt that it was a little ugly by my preferences, and that I prefer a markdown-based approach (which can then be plugged into ghpages, as normal). A custom tool could be built into the toolchain...

Concerns

I'm aware of Not Invented Here Syndrome, and I'd like to avoid that - but the power of custom features is very enticing.

[Ratstail91]

Come to think of it, if I used *! and !* as the docs wrappers, it could work pretty well.

/*!
comments converted to docs
!*/

//*!
code converted to docs
//!*

(The empty // would be omitted from the markdown.)

[Ratstail91]

Markdown Embedded C Heuristic Analyzer

MECHA

[Ratstail91]

https://github.com/Ratstail91/Toy/blob/main/tools/mecha.cpp

I have no self-control. It's really getting to be a problem.

[hiperiondev]

Doxygen can use Markdown and can be converted to Markdown:

https://www.doxygen.nl/manual/markdown.html
https://github.com/sourcey/moxygen

[Ratstail91]

I'm not super fond of the presentation of doxygen - that's my only gripe with it. But if I can use my own MECHA tool to generate markdown, the markdown can be used by both my docs branch and doxygen, correct?

I'm going to keep adding MECHA comments, and making small tweaks as I go along to clarify parts of the lang.

The docs between our branches will begin to diverge, so I recommend using toylang.com until I'm done, otherwise you might be wasting effort.

[Ratstail91]

Just committed the mecha-style comments, cdfe17a.

These work surprisingly well lol.

Edit: you can run make documentation to generate the markdown.

[hiperiondev]

I have checked the commit cdfe17a

From what I see, the documentation that already exists is transported directly as the header of the functions.
I don't really agree with that style. The code documentation should be more concise and not so explanatory, it is a reference for the programmer, not a user manual (not too long, not too explanatory, not code examples)
These references must have, in a simple and concise way, the explanation of each input and output of the function.

For example:
MECHA style:

/*!

void Toy_setInterpreterError(Toy_Interpreter* interpreter, Toy_PrintFn errorOutput)

This function sets the function called when an error occurs within the interpreter. By default, the following wrapper is used:

static void errorWrapper(const char* output) {
	fprintf(stderr, "%s", output); //no newline
}

!/
TOY_API void Toy_setInterpreterError(Toy_Interpreter interpreter, Toy_PrintFn errorOutput);

Doxygen style:

/**

• @fn void Toy_setInterpreterError(Toy_Interpreter *interpreter, Toy_PrintFn errorOutput)
• @brief Sets the function called when an error occurs within the interpreter.

•         (if you want .... more explanation!)
  

• @PARAM interpreter (what is this parameter?)
• @PARAM errorOutput (what is this parameter?)
  */

and can add a lot of tags...if you want (more permissive): https://www.doxygen.nl/manual/commands.html

---

Other structs must to resolve in tables:
for example:

/**

• @enum Toy_Opcode
• @brief Interpreter opcodes

*/
typedef enum Toy_Opcode {
TOY_OP_EOF, /**< TOY_OP_EOF */

//do nothing
TOY_OP_PASS,                      /**< TOY_OP_PASS */

//basic statements
TOY_OP_ASSERT,                    /**< TOY_OP_ASSERT */
TOY_OP_PRINT,                     /**< TOY_OP_PRINT */

//data
TOY_OP_LITERAL,                   /**< TOY_OP_LITERAL */

(....)
}

resolves in:

http://raw.githack.com/hiperiondev/uC-TOY/main/documentation/doxygen/html/dc/dad/toy__opcodes_8h.html
doxygen can be simple or as complex as you like.

Or you can create your own documentation system but you will have to implement many features that already exist in Doxygen.

On the other hand, I advise you to review Sphynx for documentation formatting and its use in Readthedocs.

Doxygen, Sphynx and Readthedocs is a de facto standard that is widely used in the industry, even the official documentation of many companies is released directly in these formats (for example ESP-IDF: https://docs.espressif.com/projects/ esp-idf/en/latest/esp32/)

Sphynx in particular allows for very "pretty" formats and multilanguage handling.
For example, this one I did in a couple of afternoons: https://berry.readthedocs.io/en/latest/

[hiperiondev]

On the other hand Doxygen is very well handled by many IDEs and the header creation is quite automatic (for example in Eclipse you can start a comment with /** and it automatically creates the whole header).

I don't use Visualstudio but have a manager for Doxygen: https://marketplace.visualstudio.com/items?itemName=cschlosser.doxdocgen

I have added all the comments in this way in a few minutes here: https://github.com/hiperiondev/uC-TOY

[hiperiondev]

Note: The functionality in the comment should be very concise and avoid too many idiomatic forms.
For example:
This function sets the function called when an error occurs within the interpreter
Here we already know that it is a function, it is not necessary to say it and we only have to comment on its functionality:
Sets the function called when an error occurs within the interpreter

[hiperiondev]

Note2: Doxygen can export in various formats, not just HTML. In particular, a script could be useful to extract the information that serves you from the XML, This software do that:
https://github.com/sourcey/moxygen
https://github.com/pcolby/doxlee
https://github.com/alandefreitas/doxybook

Or this integration:
https://github.com/boschglobal/doxysphinx
The documentation of this project is done in this way: https://boschglobal.github.io/doxysphinx/

this is an alternative format of documenting:

def read_doxyconfig(doxyfile: Path, doxygen_exe: str, doxygen_cwd: Path) -> ConfigDict:
    """Read doxyconfig and get full doxygen configuration (also with default values).

    Supplement the doxygen configuration file with the default doxygen configuration and return the final
    key value pairs as a dict.

    :param doxyfile: the doxygen configuration file to read
    :param doxygen_exe: in case one wants to execute doxygen from another directory.
    :return: a dict representing all key-value pairs defined in the final configuration
             (including warnings from the console output). The value can either be a single value or a list.
    """
    output = _compare_configs(doxyfile, doxygen_exe, doxygen_cwd)
    config = _parse_stdout(output.out)
    config["WARNINGS"] = _parse_stderr(output.err)
    return config

Can you see an example here: https://github.com/hiperiondev/uC-TOY/tree/main/documentation/doxygen/xml

[Ratstail91]

Sorry it's 2am here, so I'm not thinking too well right now.

You're right - I basically just copied over the existing docs with a few tweaks here and there, mostly because I've been tired all day, and it was just an experiment.

I am trying to be accommodating, but I've personally never liked the doxygen presentation - I just really hate the way it looks lol.

I'm not sure this experiment is really going anywhere. I'm gonna leave the docs issue for a couple days and sort it out on the weekend maybe.

[hiperiondev]

Sorry it's 2am here, so I'm not thinking too well right now.

13:50 here (Argentina UTC -3)

[Ratstail91]

@hiperiondev BTW, the deep-dive section on the site has been revised. In particular, you might be interested in Developing Toy which will eventually document everything I can about the implementation.

What about the lang would you like to know more about? The interpreter, specifically?

[hiperiondev]

To improve performance and memory consumption I would need a deep explanation of the interpreter and internal data representation (like the LUA VM documentation I gave you).

[hiperiondev]

For example:
The interpreter references Toy_keywordTypes which are strings, which is not very memory efficient, and could be replaced somehow by a table of integers (uint32_t?).
Not knowing the internal functioning of the interpreter is very difficult to find how this could be done.

[Ratstail91]

Ah, ok. Toy_keywordTypes is actually used in the interpreter to prevent functions and hooks from being injected with reserved keywords.

This only occurs once for each function/hook, and usually only at the start of the program.

As for how it works, I'll start soon, though I can't guarantee a full write up anytime soon.

[hiperiondev]

No problem. I understand that documenting the entire operation of the interpreter is quite long and tedious.
I'm waiting for you to finish it.

It is important to optimize several things because I am already noticing a large memory consumption (from the point of view of a microcontroller, of course)

[hiperiondev]

A disassembler would be an excellent tool to start with.

[Ratstail91]

Oh, I wrote a brief overview of the interpreter in developing toy, hope it helps.

[Ratstail91]

To disassemble the bytecode? huh...

There is an option within the repl to parse the bytecode's header (the -p flag).

I need to document the bytecode first, which is gonna be a challenge.

[hiperiondev]

You can't optimize the interpreter if you don't know what it interprets :-)
It is very important to know EXACTLY what each bytecode does, parameters it accepts, addressing modes, etc...

[hiperiondev]

It is a stack machine, register machine, something sui-generis?

[Ratstail91]

stack-based machine - it uses a LiteralArray for the stack, though.

[hiperiondev]

it's better, simpler to understand.

[Ratstail91]

The bytecode names should ideally specify what is happening.

[hiperiondev]

That is not very critical, perhaps changing it in the code now would be a bit cumbersome.
But if you specify each one well, how is the format in the file, if it is zero-operand or requires arguments, with that information I could quickly make a disassembler

[Ratstail91]

You'd be up for making a disassembler? That is something I'm interested in.

The individual exec* functions in the interpreter are hopefully easy to follow, especially since I always pop every argument from the stack at the beginning.

I'll see what I can do for documenting the bytecode.

[hiperiondev]

I have tried to understand the interpreter and it seems a bit "strange" as a stack machine. I made a nasty little disassembler that doesn't work very well.
I'm going to wait for the documentation of the opcodes and the file format to return to this because I find it somewhat confusing.

[hiperiondev]

Please don't forget about this!

[hiperiondev]

I recommend another book for a boring weekend, it is quite old but very educational:
https://users.ece.cmu.edu/~koopman/stack_computers/index.html

After this you will love the FORTH language :-)

You can see an example of a compact machine here:
https://github.com/hiperiondev/SM1vm

I made it a long time ago and it has a 2-step assembler and a disassembler

[Ratstail91]

Thanks, I've got them now, I'll give them a read over the weekend.

[hiperiondev]

Another good read:

https://barrgroup.com/embedded-systems/books/embedded-c-coding-standard

[Ratstail91]

@hiperiondev Hey, sorry things have been going slow on the documentation side of things. I'll try getting some basic writeups of each opcode done tomorrow. What I've done so far is here.

[hiperiondev]

I have read the documentation and it seems very clear.
What would be needed is a table of opcodes with their arguments:

OPCODE----ARG1(size)---ARG2(size)---etc