State of reasonml.org 2020-Q2 / Pt. 2

Higher Quality Docs

In our previous blogpost we talked about the rationale and general concepts of the reasonml.org project. In this post we want to highlight how we make sure that our documentation is easy to manage and easy to refactor.

Automatic Quality Assurance

When we extracted the API docs from the BuckleScript codebase into markdown files, we noticed that a substantial number of examples in the JS / Belt module did not compile when we tried to copy them in a different file (mostly due to missing context, or not fully qualified module references). One can guess how many people have been frustrated with compiler errors. For a project of the depth and scope of BuckleScript, automating away these sorts of mistakes is critical so the core team can continue focusing on delivering faster compile times and more features.

To tackle this issue, we built a simple mechanism to automatically verify every API example. It extracts all the reason example code fences from the markdown, and then compiles them with BuckleScript.

A markdown example with a code snippet might look like this:

some text with some broken `List.head` example:

```reason example
Belt.List.head([]); /* None */
List.head([1, 2, 3]); /* This should actually be Belt.List */
```

Running the test with the broken example would produce:

SH
node scripts/test-examples.js "pages/apis/javascript/latest/belt/list.mdx" 

Testing example in 'pages/apis/javascript/latest/belt/list.mdx' on line 23...
Testing example in 'pages/apis/javascript/latest/belt/list.mdx' on line 43...
Testing example in 'pages/apis/javascript/latest/belt/list.mdx' on line 57...

-------Results for 'pages/apis/javascript/latest/belt/list.mdx'----------
pages/apis/javascript/latest/belt/list.mdx:23 => ok
pages/apis/javascript/latest/belt/list.mdx:43 => failed
pages/apis/javascript/latest/belt/list.mdx:57 => ok
...
pages/apis/javascript/latest/belt/list.mdx:1054 => ok
pages/apis/javascript/latest/belt/list.mdx:1078 => ok
pages/apis/javascript/latest/belt/list.mdx:1110 => ok
-----------
Summary:
Total Examples: 56
Failed: 1
Success: 55

Tip: You can also run tests just for specific files / globs:
`node scripts/test-examples.js "pages/belt_docs/array.mdx"`

-----------
Error Preview:

  We've found a bug for you!
  pages/apis/javascript/latest/belt/list.mdx: 46:1-9

  1 │ Belt.List.head([]); /* None */
  2 │
  3 │ List.head([1, 2, 3]); /* Some(1) */

  The value head can't be found in List

As you can see, the error points you to the right error location within the list.mdx file by replacing the relative snippet line number with the absolute line number of the mdx file.

A full run of the example test suite on current master yields following results:

SH
Total Examples: 693
Failed: 0
Success: 693

Leveraging the machines do the hard work of tracking and verifying nearly 700 snippets means that the rest of us can invest in higher-level experiences.

We also created a test to verify hyperlinks that cross-reference to other relative pages, which makes it hard to link to non-existing pages within reasonml.org.

As an example, this is how a hyperlink to a non-existent page will be reported:

SH
node scripts/test-hrefs.js
-------Results for './pages/markdown-guide.mdx'----------
./pages/markdown-guide.mdx:14 => ok
./pages/markdown-guide.mdx:14 => failed / Unknown href './doesnt-exist' in line 14:52
-----------
Summary:
Total Links: 1125
Failed: 1
Success: 1125

Tip: You can also run tests just for specific files / globs:
`node scripts/test-hrefs.js "pages/belt_docs/*.mdx"`

-----------
Error Preview:
./pages/markdown-guide.mdx: Unknown href './doesnt-exist' in line 14:52

Here are the statistics for all internal cross-links within the boundaries of the documentation platform:

SH
Total Links: 1125
Failed: 0
Success: 1125

The summary reports 1125 relative links that point to different sections on reasonml.org. This includes relative hrefs such as ../javascript/my-file or "global paths" such as /docs/javascript/latest/my-file etc.

With example / hyperlink tests in place, contributors were able to jump in and fix outstanding errors in all our API docs, e.g. see PR 43, 47, 48.

This feature itself already showcases the value we provide with our new platform. Our goal is to make contributions and doc updates almost frictionless. We can even use this mechanism to test e.g. major compiler upgrades.

All the tests run automatically on CI, and since we are using NextJS to build our platform, we also offer live preview deployments for each PR which makes it easier for maintainers and contributors to spot visual regressions as well.

Why are the API Docs Maintained in Markdown?

You might ask yourself why we did the markdown extraction process in the first place. As for right now, the tool for generating documentation from code comments, called odoc, is not capable of generating simple structured data, such as JSON.

Since odoc data is pretty complex, the only output format right now is plain Html. You probably have seen odoc generated Html already, such as the Reason API docs on reasonml.github.io.

We want to achieve better UX and integrate the documentation seamlessly in our design system and application. In our next post of the series, we will explain how we'll achieve this with the aid of our doc-tools project.

Time Commitment and the Contribution Process

So far we've mostly been talking about what we did and what technical tools we build. Actually we think it's more important to talk more about you, the community, taking part in all of this.

Right now, the official documentation websites are maintained by the Reason / BuckleScript core team. Some websites are part of the source code repository, such as ReasonReact, some are separate. Issue trackers are flooded with requests, and the maintainers only have a limited amount of time to tackle them. We try to share the burden by slowly but steadily taking over the documentation part so the core team can focus on improving the compiler speed or shipping new features.

Also knowing just one project well is not enough to build a documentation platform for the whole ecosystem. For instance, it doesn't make sense to only write about Reason syntax features when most of the interesting features are coupled to the Reason JS compiler (BuckleScript).

We tried our best to raise enough funding and set our budgets in a way that allows us to allocate time and resources to work full time on the documentation for 2020. That means that we can build up competences with a holistic view on the whole Reason platform, and can spend enough time on reviewing PRs, giving feedback and allocating tasks. We also want to make sure to lower the bus factor by adding more core documentation maintainers in the future. Continuous and deep collaboration, proper guidance and mutual trust will be the foundation for this.

To wrap things up, we think that we'll be able to establish a reliable documentation focused culture to share responsibilities within the community. That said, we'll try to be clear on our processes and goals, since we believe these are key aspects for successful collaboration.

Conclusion

In this article we described how test automation across the whole Reason documentation platform can help us to proactively detect and fix errors. We also explained how we'll improve the contribution process for community curated docs.

In case you have any questions or comments, make sure to let us know in the Reason Discord #docs channel!

Happy docs browsing!

Upcoming Posts

This article is part of the "State of reasonml.org 2020-Q2" series (please stay tuned for the upcoming posts):

Part 3) Future Tools for the Community: How we'll generate API docs with our new doc-tools and how the new Playground will look like

Part 4) It's all Opinions: We'll dive into our documentation philosophy, our vision of the Reason Platform and ideas on how to tackle the "Reason Native" documentation.

Previous Posts

• Part 1) Introduction to reasonml.org