[docs] Migrate docs from AsciiDoc to Markdown

[colleenmcginnis]

Migrate docs from AsciiDoc to Markdown.
Please see docs preview

Notes:

• I deleted AsciiDoc and image files in each {beat}/docs/ directory.
• I'm not sure what to do with the AsciiDoc changelog files.

For reference, here are details about the docs migration.

[kilfoyle]

Reviewers, please let me know if you need any details about this. I assume that the failing CI checks don't matter, or the tests can be removed since they won't be applicable any longer with the new Markdown docs.

Also, just for awareness, we'll have a 1:1 mapping so that each HTML docs page now has a single Markdown source file. Previously, changes in the 'libbeat' directory were propagated to the Metricbeat docs, Filebeat docs, etc. We're making similar changes across all of the docs to reduce complexity.

[colleenmcginnis]

I would like to understand what the plans are for versioned documentation maintenance. For example, in the Asciiidoc documentation for the CEL input, when the backing library version is bumped the changes are relatively small (for example in the latest bump), but in the Markdown case, we would see 90 changes to the file based on the current state in this PR.

It's not currently possible to use a variable in a link, but I opened an issue with the docs engineering team. Feel free to add more details (or correct my description if needed!) in that issue.

[efd6]

Thanks @colleenmcginnis, that captures it.

[kilfoyle]

Another thing I noticed is that there are still some asciidoc files in x-pack/dockerlogbeat/docs/. Is this expected? Should all .asciidoc files been removed?

Nice catch! I see those are all converted to .md so I'll remove them, but I see there are lots of other .asciidoc files under the x-pack/<beatname>/docs directories so I"ll check through those too.

[kilfoyle]

I confirmed that the asciidoc and image files under the x-pack/<beatname>/docs directories were converted to markdown, so they're removed now and the buildkite/docs-build-pr CI is passing.

One thing I'm confused about: These Generated docs instructions say that docs.asciidoc files in _meta directories are used to generate docs. However, I don't find any docs.asciidoc files in the _meta directores, under beats/filebeat/_meta for example.

@belimawr Do you know if we still have generated docs? If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

[kilfoyle]

I'm not sure what to do with the AsciiDoc changelog files.

For now, I think these should remain as is. The Beats Release Notes are generated automatically so we'll need to get that process fixed up to generate Markdown. As a temporary fix, the 9.0 Beta Release Notes (in Asciidoc) were copied over into the Stack Docs repo, here.

[belimawr]

@belimawr Do you know if we still have generated docs? If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

I'm not sure... But I'll investigate and get back to you once I have an answer.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-docs upstream/migrate-docs
git merge upstream/main
git push upstream migrate-docs

[belimawr]

@belimawr Do you know if we still have generated docs? If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

@kilfoyle, yes, we still have generated docs and as far as I can tell we still need them.

While the Generated docs is a bit outdated (e.g: docs.asciidoc is not used/does not exist any more), the generated files listed there are still generated by the listed scripts.

So we have to update them to output Markdown and to stop relying on reading versions from libbeat/docs/version.asciidoc.

[kilfoyle]

Thanks @belimawr! If I should open an issue for those changes or if there's anything else I can do, please let me know.

[belimawr]

Thanks @belimawr! If I should open an issue for those changes or if there's anything else I can do, please let me know.

Yes, please, could you open an issue for that?

Given your comment:

If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

I understood you'd open the issue.

Honestly, I'm not sure who is responsible for those scripts: the Data-Plane (Beats) team or the Docs team 🙈.

Anyway, if there is anything I can help with, just ping me ;)

[bmorelli25]

Do you know if we still have generated docs? If that's still needed, I can open an issue to ask the Docs Engineering team about converting the generation scripts to output Markdown instead of Asciidoc.

Unfortunately, the Docs Eng team doesn’t have the bandwidth to update automation scripts for every engineering team 😞 . We know this work takes time to prioritize and schedule, so we can ship static docs (this point-in-time snapshot) for teams that need more runway while they update their automation. Ideally, we'd be able to do that from this repo. But it sounds like there’s some automation that needs fixing before we can merge this PR.

@kilfoyle is meeting with @pierrehilbert on Friday. I've asked to be added to that meeting to continue the discussion.

[kilfoyle]

Thanks a lot @belimawr for the investigation and @bmorelli25 for the clarification! And sorry for my confusion: I wasn't sure which team should or could own this work.

I've opened #43008 to see if we can:

1. Remove or fix the dependencies on libbeat/docs/version.asciidoc that cause the CI checks to fail.
2. Update the script that produces the generated docs to output Markdown instead of Asciidoc.

If I've missed any info in the issue please feel free to make any additions/corrections.

On Friday, Brandon and I will chat with Pierre about this issue and also the Beats Release Notes.

[mergify]

This pull request is now in conflicts. Could you fix it? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

git fetch upstream
git checkout -b migrate-docs upstream/migrate-docs
git merge upstream/main
git push upstream migrate-docs