Integration with Collection

[Altai-man]

Places where Documentable is used:

• Documentable::Search - https://github.com/Altai-man/docs.raku.org/blob/master/lib/Docky/Search.pm6
• Documentable::Config (can be replaced with a custom object of a similar structure)
• Documentable::Registry
  • compose - https://github.com/Altai-man/docs.raku.org/blob/master/lib/Docky/Host.pm6#L59-L63
  • lookup - https://github.com/Altai-man/docs.raku.org/blob/master/lib/Docky/Host.pm6#L72-L75
• Documentable::DocPage
  • parents+roles to append: https://github.com/Altai-man/docs.raku.org/blob/master/lib/Docky/Routes.pm6#L116-L118
  • Primary document as a list (or just some accessor really): https://github.com/Altai-man/docs.raku.org/blob/master/lib/Docky/Routes.pm6#L132
  • Registry.lookup - https://github.com/Altai-man/docs.raku.org/blob/master/lib/Docky/Routes.pm6#L149

Main questions to check after a migration PR is done:

• How good Collection is with concurrency? (check with a stress test)
• How fast is Collection compared to Documentable way of doing things? (to replace the code and profile)

[Altai-man]

@finanalyst ^ is the list of things where Documentable API is used directly. Documentable::Config bit can be ignored, but if all the other bits can be seamlessly replaced by your library - welcome.

[finanalyst]

Did you go from what the API can offer to create a system around it? Or did you have an input that you wanted for your system?
At present, Collection will take all the primary files and produce:

1. a html file for that POD. (The html for the file and for each FC/Block/List is configurable)
2. All items marked with X<> , including the target label (what goes in the index/glossary), and the target words (what is shown in the text to be linked to)
3. All Headings (from =head)
4. All links marked with L<> to other anchors, and provides access to them as 'internal' = inside the primary file, 'local' = inside another primary file in the collection, 'external' = to another URL.

These data are available for each file, and for the whole collection.

So, what seems to be needed as well, but which is not provided by Collection yet, is some sort of data structure to mark key words, such as role/sub/method/class etc.

This I haven't thought of before. What would be best for you? A separate html file for each occurence, or a snippet of html?

Also, suppose a key word exists in the label line of a =HEAD block, then the relevant text could be the header and the section associated with the block.
But if a key word is in the text of a block, would you want the paragraph of the block, or the whole of the =HEAD block?

I could go to Documentable and reverse engineer it, but perhaps there is stuff you would like or the data would be best presented in another form.

[Altai-man]

Did you go from what the API can offer to create a system around it? Or did you have an input that you wanted for your system?

Both are true, I believe.

This I haven't thought of before. What would be best for you? A separate html file for each occurence, or a snippet of html?

The input for this case are Pod documents, the rest of the pipeline (that is, converting to HTML) is done by Pod::To::HTML.

This is what Documentable can do, it can create HTML files, but it does not require you to ask it to create them.

So say I do Documentable.registry.lookup(kind => Kind::Type) and it returns me a list of all the Pod documents of such kind, without doing any conversions by itself. What would be an equivalent API if it's present?

[finanalyst]

I have refactored Collection considerably in the last few days.
Once Collection is run with the files from docs.raku.org/docs, they are all cached and the data from all of them is available in a return object.
The return object contains for each file

• the kinds/subkinds/categories associate with each file
• All the Table of Contents data gathered from all the =headX blocks (titles and anchors)
• All the Index (aka Glossary) data gather from the X<text | list of items > format codes, in a richer format than is used by Pod::To::HTML
• All the links, internal to a file, local to the Raku docs, external to other http urls

The return object can be obtained from the collect subroutine with :before<completion> or inside a completion subroutine. There is a CroApp plugin which shows how to use the completion data.

At present, I have not written a plugin to split the Pod6 primary files into secondary ones. That is on my TODO list.

Collection takes 176 seconds to cache all the Raku documents. That should be the same as Documentable, which uses a Module called Pod::To::Cache that I wrote, and it uses the same technique as Pod::From::Cache, which Collection uses.

Collection processes all the Raku documentation into html from cached status to fully rendered html in 50seconds. I believe it takes (or used to take) Documentable 2 minutes to process them all.

[Altai-man]

Those are great news, I'll look at if I can replace at least some calls when time allows, likely next week.

[finanalyst]

Collection is quite big so there's a learning curve. It my be a good idea to try set up some small repo with few files from doc, Collection, and then some plugins to show how it all fits together.
I have a trial set of documents (about 20).
Also, if you could add some CSS files that you would prefer that would be good. I copied the ones from docs.raku.org, and I don't like them at all. I tweaked them a bit, but not much.

[finanalyst]

I have just reworked PodRender & Collection and it took 2 minutes to render the whole site.
Caching took more time, but both Documentable and Collection use my module to cache, so no difference there.

Is there an online site using your implementation?

The Collection version is at https://raku.finanalyst.org

The colours are still not so good. I'm now working on the css.

[Altai-man]

Is there an online site using your implementation?

http://164.90.207.89:10010/

[finanalyst]

I just looked at the site from a smartphone. Really nice. I'm going to see how to imitate the site with Collection. Do I understand that the site is dynamic rather than static? If so in what way is it dynamic? I will be looking at the code on your site soon. I have still some things I want to get right in Collection / PodRender. There is a pod block called config. It should be used to propogate meta data (such as item numbering) to other blocks in lexical scope. This will allow for other things like defining language templates. Regards Richard

…

On Mon, 28 Nov 2022, 10:30 Altai-man, ***@***.***> wrote: Is there an online site using your implementation? http://164.90.207.89:10010/ — Reply to this email directly, view it on GitHub <#67 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACYZHG5LV65VNU5MPC6AUTWKSCSZANCNFSM53DTSTIQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[Altai-man]

Hi, Richard.

I believe we had this conversation long time ago, and we are repeating it again and again over time. Can that be true that some things are forgotten?

I thought, before creating this ticket, when we had a meeting, we agreed that the way to "combine" the work is to provide a drop-in (or almost) replacement API of what Documentable does, but via Collection. But not it sounds like your intention is to start working on re-inventing all the frontend from scratch. You also doesn't appear to remember the website is dynamic (you was rather disliking this a year or so ago), and I remember you writing the design doesn't match Raku, in your opinion. Can there be some misunderstanding happening?

[finanalyst]

we did have the conversation a long time ago!!! It's not that I have forgotten. On the contrary. So I went back to look at my work, found it deficient. Fixed some things, refactored, and got it to where I wanted it to be.
Now .... or at least in a few days ... I will have the time to focus on one thing.
By "dynamic website", I thought I understood the idea, thinking it was based on an SQL database. I do not like that sort of site.
However, after reading more about JAMSTACK, I have discovered a dynamic site can mean something more!
That is why I asked about what you mean by 'dynamic'.
If you look at some sample plugins you will find a Latex Equation and a map. Both of these plugins work by calling other dynamic pages. So in fact it is dynamic.
I have not written a drop-in replacement for Documentable. I wrote at first a dropin replacement for Pod::To::HTML. But Pod::To::HTML2 is now much more.
I think you have also misunderstood what I meant about Raku and Web content management. I never intended in any way to imply your site did not match Raku.

[finanalyst]

So I now have your site working with Collection. https://new-raku.finanalyst.org

I have mostly used your javascript and css. I use your search.

There is a slight difference on the 'language' page as I wanted to use a simpler flex arrangement for all the category blocks. But that looks ugly with too much white space. I'm changing that.

I have added a REPL bar, and removed the evaluation from each code snippet.

The extended search page is not complete yet, and the Types and Routines page do not match your version. To be honest I think Types and Routines need to be redesigned.

Secondary files are generated using the same algorithm as Documentable, but with only a single parse of the primary documents, not two. The whole site is generated in about two minutes.

[Altai-man]

There is a slight difference on the 'language' page as I wanted to use a simpler flex arrangement for all the category blocks. But that looks ugly with too much white space. I'm changing that.

Great.

I have added a REPL bar, and removed the evaluation from each code snippet.

Please no, it breaks the UX. If you want to have a REPL too, along with the Run button, it can be done, but it needs a better UI then. Just borrow/copypaste the look, no need to invent anything, you already done lots of effort to make all the backend going.

To be honest I think Types and Routines need to be redesigned.

Fair enough, but I am not sure if we have time to mull on it for a long time, we already wasted years. Can you please make it as it is now and then maybe have a separate ticket to improve it, while the website is actually deployed and working for people to work with?