Add spfs cli documentation

[BryceGattis]

I've got this generating docs for spfs but didn't do spk yet.

To do this, you can run:

.\spfs docs

manually from my build folder to generate the docs.

Closes #1207

TODO:

Looks like it's not showing the external commands right now. I'm assuming that's all of the subcommands defined in their own folders like spfs clean spfs enter spfs fuse etc?

[BryceGattis]

Additionally, I have an anecdote from my time trying to get this to work this evening:

At least on my machine, running the above command to generate the markdown file on my machine exports the markdown file with UTF-16 LE character encoding for some reason, so the new markdown file wasn't appearing in the sidebar, and the page looked very corrupted due to not being UTF-8 encoding. Long story short, using Notepad on Windows to see what the encoding was currently and then saving it back out with UTF-8 fixed it.

Note: This is now fixed in latest commits.

[BryceGattis]

Oh one additional thing I'm having to do manually right now, I had to add this section at the top of the generated markdown file so the new file appears correctly in the sidebar:

---
title: Clean
chapter: true
---

NOTE: This is done automatically now in latest commits.

[rydrman]

Part of me thinks that instead of integrating this into every command as an argument, we could have a single, separate command line that's used only during CI and output the help for all known commands using this syntax

let markdown: String = clap_markdown::help_markdown::<Cli>();

That way, this separate command could know where to store each output, and generate the index files and frontmatter as well, automating the whole process.

[rydrman]

From the meeting today:

• looking like the right direction for sure
• let's use a separate command entry point (not spfs itself) since we only need to run this when developing
• we need some integration with CI, ideally to generate the files before deploying the site, but I will need to look into how this would work in our current web deployment process. Aka we ideally don't want to be checking these markdown files in or having to update them manually. We could in the meantime use a lint step that runs the generation and fails if the files need to be updated and then check them in, but TBD
• We'll want this to be able to also generate docs for the other commands in the repo, assuming your intention is to programmatically manage the frontmatter and output dir (?)

[BryceGattis]

@rydrman,

I'm working to move the new spfs-docs command into its own command entry point (by making a new subfolder under crates/spfs-cli with its own Cargo.toml and src folder). However, I'm running into an issue. In my current implementation, I need to use the Opt defined in the spfs-cli-main crate so I can generate markdown for all spfs subcommands.

The problem is, that if I try to add spfs-cli-main as a dependency in my new Cargo.toml file, I get this warning:

warning: spfs-cli-docs v0.42.0 (C:\Users\bryce\Documents\spk-dev\spk\crates\spfs-cli\cmd-docs) ignoring invalid dependency `spfs-cli-main` which is missing a lib target

Do you have any suggestions on how to solve this? Is this just because it's missing lib.rs file? Should I approach this a different way?

[rydrman]

Hmm, ya you'd have to add a lib.rs file and lib target to the cargo toml for that crate so that it's both a binary and library in the eyes of cargo. Happy to provide guidance if you run into issues with that

[BryceGattis]

@rydrman,

I got the spfs-cli-main package compiling with a library target. I had to extract almost everything out of the bin.rs file to avoid circular dependencies.

Was this what you were thinking? I still need to move the command to be external to spfs itself as mentioned above, but this should allow me to do so.

[rydrman]

Yes, I think this is what I was expecting, though I'd suggest something like cmd_spfs instead of the generic bin_utils

[BryceGattis]

@rydrman,

I noticed the "external subcommands" are not included in the help text when you just run spfs:

PS E:\Code\spk> target/debug/spfs
SPK is a Package Manager for high-velocity software environments, built on SPFS. SPFS is a system for filesystem isolation, capture, and distribution.

Usage: spfs.exe [OPTIONS] <COMMAND>

Commands:
  version    Print the version of spfs
  init       Create an empty filesystem repository
  edit       Make the current runtime editable
  commit     Commit the current runtime state or a directory to storage
  config     Output the current configuration of spfs
  reset      Reset changes, or rebuild the entire spfs directory
  run        Run a program in a configured spfs environment
  tag        Tag an object
  untag      Remove tag versions or entire tag streams
  shell      Enter a subshell in a configured spfs environment
  runtime    View and manage spfs runtime information [aliases: rt]
  layers     List all layers in an spfs repository
  platforms  List all platforms in an spfs repository
  tags       List all tags in an spfs repository
  info       Display information about the current environment, or specific items
  pull       Pull one or more objects to the local repository
  push       Push one or more objects to a remote repository
  log        Log the history of a given tag over time
  search     Search for available tags by substring
  diff       Compare two spfs file system states
  ls-tags    List tags by their path [aliases: list-tags]
  ls         List the contents of a committed directory [aliases: list-dir, list]
  migrate    Migrate the data from and older repository format to the latest one
  check      Check a repositories internal integrity
  read       Output the contents of a blob to stdout [aliases: read-file, cat, cat-file]
  write      Store an arbitrary blob of data in spfs [aliases: write-file]
  help       Print this message or the help of the given subcommand(s)

Options:
  -v, --verbose...           Make output more verbose, can be specified more than once
      --log-file <LOG_FILE>  Additionally log output to the provided file [env: SPFS_LOG_FILE=]
      --timestamp            Enables timestamp in logging (always enabled in file log) [env: SPFS_LOG_TIMESTAMP=]
  -h, --help                 Print help

EXTERNAL SUBCOMMANDS:
    render       render the contents of an environment or layer
    monitor      watch a runtime and clean it up when complete

Is there a way to define those subcommands differently where they automatically appear as part of the help text? I see we're currently having to manually keep up with this here (which we are behind on it seems).

The above issue (where they are defined as external subcommands) is causing the help_markdown call I'm doing to not detect the external subcommands, so they aren't currently getting CLI documentation. Perhaps this will be fixed if we fix the above issue?

[rydrman]

In theory we could generate the list dynamically with some gymnastics but it wouldn't solve the issue with the help generation. I think we'll need to generate those separately if you want to include them in the docs

[BryceGattis]

@rydrman,

Can you elaborate a bit on what the dynamic list generation would look like? What do you think about moving towards an approach where none of our commands are "external" and they somehow can register themselves with the CLI entirely in their own crates? External subcommands don't seem to have a ton of documentation in the clap docs, so it's a bit hard for me to parse what they are doing and what the ideal use case for them is.

If this isn't something you think we should do, do you have any other TODOs I should hit for this PR?

[rydrman]

Lost track of this one, appologies

I'm not actually clear on how that would work, to be honest which is part of why we haven't done it yet. Those binaries are separated out because they are given elevated privileges when installed, and so are intentionally pulled out and only do specific things. I suppose we could use a rust build script to look at all the crates and generate the list dynamically, but that doesn't necessarily solve the issue of how you would generate documentation for everything at once. I think, the best case for the doc page is to include a note about the external subcommands in the markdown page with links to additional markdown pages for those external commands, which we can also generate.

[BryceGattis]

@rydrman One step we could do to make this more dynamic would be to make a trait that forces us to implement a run method on all of our subcommands. Then we could eliminate this match statement entirely (assuming we find a way to dynamically update the Command enum with all of our externally defined commands like clean,enter,fuse etc).

Something like

pub trait Runnable {
    fn run(&mut self, config: &spfs::Config) -> miette::Result<i32>;
}

The problem I came across though was that most of the subcommands are async but not all are. Any thoughts here?