Incorporate Unix Commands to Secrets Subcommand - `pk secrets *`

[CMCDragonkai]

Specification

Standard unix commands include things like cp, mv and ls.

This allows users to interact with secret vaults as if they were real filesystems. (And they sort of are).

Imagine (IN ORDER OF PRIORITY):

# these 2 might be merged into 1 command just `write` is sufficient
pk secrets read vault1:/a/b
pk secrets write v1:/abc

pk secrets cat vault1:/a/b
pk secrets touch vault1:/a/b
pk secrets ls vault1
pk secrets mkdir vault1:/c
pk secrets rm vault1:/a/b
pk secrets mv vault1:/a/b vault1:/a/c
pk secrets cp vault1:/a/b vault1:/a/c
pk secrets ln vault1:/a/b vault1:/d/e

# this one is special
pk secrets env

# this one is also special
pk secrets ed vault1:/a/b

# these are less important
pk secrets head vault1:/a/b
pk secrets tail vault1:/a/b
pk secrets find vault1
pk secrets grep vault1:/a/b
pk secrets sed vault1:/a/b

I believe some of these commands were already implemented before in the old PK codebase. You have things like pk secrets create and pk secrets delete.

»» ~/Projects/js-polykey/src/bin/secrets
 ♖ tree .                                                                                                          (client-refactoring) pts/5 14:47:05
.
├── create.ts
├── delete.ts
├── dir.ts
├── edit.ts
├── env.ts
├── get.ts
├── index.ts
├── list.ts
├── mkdir.ts
├── rename.ts
└── update.ts

0 directories, 11 files

I believe there's a problem with doing this. We are reinventing the wheel, and we'll never cover all the commands that Unix already has.

This has the benefit of reusing context that developers already know and remember when interacting with a Unix shell.

But how do we do this without having to rewrite all the code? Luckily it seems someone has already done this.

See: https://github.com/shelljs/shelljs

It has implemented all the major Unix shell commands in raw JS.

The only problem that their command source code currently directly imports the native Node fs:

https://github.com/shelljs/shelljs/blob/79ae14d30d7ce4064de05d41c7889885326b6754/src/ls.js#L2

If we would want to use the shelljs library, we would need to globally mock the fs as described here: shelljs/shelljs#747 (comment)

However that may be dangerous if that leaks into other places of the FS.

There is another alternative: https://github.com/dthree/cash. The difference between the 2 are: https://github.com/dthree/cash#doesnt-shelljs-do-this However I think it's even less integratable compared to shelljs.

So it seems we would need to "extract" the command code from shelljs and place it into PK directly and thus enable us to change the fs object to our Vault EFS.

Note that we do not need all shell commands, just the major ones that relate to files, and doesn't change cwd context since we don't use that. Nothing that changes permissions is relevant to us. Process control is also not relevant.

One major difference is that our commands will have to traverse both vault filesystems and the real filesystem. For example pk secrets mv vault1:/a ./a which has to take a file from a vault to the real fs. The functionality to achieve this will also end up being used to do commands between vaults. Because the efs.mv won't work between EFS instances anyway.

Additional context

• Exposing different interfaces for use of secrets within polykey Polykey#55 - Oldest issue talking about the different interfaces PK was meant to have
• Secret editor on the CLI Polykey#78 - With regards to the pk secrets ed command
• The pk secrets env command for meeting Development Environment Usecase #31 - The pk secrets env command is similar concept here replicating the Unix command env
• Pseudo file systems interface Polykey#169 - Mounting a vault as fs also sort of does this, but it is more difficult to do
• File Descriptor Interface Polykey#170 - Some discussion about file descriptor interface for PK, relevant to the pk secrets cat command and composition of PK to other commands

Sub issues

• Implement secrets write command #242
• Implement secrets cat command #243
• Implement secrets touch command #244
• Implement secrets ls command #245
• Implement secrets mkdir command #246
• Implement secrets rm command #247
• Implement secrets mv command #248
• Implement secrets cp command #249
• Implement utility that generates a file tree from a path pattern Polykey#767
• Implement a file tree serialiser/de-serialiser Polykey#768

Tasks

1. ...
2. ...
3. ...

[CMCDragonkai]

With regards to pk secrets cat and "writing a file" to a secret vault. While the cat command can be used to read out a secret, what's more difficult is putting back a secret that can be composed with other commands. Here I propose a null-command or a variant of pk secrets that opens a readable file descriptor.

Here are some examples:

# using process redirection, as pk secrets v1:/a would need to take a FD
echo 'somesecret' > >(pk secrets v1:/a)
printf 'somesecret\n' > >(pk secrets v1:/a)
echo 'somesecret' | pk secrets v1:/a

# echo 'somesecret' > pk secrets v1:/a # <- this will not work as pk secrets is a comand, not a filepath

# compare with shell commands like:
echo 'somesecret' > >(cat > abc)
echo 'somesecret' | cat > abc

Note that pk secrets v1:/a has no subcommand under secrets. It would need to open a file descriptor by attempting to read from STDIN. This takes over from the shell command cat > abc which runs cat without a file path argument which causes it to read from STDIN. However is not usable in PK since we aren't running process redirection operators > and < from inside the PK command. That is PK is not a shell. Therefore we run something different:

# starts reading from STDIN immediately
pk secrets v1:/a

As well as the ability to use cat subcommand to output to STDOUT.

# read v1:/abc and pipe it to cat, pretty simple
pk secrets cat v1:/abc | cat

At this point, one might argue that one should enable pk secrets v1:/a to be a writable file descriptor as well. That is both STDIN and STDOUT may be usable.

cat <(pk secrets v1:/abc)

Which would take over from the cat subcommand.

However I'm not sure I really like the usage of pk secrets v1:/abc as a universal file descriptor. It's nice in one way, but it seems such a "default" command to be relatively hard to use and hard to learn.

Consider that we also need another command that one can just use to put a secret in without the need to wire up other commands.

pk secrets put v1:/abc 'abc'
pk secrets write v1:/abc 'abc'
pk secrets print v1:/abc 'abc'

And as we have talked about before, we need 3 ways:

• Passing as an argument
• Passing as a file descriptor
• Passing as via the prompt

I propose a special command for this:

# this will read from STDIN, and write to v1:/abc (like reading into a secret)
pk secrets read v1:/abc

# pipe into a file descriptor
echo 'secretdata' | pk secrets read v1:/abc
pk secrets read v1:/abc <<<'secretdata'
echo 'secretdata' > >(pk secrets read v1:/abc)

So then read is the opposite of cat.

But still one more command that allows one to write a secret in easily. Here are our 3 candidates.

pk secrets put v1:/abc 'abc'
pk secrets write v1:/abc 'abc'
pk secrets print v1:/abc 'abc'

From these 3, I quite like pk secrets write. It's the opposite of read, it actually can function very similarly if pk secrets write v1:/abc was allowed. That is with no arguments, it acquires from the STDIN. In that sense, there's no need for pk secrets read at all then as it is just confusing. So write is probably the correct keyword.

[CMCDragonkai]

Some commands here due to usage over GRPC will involve client streaming or even duplex streaming. Deadlines will be important here. And if a command gets cut in the middle, the data written to EFS should be removed. That might be good idea to per-vault lock to do this. Until of course we can create a "vault"/efs snapshot. The beginnings of a vault/efs snapshot is already done in MatrixAI/js-encryptedfs#49 but it would need to be exposed to the PK vaults system. Perhaps that can help with enabling COW for the vaults MatrixAI/Polykey#172 MatrixAI/Polykey#180.

[CMCDragonkai]

Now that MatrixAI/Polykey#266 has fixed up vaults related commands, the next step is a proper review of all the secret commands, and hopefully finish up our pk secrets env PR MatrixAI/Polykey#265 so it can be merged.