alternate argument syntax for consult commands

[jdtsmith]

I make heavy use of arguments with various "two-part" consult commands (e.g. consult-rg often needs a type (e.g. -t elisp) or pattern (-g *.[ch]). The current UI for providing arguments — #<search> -- <args>#filter — is not ideal for a couple of reasons:

1. The search is kicked off prior to entering <args>, arguments which may drastically limit the search space. This can generate a massive amount of unwanted data, especially if...
2. Sometimes you only want the args. For example I use consult-fd with (e.g.) --newer 60min, and I don't need any match pattern — I just want the files modified in the last hour. Right now I have to use #. -- --newer 60min, which of course kicks off a large search of everything matching . if you type in that order. Otherwise you have to type in the args, then go back to the beginning of the prompt and add an (uninteresting) pattern.

I've considered an alternative syntax for arguments:

[<args>]@<pattern>[@<filter>]

(where @ is some separator char).

E.g. in consult-fd, --newer 60min@ would kick off a fd --newer 60min process.

But the separator here becomes a problem. First, this would obviously require args not to contain the separator character. Since consult uses any instance of leading punctuation as the separator, this would have to be refined. How will it know what separator to use? consult could take a page from corfu and let M-SPC enter a fixed separator such as a non-breaking space, and respect that as a fixed separator no matter where it occurs. This would probably also require ruling out - as a valid separator character, and, if the string starts with -, wait to kick off the search until a "real" separator is supplied. I think - is of dubious use as a separator anyway, since it basically eliminates the possibility of arguments.

So:

• #foo: as now, kicks off a process with foo as pattern.
• #foo -- -abc: same, but with args -abc.
• #foo -- -abc#filter: same but with a filter too.
• -abc@: (where @ indicates the always-on separator character) starts a search with no pattern and args -abc, only after the separator is provided.
• -abc@foo@filter: full pattern/args/filter using 2 separators.

The user could configure a "magic character" for the "always on" separator, which is always respected (and can be entered with M-SPC as a convenience for using unicode chars here). E.g. consult-alternate-separator-char (default nil).

One possible variation on this theme: if the alternate separator is configured, you could prompt with it instead of #. Everything else would remain the same, but all two part commands now become three part commands: @args@pattern@filter. This would save removing the # if you wanted the alternate syntax. I don't love this since often/usually I don't want args, and I'd have to use @@pattern.

Maybe there's a better version/simplification/refinement of this type of idea, but it definitely feels like arguments could use some love in the interface.

[minad]

I agree that the current argument interface could be improved. But I don't really find your suggestion convincing. I have to think about it more. There are also other improvement ideas, which didn't materialize so far, e.g., a transient which manipulate the arguments separately (but then they would be lost in the minibuffer history), or completion of arguments.

What about this format?

#args#input#filter

Initial input could be this:

##

Unfortunately this makes it harder to change the separator character, so its probably also not that great.

[jdtsmith]

These are fair points. ## could work. Or how about this: lead with # and iff the very next (non-blank) character is a -, convert to three-part #args#pattern#filter style. So my example would be

#--newer 60min

for "pattern free" search, or say

#-telisp#foo#bar

for a full 3-parter. Or you can go for the #foo -- -telisp#bar style as well. This might be the least disruptive.

[jdtsmith]

Continuing this after our discussion on consult-ripfd. There are a few opposing forces:

1. Most of the time, you want to just start entering search input.
2. Some of the time, you'd like to add some arguments after seeing what results you got with the bare input.
3. Some of the time (often for efficiency), you need to enter the arguments first, e.g. for readily narrowing down the file set to be searched.
4. Some of the time, you want to enter only arguments. Right now this is difficult and hard to discover: <SPC>-- is what you need (and most commands invent some input like . anyway, which is not ideal if you are changing what a search term means (regex vs. glob, for example).

Taken together, this would imply that it needs to be easy to enter either input or arguments or both, in whichever order. This suggests:

Type	Format	Note
Input only	#foo bar	as existing
Args only	#-b --other 2	the leading #- is the key
Input, then args	#foo bar -- -baz	just like now
Args, then input	#-s 1 --long -- input	reversed from now, #- is the key
Leading dashless args (only)	#-- show -s	-- is just a marker, no leading space
Input with dashes (only)	# --foo is sought	note leading space
Input containing dashes, then args	# -this-is-input -- -some -args	note leading space

[minad]

Okay, do you want to prepare a patch? Do you have a bunch of useful real world examples for command invocations (for rg/fd), which we could document in the README or wiki? I added a few examples in a4f8738. This would have to be updated of course for the new syntax.

[jdtsmith]

Happy to review and test, but rather busy with the semester now. I don't think there's any urgency.

[minad]

The new command splitter is simple:

(defun consult--command-split (str)
  "Return command argument and options list given input STR."
  (save-match-data
    (let (opts)
      (setq str (substring-no-properties str))
      ;; Find first option
      (when (string-match "\\(?:\\`\\| \\)-" str)
        (setq opts (substring str (1- (match-end 0)))
              str (substring str 0 (match-beginning 0))))
      ;; Replace backlash-escaped dashes
      (setq str (replace-regexp-in-string "\\(\\`\\| \\)\\-" "\\1-" str))
      ;; Options end with double dash
      (when (and opts (string-match " --\\(?: \\|\\'\\)" opts))
        (setq str (concat str " " (substring opts (match-end 0)))
              opts (substring opts 0 (match-beginning 0))))
      ;; Use `split-string-shell-command' here instead of
      ;; `split-string-and-unquote' since it handles more flexible input -
      ;; double quoted strings, single quoted strings and escaped spaces.
      (cons str (and opts (split-string-shell-command opts))))))

But I don't know how and where to fit the highlighting in. I wouldn't want to add this without highlighting due to the breaking change in syntax.

EDIT: New prototype with highlighting in https://github.com/minad/consult/tree/option-splitting. Now the problem is that we want highlighting only for commands and not other asynchronous sources. This still does not fit in nicely.
EDIT2: I use a separate async transformer to highlight the options.

[minad]

I am sufficiently happy with the new version. Merged in cd622ef. Please let me know if you discover issues with it. I guess we will see some bug reports anyway.

[jdtsmith]

Awesome I will give it a test and let you know if I encounter anything.

[jdtsmith]

I took a look. This works very well so far, and I'm finding it pretty intuitive. The options highlighting really helps.

A few things I noticed:

1. The # chars have the same face applied over and over:

   There are text properties here:
     consult--split       t
     face                 (consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split consult-async-split)
     rear-nonsticky       t
   

2. Couple small corrections in CHANGELOG.org:
   • escape the dash with a backslash
   • Input after two standalone dashes
3. README: Is it worth saying whether -options alone may or may not produce output (depending on whether the underlying tool requires input)?
4. I don't believe you added the field properties. I'd guess # could be prompt, then input, and options. Or maybe just prompt
5. It's a small thing, but I think -- should receive the same face as the prompt, since it also "splits" the input. Or possibly no face. It's not really part of the options.
6. Amusingly (since it's what motivated this renewed discussion) this new change broke my new consult-ripfd tool, because it composes a command string for the consult rg builder to parse in the old order, which is no longer supported.

On the last point, since the package is so new, I don't mind requiring the new version of consult, but I wonder if this will break other more established tools which munge together command lines in the old order. I.e. should consult provide backward compatibility for input -- -options for such tools? I think maybe not unless others complain.

One other little thing: fd can change its input to be glob style with -g. Right now you compile all input as a regexp. Its builder might want to check for that option and skip regex compiling if it's passed.

[minad]

Thanks for testing. Fixed a couple of things. I think -- belongs to the options so it gets this highlighting. The # splitting is on a different level.

[minad]

Regarding glob search, this does not work well with consult-fd. I have to pass --full-path to fd, which unfortunately affects the search. This means that one would have to use **/*.el as glob patterns. See also sharkdp/fd#839 (comment).

[jdtsmith]

Oh right. I disabled full-path for ripfd as it wasn't necessary.

[minad]

On the last point, since the package is so new, I don't mind requiring the new version of consult, but I wonder if this will break other more established tools which munge together command lines in the old order. I.e. should consult provide backward compatibility for input -- -options for such tools? I think maybe not unless others complain.

Yes, let's wait for complaints. Probably other tools can easily adapt.

One other little thing: fd can change its input to be glob style with -g. Right now you compile all input as a regexp. Its builder might want to check for that option and skip regex compiling if it's passed.

I added glob support.

README: Is it worth saying whether -options alone may or may not produce output (depending on whether the underlying tool requires input)?

Hmm. The documentation could be much more verbose in many way. I think as long as the tool does what the user expects, we're fine. As I had mentioned before, adding more examples for useful command lines would be most welcome (ideally per Consult command).

[jdtsmith]

Loving the new consult syntax! Thanks very much for your work on this (and all your many splendid packages!).