Binding raw escape sequences doesn't work

[friendly-bits]

Description of the problem or steps to reproduce

I'm using MobaXterm on Windows to connect to my Debian machine via SSH. The default key bindings for SelectWordLeft and SelectWordRight (CtrlShiftLeft and CtrlShiftRight) don't work. I checked in the raw mode (in micro) what it receives when I'm pressing these combinations and it says:
EventKey: Left: "\x1bOD"
EventKey: Right: "\x1bOC"

So following the instructions for binding raw escape sequences, I added this to my ~/.config/micro/bindings.json:
"\x1bOC": "SelectWordRight",
"\x1bOD": "SelectWordLeft",

But after that, when launching micro, it says "Error reading bindings.json: invalid character 'x' in string escape code".
Exactly like in this older issue #1863 .

Specifications

Commit hash: c2cebaa
OS: Debian Stretch (connecting from MobaXterm on Windows, through SSH)
Terminal: MobaXterm

[friendly-bits]

My complete bindings.json file:

{
    "Alt-/": "lua:comment.comment",
    "\x1bOC": "SelectWordRight",
    "\x1bOD": "SelectWordLeft",
    "CtrlUnderscore": "lua:comment.comment"
}

[JoeKar]

We need to check, if the prefix of x1b in bindings.go#L155 is sufficient, since @zyedidia mentioned in keybindings.md the following:

The terminal sends these events in the form of escape sequences often (but not always) starting with 0x1b.

After some local hacking I assume, that the "\x1b" at bindings.go#L155 and inside the bindings.json needs the escape of \ and thus needs to be change to "\\x1b".

[friendly-bits]

@JoeKar I tried to change the lines to:

    "\\x1bOC": "SelectWordRight",
    "\\x1bOD": "SelectWordLeft",

Now micro says at launch:

\x1bOC is not a bindable event
\x1bOD is not a bindable event

I'm not sure but maybe the cause is that micro already assigns these events to the Left and Right keys? Just guessing but this is what I gather from the initial check in raw mode which produced this output when I pressed the aforementioned key combinations:

EventKey: Left: "\x1bOD"
EventKey: Right: "\x1bOC"

Edit: Maybe I misunderstood you and you meant that something needs to be changed in the program code? If so, I'll wait for something I can test. Unfortunately, I don't know Go and I'm not acquainted with the code here, so I can't really fix this in the code.

[JoeKar]

@JoeKar I tried to change the lines to:

    "\\x1bOC": "SelectWordRight",
    "\\x1bOD": "SelectWordLeft",

Now micro says at launch:

\x1bOC is not a bindable event
\x1bOD is not a bindable event

Ah, you're right...so while changing the stuff locally I didn't realized, that escaping the backslash in the bindings.json is already enough to forward the correct string from the parsed json to micro.
Now the trick is, as already mentioned above (comment updated), that we need to adapt the bindings.go#L155 otherwise the strings.HasPrefix() fails due to unmatched prefix.

But at the end I've to ask the question, why not simply assign "Left" & "Right" to your actions?

When I understood tcell right, then it will try to handle the escaped sequence first, after no other event handling has matched. In case you now try to use more or less "known" key events then tcell will not create his raw event and thus micro wouldn't even receive a possible registered raw event.

[friendly-bits]

But at the end I've to ask the question, why not simply assign "Left" & "Right" to your actions?

Well, Left and Right are the arrow keys. I need to use them with the default actions to move around the text. My current workaround is to use AltRight and AltLeft for actions SelectWordRight and SelectWordLeft. So it's not a critical issue for me. But of course, I would prefer to use CtrlShiftRight etc because these are the defaults in most text editing applications.

[friendly-bits]

When I understood tcell right, then it will try to handle the escaped sequence first, after no other event handling has matched. In case you now try to use more or less "known" key events then tcell will not create his raw event and thus micro wouldn't even receive a possible registered raw event.

Sorry, I didn't get what you mean by that last paragraph and I don't even know what tcell is.

[JoeKar]

But at the end I've to ask the question, why not simply assign "Left" & "Right" to your actions?

Well, Left and Right are the arrow keys. I need to use them with the default actions to move around the text. My current workaround is to use AltRight and AltLeft for actions SelectWordRight and SelectWordLeft. So it's not a critical issue for me. But of course, I would prefer to use CtrlShiftRight etc because these are the defaults in most text editing applications.

Ok, check, understood. Didn't read your description carefully enough, sorry for that.

Sorry, I didn't get what you mean by that last paragraph and I don't even know what tcell is.

No worries. According to my current understanding it looks like a bug of micro (doesn't treat the hard coded escape prefix as escaped escape) and tcell (which simply converts \x1b into a string instead of adding the same escaped escape as string in front of the sequence), at least in case the raw events shall really transport strings representing the printable escaped sequence.
But I'm currently unsure, if this is all, because with my tested raw sequences I had to move the escape sequence handling in front of the normal event handling in tcell.

So long story short, it's a bit tricky to solve this issue.

[dmaluka]

One of the problems here is that JSON does not support the \x notation, so in bindings.json the raw sequence bindings need to be specified slightly differently:

    "\u001bOC": "SelectWordRight",
    "\u001bOD": "SelectWordLeft",

(Yeah, arguably it is a bug: micro does not attempt to properly parse \x-escaped strings, instead it simply relies on the JSON parser which does not support \x either, so the resulting behavior is non-intuitive and undocumented.)

The bad news here is that it is not the only problem. With the above \u notation, micro does not print any errors, and I can see that micro does call screen.Screen.RegisterRawSeq(), however the bindings themselves still don't work (i.e. nothing happens when I press the corresponding keys). I remember it used to work for me some (rather long) time ago, so I think it is a regression, either in micro or in tcell.

[friendly-bits]

@dmaluka I just tried binding this. I'm not getting any error either, and I don't get the assigned function either (SelectWordRight etc). What I get is the same as pressing the Left and Right arrow keys (which is also the behavior without the bindings).

[JoeKar]

@dmaluka

(Yeah, arguably it is a bug: micro does not attempt to properly parse \x-escaped strings, instead it simply relies on the JSON parser which does not support \x either, so the resulting behavior is non-intuitive and undocumented.)

Yep, but this is something which seems to be fixed easily (see below).

With the above \u notation, micro does not print any errors, and I can see that micro does call screen.Screen.RegisterRawSeq(), however the bindings themselves still don't work (i.e. nothing happens when I press the corresponding keys). I remember it used to work for me some (rather long) time ago, so I think it is a regression, either in micro or in tcell.

I was able with \\x to get them forwarded and registered to tcell (which needs a further improvement to not register the same raw event twice or more...) and after little dirty hacking I received the expected callback via raw event (in \\x notation) too. As a result the configured action has been performed.
So I already assumed, that this seems to be a bug on both sides (see above) and especially tcell doesn't create the raw events in the actual situation. So in case it was working in the past then the regression has been introduced somehow introduced in tcell.
Since the same situation works without double \\ in \u notation we seem to have a problem in json5 too. :(

Already prepared...

• https://github.com/zyedidia/json5/blob/master/scanner.go#L486 + 'x'
• https://github.com/zyedidia/json5/blob/master/decode.go#L1235 + additional case for 'x' reading all bytes till the next '"'

...and working to register the \x notated raw sequence.
With the newly linked PRs in json5 and tcell I was able to use the raw escape sequence of Right to register the lua comment action. micro was out of scope afterwards.

[JoeKar]

@dmaluka
Again I'd appreciate your feedback on this. ;)

[dmaluka]

I'll try to find time to take a close look into it.

In the meantime, I think I like the idea of adding \x support directly to the json parser (although it is non-standard, but hey).

...Actually there is one more thing that is not working: cannot bind raw sequences in micro via bind command.

[JoeKar]

...Actually there is one more thing that is not working: cannot bind raw sequences in micro via bind command.

I will have a look on that too, since I'm more or less comfortable with that topic.

Edit:
The \x1b support for bind was the easy part, but allowing the \u notation is much more complex, because we can't simply exchange "\u" by "\u". :(