Epic: Persisted Terminal History

[svenefftinge]

Context

• Currently, workspaces do not persist history.
• When starting a new workspace you do not get history from all previous workspaces launched in Gitpod.
• History is relied on by many developers to do their work.

Context

A terminal history is a very important productivity tool in local dev environments. With ephemeral workspaces we only have a history per session, which is inconvenient. Gitpod should persist the terminal history between workspaces for the same project and user. It's the expectation of all users coming from a local dev environment with a heavy use of terminals.

Acceptance Criteria

• Opening a new workspace will show bash history for previous workspaces
• Documentation is updated with history support

Out of scope

• Shells other than bash.

Archive

### Out of scope

• Other shells beyond bash.

This feature doesn't necessarily need to be advertised, as it is an existing expectation when working with terminals. Users will be positively surprised to see that Gitpod finally keeps history state across multiple workspaces.

Persona(s)

All users with a terminal heavy workflow.

In scope

Keep history for our default shell (bash) for the same project and user.

Out of scope

Supporting all possible shells.

[csweichel]

@akosyakov could we reuse what we built for VS Code user setting sync here?

[akosyakov]

@akosyakov could we reuse what we built for VS Code user setting sync here?

It maybe easy to use blob API from server. User setting sync is scoped to a user and versioned. It looks like here it is scoped to the same workspace and does not need support multiple versions.

keeps history state across multiple workspaces

@svenefftinge I'm not sure about this bit. If I start a completely new workspace I would expect that it is fresh. For me preserving history only between restarts of the same workspace actually makes sense. Maybe should be configurable.

[kylos101]

@akosyakov I would expect new workspaces to have empty history too. But, if I were to restart an existing one, having that history would be nice. If you "really" need something in a new workspace, I would argue that is what dotfiles are for. 😅

Another thing to consider is, how would a user delete history. Say for example, if it contained a secret they did not want persisted.

[akosyakov]

One thing which come up today that there are many clients controlling shell sessions, i.e. supervisor, VS Code, JetBrains, OpenSSH server and so on? Which one we are talking here? If all of them and we don't need to sync between different workspace then maybe the history should be preserved as a part of workspace backup? @csweichel

[csweichel]

there are many clients controlling shell sessions

We would rely on the behaviour of the shell (e.g. bash) and just backup/restore the $HISTFILE. This way we're independent of the process/client controlling the shell session, and users can get creative how they want to handle multiple sessions.

how would a user delete history. Say for example, if it contained a secret they did not want persisted.

If we just back up the histfile, users can modify this file to delete things from their history.

A few questions arise though:

• how do we know where the histfile is located? If the user is using e.g. zsh or fish the history will be stored elsewhere compared to e.g. bash. We could try and detect/guess which shell is used, look at the envvars of that shell to find the HISTFILE and in absence fall back to sensible defaults (bash: ~/.history, zsh: ~/.zsh_history).
• in what context to we restore the history? There doesn't seem to be consensus yet as to the "scope" of the terminal history. We could try and maintain history for a user, a project or a single workspace across restarts.

[axonasif]

But, if I were to restart an existing one, having that history would be nice.

@kylos101 I had the same needs and found that it's relatively easy to do from dotfiles alone [1] 😁

The idea is to create dummy hist files within /workspace/.persist for a newly started Workspace, later symlink them back to their default $HOME based locations for the first time and on every re-start from dotfiles install.sh

[akosyakov]

how do we know where the histfile is located? If the user is using e.g. zsh or fish the history will be stored elsewhere compared to e.g. bash. We could try and detect/guess which shell is used, look at the envvars of that shell to find the HISTFILE and in absence fall back to sensible defaults (bash: ~/.history, zsh: ~/.zsh_history).

Detection is rather tricky, a user can overwrite shell configuration on level of editors as well. Could not we store all?

in what context to we restore the history? There doesn't seem to be consensus yet as to the "scope" of the terminal history. We could try and maintain history for a user, a project or a single workspace across restarts.

I wonder for cross workspaces support how are we going to merge it? Last workspace wins and overwrites, always append? Should it sync in real time?

[akosyakov]

found https://github.com/rcaloras/bashhub-client maybe somebody tried

[loujaybee]

I do agree with keeping simple and limiting supported shells, but I must say that zsh is very popular. Looking into usage stats I think would be worthwhile here. Also the recent move from Apple might impact zsh adoption going forward [1].

[ghuntley]

👍 for using HISTFILE approach. My expectation is that history is scoped to the workspace and is not shared between workspaces.

[trumbitta]

I do agree with keeping simple and limiting supported shells, but I must say that zsh is very popular. Looking into usage stats I think would be worthwhile here. Also the recent move from Apple might impact zsh adoption going forward [1].

Also the default shell on MacOS for a couple years now

[trumbitta]

👍 for using HISTFILE approach. My expectation is that history is scoped to the workspace and is not shared between workspaces.

Personally, I need the history to be shared between workspaces. At the very least scoped per project. I'm 100% on board with the opening statement by @svenefftinge : "Gitpod should persist the terminal history between workspaces for the same project and user."

[svenefftinge]

I've seen several comments where people don't want to keep a history per project. I wonder why that is?
Wouldn't I run the same set of commands in all workspaces for a certain project? Isn't training the history some sort of investment that I don't want to lose because of ephemeral workspaces?

I'm still very strongly thinking it should be per project. A relatively simple solution could be to store the history in an env variable. That has the right scope (project) and can be manually deleted in case the history contains information I want to get rid of.

[arel]

As a heavy user of terminals and terminal history (especially Ctrl+R), I love this proposal. Like @svenefftinge, my expectation would be to keep history per project. (E.g., losing all my history when I open a workspace for a new branch would be inconvenient.)

I wonder for cross workspaces support how are we going to merge it? Last workspace wins and overwrites, always append? Should it sync in real time?

"Last workspace wins and overwrites" makes sense since it is similar to how bash history already works with multiple terminals. Plus, it would make it easier to delete history if there's information I want to remove.

[mteam88]

Any updates on this? Specifically: any workaround in place or theoretically possible.

[axonasif]

@mteam88 if you wish to persist bash history only, then it's fairly simple. Just create a new env var called HISTFILE at https://gitpod.io/variables with the value of /workspace/.bash_history and set the scope to */*.

Note: This will not help you to sync history across workspaces.

Okay, this wouldn't work, you need to rely on dotfiles, see below.

[spearki]

@axonasif Unfortunately it looks like HISTFILE gets truncated at workspace startup, so you only get history from the (n-1)th session, and only in task terminals (which happen to start early enough to load the history before the truncate happens).

Do you think it would be possible to fix that in lieu of an official solution here, so we have a workaround option?

[axonasif]

Hmm @spearki, wasn't aware of that line.

In that case, you could put this file to $HOME/.bashrc from your dotfiles to workaround, it's guaranteed to work. Just copy pasted from this repo with a few changes.

[spearki]

@axonasif We actually applied a very similar change to your draft PR to our own Docker image, and this is where we discovered the truncation problem. Doing it in dotfiles is also subject to the same truncation. We are now doing sed -i '/truncate/d' /ide/startup.sh as a workaround, but this is obviously an awful hack.

[axonasif]

@spearki ah, yeah, makes sense... I use fish for interactive use... so never noticed 😅
I kind of hate the fact that both bash and zsh use HISTFILE as the variable name 🙃

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mgenereu]

Does #17612 close this? I just had it start working much to my surprise!

[akosyakov]

@mgenereu only for the same workspace. We would like to support it more generally, i.e. across all user shells for instance.

[loujaybee]

Should also note this is also enabled via 3rd parties like Atuin: https://www.gitpod.io/guides/persisted-terminal-history-atuin

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.