Keeping persistent history in bash

June 11th, 2013 at 7:27 pm

For someone spending most of his time in front of a Linux terminal, history is very important. But traditional bash history has a number of limitations, especially when multiple terminals are involved (I sometimes have dozens open). Also it’s not very good at preserving just the history you’re interested in across reboots.

There are many approaches to improve the situation; here I want to discuss one I’ve been using very successfully in the past few months – a simple "persistent history" that keeps track of history across terminal instances, saving it into a dot-file in my home directory (~/.persistent_history). All commands, from all terminal instances, are saved there, forever. I found this tremendously useful in my work – it saves me time almost every day.

Why does it go into a separate history and not the main one which is accessible by all the existing history manipulation tools? Because IMHO the latter is still worthwhile to be kept separate for the simple need of bringing up recent commands in a single terminal, without mixing up commands from other terminals. While the terminal is open, I want the press "Up" and get the previous command, even if I’ve executed a 1000 other commands in other terminal instances in the meantime.

Persistent history is very easy to set up. Here’s the relevant portion of my ~/.bashrc:

log_bash_persistent_history()
{
  [[
    $(history 1) =~ ^\ *[0-9]+\ +([^\ ]+\ [^\ ]+)\ +(.*)$
  ]]
  local date_part="${BASH_REMATCH[1]}"
  local command_part="${BASH_REMATCH[2]}"
  if [ "$command_part" != "$PERSISTENT_HISTORY_LAST" ]
  then
    echo $date_part "|" "$command_part" >> ~/.persistent_history
    export PERSISTENT_HISTORY_LAST="$command_part"
  fi
}

# Stuff to do on PROMPT_COMMAND
run_on_prompt_command()
{
    log_bash_persistent_history
}

PROMPT_COMMAND="run_on_prompt_command"

The format of the history file created by this is:

2013-06-09 17:48:11 | cat ~/.persistent_history
2013-06-09 17:49:17 | vi /home/eliben/.bashrc
2013-06-09 17:49:23 | ls

Note that an environment variable is used to avoid useless duplication (i.e. if I run ls twenty times in a row, it will only be recorded once).

OK, so we have ~/.persistent_history, how do we use it? First, I should say that it’s not used very often, which kind of connects to the point I made earlier about separating it from the much higher-use regular command history. Sometimes I just look into the file with vi or tail, but mostly this alias does the trick for me:

alias phgrep='cat ~/.persistent_history|grep --color'

The alias name mirrors another alias I’ve been using for ages:

alias hgrep='history|grep --color'

Another tool for managing persistent history is a trimmer. I said earlier this file keeps the history "forever", which is a scary word – what if it grows too large? Well, first of all – worry not. At work my history file grew to about 2 MB after 3 months of heavy usage, and 2 MB is pretty small these days. Appending to the end of a file is very, very quick (I’m pretty sure it’s a constant-time operation) so the size doesn’t matter much. But trimming is easy:

tail -20000 ~/.persistent_history | tee ~/.persistent_history

Trims to the last 20000 lines. This should be sufficient for at least a couple of months of history, and your workflow should not really rely on more than that :-)

Finally, what’s the use of having a tool like this without employing it to collect some useless statistics. Here’s a histogram of the 15 most common commands I’ve used on my home machine’s terminal over the past 3 months:

ls        : 865
vi        : 863
hg        : 741
cd        : 512
ll        : 289
pss       : 245
hst       : 200
python    : 168
make      : 167
git       : 148
time      : 94
python3   : 88
./python  : 88
hpu       : 82
cat       : 80

Some explanation: hst is an alias for hg st. hpu is an alias for hg pull -u. pss is my awesome pss tool, and is the reason why you don’t see any calls to grep and find in the list. The proportion of Mercurial vs. git commands is likely to change in the very near future due to this.

Related posts:

  1. Book review: “A short history of the United States” by Edward Channing
  2. grep through code history with Git, Mercurial or SVN
  3. Book review: “A short history of nearly everything ” by Bill Bryson

8 Responses to “Keeping persistent history in bash”

  1. ripper234No Gravatar Says:

    Persistent history in bash is something I found 2 years ago while working at Google (first time actually working in Ubuntu). I shamelessly copied some of the code into a small github repo which I clone into every bash root folder on every machine I work on.

    I’m not a real bash ninja, and this is a “private” repo for my own purposes, so I don’t claim the code there is nice/documented/clean/whatnot. I also usually manually call the .scripts/bash_global.sh from .bashrc when I set up the machine.

    Do you have any other bash tricks you use?
    If you start working on a fresh ubuntu machine, what customization do you do?

  2. BjörnNo Gravatar Says:

    Great post! However, you need to set
    export HISTTIMEFORMAT=’%F %T ‘
    for the regexp to work!

  3. elibenNo Gravatar Says:

    @Ron,

    I have a private repo in which I keep all not-part-of-a-big-project code and snippets and configuration bits I’ve ever written. It has a rc_files directory with all kinds of dotfiles (bashrc, vimrc, gitconfig and so on) which I ln -sf into $HOME on each new machine. Similarly, I have a linux_bin directory with a bunch of (mostly Python, but also some bash) scripts I ln -sf into ~/bin which is on $PATH. I also have a document somewhere with a rough list of stuff not to forget when installing a new Ubuntu box.

    @Bjorn,

    Yep, that’s part of my .bashrc too, along with:

    export HISTCONTROL=ignoredups:erasedups
    export HISTSIZE=2000
    export HISTFILESIZE=2000
    export HISTTIMEFORMAT="%F %T  "
  4. Joe AbbeyNo Gravatar Says:

    OOoooo:

    echo $hostname "|" $date_part "|" "$command_part" >> ~/.persistent_history

    And:

    ln -s ~/Dropbox/persistent_history ~/.persistent_history

    Now you have all those commands persistent across hosts!

    (forgive my attempts to copy-pasta block markup…)

  5. elibenNo Gravatar Says:

    @Joe,

    Yep, this is an option if you regularly use multiple hosts you want to use in the same way.

  6. timvNo Gravatar Says:

    I’ve been using something very similar to this for a few months too. It really is a life saver. I took mine one step further and included the current working directory the command was executed in. This way I can filter bash history by the projects I was working on. Extending you config to do this is pretty straight forward. Other meta data might be useful to track as well.

  7. Mathias LaurinNo Gravatar Says:

    Interesting.

    I know you said size is not a problem for you but still, since you write your own tools, why not zip the file?
    echo $date_part "|" "$command_part" | gzip >> ~/.persistent_history.gz
    and use z{grep,more,less} or gzcat?
    i.e. alias phgrep='cat ~/.persistent_history.gz | zgrep --color' or alias phgrep='gzcat ~/.persistent_history|grep --color'.

  8. elibenNo Gravatar Says:

    @Mathias,

    I’m not sure I see the point. The problem with size would be access speed (appending, grepping), not size in itself. Zipping surely increases access speed. I suppose it only makes sense if you have a very small storage device – but that’s hard to imagine since this file really doesn’t grow very quickly (just a few MB a year or so).

Leave a Reply

To post code with preserved formatting, enclose it in `backticks` (even multiple lines)