CINXE.COM
Console productivity hack: Discover the frequent; then make it the easy
<!DOCTYPE html> <html> <head><script type="text/javascript" src="/_static/js/bundle-playback.js?v=HxkREWBo" charset="utf-8"></script> <script type="text/javascript" src="/_static/js/wombat.js?v=txqj7nKC" charset="utf-8"></script> <script>window.RufflePlayer=window.RufflePlayer||{};window.RufflePlayer.config={"autoplay":"on","unmuteOverlay":"hidden"};</script> <script type="text/javascript" src="/_static/js/ruffle/ruffle.js"></script> <script type="text/javascript"> __wm.init("https://web.archive.org/web"); __wm.wombat("http://matt.might.net:80/articles/console-hacks-exploiting-frequency/","20170430102941","https://web.archive.org/","web","/_static/", "1493548181"); </script> <link rel="stylesheet" type="text/css" href="/_static/css/banner-styles.css?v=S1zqJCYt" /> <link rel="stylesheet" type="text/css" href="/_static/css/iconochive.css?v=3PDvdIFv" /> <!-- End Wayback Rewrite JS Include --> <title>Console productivity hack: Discover the frequent; then make it the easy</title> <link rel="alternate" type="application/rss+xml" title="RSS" href="https://web.archive.org/web/20170430102941/http://matt.might.net/articles/feed.rss"/> <link rel="stylesheet" href="/web/20170430102941cs_/http://matt.might.net/css/raised-paper-2.css"/> <meta name="viewport" content="width=480, initial-scale=1"/> <link rel="stylesheet" media="screen and (max-device-width: 480px)" href="/web/20170430102941cs_/http://matt.might.net/css/raised-paper-2-handheld.css"/> <script type="text/javascript" src="/web/20170430102941js_/http://matt.might.net/matt.might.js"></script> <script type="text/javascript"> var ArticleVersion = 2 ; </script> <script> <!-- include("article-style.js"); //--> </script> <script type="text/javascript" src="/web/20170430102941js_/http://matt.might.net/articles/manifest.js"></script> <script type="text/javascript" src="/web/20170430102941js_/http://matt.might.net/articles/index-manifest.js"></script> <script type="text/javascript"> <!-- // var Key = "[an error occurred while processing the directive]"; var Pathname = location.pathname ; var PathParts = Pathname.split(/\//) ; var Key = PathParts[PathParts.length-1] ; if (Key == "") Key = PathParts[PathParts.length-2] ; //--> </script> </head> <body> <div id="body"> <div id="abstract-container" class="module"> <div id="abstract-content" class="fat-content"> <h1>Console productivity hack: Discover the frequent; then make it the easy</h1> <div> [<a href="/web/20170430102941/http://matt.might.net/articles/">article index</a>] [<script> var emailMatt = '<a href="mai'+'lto:matt-blog'+'@'+'migh'+'t.net">email me</a>' document.write(emailMatt); //--> </script>] [<a href="https://web.archive.org/web/20170430102941/http://twitter.com/mattmight">@mattmight</a>] [<a href="https://web.archive.org/web/20170430102941/http://gplus.to/mattmight">+mattmight</a>] [<a href="/web/20170430102941/http://matt.might.net/articles/feed.rss">rss</a>] </div> <p> For the expert user, nothing matches the efficacy of the command line. </p> <p> It's embarrassing that this is still true.</p> <p> Aside from multiple consoles and tab completion, the console interface hasn't evolved much in 30 years. </p> <p> The continued dominance of the command line among experts is a testament to the power of linguistic abstraction: when it comes to computing, a word is worth a thousand pictures. </p> <p> Yet, research in human-computer interaction barely acknowledges the command line's existence. It's a strange omission, since the core principles of human factors engineering still apply to the console.</p> <p> For the moment, let's consider the principle of frequency: </p> <div style="text-align: center;"> <em>The ease of performing a task <br/> should be proportional <br/> to its frequency.</em> </div> <p> If there's something you have to do a lot, you should make it easy to do.</p> <p> With a cockpit, this means that the design of a control panel should put commonly accessed controls nearest the pilot. </p> <p> In the kitchen, it means that the most frequently used cooking implements and ingredients should be within arms reach of the chef (and especially not hidden in drawers).</p> <p> In this article, I'll give a few examples of how I apply the principle of frequency to my console computing, including: </p> <ul> <li>silently logging console activity to MySQL;</li> <li>damping frequencies to eliminate defunct activities;</li> <li>mining these logs to find frequent tasks; and</li> <li>employing frequency-adaptive commands.</li> </ul> <hr/> <p> <b>Update</b>: Readers have pointed out easier ways to implement the technical bits with shell scripts. I've included these at the bottom for comparison. </p> </div> <!-- /#content --> </div> <!-- /#content-container --> <div class="module fat-container"> <div class="fat-content"> <center> <script type="text/javascript"><!-- google_ad_client = "pub-4400645483943138"; /* Header ad unit */ google_ad_slot = "8276008011"; google_ad_width = 468; google_ad_height = 60; //--> </script> <script type="text/javascript" src="https://web.archive.org/web/20170430102941js_/http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> </center> </div> </div> <div id="content-container" class="module"> <div id="article-content"> <script> Might.enableSyntaxHighlighting("Perl") ; </script> <script> Might.enableSyntaxHighlighting("Sql") ; </script> <script> Might.enableSyntaxHighlighting("Bash") ; </script> <h2>More resources</h2> <ul> <li> The best book I know of for console hacks is <a href="https://web.archive.org/web/20170430102941/http://www.amazon.com/gp/product/0596004613?ie=UTF8&tag=ucmbread-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=0596004613">Linux Server Hacks: 100 Industrial-Strength Tips and Tools</a><img src="https://web.archive.org/web/20170430102941im_/http://www.assoc-amazon.com/e/ir?t=ucmbread-20&l=as2&o=1&a=0596004613" width="1" height="1" border="0" alt="" style="border:none !important; margin:0px !important;"/>. It's loaded with fiendishly creative scripts and shortcuts. (Almost all of the hacks apply to any Unix-derived OS, including Mac OS X.) </li> <li> It turns out that my colleague <a href="https://web.archive.org/web/20170430102941/http://www.cs.utah.edu/~eeide/">Eric Eide</a> did a <a href="https://web.archive.org/web/20170430102941/http://www.cs.utah.edu/~eeide/pubs/thesis.ps.gz">masters thesis</a> on an adaptive interface to a Unix shell, Valet. Valet uses a mixture of system knowledge and heuristics to "detect and correct the kinds of mistakes that experienced users make most frequently: typographical errors, file location errors, and minor syntactic errors." Neat! </li> </ul> <h2>Logging and mining console activity</h2> <p> Before you can exploit the principle of frequency, you need an unbiased record of what it is that you do most frequently. You might even be surprised by what your most frequent tasks are. </p> <p> Fortunately, shells like bash already have some of that data in the form of the <code>history</code> command. The <code>history</code> command prints a list of recent commands. bash stores this history (by default) in <code>~/.bash_history</code>. </p> <p> You should periodically examine your frequently used commands, and find ways to execute them quickly. In many cases, a one-letter alias (e.g. <code>e</code> for <code>emacs</code>) is surprisingly effective. </p> <p> To find frequent commands, you'll need a script to compute counts for the contents of the history file. For this, I created a perl script called <code>frequency</code>: </p> <pre class="brush: pl; gutter: false;"> my %counts = () ; while (my $cmd = <STDIN>) { chomp $cmd ; if (!$counts{$cmd}) { $counts{$cmd} = 1 ; } else { $counts{$cmd}++ ; } } foreach $k (keys %counts) { my $count = $counts{$k} ; print "$count $k\n" ; } </pre> <p> It counts the number of times that each line occurs on stdin, and then dumps out a report. </p> <p> For example: </p> <pre> $ cat ~/.bash_history | frequency | sort -rn | head 78 ls 25 recent 17 ls -l 16 mutt 12 cd matt.might.net 12 cd 10 echo hi 9 emacs .bash_profile 7 cd articles/ 5 sudo perl -MCPAN -e 'install DBD::mysql' ; </pre> <p> This example is clearly representative of about a day's worth of my recent activity, since my history only goes back for 500 lines. To get a truly representative sample, you'll probably want to increase your history size. Inside your <code>.bash_profile</code>, you should add: </p> <pre> export HISTFILESIZE=10000 # Record last 10,000 commands export HISTSIZE=10000 # Record last 10,000 commands per session </pre> <p> At the same time, while it's useful to look at the full commands, it's more instructive to look at just the commands themselves. For this, you can use <code>cut</code>: </p> <pre> $ cat ~/.bash_history | cut -d" " -f1 | frequency | sort -rn | head 84 ls 67 cd 48 recent 47 history 37 sudo 26 emacs 22 echo 15 mutt 15 ecmd 13 r </pre> <p> It shouldn't be any surprise to an experienced console user that nearly a third of all my commands are to list files and change directories. Since these commands are so common, I'll focus on optimizing those two. That's where my biggest "bang for buck" comes from. </p> <h2>Logging directories to MySQL</h2> <p> Switching directories is a common console task. In any given week, however, perhaps 80-90% of a your time will be spent in a small working set directories. This frequency provides a chance to optimize. </p> <p> What you should do is log the directories visited, and then create a script for quickly switching to one of the most visited directories. </p> <p> To log directories, you can use (1) a MySQL database to store the data, and (2) the <code>PROMPT_COMMAND</code> environment variable to insert a script into the console that runs every time a console prompt displays. </p> <p> (Yes, this is an abuse of the <code>PROMPT_COMMAND</code> variable, but that's OK.) </p> <p> Setting up the table is straightforward: </p> <pre class="brush: sql; gutter: false;"> CREATE TABLE dircounts (path VARCHAR(255) PRIMARY KEY NOT NULL, count INT NOT NULL DEFAULT 0, time TIMESTAMP NOT NULL) </pre> <p> Then, you need a script, <code>markdir.pl</code>, to mark the current directory: </p> <pre class="brush: pl; gutter: false;"> use strict; use warnings; use DBI; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Connection failed: $DBI::errstr"; my $qpwd = $dbh->quote($ENV{'PWD'}) ; $dbh->do("INSERT INTO dircounts(path,count) ". "VALUES ($qpwd,1) ". "ON DUPLICATE KEY UPDATE count=count+1") ; $dbh->disconnect(); </pre> <p> Before each time the prompt is displayed, bash evaluates the contents of the environment variable <code>PROMPT_COMMAND</code>. This variable is just the right hook to monitor the current directory; for instance, in <code>~/.bash_profile</code>, we can add: </p> <pre class="brush: bash; gutter: false;"> export LASTDIR="/" function prompt_command { # Record new directory on change. newdir=`pwd` if [ ! "$LASTDIR" = "$newdir" ]; then /path/to/markdir.pl fi export LASTDIR=$newdir } export PROMPT_COMMAND="prompt_command" </pre> <h3>Damping the frequency</h3> <p> You should damp the counts over time, so that if you spend one week accessing a directory a lot, but never again thereafter, it disappears from the frequent directories set quickly. Otherwise, you could have a defunct directory that sticks around for a long time after you've finished a major project. </p> <p> To do this, I recommend a cronjob that runs a damping script once a week: </p> <pre class="brush: pl; gutter: false;"> use strict; use warnings; use DBI; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Could not connect to database: $DBI::errstr"; my $qpwd = $dbh->quote($ENV{'PWD'}) ; # Cut all counts in half: $dbh->do("UPDATE dircounts SET count = count/2"); $dbh->disconnect(); </pre> <p> The script exponentially decays counts, so that as soon as you stop using a directory regularly, its relative importance fades quickly. </p> <p> Run <code>crontab -e</code> and then add: </p> <pre> 0 0 * * 0 /path/to/damping/script </pre> <h2>A frequency-adaptive change directory</h2> <p> With a database of all directory visits, it's easy to write a perl script, <code>f</code>, that lists these directories and permits changing to them with just a couple keystrokes: </p> <pre class="brush: pl; gutter: false;"> #!/usr/bin/env perl use strict; use warnings; use DBI; my $target = shift ; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Could not connect to database: $DBI::errstr"; my $sth = $dbh->prepare("SELECT path ". "FROM dircounts ". "ORDER BY count DESC LIMIT 7") ; $sth->execute() ; my $count = 1 ; if (!$target) { print "+ change to directory with: f <number>\n" ; } while (my @row = $sth->fetchrow_array) { if ($target) { if ($count == $target) { $sth->finish() ; $dbh->disconnect(); system(". cdto '@row'") ; exit ; } } else { my $home = $ENV{'HOME'} ; my $path = "@row" ; $path =~ s/$home/~/ ; print "${count}: $path\n" ; } $count++ ; } $dbh->disconnect(); </pre> <p> For example: </p> <pre> $ f + change to recent directory with: f <number> 1: ~ 2: ~/family/bertrand 3: ~/ucombinator/private/grants 4: ~/ucombinator/private/papers 5: ~/ucombinator/private/projects 6: ~/matt.might.net 7: ~/talks </pre> <h2>A recency-adaptive change directory</h2> <p> Another useful perl script, <code>r</code>, lists the most recently accessed directories. </p> <pre class="brush: pl; gutter: false;"> #!/usr/bin/env perl use strict; use warnings; use DBI; my $target = shift ; # DB settings my $host = "localhost" ; my $database = "..." ; my $user = "..." ; my $password = "..." ; my $dbh = DBI->connect("DBI:mysql:$database", "$user", "$password") || die "Could not connect to database: $DBI::errstr"; my $span = 1000 * 60 * 60 * 7 ; my $q = "SELECT path ". "FROM dircounts ". "WHERE UNIX_TIMESTAMP(time) > UNIX_TIMESTAMP(now()) - $span ". "ORDER BY time DESC LIMIT 7" ; my $sth = $dbh->prepare($q) ; $sth->execute() ; my $count = 1 ; if (!$target) { print "+ change to directory with: r <number>\n" ; } while (my @row = $sth->fetchrow_array) { if ($target) { if ($count == $target) { $sth->finish() ; $dbh->disconnect(); system(". cdto '@row'") ; exit ; } } else { my $home = $ENV{'HOME'} ; my $path = "@row" ; $path =~ s/$home/~/ ; print "${count}: $path\n" ; } $count++ ; } $dbh->disconnect(); </pre> <h2>List recent files on directory change</h2> <p> In addition to changing directories, my other most common task was listing directory contents. Looking at the history file, I saw that an <code>ls</code> usually comes right after a <code>cd</code>. </p> <p> This indicates that automatically listing directory contents after a directory switch should save time. </p> <p> But, instead of listing <em>all</em> of the directory contents, it's more useful list only the most recently used files. </p> <p> To do this, we need to augment <code>PROMPT_COMMAND</code> to check if the directory has changed, and if so, to run a command like <code>ls -t | head -7</code>, which tells us the 7 most recently modified files in the directory: </p> <pre class="brush: bash; gutter: false;"> export LASTDIR="/" function prompt_command { # Record new directory on change. newdir=`pwd` if [ ! "$LASTDIR" = "$newdir" ]; then /path/to/markdir.pl # List contents: ls -t | head -7 fi export LASTDIR=$newdir } export PROMPT_COMMAND="prompt_command" </pre> <p> Recency is actually a good approximation for frequency. Seeing the seven most recently modified files in a directory tends to quickly jog my mind and remind me where I left off. </p> <h2>Switching to the last directory on login</h2> <p> Another comman pattern in my work-flow is to open a second terminal window and then move to the directory I'm currently in. Once again, this change directory step should be eliminated by having new terminals open to the most recently used directory. Modifying <code>~/.bash_profile</code> yet again provides the desired behavior. </p> <pre class="brush: bash; gutter: false;"> # Change to most recently used directory: if [ -f ~/.lastdir ]; then cd "`cat ~/.lastdir`" fi export LASTDIR="/" function prompt_command { # Remember where we are: pwd > ~/.lastdir # Record new directory on change. newdir=`pwd` if [ ! "$LASTDIR" = "$newdir" ]; then /path/to/markdir.pl # List contents: ls -t | head -7 fi export LASTDIR=$newdir } export PROMPT_COMMAND="prompt_command" </pre> <h3>The cdto script</h3> <p> The scripts above use a <code>cdto</code> command, which changes to a new directory from within a script. To accomplish this, <code>cdto</code> actually writes to the file <code>~/.lastdir</code>, and starts a new shell. </p> <pre class="brush: bash; gutter: false;"> #!/bin/bash echo + Switching to $* echo + Press CTRL-D to return to `pwd` echo $* > ~/.lastdir bash --login </pre> <p> The advantage of this is that a user can quickly change to a new directory, do work and "pop" back to the old directory by exiting the shell. </p> <h2>Recent files in emacs</h2> <p> Emacs can remember recent files as well. Add the following to <code>~/.emacs</code>: </p> <pre> (require 'recentf) (recentf-mode 1) (setq recentf-max-menu-items 25) (global-set-key "\C-x\ \C-r" 'recentf-open-files) </pre> <p> Then <code>C-x C-r</code> to see recently opened files. </p> <h2>Conclusion</h2> <p> The principle of frequency isn't profound; but it is effective. Just because the console is a powerful way to compute doesn't mean we shouldn't look for ways to make it more efficient. Logging activity and then mining that data is a low-cost way for console users to discover their most frequent tasks. And, once those tasks are discovered, users can devise methods (or additional shell scripts) to make them go faster. </p> <h2>Update: Easier ways</h2> <p> One of the beautiful things about Unix is that you can use it for a decade and still find new things to learn every day. </p> <p> Multiple readers have pointed out that much of these scripts could be replaced with simple, elegant shell hacks. </p> <p> For instance, you can implement <code>frequency</code> as: </p> <pre> $ sort | uniq -c | sort -g </pre> <p> And, you can count commands with: </p> <pre> $ history | cut -c8- | sort | uniq -c | sort -rn | head </pre> <p> For jumping between directories, try <a href="https://web.archive.org/web/20170430102941/https://github.com/joelthelion/autojump">autojump</a>. </p> <h2>Related pages</h2> <ul> <script> RenderTagLinks("productivity") ; </script> </ul> <script> <!-- SyntaxHighlighter.all() ; //--> </script> <hr/> <div id="footer-links"> [<a href="/web/20170430102941/http://matt.might.net/articles/">article index</a>] [<script> var emailMatt = '<a href="mai'+'lto:matt-blog'+'@'+'migh'+'t.net">email me</a>' document.write(emailMatt); //--> </script>] [<a href="https://web.archive.org/web/20170430102941/http://twitter.com/mattmight">@mattmight</a>] [<a href="https://web.archive.org/web/20170430102941/http://gplus.to/mattmight">+mattmight</a>] [<a href="/web/20170430102941/http://matt.might.net/articles/feed.rss">rss</a>] </div> </div> <!-- /#content --> </div> <!-- /#content-container --> <div id="footer-ad" class="module fat-container"> <div class="fat-content"> <center> <script type="text/javascript"><!-- google_ad_client = "pub-4400645483943138"; /* Article footer banner */ google_ad_slot = "3531754286"; google_ad_width = 468; google_ad_height = 60; //--> </script> <script type="text/javascript" src="https://web.archive.org/web/20170430102941js_/http://pagead2.googlesyndication.com/pagead/show_ads.js"> </script> </center> </div> <!-- /footer-ad --> </div> <!-- /footer-ad-container --> <div id="footer-linode" class="module fat-container"> <div class="fat-content"> <center> matt.might.net is powered by <b><a href="https://web.archive.org/web/20170430102941/http://www.linode.com/?r=bf5d4e7c8a1af61855b5227279a6744c3bde8a8a">linode</a></b> | <a href="/web/20170430102941/http://matt.might.net/articles/legal/">legal information</a> </center> </div> </div> </div> <!-- /#body --> <script type="text/javascript"> var gaJsHost = (("https:" == document.location.protocol) ? "https://web.archive.org/web/20170430102941/https://ssl." : "https://web.archive.org/web/20170430102941/http://www."); document.write(unescape("%3Cscript src='" + gaJsHost + "google-analytics.com/ga.js' type='text/javascript'%3E%3C/script%3E")); </script> <script type="text/javascript"> var pageTracker = _gat._getTracker("UA-3661244-1"); pageTracker._trackPageview(); </script> </body> </html> <!-- FILE ARCHIVED ON 10:29:41 Apr 30, 2017 AND RETRIEVED FROM THE INTERNET ARCHIVE ON 08:23:49 Dec 11, 2024. JAVASCRIPT APPENDED BY WAYBACK MACHINE, COPYRIGHT INTERNET ARCHIVE. ALL OTHER CONTENT MAY ALSO BE PROTECTED BY COPYRIGHT (17 U.S.C. SECTION 108(a)(3)). --> <!-- playback timings (ms): captures_list: 0.76 exclusion.robots: 0.034 exclusion.robots.policy: 0.02 esindex: 0.016 cdx.remote: 9.539 LoadShardBlock: 263.017 (3) PetaboxLoader3.datanode: 125.429 (4) PetaboxLoader3.resolve: 1581.53 (3) load_resource: 1539.056 -->