Submit Hint Search The Forums LinksStatsPollsHeadlinesRSS
14,000 hints and counting!

Describe all Unix apps in current PATH UNIX
Previous hints have mentioned that you can get a list of Unix-level commands in the Terminal via auto-completion features of the shell, and that you can get a one-line synopsis of a specified Unix command with the whatis command -- and that this can be used to search for a desired command.

The hint presents a way to generate a list of all of the Unix commands in your execution PATH (see this Unix FAQ for an explanation of the execution PATH variable) with a one-line synopsis for each program (assuming this one-line description is available). This is accomplished via a rather complicated pipeline of commands:

echo $PATH | sed -e 's/:/ /g' | xargs -J % find % -maxdepth 1 \( -type f -or -type l \) | xargs basename | sort | uniq | xargs whatis 2> /dev/null | grep -E '\((1|1m|6|8)\)' | perl -ne '($name, $descrip) = m/^(.*?)\s+- (.*)$/; $name =~ s/\((1|1m|6|8)\)//g; printf("%-20s - %s\n", $name, $descrip)'

Here are the first several lines of output from the above command when run on my Mac:
SystemStarter    - Start, stop, and restart system services
Xmark            - summarize x11perf results
Xnest            - a nested X server
Xquartz          - X window system server for Quartz operating system
Xvfb             - virtual framebuffer X server for X Version 11
a2p              - Awk to Perl translator
...
If you want the output of the above command to be saved into a file instead of appearing in the Terminal window, add a greater-than sign and then the name of the file you want to create at the end of the command.

For example, if you want to save the output into a file named unix_commands.txt in your Documents folder, add
> ~/Documents/unix_commands.txt
at the end of the above command:

echo $PATH | sed -e 's/:/ /g' | xargs -J % find % -maxdepth 1 \( -type f -or -type l \) | xargs basename | sort | uniq | xargs whatis 2> /dev/null | grep -E '\((1|1m|6|8)\)' | perl -ne '($name, $descrip) = m/^(.*?)\s+- (.*)$/; $name =~ s/\((1|1m|6|8)\)//g; printf("%-20s - %s\n", $name, $descrip)' > ~/Documents/unix_commands.txt

If you just want to use the above command to get the list of commands with descriptions, that's all you need to know. The rest of this hint will explain how the above command works by dissecting it into its various parts.

A pipeline of Unix commands is a series of commands separated by the pipe symbol (a vertical bar: |). The output of each command is used as input for the next command in the sequence. The first part of the above command is:
echo $PATH
This outputs the colon-separated list of directories (folders) that are in your execution PATH. The next stage of the command pipes this output into:
sed -e 's/:/ /g'
which changes each colon (:) into a space. The next stage of the pipeline pipes the resulting space-separated list of directories into the find command via the xargs command:
xargs -J % find % -maxdepth 1 ( -type f -or -type l )
The -J % option to the xargs command tells it to substitute the strings it receives as input at the place designated by % in the subsequent (find) command. This means that we are starting the search of the find command from the list of directories that we got from the PATH variable. For each directory in the PATH, we find all files or symbolic links (-type f -or -type l) that are in that directory. (The -maxdepth 1 tells it not to look in sub-directories.) We need to use the xargs command here because the find command expects to get the list of directories as command-line arguments, not as input.

One important note: we are making the simplifying assumption that none of the directories in the PATH have spaces in their names. That is usual for Unix directories.

The output from the find command will be a list of the files that are in the directories in the PATH. Each entry in this list will be a full pathname -- i.e. it will include the names of the directories where the file is. (e.g. one entry in this list will be /usr/bin/cmp). The next stage of the pipeline pipes this list of file pathnames into the basename command via the xargs command:
xargs basename
This gets rid of the directory information and just leaves us with a list of filenames (e.g. cmp instead of /usr/bin/cmp). We then pipe this list of files into the sort command, followed by the uniq command. This sorts the list of files into alphabetic order and eliminates any duplicates. And then we pipe that list of files into the whatis command via the xargs command:
sort | uniq | xargs whatis
The output from the whatis command is a series of lines (of descriptive info about the commands) from the (previously generated) whatis database. Some of these lines describe functions and other items that happen to be named the same as the commands we are interested in. There might be some irrelevant errors from the whatis command, so we discard any error messages by redirecting the standard-error stream (#2) to /dev/null:
2> /dev/null
The info we are interested in comes mostly from section 1 of the man pages (section 1 describes Unix commands, sections 2 and 3 describe functions useful to programmers, etc), but some commands are listed in section 1m or section 6, and several are in section 8, and so we pipe the results into the grep command to filter out only those lines that refer to the sections we are interested in:
grep -E '((1|1m|6|8))'
This also gets rid of the lines relating to commands for which whatis has no description. (This means that our final result won't be a complete list of all commands in the PATH, but will only show those commands for which a one-line description is available.)

The last stage in our pipeline is the following Perl command which removes the now-useless references to section 1 from the lines and reformats the lines for easy readability:

perl -ne '($name, $descrip) = m/^(.*?)\s+- (.*)$/; $name =~ s/\((1|1m|6|8)\)//g; printf("%-20s - %s\n", $name, $descrip)'

This first extracts the command name and description into separate Perl variables ($name, $descrip) and then gets rid of all occurrences of the man page section numbers from the command names. It then prints each command name in a field of width 20 (so that the subsequent descriptions all line up nicely) followed by a dash and then the description.

If you want to understand the stages of this pipeline better, you could remove the stages one by one starting from the right side, running the command each time to see what the output is.

[robg adds: Sorry for the use of the div blocks for the code, as I know they're not universally liked. However, in this case, it was the best way to present the long, complex string such that it could be easily copied and pasted.]
    •    
  • Currently 2.25 / 5
  • 1
  • 2
  • 3
  • 4
  • 5
  (4 votes cast)
 
[12,230 views]  

Describe all Unix apps in current PATH | 14 comments | Create New Account
Click here to return to the 'Describe all Unix apps in current PATH' hint
The following comments are owned by whoever posted them. This site is not responsible for what they say.
Describe all Unix apps in current PATH
Authored by: merlyn on Jan 09, '07 08:12:16AM
You know, I hear people say how ugly Perl is, but let's compare the Perl equivalent to the massively messy thing just posted:

#!/usr/bin/env perl
use strict;
$|++;

my %commands;
for my $dir (split /:/, $ENV{PATH}) {
  next unless opendir DIR, $dir;
  for my $command (readdir DIR) {
    $commands{$command}++ if -f "$dir/$command" and -x _;
  }
}

for my $command (sort keys %commands) {
  for (`whatis $command`) {
    my ($key, $detail) = split /\s+- / or next;
    next unless $key =~ s/\((1|1m|6|8)\)//g;
    printf "%20s - %s", $key, $detail;
  }
}
Not only that, but it probably runs hella faster. :)

[ Reply to This | # ]
Describe all Unix apps in current PATH
Authored by: merlyn on Jan 09, '07 08:15:43AM
Oh wait, that's not hella faster, because it still invokes whatis repeatedly (and with a whitespace problem). This is hella faster:

#!/usr/bin/env perl
use strict;
$|++;

my %commands;
for my $dir (split /:/, $ENV{PATH}) {
  next unless opendir DIR, $dir;
  for my $command (readdir DIR) {
    $commands{$command}++ if -f "$dir/$command" and -x _;
  }
}

open WHATIS, "-|", "whatis", sort keys %commands or die;
while (<WHATIS>) {
  my ($key, $detail) = split /\s+- / or next;
  next unless $key =~ s/\((1|1m|6|8)\)//g;
  printf "%20s - %s", $key, $detail;
}


[ Reply to This | # ]
Describe all Unix apps in current PATH
Authored by: gshenaut on Jan 09, '07 08:57:51AM
Well, formatting helps a little:
echo $PATH \
| sed -e 's/:/ /g' \
| xargs -J % find % -maxdepth 1 \( -type f -or -type l \) \
| xargs basename | sort | uniq \
| xargs whatis 2> /dev/null \
| grep -E '\((1|1m|6|8)\)' \
| perl -ne '($name, $descrip) = m/^(.*?)\s+- (.*)$/; $name =~ s/\((1|1m|6|8)\)//g; printf("%-20s - %s\n", $name, $descrip)'
Greg Shenaut

[ Reply to This | # ]
Sorry, bash wins.
Authored by: bm on Jan 25, '07 11:46:07PM
merlyn:
That's not faster at all. Even with an added command in bash (see below), 'time' reports:
  bash
real    0m44.014s
user    0m11.898s
sys     0m25.316s

  perl
real    0m55.462s
user    0m0.302s
sys     0m0.172s
One caveat: that's bash 3.2.9 -- the latest, while perl remains at 5.8.7;
..the bash-code used an additional 'sed' to escape whitespacing in $PATH.

[ Reply to This | # ]
pedagogy
Authored by: hayne on Jan 09, '07 10:57:04AM

As a tool to produce the list of commands with descriptions, a pure Perl solution is obviously better. And that's probably how I would have done it had I started out to produce such a tool. But this complicated pipeline of commands started off life in a discussion about getting a list of commands in your PATH (with no descriptions) and grew from there.

I thought it was valuable pedagogically to leave it as a pipeline since that way readers can experiment (in the way I mentioned at the end of the article) to see what results with different commands.
This sort of experimentation would be less likely had I just presented a Perl script.

I think for many readers, the idea of a pipeline of commands and the explanations of the pipeline stages may be more valuable than the end tool.



[ Reply to This | # ]
pedagogy
Authored by: ccjensen on May 21, '08 08:00:53AM

I know this is 'old' news, but thanks for the script and BRILLIANT explanation of each step. I am revising for an exam which will require us to write pipelines, and this was most helpful!



[ Reply to This | # ]
Describe all Unix apps in current PATH
Authored by: adrianm on Jan 09, '07 08:40:17AM
Not *exactly* equivalent, but surely this is a lot quicker and simpler?

man -k 1
(ie show all commands in section 1 of the manual)

For me, this returns 1091 items compared to 1221 from:


find ${PATH//:/ } -type f -print0 | xargs -0 basename | sort | uniq
so perhaps good enough.

---
~/.sig: not found

[ Reply to This | # ]

Describe all Unix apps in current PATH
Authored by: ernst_mulder on Jan 09, '07 08:40:52AM

Why so difficult? This is how I would do it in my favourite shell, tcsh:

foreach i ( $path )
whatis `ls $i` | grep -e "- "
end



[ Reply to This | # ]
Describe all Unix apps in current PATH
Authored by: Hal Itosis on Jan 09, '07 10:19:59AM
Sorta like:

apropos '(1)'

-HI-

[ Reply to This | # ]
Describe all Unix apps in current PATH
Authored by: ernst_mulder on Jan 18, '07 05:48:49AM

Darn, hadn't thought of that!



[ Reply to This | # ]
Help finding the right command
Authored by: jecwobble on Jan 09, '07 08:52:02AM

This is all great, but when I'm trying to remember which command does the task I'm trying to accomplish at the moment, I don't want to have to visually scan through several hundred commands.

I'm away from my Mac at the moment, so I don't know how responsive any of these options are, but I suppose I could write a script that incorporates one of these options along with perhaps egrep to whittle the list down to say, any commands dealing with PDFs for instance.

Aside from that, anyone know how to find the right command quickly?



[ Reply to This | # ]
Help finding the right command
Authored by: cleanhead on Jan 09, '07 04:42:56PM
Help finding the right command
Authored by: calum on Jan 10, '07 06:33:43PM

"man -k pdf" or "apropos pdf" will show you all the commands related to pdf handling, for example.



[ Reply to This | # ]
Describe all Unix apps in current PATH
Authored by: vocaro on Jan 09, '07 10:52:12AM

Sorry for the use of the div blocks for the code, as I know they're not universally liked.

What's so bad about div blocks for code? I like it that way a lot better than the usual way this site posts code, which IIRC is via iframes (yech).



[ Reply to This | # ]