Unix tip #2: explicit for loops, command substitution

A lot of unix commands are designed to operate on a large number of files at once.  For instance, the move file command, mv, has as its (simplified) arguments

mv file 1 [file 2, ... file n] destination

It’s for that reason that you can do something like

mv *.jpg ~/Desktop/Images

and have all the jpegs in the current working directory moved to the Images folder on the Desktop.  So many unix commands are set up that way that you might not need to know how to explicitly iterate (loop).  I was forced to learn this syntax when I had a large number of .tar.gz (roughly equivalent to zip files for those more familiar with Windows land) files to decompress.

The command I usually use to extract the contents of a zipped file is

tar -xzvf /path/to/tar/file

This expects a single file path; doing something like tar -xzvf *.tar.gz will not work.

Nick@Macintosh-2 ~/Desktop/TarExample$ ls
foo.tar.gz  foo2.tar.gz
Nick@Macintosh-2 ~/Desktop/TarExample$ tar -zxvf *.tar.gz
tar: foo2.tar.gz: Not found in archive
tar: Error exit delayed from previous errors

As you can see, this isn’t going to work.  Instead we need an explicit loop.  The general syntax is

for i in [iterable]; do [command with variable $i]; done

For instance,

Nick@Macintosh-2 ~/Desktop/TarExample$ for i in 1 2 3 4 5; do echo $i; done
1
2
3
4
5

Recalling our last unix tip, we could replace this with

 Nick@Macintosh-2 ~/Desktop/TarExample$ for i in {1..5}; do echo $i; done
 1
2
3
4
5

The iterable list is whitespace separated.  This is very important for what I’m about to show to you next.

If you’re familiar with basic Unix functionality, you know that you list the contents of a directory with the ls command.  Let’s do that here.

Nick@Macintosh-2 ~/Desktop/TarExample$ ls foo.tar.gz  foo2.tar.gz
If you’ll notice, these are exactly the filenames we need to pass into the tar command.  Let’s try with echo first.

for i in ls; do echo $i; done ls
Well, that didn’t work.  What’s going on?  Turns yout you need to add backticks (the key to the left of the 1 key) around the ls command; otherwise bash treats it as text.

or i in `ls`; do echo $i; done foo.tar.gz foo2.tar.gz
We can go ahead and replace the echo command with our tar command:

 Nick@Macintosh-2 ~/Desktop/TarExample$ ^echo^tar -xzvf

 for i in `ls`; do tar -xzvf $i; done
 a.txt
 b.txt
 a.txt
 b.txt

The stdout here shows the contents that were extracted from the .tar.gz files.

But what is that syntax?

 ^echo^tar -xzvf

?  This is another neat feature of bash; you can repeat the last command, textually substituting the second command for the first.  I could have just as easily hit the up key, moved my cursor, deleted echo, replaced it with tar -xzvf, but this is faster to type for me.

Just for another example,

 echo "Hello"
 Hello
 Nick@Macintosh-2 ~/Desktop/TarExample$ ^Hello^World
 echo "World"
 World

In actuality, I would not use `ls`; what happens if there were things other than .tar.gz files in the directory?  We’d be calling the tar command with the incorrect arguments.  Instead we only want it to affect the files ending in.tar.gz; this is a place where the * wildcard comes in handy.

Nick@Macintosh-2 ~/Desktop/TarExample$ ls
a.txt       b.txt       foo.tar.gz  foo2.tar.gz

 Nick@Macintosh-2 ~/Desktop/TarExample$ ls *.tar.gz
 foo.tar.gz  foo2.tar.gz

So I can use this in my earlier command,

 for i in `ls *.tar.gz`; do tar -xzvf $i; done

Note that you can avoid the use of backticks if you use plain wildcard expansion:

 for i in *.tar.gz; do tar -xzvf $i; done

The reason that this works without the use of backticks is that the ls text is a command that needs to be run by the shell; the * is an expression that is evaluated earlier in the pipeline.  Read more about globbing.

Nested for loops

I haven’t had a need to nest for-loops yet, but you can if you wish.

for i in {1,2,3}; do for j in {3,4,5}; do echo $i $j; done; done
1 3
1 4
1 5
2 3
2 4
2 5
3 3
3 4
3 5

Conclusion:

I have shown you how to explicitly iterate over lists in bash, how to use wildcard matching to restrict the set of objects returned by command, and how to replace one piece of the last command with another.  In most cases you will not need to explicitly iterate over lists, due to the way many unix commands are written, but it’s a useful skill to have nonetheless.