Most powerful examples of Unix commands or scripts every programmer should know

Question

There are many things that all programmers should know, but I am particularly interested in the Unix/Linux commands that we should all know. For accomplishing tasks that we may

Answer 1

Some way to search (multiple) badly formatted log files, in which the search string may be found on an "orphaned" next line. For example, to display both the 1st, and a concatenated 3rd and 4th line when searching for id = 110375:

[2008-11-08 07:07:01] [INFO] ...; id = 110375; ...
[2008-11-08 07:07:02] [INFO] ...; id = 238998; ...
[2008-11-08 07:07:03] [ERROR] ... caught exception
...; id = 110375; ...
[2008-11-08 07:07:05] [INFO] ...; id = 800612; ...

I guess there must be better solutions (yes, add them...!) than the following concatenation of the two lines using sed prior to actually running grep:

#!/bin/bash

if [ $# -ne 1 ]
then
  echo "Usage: `basename $0` id"
  echo "Searches all myproject's logs for the given id"
  exit -1
fi  

# When finding "caught exception" then append the next line into the pattern
# space bij using "N", and next replace the newline with a colon and a space
# to ensure a single line starting with a timestamp, to allow for sorting
# the output of multiple files:
ls -rt /var/www/rails/myproject/shared/log/production.* \
  | xargs cat | sed '/caught exception$/N;s/\n/: /g' \
  | grep "id = $1" | sort

...to yield:

[2008-11-08 07:07:01] [INFO] ...; id = 110375; ...
[2008-11-08 07:07:03] [ERROR] ... caught exception: ...; id = 110375; ...

Actually, a more generic solution would append all (possibly multiple) lines that do not start with some [timestamp] to its previous line. Anyone? Not necessarily using sed, of course.

Answer 2

I use this so much I am actually ashamed of myself. Remove spaces from all filenames and replace them with an underscore:

[removespaces.sh]

#!/bin/bash
find .  -type f -name "* *" | while read file
do
   mv "$file" "${file// /_}"
done

Answer 3

My personal favorite is the lsof command.

"lsof" can be used to list opened file descriptors, sockets, and pipes. I find it extremely useful when trying to figure out which processes have used which ports/files on my machine.

Example: List all internet connections without hostname resolution and without port to port name conversion.

lsof -i -nP

http://www.manpagez.com/man/8/lsof/

Answer 4

The fact you can use -name and -iname multiple times in a find command was an eye opener to me.

[findplaysong.sh]

#!/bin/bash
cd ~
echo Matched...
find /home/musicuser/Music/ -type f  -iname "*$1*" -iname "*$2*" -exec echo {} \;
echo Sleeping 5 seconds
sleep 5
find /home/musicuser/Music/ -type f  -iname "*$1*" -iname "*$2*" -exec mplayer {} \;
exit

Answer 5

To run in parallel several processes without overloading too much the machine (in a multiprocessor architecture):

NP=`cat /proc/cpuinfo | grep processor | wc -l`
#your loop here
    if [ `jobs | wc -l` -gt $NP ];
    then
         wait
    fi
    launch_your_task_in_background&
#finish your loop here

Answer 6

The power of this tools (grep find, awk, sed) comes from their versatility, so giving a particular case seems quite useless.

man is the most powerful comand, because then you can understand what you type instead of just blindly copy pasting from stack overflow.

Example are welcome, but there are already topics for tis. My most used :

grep something_to_find * -R

which can be replaced by ack and

find | xargs 

find with results piped into xargs can be very powerful