Problem

You find yourself standing outside of a perfect maze. A maze is defined as "perfect" if it meets the following conditions:

1. It is a rectangular grid of rooms, R rows by C columns.
2. There are exactly two openings on the outside of the maze: the entrance and the exit. The entrance is always on the north wall, while the exit could be on any wall.
3. There is exactly one path between any two rooms in the maze (that is, exactly one path that does not involve backtracking).

You decide to solve the perfect maze using the "always turn left" algorithm, which states that you take the leftmost fork at every opportunity. If you hit a dead end, you turn right twice (180 degrees clockwise) and continue. (If you were to stick out your left arm and touch the wall while following this algorithm, you'd solve the maze without ever breaking contact with the wall.) Once you finish the maze, you decide to go the extra step and solve it again (still always turning left), but starting at the exit and finishing at the entrance.

The path you take through the maze can be described with three characters: 'W' means to walk forward into the next room, 'L' means to turn left (or counterclockwise) 90 degrees, and 'R' means to turn right (or clockwise) 90 degrees. You begin outside the maze, immediately adjacent to the entrance, facing the maze. You finish when you have stepped outside the maze through the exit. For example, if the entrance is on the north and the exit is on the west, your path through the following maze would beWRWWLWWLWWLWLWRRWRWWWRWWRWLW:

If the entrance and exit were reversed such that you began outside the west wall and finished out the north wall, your path would be WWRRWLWLWWLWWLWWRWWRWWLW. Given your two paths through the maze (entrance to exit and exit to entrance), your code should return a description of the maze.

Como sempre, o Google Code Jam especifica o formato do arquivo de saída desejado para o programa. Deixo você visitarem a página do site para ver os detalhes. Segue meu programa, novamente em shell+awk (eu consigo desenvolver bem rapidamente usando awk, em particular quando usam-se arrays e carateres)


#!/bin/bash

dbg=0
if [ ! -z "$*" ]; then dbg=1 ; fi

awk -v dbg=$dbg 'FNR==1 { cases=$1; cn=0; next }
{
delete x
cn++
if (cn>cases) next
print "Case #"cn":"
r=0
c=1
inout=$1
outin=$2
li=length(inout)
lo=length(outin)
dv=1
dh=0
dir=1
dd[1]=2
dd[2]=4
dd[3]=1
dd[4]=8
mw=1
me=1
ms=0
if (dbg) print "init: dv=1 dh=1 dir=1 r=0 c=1"
for (i=1;i<=li;i++) {
a=substr(inout,i,1)
if (a=="W") {
x[r,c,dir]=1
if (dbg) print "("r","c","dir")=1"
r=r+dv
c=c+dh
dirop=((dir+2)%4)+(dir==2)*4
x[r,c,dirop]=1
if (dbg) print "("r","c","dirop")=1"
} else if (a=="L") {
dir=dir-1+4*(dir==1)
} else if (a=="R") {
dir=(dir%4)+1
}
dv=(dir==1)-(dir==3)
dh=(dir==4)-(dir==2)
if (i

if (r>ms) ms=r
if (c if (c>me) me=c
}
if (dbg) print "("a") dv="dv" dh="dh" dir="dir" r="r" c="c
}
# going back: return
dir=((dir%4)+1)%4+1
dv=(dir==1)-(dir==3)
dh=(dir==4)-(dir==2)
if (dbg) print "back: dv="dv" dh="dh" dir="dir" r="r" c="c
for (i=1;i<=lo;i++) {
a=substr(outin,i,1)
if (a=="W") {
x[r,c,dir]=1
if (dbg) print "("r","c","dir")=1"
r=r+dv
c=c+dh
dirop=((dir+2)%4)+(dir==2)*4
x[r,c,dirop]=1
if (dbg) print "("r","c","dirop")=1"
} else if (a=="L") {
dir=dir-1+4*(dir==1)
} else if (a=="R") {
dir=(dir%4)+1
}
dv=(dir==1)-(dir==3)
dh=(dir==4)-(dir==2)
if (i
if (r>ms) ms=r
if (c if (c>me) me=c
}
if (dbg) print "("a") dv="dv" dh="dh" dir="dir" r="r" c="c
}
for (r=1; r<=ms; r++) {
for (c=mw;c<=me;c++) {
v=dd[1]*x[r,c,1]+dd[2]*x[r,c,2]+dd[3]*x[r,c,3]+dd[4]*x[r,c,4]
printf substr("0123456789abcdef",1+v,1);
}
print ""
}
if (dbg) {
for (r=0; r<=ms; r++) {
for (c=mw;c<=me;c++) {
if (r>0 && x[r,c,2]==0) printf "|" >"/dev/stderr"
else printf " " >"/dev/stderr"
if (x[r,c,1]==0) printf "_" >"/dev/stderr"
else printf " " >"/dev/stderr"
if (r>0 && c==me && x[r,c,4]==0) printf "|" >"/dev/stderr"
}
print "" >"/dev/stderr"
}
}
}' src.in > src.out


Eu inclui uma opção de "debug" no programa (basta passar qualquer argumento ao programa) que apresenta passo-a-passo a travessia do labirínto, e também representa "graficamente" (em modo de texto) o labirínto. Cuidado que testando o programa com os labiríntos de verdade propostos no concurso, muitos estão de grande largura e a representação dos mesmos fica meio "quebrada"...

Abraços,
Phil.

If you have the same problem, here's a fast-and-dirty solution.

1. export your Firefox bookmarks in HTML format, in a file called, say, bookmarks.html
2. using a bash shell, extract the feed lines into a file called feeds.xml doing

    grep FEEDURL bookmarks.html > feeds.xml

3. create the following script called "scriptfeed.sh"

   #!/bin/bash
   echo "<?xml version=\"1.0\" encoding=\"UTF-8\"?>"
   echo "<opml version=\"1.0\">"
   echo "<head><title>RSS Subscriptions</title></head>"
   echo "<body>"
   while read line
   do
           temp=`echo $line | awk -F"<" {'print $3;'}`
           feed=`echo $temp | awk -F"\"" {'print $2;'}`
           url=`echo $temp | awk -F"\"" {'print $4;'}`
           text=`echo $temp | awk -F">" {'print $2;'}`
           echo "<outline text=\"$text\""
           echo "  title=\"$text\""
           echo "  type=\"rss\""
           echo "  xmlUrl=\"$feed\""
           echo "  htmlUrl=\"$url\"/>"
   done < feeds.xml
   echo "</body>"
   echo "</opml>"

4. Give the script the right to execute

   chmod +x scriptfeed.sh

5. Launch the script on the output file, say, exp.xml

   ./scriptfeed.sh > exp.xml

I know, it's dirty and tricky, but it works :-P

Hi,

Here the placement for Application Designer.
Req. ID:
Req-AD-04 - Application Designer
Primary Skills:
LDAP,Database systems,Daytona data management,SQL,UNIX and C development and web development,HTML, JavaScript, Perl, UNIX, Linux…
Secondary Skills:
LDAP, Windows, SQL, AWK, C, CGI..

Description:
Dynamic Development Role with Telecom Leader! Project Name Application Designer, Fraud Management Project Description: Global Fraud Management system which is a continuous analytical engine and case manager that uses pattern recognition to detect fraud.

Job Description:

Developer will create new applications and enhance existing application using C, CGI, JavaScript, Perl, Shell, AWK, HTML, and SQL. Only submit extremely technically qualified candidates. Also desired:LDAPDatabase systems Daytona data managementBasic knowledge of database systemsDesign.

SQL Database concept:

Must be expert in all aspects of UNIX and C development and web development.Responsible for development and maintenance of mission-critical applications that support high-visibility client organizations and external customers on a24×7x365 basis. Technical assessment will be done on candidates. Background check will be required before the candidate can start the assignment.

REQUIRED SKILLS:

HTML, JavaScript, Perl, UNIX, Linux

PREFERRED SKILLS:

LDAP, Windows, SQL, AWK, C, CGI

Rate:DOE,

Job Type: Contract,

Total Exp: 6+Yrs,

Duration: 6+ Months,

Number Of Openings: 1,

Location: Florham Park, NJ

Contact Us:

KBS consultants

Flat H,Kulothungan Apts,

No, 5 Natesan Road,

Ashoknagar,

Chennai 600 083

India

Phone: +91-44 2489 5341 / 2371 9622

Visit us:

Email:www.kbsconsultants.com

www.kbsconsultants.org.in/

www.kbsconsultants.net.in/

Often used and under appreciated, the pipe feature in unix/linux/dos has to be my favorite tool in incident response and forensics.

Need the device at /dev/sda imaged with progress indicators and an md5sum?

Need a summary of the unique hosts from Internet Explorer's index.dat history file?

No other feature I can think of makes it easier to quickly analyze data. There's even a wiki entry dedicated to pipe 101 exploration.

Here's my top 3 bash shell pipelines I use every day on my gentoo linux workstation:

No doubt if you've been around the block more than a few times you've got your own pipelines you can't live without. If you're open to it, share them so folks can pick up something new or add to their favorites.

Links to more info:

Jeff Bryner , GCFA Gold #137, also holds the CISSP and GCIH certifications, occasionally teaches for SANS and performs forensics, intrusion analysis, and security architecture work on a daily basis.

Many standard utilities (rdist, make, cron, etc.) allow you to specify a command to run at a certain time. Usually, this command is simply passed to the Bourne shell, which means that you can execute whole scripts, should you choose to do so. Steve Bourne, wrote the Bourne shell which appeared in the Seventh Edition Bell Labs Research version of Unix.

Lastly, UNIX runs Bourne shell scripts when it boots. If you want to modify the boot-time behavior of a system, you need to learn to write and modify Bourne shell scripts. It said there are 95% shell code written in Bourne shell. Code written in Bourne shell is compatible with shell scripting like ksh, bash & zsh. That means with minimal or no change one could port shell script written in Bourne shell to ksh or bash or zsh.

*Bash is popular among Linux users.

There are couple of other shells available which I will not be talking are C shell ( Used by C program to create programming rich shell) this is incompatible to Bourne shell. The C shell, csh, and its variant tcsh is a fine interactive shell (I use tcsh), but is a lousy shell for writing scripts.

Lets understand what is an executable files, they also know as program or binary executables, This files are machine readable files and human eyes can not make sense out of it. Others only contain text, knows as script. They are interpreter scripts like awk, sed, perl and many more.

Learning Shell Scripting

#!/bin/sh

The first line of any script must begin with #!, followed by the name of the interpreter. Some versions of UNIX allow whitespace between #! and the name of the interpreter. Others do not. Hence, if you want your script to be portable, leave out the blank.

A script, like any file that can be run as a command, needs to be executable: save this script as rotatelog and run

chmod +x rotatelog
to make it executable.

You can now run it by running
./rotatelog

Unlike some other operating systems, UNIX allows any program to be used as a script interpreter. This is why people talk about ``a Bourne shell script'' or ``an awk script.'' One might even write a more script, or an ls script (though the latter wouldn't be terribly useful). Hence, it is important to let Unix know which program will be interpreting the script.

When Unix tries to execute the script, it sees the first two characters (#!) and knows that it is a script. It then reads the rest of the line to find out which program is to execute the script. For a Bourne shell script, this will be /bin/sh. Hence, the first line of our script must be

#!/bin/sh

After the command interpreter, you can have one, and sometimes more, options. Some flavors of Unix only allow one, though, so don't assume that you can have more.

Variables

sh allows you to have variables, just like any programming languages. Variables do not need to be declared.

To set a sh variable, use VAR=value

And to use the value of the variable later, use $VAR or ${VAR}

The latter syntax is useful if the variable name immediately followed by other text:

#!/bin/sh

COLOR=yellow

echo This looks $COLORish

echo This seems ${COLOR}ish

prints

This looks

This seems yellowish

There is only one type of variable in sh: strings.

This is somewhat limited, but is sufficient for most purposes.

Local vs. environment variables

A sh variable can be either a local variable or an environment variable. They both work the same way; the only difference lies in what happens when the script runs another program (which, as we saw earlier, it does all the time).

Environment variables are passed to subprocesses. Local variables are not.

By default, variables are local. To turn a local variable into an environment variable,

use

export VAR

Here's a simple wrapper for a program:

#!/bin/sh

NETSCAPE_HOME=/usr/imports/libdata

CLASSPATH=$NETSCAPE_HOME/classes

export CLASSPATH $NETSCAPE_HOME/bin/netscape.bin

Here, NETSCAPE_HOME is a local variable; CLASSPATH is an environment variable. CLASSPATH will be passed to netscape.bin (netscape.bin uses the value of this variable to find Java class files); NETSCAPE_HOME is a convenience variable that is only used by the wrapper script; netscape.bin doesn't need to know about it, so it is kept local.

The only way to unexport a variable is to unset it:

unset VAR

This removes the variable from the shell's symbol table, effectively making as if it had never existed; as a side effect, the variable is also unexported.

Also, if you have a function by the same name as the variable, unset will also delete that function. Since you may want to use this variable later, it is better not to define it in the first place.

Also, note that if a variable was passed in as part of the environment, it is already an environment variable when your script starts running. If there is a variable that you really don't want to pass to any subprocesses, you should unset it near the top of your script. This is rare, but it might conceivably happen.

If you refer to a variable that hasn't been defined, sh substitutes the empty string.

#!/bin/sh

echo aaa $FOO bbb

echo xxx${FOO}yyy

prints

aaa bbb xxxyyy

Special variables

sh treats certain variables specially: some are set for you when your script runs, and some affect the way commands are interpreted.

Command-line arguments

The most useful of these variables are the ones referring to the command-line arguments. $1 refers to the first command-line argument (after the name of the script), $2 refers to the second one, and so forth, up to $9.

If you have more than nine command-line arguments, you can use the shift command: this discards the first command-line argument, and bumps the remaining ones up by one position: $2 becomes $1, $8 becomes $7, and so forth.

The variable $0 (zero) contains the name of the script (argv[0] in C programs).

Often, it is useful to just list all of the command-line arguments.

For this, sh provides the variables $* (star) and $@ (at). Each of these expands to a string containing all of the command-line arguments, as if you had used $1 $2 $3...

The difference between $* and $@ lies in the way they behave when they occur inside double quotes: $* behaves in the normal way, whereas $@ creates a separate double-quoted string for each command-line argument.

That is, "$*" behaves as if you had written "$1 $2 $3", whereas "$@" behaves as if you had written "$1" "$2" "$3".

Finally, $# contains the number of command-line arguments that were given.

Other special variables

$? gives the exit status of the last command that was executed. This should be zero if the command exited normally.

$- lists all of the options with which sh was invoked. See sh(1) for details.

$$ holds the PID of the current process.

$! holds the PID of the last command that was executed in the background.

$IFS (Input Field Separator) determines how sh splits strings into words.

Quasi-variable constructs

The ${VAR} construct is actually a special case of a more general class of constructs:

${VAR:-expression}

Use default value: if VAR is set and non-null, expands to $VAR. Otherwise, expands to expression.

${VAR:=expression}

Set default value: if VAR is set and non-null, expands to $VAR. Otherwise, sets VAR to expression and expands to expression.

${VAR:?[expression]}

If VAR is set and non-null, expands to $VAR. Otherwise, prints expression to standard error and exits with a non-zero exit status.

${VAR:+expression}

If VAR is set and non-null, expands to the empty string. Otherwise, expands to expression.

${#VAR}

Expands to the length of $VAR.

The above patterns test whether VAR is set and non-null. Without the colon, they only test whether VAR is set.

Pattern-matching

sh supports a limited form of pattern-matching. The operators are

* Matches zero or more characters.

? Matches exactly one character.

[range] Matches any character in range.

Range can be either a list of characters that match, or two endpoints separated by a dash: [ak3] matches either a, k, or 3; [a-z] matches any character in the range a through z; [a-mz] matches either a character in the range a through m, or z. If you wish to include a dash as part of the range, it must be the first character, e.g., [-p] will match either a dash or p.

When an expression containing these characters occurs in the middle of a command, sh substitutes the list of all files whose name matches the pattern. This is known as ``globbing.'' Otherwise, these are used mainly in the case construct.

As a special case, when a glob begins with * or ?, it does not match files that begin with a dot. To match these, you need to specify the dot explicitly (e.g., .*, /tmp/.*).

Note to MS-DOS users: under MS-DOS, the pattern *.* matches every file. In sh, it matches every file that contains a dot.

Quoting

If you say something like

echo * MAKE $$$ FAST *

it won't do what you want: first of all, sh will expand the *s and replace them with a list of all the files in the current directory. Then, since any number of tabs or blanks can separate words, it will compress the three spaces into one. Finally, it will replace the first instance of $$ with the PID of the shell. This is where quoting comes in.

sh supports several types of quotes. Which one you use depends on what you want to do.

Backslash

Just as in C strings, a backslash (``\'') removes any special meaning from the character that follows. If the character after the backslash isn't special to begin with, the backslash has no effect. The backslash is itself special, so to escape it, just double it: \\.

Single quotes

Single quotes, such as 'foo' work pretty much the way you'd expect: anything inside them (except a single quote) is quoted.

You can say

echo '* MAKE $$$ FAST *'

and it'll come out the way you want it to.

Note that a backslash inside single quotes also loses its special meaning, so you don't need to double it. There is no way to have a single quote inside single quotes.

Double quotes, such as

"foo"

preserve spaces and most special characters. However, variables and backquoted expressions are expanded and replaced with their value.

If you have an expression within backquotes (also known as backticks), e.g.,

`cmd`

the expression is evaluated as a command, and replaced with whatever the expression prints to its standard output. Thus,

echo You are `whoami`

prints

You are irfan

Built-in commands

sh understands several built-in commands, i.e., commands that do not correspond to any program. These commands include:

{ commands ; }, ( commands )

Execute commands in a subshell. That is, run them as if they were a single command. This is useful when I/O redirection is involved, since you can pipe data to or from a mini-script inside a pipeline.

The { commands; } variant is somewhat more efficient, since it doesn't spawn a true subshell. This also means that if you set variables inside of it, the changes will be visible in the rest of the script.

: (colon)

Does nothing. This is generally seen as

: ${VAR:=default}

. filename

The dot command reads in the specified filename, as if it had occurred at that place in the script.

bg [job], fg [job]

bg runs the specified job (or the current job, if none is specified) in the background. fg resumes the specified job (or the current job, if none is specified) in the foreground.

Jobs are specified as %number. The jobs command lists jobs.

cd [dir]

Sets the current directory to dir. If dir is not specified, sets the current directory to the home directory.

pwd

Prints the current directory.

echo args

Prints args to standard output.

eval args

Evaluates args as a sh expression. This allows you to construct a string on the fly (e.g., using a variable that holds the name of a variable that you want to set) and execute it.

exec command

Runs the specified command, and replaces the current shell with it. That is, nothing after the exec statement will be executed, unless the exec itself fails.

exit [n]

Exit the current shell with exit code n. This defaults to zero.

kill [-sig] %job

Send signal sig to the specified job. sig can be either numeric or symbolic. kill -l lists all available signals. By default, sig is SIGTERM (15).

read name...

Reads one line from standard input and assigns it to the variable name. If several variables name1, name2, name3 etc. are specified, then the first word of the line read is assigned to name1, the second to name2, and so forth. Any remaining words are assigned to the last variable.

set [+/-flag] [arg]

With no arguments, prints the values of all variables.set -x turns on the x option to sh; set +x turns it off.set args... sets the command-line arguments to args.

test expression

Evaluates a boolean expression and exits with an exit code of zero if it is true, or non-zero if it is false. See test for more details.

trap [command sig]...

If signal sig is sent to the shell, execute command. This is useful for exiting cleanly (e.g., removing temporary files etc.) when the script is interrupted.

ulimit

Print or set system limits on resource usage.

umask [nnn]

Sets the umask to nnn (an octal number). With no argument, prints the current umask. This is most useful when you want to create files, but want to restrict who can read or write them.

wait [n]

Wait for the background process whose PID is n to terminate. With no arguments, waits for all of the background processes to terminate.

Bear in mind that the list of builtins varies from one implementation to another, so don't take this list as authoritative.

Flow control

sh supports several flow-control constructs, which add power and flexibility to your scripts.

if

The if statement is a simple conditional. You've seen it in every programming language. Its syntax is

if condition ; then
commands
[elif condition ; then
commands]...
[else
commands]
fi

That is, an if-block, optionally followed by one or more elif-blocks (elif is short for ``else if''), optionally followed by an else-block, and terminated by fi.

The if statement pretty much does what you'd expect: if condition is true, it executes the if-block. Otherwise, it executes the else-block, if there is one. The elif construct is just syntactic sugar, to let you avoid nesting multiple if statements.

#!/bin/sh

myname=`whoami`

if [ $myname = root ]; then

echo "Welcome to FooSoft 3.0"

else

echo "You must be root to run this script"

exit 1

fi

The more observant among you (or those who are math majors) are thinking, ``Hey! You forgot to include the square brackets in the syntax definition!''

Actually, I didn't: [ is actually a command, /bin/[, and is another name for the test command. See below for details.

This is why you shouldn't call a test program test: if you have ``.'' at the end of your path, as you should, executing test will run /bin/test.

The condition can actually be any command. If it returns a zero exit status, the condition is true; otherwise, it is false. Thus, you can write things like

#!/bin/sh

user=arnie

if grep $user /etc/passwd; then

echo "$user has an account"

else

echo "$user doesn't have an account"

fi

while

The while statement should also be familiar to you from any number of other programming languages. Its syntax in sh is

while condition; do
commands
done

As you might expect, the while loop executes commands as long as condition is true. Again, condition can be any command, and is true if the command exits with a zero exit status.

A while loop may contain two special commands: break and continue.

break exits the while loop immediately, jumping to the next statement after the done.

continue skips the rest of the body of the loop, and jumps back to the top, to where condition is evaluated.

for

The for loop iterates over all of the elements in a list. Its syntax is

for var in list; do
commands
done

list is zero or more words. The for construct will assign the variable var to each word in turn, then execute commands. For example:

#!/bin/sh

for i in foo bar baz "do be do"; do

echo "$i"

done

will print

foo bar baz do be do

A for loop may also contain break and continue statements. They work the same way as in the while loop.

Case

The case construct works like C's switch statement, except that it matches patterns instead of numerical values. Its syntax is

case expression in
pattern)
commands
;;
...
esac

expression is a string; this is generally either a variable or a backquoted command.

pattern is a glob pattern (see globbing).

The patterns are evaluated in the order in which they are seen, and only the first pattern that matches will be executed. Often, you'll want to include a ``none of the above'' clause; to do this, use * as your last pattern.

IO

A command's input and/or output may be redirected to another command or to a file. By default, every process has three file descriptors: standard input (0), standard output (1) and standard error (2). By default, each of these is connected to the user's terminal.

However, one can do many interesting things by redirecting one or more file descriptor:

< filename

Connect standard input to the file filename. This allows you to have a command read from the file, rather than having to type its input in by hand.

> filename

Connect standard output to the file filename. This allows you to save the output of a command to a file. If the file does not exist, it is created. If it does exist, it is emptied before anything happens.

(Exercise: why doesn't cat * > zzzzzzz work the way you'd expect?)

>> filename

Connects standard output to the file filename. Unlike >, however, the output of the command is appended to filename.

<<word

This construct isn't used nearly as often as it could be. It causes the command's standard input to come from... standard input, but only until word appears on a line by itself. Note that there is no space between << and word.

This can be used as a mini-file within a script, e.g.,

cat > foo.c <<EOT

#include <stdio.h>

main() {

printf("Hello, world!\n");

} EOT

It is also useful for printing multiline messages, e.g.:

line=13 cat <<EOT An error occurred on line $line. See page 98 of the manual for details. EOT

As this example shows, by default, << acts like double quotes (i.e., variables are expanded). If, however, word is quoted, then << acts like single quotes.

<&digit

Use file descriptor digit as standard input.

>&digit

Use file descriptor digit as standard output.

<&-

Close standard input.

>&-

Close standard output.

command1 | command2

Creates a pipeline: the standard output of command1 is connected to the standard input of command2. This is functionally identical to

command1 > /tmp/foo
command2 < /tmp/foo

except that no temporary file is created, and both commands can run at the same time

There is a proverb that says, ``A temporary file is just a pipe with an attitude and a will to live.'' Any number of commands can be pipelined together.

command1 && command2

Execute command1. Then, if it exited with a zero (true) exit status, execute command2.

command1 || command2

Execute command1. Then, if it exited with a non-zero (false) exit status, execute command2.

If any of the redirection constructs is preceded by a digit, then it applies to the file descriptor with that number, rather than the default (0 or 1, as the case may be). For instance,

command 2>&1 > filename

associates file descriptor 2 (standard error) with the same file as file descriptor 1 (standard output), then redirects both of them to filename.

This is also useful for printing error messages:

echo "Danger! Danger Will Robinson!" 1>&2

Note that I/O redirections are parsed in the order they are encountered, from left to right. This allows you to do fairly tricky things, including throwing out standard output, and piping standard output to a command.

Functions

When a group of commands occurs several times in a script, it is useful to define a function. Defining a function is a lot like creating a mini-script within a script.

A function is defined using

name () {
commands
}

and is invoked like any other command:

name args...

You can redirect a function's I/O, embed it in backquotes, etc., just like any other command.

One way in which functions differ from external scripts is that the shell does not spawn a subshell to execute them. This means that if you set a variable inside a function, the new value will be visible outside of the function.

A function can use return n to terminate with an exit status of n. Obviously, it can also exit n, but that would terminate the entire script.

Function arguments

A function can take command-line arguments, just like any script. Intuitively enough, these are available through $1, $2... $9 just like the main script.

Useful utilities

There are a number of commands that aren't part of sh, but are often used inside sh scripts. These include:

basename

basename pathname prints the last component of pathname:

basename /foo/bar/baz

prints

baz

dirname

The complement of basename: dirname pathname prints all but the last component of pathname, that is the directory part: pathname:

dirname /foo/bar/baz

prints

/foo/bar

/bin/[ is another name for /bin/test. It evaluates its arguments as a boolean expression, and exits with an exit code of 0 if it is true, or 1 if it is false.

If test is invoked as [, then it requires a closing bracket ] as its last argument. Otherwise, there must be no closing bracket.

test understands the following expressions, among others:

-e filename

True if filename exists.

-d filename

True if filename exists and is a directory.

-f filename

True if filename exists and is a plain file.

-h filename

True if filename exists and is a symbolic link.

-r filename

True if filename exists and is readable.

-w filename

True if filename exists and is writable.

-n string

True if the length of string is non-zero.

-z string

True if the length of string is zero.

string

True if string is not the empty string.

s1 = s2

True if the strings s1 and s2 are identical.

s1 != s2

True if the strings s1 and s2 are not identical.

n1 -eq n2

True if the numbers n1 and n2 are equal.

n1 -ne n2

True if the numbers n1 and n2 are not equal.

n1 -gt n2

True if the number n1 is greater than n2.

n1 -ge n2

True if the number n1 is greater than or equal to n2.

n1 -lt n2

True if the number n1 is less than n2.

n1 -le n2

True if the number n1 is less than or equal to n2.

! expression

Negates expression, that is, returns true iff expression is false.

expr1 -a expr2

True if both expressions, expr1 and expr2 are true.

expr1 -o expr2

True if either expression, expr1 or expr2 is true.

( expression )

True if expression is true. This allows one to nest expressions.

Note that lazy evaluation does not apply, since all of the arguments to test are evaluated by sh before being passed to test. If you stand to benefit from lazy evaluation, use nested ifs.

echo

echo is a built-in in most implementations of sh, but it also exists as a standalone command.

echo simply prints its arguments to standard output. It can also be told not to append a newline at the end: under BSD-like flavors of Unix, use

echo -n "string"

Under SystemV-ish flavors of Unix, use

echo "string\c"

awk

Awk (and its derivatives, nawk and gawk) is a full-fledged scripting language. Inside sh scripts, it is generally used for its ability to split input lines into fields and print one or more fields. For instance, the following reads /etc/passwd and prints out the name and uid of each user:

awk -F : '{print $1, $3 }' /etc/passwd

The -F : option says that the input records are separated by colons. By default, awk uses whitespace as the field separator.

sed

Sed (stream editor) is also a full-fledged scripting language, albeit a less powerful and more convoluted one than awk. In sh scripts, sed is mainly used to do string substitution: the following script reads standard input, replaces all instances of ``foo'' with ``bar'', and writes the result to standard output:

sed -e 's/foo/bar/g'

The trailing g says to replace all instances of ``foo'' with ``bar'' on a line. Without it, only the first instance would be replaced.

tee

tee [-a] filename reads standard input, copies it to standard output, and saves a copy in the file filename.

By default, tee empties filename before it begins. With the -a option, it appends to filename.

Debugging

Unfortunately, there are no symbolic debuggers such as gdb for sh scripts. When you're debugging a script, you'll have to rely the tried and true method of inserting trace statements, and using some useful options to sh:

The -n option causes sh to read the script but not execute any commands. This is useful for checking syntax.

The -x option causes sh to print each command to standard error before executing it. Since this can generate a lot of output, you may want to turn tracing on just before the section that you want to trace, and turn it off immediately afterward:

set -x # XXX - What's wrong with this code? grep $user /etc/passwd 1>&2 > /dev/null set +x

Actually, these are the 2nd 10, the first already documented by Michael Stutz - complete with link in the article. This set of 10 were selected to help you break the bad habit or two you have allowed to linger on. (Other than this one or two, you are a perfect programmer, and we would expect nothing else from someone as sharp and professional as you are.)

A couple examples - OK, all 10:

• Use file name completion.
• Use history expansion.
• Reuse previous arguments.
• Manage directory navigation with pushd and popd.
• Find large files.
• Create temporary files without an editor.
• Use the curl command-line utility.
• Make the most of regular expressions.
• Determine the current user.
• Process data with awk.

So now you can be perfect.

ifconfig eth0 | grep 'inet Adresse' | awk -F ':' '{ print $2 }' | awk -F ' ' '{ print $1 }'
Ausgabe ist einfach die IP-Adresse des Interfaces eth0.

ping $(ifconfig eth0 | grep 'inet Adresse' | awk -F ':' '{ print $2 }' | awk -F ' ' '{ print $1 }')
Pingt die rausgefilterte Adresse an. (Bringt zwar nicht viel, weils die eigene IP ist, aber egal :D Der Syntax-Lerneffekt zählt!)

Ich bin aber der Meinung, dass man vor allem das zweite awk schöner gestalten kann!

EDIT: ifconfig eth0 | grep 'inet Adresse' | awk -F ':' '{ print $2 }' | awk '{ print $1 }'
Find ich persönlich noch besser!

then, like how a person with hammer sees every problem as nail, I started looking out for problems that i could solve using my newly acquired weapon. there were plenty. not just problems, but could do quite a bit of new things that would be too difficult to do with java and bit difficult with python. Above all I like awk's simplicity. After all it is written by 3 demigods (Aho, Weinberger & kernighan)

Let me end this post by writing a few words on one of my awk exploits. In the legacy code that we have, the SQL statements are spewed all over the code. MVC has been violated to the extent possible. Whenever we add a new column to a table, we have a tough time figuring out all jsp that has code to access this table and make changes to it.

So I wrote an awk script that would provide such a report 1. In each jsp file, the db tables used.

2. For each db table, the number of jsp files that accesses this table.

3. In both the cases the line number and the CRUD information.

It is really worth spending some time learning and becoming comfortable with any scripting language. The productivity gains are enormous and it is more fun writing these scripts. In the forthcoming posts, I will write in more detail about some of the scripts.

But, when you work for a company which use a prehistoric version of awk, you could use my script (it is more a hack than a real program) :

#!/usr/bin/env zsh
if (($#<1)); then
    {
        echo "usage : $(basename $0) script.awk arg1 ... argN"
        echo "provide @include \"path/to/awk/file\" directive"
        echo "usefull for libraries"
    } >&2
    exit 1
fi

tmpScriptAwk="/tmp/$(basename $1)"

# see my blog entry about this function
function matche
{
    echo "$1" | egrep "$2" > /dev/null
}

# go into the same directory as the script (to find the good path)
cd $(dirname $1)

# read the script
IFS='
'
for ligne in $(cat $(basename $1) | sed 's/\\/\\\\/g'); do
    # transform @include "path/to/file" by the content of path/to/file.awk
     if matche "$ligne" "^@include \".*\"" ; then
         nomFic=$(echo $ligne | perl -p -e 's#\@include "(.*)"#$1#' )
        # verify if the file to include is readable
         if [ ! -r "$nomFic" ]; then
            # add the .awk prefix (as it is not necessary)
             nomFic="$nomFic.awk"
             if [ ! -r "$nomFic" ]; then
                 echo "$nomFic n'est pas accessible en lecture" >&2
                 exit 1
             fi
         fi
        # write the content of the included file
         cat $nomFic
     else
         echo $ligne
     fi
done > $tmpScriptAwk
shift
# run the script with the argument
awk -f $tmpScriptAwk "$@"

Hope this help, ;-)

Gnarls Barkley's "Who's Gonna Save My Soul".
I'm kinda into the fact that no one is creeped out by the little heart just walking heart, but rather is looking at it with empathy. Except well for Cee-lo.

Ciara's "Oh".
Yeah, I know she's kinda played out by this point. But it's summertime damnit. Missy and Timbaland aren't sending out enough young, talented chanteuses (ie Tweet, Aliyah, Ashanti, etc.) for me to listen to on the regular.

Rick Ross' "Everyday I'm Hustlin".

Flo Rida (feat. Wil.I.Am and Fergie) "Hands in the Ayer"

Jermaine Stewart's "We Don't Have to Take Our Clothes Off"
Yeah, I've got nothing. I refuse to be ashamed though-- that song is so damn catchy. It's AWESOME.

Wondering where I'm going? Well...

Andrew W K's sing-songing along to "I Love New York City" at a lecture in front of 800 students.

Now, what you gonna do?

T.I. "Bring 'Em Out" (of course).

The you can use this command:

# df -k | egrep -v 'kbytes|proc|swap|mnttab|fd' | awk '{ SUM += $4} END { print SUM }'

Basically it will print the total space available on your machine. If you need total or used space, use $2 or $3 respectively in awk part of the command.

Also, if you want Mb or Gb, you can just ... { print SUM/1024/1024 }'

What a silly idea! While Python may be great for "enterprise-class"^[1] log analysis, nothing beats awk for one-liners. Take a few examples off the top of my head...

Who's trying to hack me?

$ zcat /var/log/auth.log.*.gz  | awk '$6 == "Invalid" { print $8 }' | sort | uniq -c | sort -n -r | head -n 30
     79 admin
     71 test
     52 user
     43 michael
     40 alex
     39 guest
     32 oracle
     30 www
     30 dave
     28 info
     26 sales
     25 web
     25 ben
     23 victoria
     23 paul
     23 httpd
     23 adam
     22 john
     21 shop
     21 mike
     21 ftp
     21 david
     21 caroline
     21 amanda
     20 toor
     20 server
     20 samba
     20 linux
     20 danny
     20 claire

Most interesting... Nobody bothers to try root, but apparently someone's used toor before. Also, I see a mix of common first names as well as known linux service names (httpd, ftp, etc). My question is... are there that many sysadmins named caroline?

Where are the Bastards Coming From?

$ zcat /var/log/auth.log.*.gz | awk '$6 == "Invalid" { print $10 }' | sort | uniq -c | sort -n -r
   5170 80.237.205.72
   1243 212.112.227.139
   1040 216.190.237.68
    336 193.137.179.181
    220 200.168.28.21
    132 222.128.249.253
     94 196.200.90.99
     64 200.105.16.242
     60 211.104.85.236
     44 61.192.163.188
     13 200.11.76.170
      6 210.100.157.9
      6 124.135.192.2
      5 222.69.93.27
      5 222.189.238.179
      3 70.97.158.195

Wow, 80.237.205.72 is a really persistent little bugger. Upon looking closer, I see all of the attempts were on a single day. Let's see the latency between attempts:

$ zcat /var/log/auth.log.*.gz  | awk '
$6 == "Invalid" && $10 == "80.237.205.72" {
    oldsec = sec;
    split($3, time, ":");
    sec = time[3] + 60 * (time[2] + 60 * time[1]);
    if (oldsec > 0) {
        print sec - oldsec;
    }
}' | sort -n | uniq -c
    267 2
   3366 3
   1457 4
     23 5
     19 6
     15 7
      1 8
      4 9
      1 10
      5 11
      1 13
      3 14
      2 15
      2 16
      2 24
      1 44

So basically, throughout the day, every 3 seconds someone was trying to log in.

Anyways, I thought I would have something more interesting from awk, but this'll have to suffice.

Footnote [1] Whatever that means

AWK liznąłem trochę w pierwszym semestrze studiów na Językach Formalnych, jednak wtedy nie zdawałem sobie sprawy z możliwości, jakie niesie ta maszynka i ile przy pisaniu programów w tym języku może być zabawy : ) Nie będę się rozwodził tutaj nad tym, co to jest AWK, ani dawał "miniporadnika" ( bo sami zobaczycie za chwilę ), za to odsyłam Was tutaj, znajdziecie tam dużo ciekawych informacji. Ale wracając do tematu pokażę Wam, jak można się bawić z AWK : ) Od zawsze podobały mi się emoticony na deviancie i pomyślałem, że fajnie byłoby mieć je u siebie, nie zapisując każdej po kolei prawym klawiszem myszy. Zapisałem więc sobie źródło strony z emoticonami w pliki in.html i przejrzałem kod. Okazało się, że link do emotki znajduje się zawsze na 4. polu w wierszu. Aby wyciągnąć to pole do osobnego pliku piszemu:

awk '{ print $4 }' in.html > krok1

Teraz mamy w pliku krok1 mniej tekstu, natomiast nas interesują wszystkie niepuste linie, które zawierają mniej więcej taką wartość:

• src="http://e.deviantart.com/emoticons/f/film.gif"

Czyli zaczynające się od wyrażenia regularnego "src", idąc tym tropem dajemy:

awk '/^[src]/ {print;}' krok1 > krok2

Pozbyliśmy się właśnie wszystkich śmieci, które były poza linijkami nas interesującymi, dalej robimy tak:

awk 'gsub(/(\")|(src=)/,"",$0)' krok2 > krok3

Śpieszę z tłumaczeniem  - gsub to funkcja która szuka wzorca w naszym przypadku jest to " (cudzysłowie) oraz "src=" i zamienia je na... no własnie na nic, więc po prostu wycina. Tak więc teraz w pliku krok3 znajdują się same linki postaci:

• http://e.deviantart.com/emoticons/f/film.gif

Teraz pozostaje nam już tylko ściągniecie naszych emotek : )

awk 'system("wget \""$0"\" ")' krok3

I na ekranie pokaże się wynik pracy programu wget, po wylistowaniu katalogu ukażą się nam nasze emotikonki : ) AWK może być wykorzystywane w wielu bardzo różnych przypadkach, łączyć je można z poleceniami systemowymi oraz skryptami w bashu, co daję duuuużo możliwości. W zakładce pliki jest "skrypt" w bashu, dzięki któremu możemy zautomatyzować ten proces. : )
Z tego miejsca chciałbym serdecznie podziękować ZoczuS'owi za inspirację, zaszczepienie we mnie zainteresowania do awk, cierpliwość oraz wielkrotne rozjaśnianie zawiłości awk : ) Zachęcam Was również do zainteresowania się awk, bo ja na tym na pewno nie poprzestanę.

[sourcecode lang='python']#!/usr/local/bin/bash

# psupgrade: Aggiorna i port rilevati come insicuri da portaudit(1) utilizzando portmanager(1).

me=$(basename $0)

awk=$(which awk)
echo=$(which echo)
grep=$(which grep)
pkg_info=$(which pkg_info)
portaudit=$(which portaudit)
portmanager=$(which portmanager)
sed=$(which sed)

for tool in $awk $echo $grep $pkg_info $portaudit $portmanager $sed; do
if [ ! -x $tool ]; then
echo "Non è possibile eseguire $me perchè manca uno dei seguenti tool:"
echo "awk\n\grep\pkg_info\nportaudit\nportmanager".
echo "Assicuratevi che siano installati in una directory del PATH di sistema prima di rieseguire lo script."
exit 1
fi
done

echo "$me: Rilevo quali pacchetti necessitano di un aggiornamento..."
packages=$($echo -n $($portaudit -a | $grep "Affected package:" | $awk '{print $3}'))

echo "$me: Riferisco i pacchetti ai port d'origine..."
ports=$($echo -n $($pkg_info -o $packages | $grep -A 1 "Origin:" | $sed -e "s/Origin://g" -e "s/--//g"))

echo "$me: Lancio $portmanager..."
$portmanager $ports[/sourcecode]

After some quick Googling, I found Delete all Snapshots which made me hold off on it until I had some more information about our snapshots. This number will just increase daily until they're committed.

So in order to plead a case with our SAN admin, I've got to have some information. How much space is being taken up on our LUN with regards to snapshots?

bash to the rescue!

[root@host LUN] ls -laR | grep delta | awk '{ SUM += $5} END { print SUM/1024/1024 }'

(Watch out for apostrophes etc if pasting this.)

Win.

In a more "awkish" venture, my old boss who was now at a company that made microprocessor hardware emulators, asked me if I could do a project for him. They were converting from a 6809-based emulator to a 68000-based one and needed to convert thousands of lines of instruction set encode and decode tables from 6809 assembly language to C. He had estimated a month to do the task assuming it would be done in C. I did it in about three days using awk and sed.

...

So, I was now a C programmer and an awk scripter. The problem was that there was a gap in my toolset. Unlike Fortran V where I even found a way to build and execute assembly language code within the Fortran framework, if I wanted to do anything other than process characters, I had to write the code in C. In most cases this wasn't bad but sometimes 99% of a task could easily be done in a few lines of awk but would require a lot of C.

Well, in 1999 I had the opportunity to interview Guido von Rossum, the father of Python. While I had already been playing with Python, my chat with Guido (a lot more than an interview) inspired me to get more serious. The interview actually addresses what is wrong with awk and Perl so I won't go into more detail here.

While my life in the last almost ten years has not been primarily programming, I do turn to Python as my tool of choice. Some of what I have done can be called scripting but not all. For example, I implemented a credit card processing system for Visanet (the nastiest of all possible credit card protocols) in Python. I also have done some work with Karrigell, a web framework implemented in Python.

Ok, I talked about awk's limitations and Guido helped me beat up Perl in the interview. Note that my default comment about Perl is that if you don't know regular expressions and UNIX/Linux shell programming, Perl is not the best way to get to knowing a scripting language. So, what's left? The obvious choice is Ruby.

Well, this is where my long, boring history of my Pascal experience fits in. First, let me say that I like Ruby. It is a nicely-designed and very clean language. If you don't already know Python, you should seriously give it a try. But, is it better than Python? I think not.

Ruby, while not really new, just hasn't done as much work as Python. Much like Pascal 30 years ago, Ruby looks good on paper. Pascal's claim to fame was that there were no surprises. Ruby's is that it is pure object oriented. Those are both good things. But, as Ruby does more heavy lifting, it seems likely to me that it will have to evolve. That is, its purity will get subverted much like the purity of Pascal did.

In any case, I will stick with Python. It has certainly evolved over the years but that evolution feels like taking a solid base, seeing what makes it more useful and cleanly handling the evolution. To me, Python 2.5 is easier to explain to someone that Python 1.5.2.

A very interesting piece; take a look!

rm `find . -type f -exec file '{}' \; | grep -i linux | awk -F: '{print $1}'`

I mean what happened to the high tops, dirty white jeans, and long, flowing hair? It just caught me off guard. After googling* his new look-- I am now finding myself knee-deep in what might be an AWK conspiracy thoery. Rumor has it that WK is a "pawn/actor" being controlled by Steev Mike who may or may not be his father. Also that he may or may not be wearing a hair piece. No words.

See Steev Mike and his Wikipedia page on "Multiple A.W.K.'s". How far does the rabbit hole go?!

I remember Andrew W. Boner.

Goodnight, sweet prince.

* Sidenote: It's almost impossible to google goggles btw. I keep getting "did you mean Google?"

Awk:

awk -F '[\\t ]*([&]|\\\\\\\\)[\\t ]*' 'BEGIN { maxcol=0 } \
  { for (i=1; imaxcol) \
  maxcol=NF-1 } END { for (c=1; c<=maxcol; ++c) { \
  colwidth[c]=0; for (r=1;rcolwidth[c]) colwidth[c]= \
  length(cell[r,c]); }; for (r=1; r<=NR; ++r) { printf \
  "%s%"(colwidth[1]-length(cell[r,1]))"s", cell[r,1], ""; \
  for (c=2; c<=maxcol; ++c) printf " & %"colwidth[c]"s", \
  cell[r,c]; print " \\\\" } }'

Ruby: Full source available to play with.

# Splits the arrays contained in the strarr parameter into
# arrays containing the individual table cell contents.
def totable(strarr)
    strarr.map do |line|
      line.gsub('\\','').split('&').map {|s| s.strip}
    end
end

# Given a two multidimensional array of strings, this method
# returns an array with the maximum string sizes of the rows
# of the original array.
def rowwidths(tbl)
    tbl.map {|x| x.map {|y| y.size}}.map {|x| x.max}
end

# Given an Enumerable object containing LaTeX table source code
# lines, this method returns a pretty printed version of it (in
# one single string).
def prettyprint(strarr)
    tbl = totable(strarr)
    # transpose
    transposed = tbl.transpose
    colwidths = rowwidths(transposed)
    # on the transposed table: expand rows of strings
    # to full string size
    transposed = (0...transposed.size).map do |i|
      transposed[i].map {|x| sprintf("%#{colwidths[i]}s",x)}
    end
    # transpose back
    tbl = transposed.transpose
    tbl.map {|x| x.join(' & ') + " \\\n"}.join()
end

It's interesting to see where you end up when pushing closures to the limit to make code more concise. No explicit looping beyond that border. Is this the limit? Can you make the code more readable without going back to for-loops?

//cuidado com directorios que contenham espaços doens't work with spaces ;

find /home/ -uid 0 -exec ls -l \{\} \;

fgrep 'gold' /var/log/ -R | grep -i something

or regexp style search
egrep '(err|fail|warning)' /var/log/* > `date +%F`.txt

ls -l  | sed -e 's/andre/netcrash/gi'

#!/usr/bin/perl

print "hello\n";
$dir="/home/andre";
if ( (!(-d $dir."/Maildir")) ){
`/usr/bin/maildirmake.maildrop $dir"/Maildir"`;

}

bash
[03:42:08]andre@garden:~$ for i in `seq 10`; do echo $i ; done;

Haven't been using cut that mutch but, also a good weapon of choice.

Simple things that evolve to more complex tools :D

The more I learn about AWK, the more I love it. It's just that simple. I've been trying its powers a bit and came with an awk-mazing (and probably useless, but still exciting) five-lines (sans comments) cute script that emulates the most popular feature (the "-e" option) of our good ol' boy apropos.
Enter lazyman!

#!/bin/bash
#
# lazyman.sh - displays the name section of some program’s man
# page.
#
# DESCRIPTION
# this tiny script shows the NAME section of a given program’s
# manual page, displaying an output quite similar to
# “apropos -e” but directly calling “man” instead of querying
# against “mandb”.
#
params=$(echo $@ | awk 'END{print NF}')
case $params in
1) man $1 | awk '/NAME/{getline;print}' ;;
*) echo "usage: lazyman.sh [program]" ;;
esac

That's all folks! I hope you've enjoyed it and thanks for reading!

Segue o código-fonte.

#!/bin/bash

set -- `getopt ':sgboh' $*`
stat=0
graf=0
book=0
while [ "$1" != "--" ]; do
        case "$1" in
        -s)
                stat=1
                ;;
        -g)
                graf=1
                ;;
        -b|-o)
                book=1
                ;;
        -h)
                echo 'Usage: book [-s|-g|-b] name'
                exit
                ;;
        esac
        shift
done
shift

if [ $book -eq 0 -a $graf -eq 0 -a $stat -eq 0 ]; then stat=1 ; fi
fn="$1"
if [ ! -f ~/$fn ]; then fn=$fn.htm ; fi

awk -v stat="$stat" -v book="$book" -v graf="$graf" '
/Ofertas de Compra/ { mode=1 ; get=0 ; tb=0 }
/Ofertas de Venda/ { mode=2  ; get=0 ; tb=0 }
mode==0 { next }
/<tr>/ {get=1}
get==1 && /td_informacoes/ {
        gsub("^M","")
        line=$0 ; gsub("\\.|R\\$ *|[\t ]*<[^>]+>[ \t]*","",line)
        gsub("[ \t]+"," ",line)
        if (corr[mode,tb]=="") corr[mode,tb]=line
        else if (qt[mode,tb]=="") qt[mode,tb]=line
        else if (vl[mode,tb]=="") vl[mode,tb]=line
}
/<\/tr>/ {
        get=0
        if (corr[mode,tb]!="") {
                if (match(qt[mode,tb],"k")) {
                        qq=qt[mode,tb]
                        qq=substr(qq,0,length(qq-1))
                        qt[mode,tb]=qq*1000
                }
                if (match(qt[mode,tb],"M")) {
                        qq=qt[mode,tb]
                        qq=substr(qq,0,length(qq-1))
                        qt[mode,tb]=qq*1000000
                }
                if (match(qt[mode,tb],"B")) {
                        qq=qt[mode,tb]
                        qq=substr(qq,0,length(qq-1))
                        qt[mode,tb]=qq*1000000000
                }
                tb++
                bk[mode]=tb
                if (tb>tm) tm=tb
        }
}
END {
        mi[1]=vl[1,0]
        mi[2]=vl[2,0]
        for (i=0;i<tm;i++) {
                if (book==1) {
                        if (corr[1,i]!="") printf corr[1,i]";"qt[1,i]";"vl[1,i]";"
                        else printf ";;;"
                        if (corr[2,i]!="") printf corr[2,i]";"qt[2,i]";"vl[2,i]";"
                        else printf ";;;"
                }
                mx[1]=mx[1]+qt[1,i]*vl[1,i]
                qx[1]=qx[1]+qt[1,i]
                if (vl[1,i]<mi[1] && vl[1,i]>0) mi[1]=vl[1,i]
                if (vl[1,i]>ma[1]) ma[1]=vl[1,i]
                if (vl[2,i]<mi[2] && vl[2,i]>0) mi[2]=vl[2,i]
                if (vl[2,i]>ma[2]) ma[2]=vl[2,i]
                mx[2]=mx[2]+qt[2,i]*vl[2,i]
                qx[2]=qx[2]+qt[2,i]
                qxm=(qx[1]<qx[2])?qx[1]:qx[2]
                if (book==1) print ""
        }
        qqm[1]=qqm[2]=bkm=0
        vl[2,-1]=vl[2,0]
        for (i=0;i<tm;i++) {
                if (qqm[1]<qxm && vl[1,i]>vl[1,i-1]*0.60) {
                        mim=vl[1,i]
                        bkm++
                        if (qqm[1]+qt[1,i]>qxm) {
                                mxm=mxm+(qxm-qqm[1])*vl[1,i]
                                qqm[1]=qxm
                        } else {
                                mxm=mxm+qt[1,i]*vl[1,i]
                                qqm[1]=qqm[1]+qt[1,i]
                        }
                }
                if (qqm[2]<qxm && vl[2,i]<vl[2,i-1]*1.40) {
                        mam=vl[2,i]
                        bkm++
                        if (qqm[2]+qt[2,i]>qxm) {
                                mxm=mxm+(qxm-qqm[2])*vl[2,i]
                                qqm[2]=qxm
                        } else {
                                mxm=mxm+qt[2,i]*vl[2,i]
                                qqm[2]=qqm[2]+qt[2,i]
                        }
                }
        }
        qc=qv=0
        ic=iv=0
        i=1
        while (ic<tm && iv<tm) {
                if (qc==0) {
                        qc=qt[1,ic]
                        if (qc>0 && vl[1,ic]<cv*0.60) qc=0
                        cc=vl[1,ic]
                        ic++
                }
                if (qv==0) {
                        qv=qt[2,iv]
                        if (qv>0 && vl[2,iv]>cc*1.40) qv=0
                        cv=vl[2,iv]
                        iv++
                }
                if (qc==0 || qv==0) break;
                if (qc>qv) {
                        cc=int(100*(cc*qc+cv*qv)/(qc+qv)+0.5)/100
                        if (graf==1) print i ";" qv ";" cc
                        qc=qc-qv
                        qv=0
                } else {
                        cv=int(100*(cc*qc+cv*qv)/(qc+qv)+0.5)/100
                        if (graf==1) print i ";" qc ";" cv
                        qv=qv-qc
                        qc=0
                }
                i++
        }
        mmm=mxm/(2*qxm)
        mm[1]=mx[1]/qx[1]
        mm[2]=mx[2]/qx[2]
        gm=(mx[1]+mx[2])/(qx[1]+qx[2])
        if (stat==1) {
                print "-------------"
                printf("          Media    Min    Max Quantidade        Valor Ordens\n")
                printf("Compra:  %6.2f %6.2f %6.2f %10d %12d %6d\n",mm[1],mi[1],ma[1],qx[1],mx[1],bk[1])
                printf("Venda:   %6.2f %6.2f %6.2f %10d %12d %6d\n",mm[2],mi[2],ma[2],qx[2],mx[2],bk[2])
                printf("Geral:   %6.2f %6.2f %6.2f %10d %12d %6d\n",gm,mi[1],ma[2],qx[1]+qx[2],max[1]+mx[2],bk[1]+bk[2])
                printf("Melhor:  %6.2f %6.2f %6.2f %10d %12d %6d\n",mmm,mim,mam,2*qxm,mxm,bkm)
                printf("Media:   %6.2f\n",(mm[1]+mm[2])/2)
        }
}
' ~/$fn

Eu não vou entrar nos detalhes de programação, mas o programa efetua o seguinte tratamento:

1. Ele analisa um página web (.htm) contendo os dados do book de oferta, página formatada pela corretora ABN AMRO, e salva no navegador (firefox no caso, mas a opção também existe no Internet Explorer). Só interessa a página, então não precisa salvar as imagens e outros arquivos anexos.
   A partir dessa página, ele extrai a lista de ordens de compra e venda.
2. Se a opção -b é dada na linha de comanda (ex: book -b rdcd3), a lista de ordens é mostrada, num formato importável no excel (6 colunas: banco comprador, nº de ações, valor, banco vendedor, nº de ações, valor; o separador é o ponto-virgula; os separadores de milhares estão removidos, e os fatores (k[ilo],M[ilhão],B[ilhão]) são convertidos para só sobrar números)
3. Se a opção -s é informada (ou se nenhuma opção é dada), o programa gera algumas estatísticas:
   - Media, Mínimo, máximo, quantidade e valor total das ofertas de compra
   - Media, Mínimo, máximo, quantidade e valor total das ofertas de venda
   - Media, Mínimo, máximo, quantidade e valor total das ofertas
   - Mesma coisa, mas só satisfazendo o número de transações possíveis de acordo com a demanda e a oferta (se tiver só 100 ações para vender e 2000 para comprar, só se trocaram 100...)
   - Media entre a media de compra e a media de venda
4. Enfim se a opção -f for especificada, o programa gera uma lista de transações possíveis (número da transação;número de ações;valor) baseada no seguinte raciocínio: efetuamos a correspondência entre a melhor oferta de compra e a melhor oferta de venda, calculando o preço ponderado e efetuando a troca do número possível de ações. Com o que sobra para comprar ou vender, consideramos o novo preço e correspondemos com a próxima melhor venda ou compra, e assim em seguinte... Isso nós da uma lista de transações prováveis. Como podem entrar outras ofertas no intervalo, essa estimativa deve ser considerada mais como uma tendência que como uma previsão, mas dá para pegar o resultado e grafar com o excel...

[leon@logos ~]$ history | awk '{a[$2]++} END {for(i in a){print a[i] " " i}}' | sort -nr | head
1504 sudo
611 ls
502 pacman
462 cd
182 vi
172 mv
154 rm
148 man
127 echo
47 rename

Now... trying to understand that evil-looking mess First, history prints out the command history in the format <No.> <Command>. Then the output is pipelined to awk. a[$2] takes the 2nd field of the output, the commands, and stores them in an array called a[] as indices. That's right... indices can be strings in awk. See the GNU Awk User's Guide. ++ adds 1 to a command's corresponding element each time that command is encountered. So the structure of a[] will look like this:

index element
ls 611
cd 462
vi 182

After everything is processed, the END block prints out the whole array, with indices and elements seperated by a whitespace. sort rearranges the output so the biggest numbers come up first. Finally head crops the output to the first 10 lines.

It's quite interesting looking at my own results. I use sudo almost twice as often as ls. And I don't remember using echo that much...

Note, however, this result is an incomplete statistics. Because awk '{a[$2]++}...' uses whitespaces as delimiter and therefore only picks up the first word of a command line and throws the rest away. As a result commands following a pipe or inside a parameter substitution are ignored.

Luego de discutir sobre el tema por un rato con FCR, sugirió hacer algo muy sucio con ls -v, así que recordé que ls usa la función strverscmp() de glibc, lamentablemente no es estándar, y hacer un programa en C para esto no me hacía mucha gracia, así que decidí remangarme e implementar algo similar en AWK:

[sourcecode language='jscript']
#!/bin/awk -f
BEGIN {
FS = "[-_.]"
la = split(ARGV[1], a)
lb = split(ARGV[2], b)
l = (la < lb) ? la : lb
o = la - lb

for (i = 1; i <= l; i++) {
if (a[i] != b[i]) {
o = a[i] - b[i]
break
}
}

print o
}
[/sourcecode]

¿No es hermoso?