Category Archives: Scripts and programs

How to split a string using Awk

Posted on by

Using Awk a string may be split at least using two different methods.

Let’s split the strings given in file foo. The file consists of 5 columns separated by a comma (,) so we will split the lines at “,”.

Split
awk ‘{split($0,array,”,”)}{print array[1],array[2],array[3],array[4],array[5]} foo

-F
awk -F “,” ‘{print $1,$2,$3,$4,$5}’ foo

Where foo is a file such as this e.g:
1005.709115,-3.974680,0.081765,3.265020,Wed 23 Nov 2011 11:27:14 AM UTC
1006.709004,-3.974761,0.084558,3.265004,Wed 23 Nov 2011 11:27:15 AM UTC
1007.708987,-3.973002,0.087218,3.265001,Wed 23 Nov 2011 11:27:16 AM UTC
1008.709055,-3.969182,0.089719,3.265034,Wed 23 Nov 2011 11:27:17 AM UTC

Find minimum and maximum using AWK

Posted on by

Let’s assume you want to find the minimum and/or maximum of a column of numbers in a given file using AWK.

Assume we have an input file foo with f.ex. line number in first column and in the second ($2) and third column ($3 in awk) we have the values of interest.

File: foo

1 2 40
2 3 23
3 15 -12
4 8 -1
5 11 0

Use the following awk commando find minimum and maximum of the columns of interest:

awk ‘NR == 1 {max=$2 ; min=$2} $2 >= max {max = $2} $2 <= min {min = $2} END { print "Min: "min,"Max: "max }' foo

or

awk ‘BEGIN{ max = -999999999 ; min = 9999999999 } $2 > max {max = $2} $2 < min {min = $2} END{ print "Min: ",min, "Max: ",max }' foo

The result will be for column 2:

Min: 2 Max: 15

and for column 3:

Min: -12 Max: 40

Standard deviation

Here you may see how to calculate the standard deviation using AWK.

Average or arithmetic mean

Here you may see how to calculate the average or arithmetic mean using AWK.

Pythons Simple HTTP Server

Posted on by

There is a nice little feature in Python that a friend of mine showed me. You may use Python as a Web Server… and that is right out of the box. No additional programs to install or any tweaking at all 🙂

Move to the folder you want to make available for others to see.
Then type
python -m SimpleHTTPServer

If you want to use a special port you may type something like this:

python -m SimpleHTTPServer 8713

Others may browse your directory and directories below.
If you haven’t specified a port, in your browser type :
http://your.ip:8000/

Or, if you have specified a port
http://your.ip:8713/

Enjoy 🙂

Calculate standard deviation using AWK

Posted on by

The standard deviation ? (sigma) is the square root of the average value of (X – ?)2.

In the case where X takes random values from a finite data set x1, x2, …, xN, with each value having the same probability, the standard deviation is

  where mu

Assume we have an input file foo with f.ex. line number in first column and in the second column ($2 in awk) we have the values of interest.

File: foo

1 2
2 3
3 6
4 8
5 11

Use the one of the following awk commandos to calculate the standard deviation

awk ‘{sum+=$2; array[NR]=$2} END {for(x=1;x<=NR;x++){sumsq+=((array[x]-(sum/NR))^2);}print sqrt(sumsq/NR)}’ foo

awk ‘{sum+=$2;sumsq+=$2*$2} END {print sqrt(sumsq/NR – (sum/NR)^2)}’ foo

The result is

3.28634

Average

Here you may find how to calculate the average or arithmetic mean using AWK.

Minimum and maximum

Here you may find how to calculate the minimum and maximum values using AWK.

Hard Drive Crashes … foremost to the rescue

Posted on by

If your hard drive crashes … you may use foremost to retrieve your documents, photographs and so on.

Foremost is a linux tool for conducting forensic examinations. Although intended for law enforcement purposes, it may be useful to other members of the community. Foremost reads through a file, such as a dd image file or a disk partition and extracts files.

Foremost is available for most *nix platforms and Windoze via CygWin. Foremost is useful when you can’t mount a ddrescue image.

Execute the following command:

foremost -t all -i my_crashed_harddrive.dd

… and all your documents should hopefully be rescued 🙂

SSH and RSYNC without password

Posted on by

Tried of typing your password every time you use ssh or rsync? If so, you should create a local ssh key for your computer and append your key to the remote computers you usually login to.

1. Generate your ssh key on your local computer:

local> ssh-keygen -t rsa

2. If prompted for file name to store your key, select the default (id_rsa).

3. Do not enter a password when asked. This will generate a password-less key and a public key. If you have   selected the default names, then those will be: id_rsa (password-less key) and id_rsa.pub (public key)

4. Copy the public key (id_rsa.pub) to the machine you want to ssh/rsync to. Avoid overwriting your RSA key at your remote host by renaming it first.

local> cp id_rsa.pub yourlocal_key.pub

local> scp yourlocal_key.pub you@remote-machine:

remote> cat yourlocal_key.pub >> .ssh/authorized_keys

remote> rm yourlocal_key.pub (cleaning up)

And now your remote computer will accept a ssh login without prompting you for a password.

How to create a MySQL database

Posted on by

Here we will show how to install and create a MySQL database. We will also create a MySQL user with access to our new MySQL database. We will GRANT our user all privileges on using the database.

Install mysql

Install mysql-server from your distro or using apt-get install mysql-server. During installation you a password for root is set.

If you get this error message you should install mysql-server.
> mysql
ERROR 2002 (HY000): Can’t connect to local MySQL server through socket ‘/var/run/mysqld/mysqld.sock’ (2)

Connect to MySQL

After you have installed mysql you should be able to connect to mysql.
> mysql
ERROR 1045 (28000): Access denied for user ‘xxx’@’localhost’ (using password: NO)
This error is due to no user with name ‘xxx’.

Connect by providing username and password:
> mysql -h localhost -u root -p
Enter password:
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 34
Server version: 5.0.75-0ubuntu10 (Ubuntu)

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the buffer.

mysql>

End your session by typing either quit or exit at mysql>

Create a MySQL database

We are connected to mysql as root.
In the next line we create a mysql database with name ‘wordpress_db’

mysql> create database wordpress_db;
Query OK, 1 row affected (0.00 sec)

mysql>

Create a MySQL user

Let’s create a user ‘wp_user’ with a given password ‘mypassword’ at ‘localhost’.

mysql> CREATE USER ‘wp_user’@’localhost’ IDENTIFIED BY ‘mypassword’;
Query OK, 0 rows affected (0.00 sec)

mysql>

Grant user all priveleges on your MySQL database

mysql> grant all on wordpress_db.* to ‘wp_user’@’localhost’;
Query OK, 0 rows affected (0.00 sec)

mysql>

Calculate average using AWK

Posted on by

The average or arithmetic mean is given as

Assume we have an input file foo with f.ex. line number in first column and in the second column ($2 in awk) we have the values of interest.

File: foo

1 2
2 3
3 6
4 8
5 11

Use the following awk commando to calculate average or arithmetic mean

awk ‘{s+=$2} END{print “Sum: “s, “\nNumber of lines: “NR, “\nAverage: “s/(NR)}’ foo

The result will be:

Sum: 30
Number of lines: 5
Average: 6.0

Standard deviation

Here you may see how to calculate the standard deviation using AWK.

Minimum and maximum

Here you may find how to calculate the minimum and maximum values using AWK.

Merge two files using AWK

Posted on by

We may use AWK to easily join two files. Assume we have input files foo and foo2

Content of foo

OSL 59.9 10.9
TRD 62.7 11.8

… and foo2

OSL 25 m
TRD 36 m

Join the two files with AWK commando

awk ‘NR==FNR{a[FNR]=$0;next} {print a[FNR],$2,$3}’ foo foo2

or

awk ‘{str=$2;str2=$3; getline < "foo";printf "%3s %3.1f %3.1f %2f %1s\n", $1,$2,$3,str,str2}' foo2

Output will be:

OSL 59.9 10.9 25 m
TRD 62.7 11.8 36 m