Some great tips on unix system.

1. Joining 2 consecutive lines using sed:

sed '$!N;s/\n/ /' myfile or paste -s -d" \n" myfile

2. The man program searches through manual pages and displays the first page whose name matches topic.

The basic form for the man command is
man topic
man -f gives single line description of what the command does

man -f topic

eg: man -f who

who(1) - who is on the system

3. leave command:

$ leave
When do you have to leave? 0430
Alarm set for Mon Nov 27 04:30:16 2006

Give leave the time when you have to leave. It runs in the background, and five minutes before that given time, it outputs on your terminal a reminder for you to leave. It does this again one minute before the given time if you're still logged in, and then at the time itself -- and from then on, it keeps sending reminders every minute until you log out (or kill the leave process).

4. sort with -M option:

Compare fields as months. The first three nonblank characters are folded (see -i option) to uppercase and compared. Thus January is compared as JAN. JAN precedes FEB, and fields not containing months precede JAN.

5. To stop all of your processes
and log yourself off, enter:

kill -kill 0

6. FILE SPACING:

# double space a file
sed G filename

double space a file which already has blank lines in it. Output file
should contain no more than one blank line between lines of text.

# triple space a file
sed 'G;G' filename

7. Multiple commands with -e command

combining multiple commands can be done by using a -e before each command:

sed -e 's/a/A/' -e 's/b/B/' <old >new

8. Backreferences - Remembering patterns with $, $ and \1

To search for two identical letters, use "$[a-z]$\1." You can have 9 different remembered patterns. Each occurrence of "\(" starts a new pattern. The regular expression that would match a 5 letter palindrome, (e.g. "radar"), would be

grep '$[a-z]$$[a-z]$[a-z]\2\1' filename

9. sed options:
Sometimes you want to search for a pattern and add some characters, like parenthesis, around or near the pattern you found. The solution requires the special character "&." It corresponds to the pattern found.

10. Restricting to a line number in sed:

The simplest restriction is a line number. If you wanted to delete the first number on line 3, just add a "3" before the command:

sed '3 s/[0-9][0-9]*//' <file >

11. IFS: Internal field separator:

The shell uses the value stored in IFS, which is the space, tab, and newline characters by default, to delimit words for the read and set commands.

12. grep -x "see" file_a>file_b 1>&2

grep -x : Select only those matches that exactly match the whole line.

13. In the above case both standard error and output is placed in standard error file.

14. tr command:

tr copies the standard input to the standard output with substitution or deletion of selected characters. Input characters in set1 are mapped to corresponding characters in set2.

tr [ options ] [ set1 [ set2 ] ]

tr -s :
Replaces any character specified in string1 that
occurs as a string of two or more repeating
characters as a single instance of the character
in string2.

15. grep options:

-v, --invert-match
Invert the sense of matching, to select non-matching lines.

16. w in unix:

w - show who is logged on and what they are doing

displays information about the users currently on the machine, and their processes. The header shows, in this order, the current time, how long the system has been running, how many users are currently logged on, and the system load averages for the past 1, 5, and 15 minutes.

17. grep options:

                     -c, --count
      Suppress normal output; instead print a count of matching lines
      for each input file.

18. awk options:

This just shifts the last colum to the first position sort it

awk '{ printf $NF;$NF = "" ;printf " "$0"\n" }' | sort

19. Multiple sort keys may be used on the same command line. The first key is used to sort the lines. If two lines have the same value in the same field, sort uses the next set of sort keys to resolve the equal comparison. For example,

     sort +4 -5 +1 -2 infile

means to sort based on field 5 (+4 -5). If two lines have the same value in field 5, sort those two lines based on field 2 (+1 -2).

20. grep options:

-i, --ignore-case
Ignore case distinctions in both the PATTERN

21. grep options:

-n, --line-number
Prefix each line of output with the line number within its input
file.

22. Variable assignment:

Variables can be assigned values previously stored or if not present any other value.

23. tee :

tee - read from standard input and write to standard output and files

tee -a :

--append
append to the given FILEs, do not overwrite

24. egrep:

instead of grep -e egrep can be used.

egrep [A-Z]\{3\}

matches 3 upper case characters.

25. To find whether the file is old:

find [path_list] -atime n

where path_list is the path in which the file is to be searched,

This gives the File which was accessed n*24 hours ago.

26. sed options:

delete the last line of a file

sed '$d' file

delete the first line of a file

sed '1d' file

27. To get the 3rd line in the file:

head -3 file | tail -1

28. To get only today`s date:

date +%j

Similarly

date +%m gives current month

date +%y gives current year

29. Joining 2 consecutive lines using sed:

sed '$!N;s/\n/ /' myfile or paste -s -d" \n" myfile

30. uptime

This command gives the current time, the length of time the system has been up, the number of users logged on to the system, and the average number of jobs in the run queue over the last 1, 5, and 15 minutes.

31. To get the first and second field in a file where the fields are seperated by commas,

awk -F, '{print $1,$2}' filename

To get the first and second field in reverse order,

awk -F, '{print $2,$1}' filename

awk prints these fields in all the lines.

32. NF: number of fields in a line

NR : number of records (i.e no. of lines in that file)

To get only the line which hav more than 4 fields give, awk 'NF > 4' filename

To print the line number at the starting of each line, awk '{print NR,$0}' filename

I have used F, in the command as the delimiter is comma.

33. To get the sum of all the file sizes present in a directory, just go to that directory use this command:

ll | awk '{for (i=1; i<=NF; i++) s=s+$5}; END{print s}'

This command iteratively sums up all the fifth field values i.e size of the files and prints the sum.

34. The ps command reports information about active processes. If no options are specified, you will see information about only those processes connected to your current terminal session.

ps -e

shows information about all processes currently running on the system

35. To list only the directories:

ll | grep ^d

36. To search for a pattern recursively in all the directories from the current location:

find . | xargs egrep -li "pattern"

37. To list only the directories:

ll | grep ^d

38. Removing a directory:

rm -r directory

Copying a directory:

cp -r old_dir new_dir

39. Precede each line by its line number FOR ALL FILES TOGETHER, with tab.

awk '{print NR "\t" $0}' filename

40. print the sums of the fields of every line
awk '{s=0; for (i=1; i<=NF; i++) s=s+$i; print s}' filename

41. add all fields in all lines and print the sum
awk '{for (i=1; i<=NF; i++) s=s+$i}; END{print s}'

42. delete ALL blank lines from a file (same as "grep '.' ")
awk NF
awk '/./'

43. grep for AAA and BBB and CCC (in that order)
awk '/AAA.*BBB.*CCC/'

44. tr command to change upper case letters to lower case (vice-versa)

echo $VAR|tr '[A-Z]' '[a-z]'

45. cat options:

cat -n displays the output of a file with line numbers

46. To copy fields 3 and 7 from input to output (3 string, 7 float):

awk '{printf"%s\t%10.4f\n", $3, $7}' < input > output

47. To add a column of numbers:

awk '{sum=sum+$1;printf("sum: %d\n", sum)}' < input | tail -1

48. To calc the average of numbers in a file:

egrep "^:2" *out | egrep total | cut -c44- | \

awk '{sum += $1;total += 1;printf"avg = %.4f\n", sum/total}' | \

tail -1

49. To find 'word' in files owned by 'username' from location '.':

find . -user username -print -exec \grep word '{}' \; > output &

50. To find all files that do not end in 'pdb' in location '.':

find . $ ! -name \*pdb $ -print

51. To find all files that do not end in 'pdb' in location '.' and execute a

command on each file found:

find . $ ! -name \*pdb $ -exec \anneal2pdb '{}' \;

52. To remove all files 'garb':

find ~ -name garb -print -exec \rm '{}' \;

53. To report the file size of all files bigger than 2 meg and older than 30

days.

find . -type f -size +4096 -atime +30 -exec \du -sk '{}' \;

54. To search with a wildcard:

find . -name goob\* -print

55. To manually nudge NFS:

umount /iris1/a (or whatever)

mount -a -t nfs

exportfs -a -v

56. To encrypt the file 'input' to a file 'output':

crypt < input > output

enter key: (enter a password)

To decrypt the file 'output' to a file 'done':

crypt < output > done

enter key: (enter the password used to encrypt 'output')

57. To convert tabs to spaces:

expand -tabstop filename.in > filename.out

expand -3 garb > garb.out

will convert tabs to three spaces

To convert spaces to tabs:

unexpand filename

58. To check the print que:

lpq

To remove a print job:

lprm <job # from lpq>

To check the printer:

lpc

59. To remove duplicate lines from a file:

uniq filename > out

60. To mail to compuserve:

If the username is '70000,200', mail to:

mail 70000.200@compuserve.com

note the period instead of the comma.

To mail from compuserve:

>INTERNET Loopy@cup.portal.com

61. To create a text file from a manual page:

man command | expand > file

vi file

:1,$s/.^V^H//g

62. To compare two files:

Flags 1, 2, or 3 suppress printing of the corresponding column. Thus

comm -12 prints only the lines common to the two files; comm -23 prints

only lines in the first file but not in the second; comm -123 prints

nothing.

63. To run a string of commands over and over:

csh> sh

$ while :

> do

> command

> done

When done, ^C to quit. ^D to return to csh.

64. To get a list of 'fsu.edu' machines:

UNIX> nslookup

> ls fsu.edu > file

> exit

65. To get a machine name from an IP address:

UNIX> nslookup

> set type=ptr

> 34.12.186.128.in-addr.arpa

Server: magnet.fsu.edu

Address: 146.201.250.2

34.12.186.128.in-addr.arpa name = weirdscience.chem.fsu.edu

66. To add line numbers to the beggining of a file:

grep -n . input > output

paste x y > z

67. To move (or copy) a directory heirarchy:

cd fromdir

tar cf - . | (cd todir; tar xfBp -)

68. To copy a directory heirarchy exactly:

cd fromdir; tar cBf - . | (cd todir; tar xpBf -)

69. To restart the sendmail process:

kill the original process

/usr/lib/sendmail -bd -q1h

70. To alias rm so that it lists the files to be removed and asks for

confirmation:

alias rm \

'/bin/ls -AsF \!* && echo -n "Remove? " && if ($< == y) /bin/rm -rf \!*'

71. To remove blank lines from a file:

sed -e '/^$/d' file.in > file.out

72. To find all subdirectories find . -type d –print

73. Comparing two directories dircmp -d -s dir1 dir2

74. To replace text in multiple files sed -e 's/oldtext/newtext/' -i file1 file2

75. timex options:

-p[fhkmrt]: List process accounting records for command and

all its children. The suboptions f, h, k, m, r,

and t modify the data items reported. They behave

as defined in acctcom(1M). The number of blocks

read or written and the number of characters

transferred are always reported

76. timex options:

-o: Report the total number of blocks read or written

and total characters transferred by command and

all its children. This option works only if the

process accounting software is installed

77. ex command

Replace characters in a file or delete them without explicitly opening the file.

78. Variable manipulation:

Eg: FILE=XXX01.PNnnnnnn.DIR

echo ${FILE%.*}

output:

XXX01.PNnnnnnn

79. Grep options:

-s : Suppress error messages about nonexistent or unreadable files.

80. Do you hate users who put spaces in file names, preventing you from doing any sort of scripting?

Well then this is the little one-liner for you to pop in a shell script then...

for i in $1 ; do mv "$i" `echo $i | sed 's/ /_/g'` ; done

81. sed options:

To replace characters according to its position in the line.

To replace characters more than 255th position, use accordingly as the following example

sed -e 's/^$.\{200\}.\{71\}$Blore/\1Mlore/'

82. dc command:

Short for desk calculator, dc is an arbitrary precision arithmetic package.

DC COMMANDS

p: prints current results

q: quit

83. dc options:

c : Clear all values on the stack

84. dc : more options

v : Take square root

85. Printing all the fields in a file within double quotes.

there are 2 ways of doing it..

as explained in the example

a. cat file|awk ‘{print “\””$1”\””,”\””$2”\””}’

b. cat file|awk ‘{printf(“\”%s\”,\”%s\”\n”,$1,$2)}’

86. find options:

If you want to list out all the files except a particular file, use

find < path > ! -name <filename >

87. awk functionality:

Searching happens within "//" and actions within "{}".

awk '/search pattern/ {print}'

88. awk functionality

Match in a particular field:

Awk autosplits a line on whitespace as default. The fields are stored in $1 through $NF and the whole line is in $0. One can match or not match an individual field

Eg: cat infile

Fred notok 3000

Cat infile|awk ‘ $1 ~ /Fred/ && $2!~ /ok/ {print “Fred has not paid $3”}’

89. awk options:

cat file|awk -F"," '{ $NF = "" ;print }'

To delete the last field in every line

90. awk options:

To delete the last field in the file and also print the output as the original file with the delimiter

cat file|awk -F"," '{ OFS=","; $NF = "" ;print }'

91. To get the time stamp of a file in hh:mm:ss format:

echo $file | cpio -o 2>/dev/null|cpio -ivt 2>/dev/null|awk '{print $7,$4,$5,$6}'

92. If you ever have to delete many, many files but rm returns a "too many arguments" error, use find and xargs to save the day:

find . | grep 'your_filename_filter' | xargs rm -f

93. Defining directory trees with one command

Mkdir –p tmp/a/b/c

94. Print in reverse order the fields of every line:

   cat file | awk ' { for (i=NF ; i>0 ; i=i-1) printf( "%s ", $1)
            printf("\n")
     } '

95. Control M character check in a file can be accomplished through the following small code:

grep -q $(echo "\r\c") infile
STAT=${?}
if [[ ${STAT} -eq 0 ]]
then
echo "CR Found"
fi

96. If the control M characters present in the file is intended to be removed,

dos2ux test1 > test2 is the perfect command for tat.

where test1 is the file which contains control M characters, test2 is the output file.

97. Interesting ls option:

ls -rt1 will give last modified file first in a list. Unlike ls -lrt which gives all the other fields, like permissions,etc, this option will give only the file name in the form of a list

98.

Maths calculations can be done using awk. Floating point output can be got.

awk -v VAR=40 -v MULT=2002.4 'BEGIN{printf("%.2f\n",(VAR/100)*MULT)}'

800.96

99. du -a |grep *.txt

This will locate all the files

with the extension .txt in the

current directory.

100. To print last two fields in a file.

cat sep|awk 'NF>2 {print $(NF-1),$NF}'

Rakesh Goel

Tips on Unix System

7. Multiple commands with -e command

8. Backreferences - Remembering patterns with \(, \) and \1

tr copies the standard input to the standard output with substitution or deletion of selected characters. Input characters in set1 are mapped to corresponding characters in set2.

Total Pageviews

Blog Archive