Welcome, Guest. Please Login
 
  HomeHelpSearchLogin FAQ Radified Ghost.Classic Ghost.New Bootable CD Blog  
 
Pages: 1 2 3 4 5
Send Topic Print
Using the Unix/Linux shell command-line (Read 114594 times)
Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #15 - Jul 22nd, 2009 at 7:24pm
 
MrMagoo wrote on Jul 22nd, 2009 at 6:44pm:
I know some admins who work on huge (200 CPU) Solaris servers 

Didn't know such animals existed. All that power.

there are some prgms i keep hearing about, such as sed, grep, awk (i think), but i am not there yet.
 
WWW  
IP Logged
 

Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #16 - Jul 22nd, 2009 at 7:31pm
 
06-02 - Input (Not many programs take input)

By default, programs take their input from the keyboard
Filters can take input from either:
1. keyboard
2. file
3. another program
to feed input to a program from a file, we use "<"
e.g. "wc < file.txt"
the "<" sign tells the shell to feed the contents of the file named "file.txt" into the wc program
can also write "wc file.txt"
In this case, the file.txt is an argument
wc = an example of a 'filter'
 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: Using the Unix/Linux shell command-line
Reply #17 - Jul 22nd, 2009 at 8:09pm
 
Rad.in.UbuntuVM wrote on Jul 22nd, 2009 at 7:24pm:
there are some prgms i keep hearing about, such as sed, grep, awk (i think), but i am not there yet.

I use grep daily.  I use awk on occasion. 

I've never used sed.  That's not to say it isn't useful - there are several times it could have been helpful - it just doesn't happen often enough that I have learned how to use it.
 
WWW  
IP Logged
 
Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #18 - Jul 22nd, 2009 at 10:50pm
 
06-03 - Standard Input and Standard Output

Input:
1. Keyboard
2. File
3. Program

Output:
1. Screen
2. File
3. Program

* Unix philosophy: The people who designed Unix utilities created many simple utilities that can be joined together as necessary.
* They provided the Unix utilities as building blocks that can be put together and used in more complex tasks.
* This roll-your-own approach is the foundation of Unix shell scripting and is a powerful way of specifying tasks.
 
WWW  
IP Logged
 
Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #19 - Jul 22nd, 2009 at 11:39pm
 
06-04 - Intro to Filters

1. Takes standard input (from keyboard, file or another program .. after the program has started)
2. Processes data inputted
3. Produces output based on input

wc is a filter (word count). It does all 3.
Filters are text processing devices
 
WWW  
IP Logged
 
Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #20 - Jul 23rd, 2009 at 12:35am
 
06- 05 - Common Filters

more (pagination) e.g. "who | more" or "more file.txt"
grep (removes lines that do not contain certain text)
[ grep is so useful that an entire module will be devoted to it. ]
sort (sorting) e.g. "who | sort" or "sort file.txt" (sorts alphabetically) or "who | sort | more"
[ when sorting a file, only the on-screen data is sorted. The file itself remains unchanged. ]
wc (counts lines, words and/or characters)
tee (duplicates by writing to both the screen and a file) e.g. "who | tee wholist.txt"
sed (basic editing .. a rather complex filter, could take an entire chaper, more powerful processing than sort or grep)
awk (most compex & powerful of all filters, does anything you might want, could take an entire course)
[ awk has its own programming language ]
cat (simplest of all filters, involves no processing, echoes input to screen) e.g. "who | cat" or "cat file.txt"
 
WWW  
IP Logged
 

MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: Using the Unix/Linux shell command-line
Reply #21 - Jul 23rd, 2009 at 1:19am
 
Rad.in.UbuntuVM wrote on Jul 23rd, 2009 at 12:35am:
cat (simplest of all filters, involves no processing, echoes input to screen) e.g. "who | cat" or "cat file.txt"

This is oversimplified.  I'm going to add a little detail.

cat was originally designed to to conCATenate multiple files.  In other words, the results of 'cat file1 file2 file3' should be a single file containing each of the 3 input files one after another.  Now, that by itself isn't often incredibly useful, and there are lots of ways to achieve the same effect.  However, by default, cat sends its output to stdout, which is usually the tty you are on, which means the text gets printed on the screen. 

So, cat often gets used to simply dump text onto the screen.  While this is useful, it is a side-effect of its original design.

I do use it often for this simplified purpose, but it always give me a slight twinge of guilt.  It feels like I'm somehow cheating since I never use it for what it was actually intended for.
 
WWW  
IP Logged
 
Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #22 - Jul 23rd, 2009 at 7:43pm
 
06-06 - Searching for Text in Files (grep)

grep is a particularly useful filter
grep wll print all lines that contain the text specified
actually (to be specific) grep REMOVES lines that do not contain the text specified
so grep is a true FILTER, by filtering out all lines that don't contain the text specified
syntax: "grep pattern filename(s)"
pattern = the text you search for
e.g. "grep rad *.html" will display all the lines that contain the word 'rad' in all the html files
grep uses its own set of wildcards
grep's wildcards are consistent with all program's in Unix that use Regular Expressions, such as vi and sed.
the shell does NOT use Regular Expressions
Regular Expressions have their own particular set of wildcards.
use single quotes to enclose any searched-for terms that contain special characters.
e.g. "grep '<h3' index2.html" - The < sign is a special character.
If you're searching for only regular text words (with no special characters), single quotes are not required.
But good idea to enclose EVERYTHING you search for in grep inside single quotes
when searching for text thru a big set of directories & subdirectories, or even the entire computer, then good idea to use grep in conjunction with find.
e.g  "find . -name '*.txt' -exec grep 'hello' {} \;"
notice no piping going on. Simply using 2 programs in conjunction with each other.
Read the grep man pages, particularly in relation to the Regular Expressions.
 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: Using the Unix/Linux shell command-line
Reply #23 - Jul 24th, 2009 at 12:15am
 
Rad.in.UbuntuVM wrote on Jul 23rd, 2009 at 7:43pm:
notice no piping going on. Simply using 2 programs in conjunction with each other.

There is no piping going on because the find command has the '-exec' option, which is being used here.  It's a really nice option, allowing you to do things like:
find /var/log -atime +90 -exec rm \;
which finds any file in the /var/log directory which hasn't been read in more than 90 days and deletes it.  You could achieve a similar effect by piping the output of find to xargs (find /var/log -atime +90 | xargs rm) but the -exec option makes it a little more streamlined.

Rad.in.UbuntuVM wrote on Jul 23rd, 2009 at 7:43pm:
Read the grep man pages, particularly in relation to the Regular Expressions. 

The man pages probably won't do Regular Expressions justice.  They are worth some study on their own.

There aren't many guides on the Internet that give a clear introduction to Regular Expressions.  Here are a few I found in a brief search, and my sincere thanks would go out to anyone who can find anything more clearly written:

http://www.amk.ca/python/howto/regex/
http://www.regular-expressions.info/tutorial.html

The Wikipedia example page is helpful for learning:
http://en.wikipedia.org/wiki/Regular_expression_examples

and here is a user-contributed library:
http://regexlib.com/

The best book, for those who have time, money, and a need to *really* understand this, is by O'Reilly:

http://oreilly.com/catalog/9780596528126/

O'Reilly has outstanding books on many Unix and computer science subjects.  I've read several, and found them all worth my time.
 
WWW  
IP Logged
 
Rad.Test
Technoluster
***
Offline


Rad's non-Admin test-profile
in Firefox

Posts: 108


Back to top
Re: Using the Unix/Linux shell command-line
Reply #24 - Jul 24th, 2009 at 5:18pm
 
I've been hearing about Regular Expressions more and more. And it seems many languages use/support them, no? (such as PHP)

So it seems time spent learning RegEx would be time well spent, no?

Nigel was the one who first suggested I pay particular attention to RegEx.

I seem to learn best with the multi-pass approach .. first a quick review to get a general idea of what's what. Then a more involved look/see. Progressing with breaks in between to allow for 'settling' and development of a format for more involved learning.
 
 
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: Using the Unix/Linux shell command-line
Reply #25 - Jul 24th, 2009 at 5:34pm
 
Rad.Test wrote on Jul 24th, 2009 at 5:18pm:
And it seems many languages use/support them, no? (such as PHP)

Yup.  PHP, Perl, Ruby, grep, sed, and too many more to list.  Regular expressions are very useful and very portable.

Rad.Test wrote on Jul 24th, 2009 at 5:18pm:
Nigel was the one who first suggested I pay particular attention to RegEx.

I think that Nigel also made a point that with PHP, everything you are doing breaks down to processing text patterns if you break it apart far enough.  Therefor, if you understand RegEx, you will have an idea what is going on underneath the PHP code you write.

Rad.Test wrote on Jul 24th, 2009 at 5:18pm:
I seem to learn best with the multi-pass approach

I've found that one of my strongest strengths is understanding how I learn.  When I can find material that matches up with my learning style, I can absorb an incredible amount of information quickly.  Unfortunately, finding material that fits my unique learning style often takes almost as long as working through said material, so I have to balance my time looking for the right way to learn with actually learning.

Of course, teaching something and getting practical experience in it are the BEST ways to learn, but both can be time consuming.   Maybe someday we'll make a whole thread on it, but for now, you might find the cone of learning interesting:

http://docs.google.com/gview?a=v&q=cache:C9L5WXRRopIJ:www.public-health.uiowa.ed...

I'm not sure if I think it fits me 100%, but it does put learning/teaching styles into perspective.
 
WWW  
IP Logged
 

Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #26 - Jul 24th, 2009 at 7:01pm
 
MrMagoo wrote on Jul 24th, 2009 at 5:34pm:
for now, you might find the cone of learning interesting:

http://docs.google.com/gview?a=v&q=cache:C9L5WXRRopIJ:www.public-health.uiowa.ed....

I would call that a 'pyramid' .. of learning.

Suks that reading seems to yield the least, with only 10%.

The Head First series of books specifically addresses the concept of learning and begins each text with a chapter on why the book is designed the way it is.

http://blogs.radified.com/2007/07/learning_theory_head_first_book_series_oreilly...

I enjoy learning about learning. I normally read with a pen. The mere act of deciding what to underline (the important parts) and what to leave alone seems to improve retention. By getting me actively involved.

Really important ideas/concepts get a star. Things I'm not clear on get a question mark. Things worth pondering get a "Hmmm" (in the margin) Etc.

From what I've read, we should try to be *emotionally* involved in learning, as emotions seem to aid retention. Being RELAXed is also another concept I've heard of .. as is "frequent breaks" (as we learn better at the beginning & end of a learning session).

But best of all, I think, is being INTERESTED in the subject matter. No?

If you are lost in the wild, and hungry, I bet a book on edible plants would be very easy to read.
 
WWW  
IP Logged
 
Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #27 - Jul 24th, 2009 at 7:08pm
 
MrMagoo wrote on Jul 24th, 2009 at 5:34pm:
I think that Nigel also made a point that with PHP, everything you are doing breaks down to processing text patterns if you break it apart far enough.Therefor, if you understand RegEx, you will have an idea what is going on underneath the PHP code you write.

Not sure exactly what you are referring to. Example?
 
WWW  
IP Logged
 
Rad.in.UbuntuVM
Radmeister
**
Offline


Rad in Ubuntu 9.04 Jaunty
Virtual Machine

Posts: 92


Back to top
Re: Using the Unix/Linux shell command-line
Reply #28 - Jul 24th, 2009 at 7:24pm
 
Is RegEx like a search prgm, or just the characters used in the strings that you search for?

In other words, do you have to use RegEx in conjunction with another prgm?
 
WWW  
IP Logged
 
Rad
Radministrator
*****
Offline


Sufferin' succotash

Posts: 4090
Newport Beach, California


Back to top
Re: Using the Unix/Linux shell command-line
Reply #29 - Jul 24th, 2009 at 7:29pm
 
06-07 - Standard Error

Each program produces 2 separate sets of output
They both go to the screen by default
Do it appears to be 1 set
First set = standard output
Second set = standard error
Standard error contains any error messages a program might've produced.
Standard error can go to either:
1. screen (default output)
2. file
(can't go to another program)
To redirect standard error to a file, use "2>" (no space between 2 and >)
btw - standard output can be redirected to a file by using "1>"
but the 1 is assumed by default, so nobody uses it.
So .. any command's output can be redirected to 2 separate files, if necessary.
Or 1 can go to a file, while the other goes to the screen
/dev/null = special file, like a black hole. anything sent there disappears into thin air. 
e.g. "find / -name xyz 2> /dev/null" redirects all error messages (many "Permission denied" messages) to black hole /dev/null and displays only valid finds (files or directories named 'xyz').

This was the most boring and seemingly useless module yet. While good to know, can't see how I would ever actually use this info.

next one looks better > Process Control
 
WWW  
IP Logged
 
Pages: 1 2 3 4 5
Send Topic Print