--- title: "Linux Command Lines Notes" date: 2023-01-18T14:42:15-06:00 draft: false medium: notes asset: false tags: description: --- ## Wild Card The `*` acts as a wild card. If you were to select a number of files that shared a characteristic, you can use the `*` in its place. ```shell touch file1 file2 file3 file4 ls >file1 file2 file3 file4 chmod 777 file* ``` This would have the terminal check each file that met that starting variable and apply the action requested to it. But assume that another file was in there that we didn't need to change the permissions on. We could combine the wild card and history operators to pull the variable applied to one command into another. ```shell touch file1 file2 file3 file4 ls >file1 file2 file3 file4 file12_DO_NOT_CHANGE chmod 777 !-2* ``` This would change the permissions of only the files named by the `touch` command. | Wildcard | Meaning | | --------------- | ------------------------------------------------------------------ | | * | Matches any character | | ? | Matches any single character | | \[characters\] | Matches any character that is a member of the set *characters* | | \[!character\] | Matches any character that is not a member of the set *characters* | | \[\[:*class*:\]\] | Matches any character that is a member of the specified *class* | ### Commonly used Character Classes | Character Class | Meaning | | --------------- | ---------------------------------- | | \[:alnum:\] | Matches any alphanumeric character | | \[:alpha:\] | Matches any alphabetic character | | \[:digit:\] | Matches any numeral | | \[:lower:\] | Matches any lowercase letter | | \[:upper:\] | Matches any uppercase letter | ### Wildcard Examples | Pattern | Matches | | ------------------------------ | ----------------------------------------------------------------------------- | | * | All Files | | *g*\* | Any file beginning with *g* | | *b*\*.txt | Any file beginning with *b* followed by any characters and ending with *.txt* | | *Data*??? | Any file beginning with "Data" followed by exactly three characters | | \[*abc*\]\* | Any file beginning with either an *a*, *ab*, or *ac* | | *BACKUP*.\[0-9\]\[0-9\]\[0-9\] | Any file beginning with *BACKUP*. followed by exactly three numerals | | \[\[:upper\]\]* | Any file beginning with an uppercase letter | | \[!\[:digit:\]\]* | Any file not beginning with a numeral | | \*\[\[:lower:\]*123*\] | Any file ending with a lowercase letter or the numerals *1*, *2*, *3* | ![[Command Line Tricks That Make Me the Coolest Guy in the Office#Wild Card]] ## Part III -- Common Tasks and Essential Tools ### Package Management ### Storage Media ### Networking **wget** allows you to install programs or files from the command line, particularly for remote accessed servers. #### SSH Similar to the *File Transfer Protocol* (FTP), remote communications were initially performed without login or password encryption. SSH solves this problem by providing two key features: 1. It authenticates that the remote host is who it says it is (thus preventing so-called man-in-the-middle attacks) 2. It encrypts all of the communications between the local and remote hosts. An SSH server runs on the remote host, listening for incoming connections, by default, on port 22, while an SSH client is used on the local system to communicate with the remote server. ### Searching for Files **Locate** is a terminal program that provides a search mechanism for finding files based solely on name. ```shell ~> locate bin/zip ___ /usr/bin/zip /usr/bin/zipcloak /usr/bin/zipgrep /usr/bin/zipinfo/usr/bin/zipnote /usr/bin/zipsplit ___ ``` #### Find **Find** searches a given directory (and its subdirectories) for files based a variety of attributes. *Options*, *tests*, and *actions* serve as the specific criteria for the search. ##### Tests | File type | Description | | --------- | ----------------------------- | | b | Block special device file | | c | Character special device file | | d | Directory | | f | Regular File | | l | Symbolic link | ```Shell find ~ -type f ``` ###### Name ```Shell find ~ -type f -name "*.md" ``` ###### File Size | Character | Unit | | --------- | ------------------------------------------------------------ | | b | 512-byte blocks. This is the default if no unit is specified | | c | bytes | | w | 2-byte words | | k | Kilobytes (units of 1,024 bytes) | | M | Megabytes (units of 1,048,576 bytes) | | G | Gigabytes (units of 1,073,741,824 bytes) | ```Shell find ~ -type f -name "*.md" -size +10c ``` ##### Operator | Operator | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------ | | -and / -a | Match if tests on both sides of the operator are true. Note that when no operator is present, -and is implied by default | | -or / -o | Match if a test on either side of the operator is true | | -not / ! | Match if the test following the operator is false | | ( ) | Group tests and operators together | ```shell find ~ \( -type f -not -perm 0600 \) -or \( -type d -not -perm 0700 \) ``` ### Archiving and Backup #### gzip `gzip` is a lossless[^1] compression program that compress one or more files that attaches a `.gz` to end of it. ### Regular Expressions **Regular Expressions**- symbolic notations used to identify patterns in text #### grep As looked at in [[Don't Use Cat]], the grep tool provides an incredible text find command from within the terminal. `grep` stands for "Global Regular Expression Print", and is can be used with *metacharacters* to accomplish incredible feats. #### Metacharacters | Character | Function | | --------- | ----------------------------------------------- | | . | Substitutes as any character | | [[The Linux Command Line#Anchors\|^ (caret)]] | Filters by the beginning of the line | | [[The Linux Command Line#Anchors\|$]] | Filters by the end of the line | ##### The Any Character The period character (.) is used to match any character. This is why if you were to filter by file type having a `'.zip'` could result in anything from `bunzip`, `bzip2recover`, `unzip`, `prezip-bin`, etc. ==**Interestingly, if there were a `zip` file in our directory, it wouldn't appear in our results because it increased the length of the requirement for our regular expression.**== Since `.zip` is four characters, `zip` will not come up as a result since it's three characters. ##### Anchors ```shell [me@linuxbox ~]$ grep -h '^zip' dirlist*.txt zip zipcloak zipgrep zipinfo zipnote zipsplit [me@linuxbox ~]$ grep -h 'zip$' dirlist*.txt gunzip gzip funzip gpg-zip preunzip prezip unzip zip [me@linuxbox ~]$ grep -h '^zip$' dirlist*.txt zip ``` ##### Bracket Expressions and Character Classes Bracket expressions allow you to match a single character from a specified set of characters. With brackets, you can include metacharacters except `^` and `-` as they serve to indicate negation and a character range, respectively. ```shell [me@linuxbox ~]$ grep -h '[bg]zip' dirlist*.txt bzip2 bzip2recover gzip ``` ###### Negation IF the first character in a bracket expression is a caret (`^`), the remaining characters are taken to be a set of characters that must not be present at the given character position. ```shell [me@linuxbox ~]$ grep -h '[^bg]zip' dirlist*.txt bunzip2 gunzip funzip gpg-zip preunzip prezip prezip-bin unzip unzipsfx ``` *Note*: just `zip` didn't appear in the results. ==A negated character set still requires a character at the given position, but the character must not be a member of the negated set.== Also, a caret will only invoke negation if it is the first character of the bracket. ### Text Processing #### Sort | Option | Long Option | Description | | ------ | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -b | --ignore-leading-blanks | By default, sorting is performed on the entire line, starting iwth the first character in the line. This option causes sort to ignore leading spaces in lines an calculates sorting based on the first non-whitespace character on the line | | -f | --ignore-case | Make sorting case-insensitive | | -n | --numeric-sort | Using this option allows sorting to be performed on numeric values rather than alphabetic values | | -r | --reverse | Sort in reverse order; descending rather than ascending | | -k | --key=*field1*\[,*field2*\] | Sort based on a key field located from *field1* to *field2*. | | -m | --merge | Merge multiple files into a single sorted result without performing any additional sorting | | -o | --output=*file* | Send sorted output to *file* rather than standard output | | -t | --field-separator=*char* | Define the field-separator character. By default fields are separated by spaces or tabs | *field* comes from [[Accounting Information systems|database terminology]]; in essence, a cell on a spreadsheet and each row in a *record*. If a command result were to be in tabular format, you can specify a field number and the sort command will pull from that column. ```shell ls -l /usr/bin | sort -nrk 5 | head ``` The above command will have list the complete information about each entry in the `usr/bin` and sort all entries in the fifth column by number and from highest to lowest; then it will print the first resulting ten. `sort` also allows you to apply different sort rules to each field. We could sort the data by applying multiple keys and rules to each key, such as `-k 1,1 -k 2n`. This would sort *field1* alphabetically and sort *field2* numerically. On top of multiple filters, you can sort select portions of a data feed. For instance, if you were to filter by date, you would specify `field.character_number`. ![](https://i.imgur.com/UptFGs1.png) Now assume that there are no spaces between the records, but they share a separating character (`.`,`-`,`_`,`,`). The `-t` option allows you to define a separator character. ![](https://i.imgur.com/d9iO3rn.png) #### Uniq | Option | Long Option | Description | | ------ | ----------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | -c | --count | Output a list of duplicate lines preceded by the number of times the line occurs | | -d | --repeated | Output only repeated lines, rather than unique lines | | -f *n* | --skip-fields=*n* | Ignore *n* leading fields in each line. Fields are separated by whitespace as they are in `sort`; however, unlike it, `uniq` has no option for setting an alternate field separator | | -i | --ignore-case | Ignore case during the line comparisons | | -s *n* | --skip-chars=*n* | Skip the leading *n* characters of each line | | -u | --unique | Output only unique lines. Lines with duplicates are ignored | ### Slicing and Dicing #### Cut | Option | Long Option | Description | | ---------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | | -c *list* | --characters=*list* | Extract the portion of the line defined by *list*. The list may consist of one or more comma-separated numerical ranges | | -f *list* | --fields=*list* | Extract one more fields from the line as defined by *list*. The list may contain one or more fields or field ranges separated by commas | | -d *delim* | --delimiter=*delim* | When -f is specified use *delim* as the field delimiting character. By default, fields must be separated by a single tab character | | | --complement | Extract the entire line of text, except for those potions specified by -c and/or -f | `cut` is used to extract a section of text from a line and output the extracted section to standard output. It can accept multiple file arguments or input from standard input. With the cut materials, you can use the `paste` function to add the cut data to a data collection in question. #### Join - Join Lines of Two Files on a Common Field `join` acts similar to `paste`, but it goes beyond just adding the field to the data set. `join` is an operation usually associated with *relational databases* where data from multiple tables with a shared key field is combined to form a desired result. ### Compiling Programs Compiling is the process of translating source code into the native language of the computer's processor (generally X86 or ARM). *Machine language* is numeric code that describes extremely small operations, such as "add this byte", "point to this location in memory", or "copy this byte". *Assembly language* simplified this (a little) by using character *mnemonics* such as `CPY` for copy and `MOV` for move. Programs written in assembly are processed into machine language by an *assembler* program. Both of these methods are *low-level programming languages* as they interact directly with the operation and processing of the computer. Programs written in *high-level programming languages* are converted into machine language by processing them with another program, called a *compiler*. ### Printf `printf` is a stupidly important command that does not take input (natively, see [[The Linux Command Line#xargs|xargs]]). Instead, it is called to *print formatted* text as an echo for simplicity and efficiency. The formatted string can contain things like literal text, escape characters, and sequences beginning with the `%` character, which are called *conversion specifications*. | Specifier | Description | | --------- | ----------------------------------------------------------------------------- | | d | Format a number as signed decimal integer | | f | format and output a floating-point number | | o | format an integer as an octal number | | s | format a string | | x | Format an integer as a hexadecimal number using lowercase a to f where needed | | X | Same as `x` but use uppercase letters | | % | Print a literal % symbol (i.e, specify \%\%) | Several optional components may be added to the conversion specifier to adjust its output. A complete conversion specification may consist of the following: ```shell %[flags][width][.precision]conversion_specification ``` | Component | Description | | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | flags | There are five different slags: | | | #: Use the *alternative format* for output. This varies by data type. For o (octal number) conversion, the output is prefixed with o. For `x` or `X` (hexidecimal number) conversions, the output is presfixed with 0x or 0X, respecively | | | 0 (zero): Pad the output with zeros. This means that the field will be filled with leading zeros, as in 000380 | | | - (dash): Left-align the output. By default, `printf` right-aligns output | | | ' ' (space): Produce a leading space for positive numbers | | | + (plus sign): Sign positive numbers. By default, `printf` only signs negative numbers | | width | A number specifying the minimum field width | | .precision | For floating-point numbers, specify the number of digits of precision to be output after the decimal point. For string conversion, *precision* specifies the number of characters to output | ## Part IV - Writing Shell Scripts ### General Script File Format ```shell #!/bin/bash # This is our first script echo 'Hello World!' ``` The `#!` is called a *shebang* and is used to tell the kernel the name of the interpreter that should be used to execute the script that follows. Once the script is generated, you need to make it executable with the `chmod` program. Generally you'd want to set them to 755 for everyone to execute or 700 for scripts that only the owner can execute. ![[Remove Permission Requirements]] With the permissions set, you can execute the script. ```shell [me@linuxbox ~]$ ./hello_world Hello World! [me@linuxbox ~]$ hello_world bash: hello_world: command not found ``` There is nothing different from a script you make vs one that is already installed. The difference is in the location. The `PATH` environment variable is how the system searches a list of directories each time it needs to find an executable program, if no explicit path is specified. You can find the set paths by using `echo $PATH`. ```shell [jed_hed@JedHedPC] echo $PATH /home/jed_hed/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/bin/vendor_perl:/usr/bin/core_perl ``` So long as our script is located in one of the above directories, our problem is solved. ### Starting a Project To add a variable to a project, declare the variable name and set it equal to the desired value. Once a variable is specified, it can be referred to by calling `$VARIABLE_IN_QUESTION`. There is no inherent difference between variables and constants in shell scripting, so generally people specify between the two by making variables **lowercase** and constants **uppercase**. >You can change a selection from lowercase to uppercase (or vise versa) in vim by going into visual mode, selecting the word(s) in question, and pressing either `U` or `u`. This function can also be performed by pressing the `~` key and it will invert the casing of all selected characters. If you want to perform this same action outside of visual mode, type `g`, the respective casing key, and the location keys similar to other functions (`a` around `w` word). ^ea43e9 Unlike some other programming languages, the shell **does not care about the type of data assigned to a variable**; it treats them all as strings. ```shell a=z # Assign the string "z" to variable a. b="astring" # Embedded spaces must be within quotes. c="a string and $b" # Other expansions such as variables can be # expanded into the assignment. d="$(ls -l foo.txt)" # Results of a command. e=$((5 * 7)) # Arithmetic expansion. f="\t\ta string\n" # Escape sequences such as tabs and newlines. ``` Multiple variable assignments may be done on a single line. #### Here Documents *Here Document* or *here script* is an additional form of I/O redirection in which we embed a body of text into our script and feed it into the standard input of a command. It works like this: ```shell command << token text token ``` *Command* is the name of a command that accepts standard input and *token* is a string used to indicate the end of the embedded text. Generally, you will put a `cat << _EOF_` towards the start of the form and ending the document with the `_EOF_`. This will concatenate the text from the bottom element to the top. The reason this can be helpful is because it allows you to embed quotes freely within the document. ![[Command Line Tricks That Make Me the Coolest Guy in the Office#TL;DR]] ### Top-Down Design [[Software Architecture|Top-down design]] is a mentality in which each function has one sole action that contributes to the program. Like most other programming languages, shell scripts permit *functions* to be used to execute commands. Functions can have carry their own *local variables* that can only be called to from within the function in question. Variables outside of the functions are *global variables* and can be referred to anytime after they have been defined. ```shell #!/bin/bash # local-vars: script to demonstrate local variables foo=0 # global variable foo funct_1 () { local foo # variable foo local to funct_1 foo=1 echo "funct_1: foo = $foo" } funct_2 () { local foo # variable foo local to funct_2 foo=2 echo "funct_2: foo = $foo" } echo "global: foo = $foo" funct_1 echo "global: foo = $foo" funct_2 echo "global: foo = $foo" ___ [me@linuxbox ~]$ local-vars global: foo = 0 funct_1: foo = 1 global: foo = 0 funct_2: foo = 2 global: foo = 0 ``` ### Branching with IF ```shell x=5 if [ "$x" -eq 5 ]; then echo "x equals 5." else echo "x does not equal 5." fi # an if statement must end with a "fi" ``` There are two equivalent forms of boolean query: `test *expression*` and `[*expression*]`. It's interesting to note that both `test` and `[` are actually commands. #### String Expressions | Expression | Is true if: | | ---------------------- | --------------------------------------------------------------------------------------------------------------------------- | | *string* | *string* is not null | | -n *string* | the length of *string* is greater than zero | | -z *string* | the length of *string* is zero | | *string*1 = *string*2 | *string*1 and *string*2 are equal. | | *string*1 == *string*2 | Single or double equal signs may be used. The use of double equal signs is greatly preferred, but it is not POSIX complaint | | *string*1 != *string*2 | *string*1 and *string*2 are not equal | | *string*1 > *string*2 | *string*1 sorts after *string*2 | | *string*1 < *string*2 | *string*1 sorts before *string*2 | #### Integer Expressions | Expressions | Is true if: | | ------------------------- | ------------------------------------------------- | | *integer*1 -eq *integer*2 | *integer*1 is equal to *integer*2 | | *integer*1 -ne *integer*2 | *integer*1 is not equal to *integer*2 | | *integer*1 -le *integer*2 | *integer*1 is less than equal to *integer*2 | | *integer*1 -lt *integer*2 | *integer*1 is less than *integer*2 | | *integer*1 -ge *integer*2 | *integer*1 is greater than or equal to *integer*2 | | *integer*1 -gt *integer*2 | *integer*1 is greater than *integer*2 | #### Integer vs File and Pathname `[[ ]]` are useful for evaluating file and pathnames while `(( ))` are used for integer expressions. ```shell x=-5 if ((INT == 0)); then echo "INT is zero" else if ((INT < 0)); then echo "INT is negative." else echo "INT is positive." fi if (( ((INT % 2)) ==0)); then echo "INT is even." else echo "INT is odd." fi fi ``` #### Combining Expressions | Operation | test | \[\[ \]\] and (()) | | --------- | ---- | ------------------ | | AND | -a | && | | OR | -o | \|\| | | NOT | ! | ! | ![[Never say "If" writing a Bash script -- Linux PSCT]] ### Reading Keyboard Input `read [-options] [variable]`. `read` is used to read a single line of standard input that can be assigned to a variable. ```shell echo -n "Please enter an integer ->" read int if ((INT == 0)); then echo "INT is zero" else if ((INT < 0)); then echo "INT is negative." else echo "INT is positive." fi if (( ((INT % 2)) ==0)); then echo "INT is even." else echo "INT is odd." fi fi ``` If no variables are listed after the `read` command, a shell variable, `REPLY` will be assigned all the input. #### Read Options | Option | Description | | -------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | -a *array* | Assign the input to *array*, starting with index zero | | -d *delimiter* | The first character in the string *delimiter* is used to indicated the end of input, rather than a newline character | | -e | Use Readline to handle input. This permits input editing in the same manner as the command line | | -i *string* | Use *string* as a default reply if the user simply presses `ENTER`. Requires the -e option | | -n *num* | Read *num* characters of input, rather than an entire line | | -p *prompt* | Display a prompt for input using the string *prompt* | | -r | Raw mode. Do not interpret backslash characters as escapes | | -s | Silent mode. Do not echo characters to display as they are typed. This is useful when inputting passwords and other condiential information. | | -t *seconds* | Timeout. Terminate input after *seconds*. `read` returns a non-zero exit status if an input times out | | -u *fd* | Use input from file descriptor *fd*, rather than standard input | #### IFS Generally, the shell performs word-splitting when provided to `read`: each word is separated into different items by one or more spaces. ```shell FILE=/etc/passwd read -p "Enter a username > " user_name file_info="($grep "^user_name:" $FILE)" if [ -n "$file_info" ]; then IFS= ":" read user pw uid grid name home shell <<< "$file_info" echo "User = '$user'" echo "UID = '$uid'" echo "GID = '$gid'" echo "Full Name = '$name'" echo "Home Dir. = '$home'" echo "Shell = '$shell'" else echo "No such user '$user_name'" >&2 exit 1 fi ``` The `IFS` line consists of three parts: a variable assignment, a `read` command with a list of variable names as arguments, and a *here string*. The here string allocates the data to each of the expressed variables. #### Menus In menu-driven programs, the user is presented with a list of choices and is asked to choose one. ```shell #!/bin/bash # read-menu: a menu driven system information program clear echo " Please Select: 1. Display System Information 2. Display Disk Space 3. Display Home Space Utilization 0. Quit " read -p "Enter selection [0-3] > " if [[ "$REPLY" =~ ^[0-3]$ ]]; then if [[ "$REPLY" == 0 ]]; then echo "Program terminated." exit fi if [[ "$REPLY" == 1 ]]; then echo "Hostname: $HOSTNAME" uptime exit fi if [[ "$REPLY" == 2 ]]; then df -h exit fi if [[ "$REPLY" == 3 ]]; then if [[ "$(id -u)" -eq 0 ]]; then echo "Home Space Utilization (All Users)" du -sh /home/* else echo "Home Space Utilization ($USER)" du -sh "$HOME" fi exit fi else echo "Invalid entry." >&2 exit 1 fi ``` ### Flow Control: Looping with While/Until #### While ```shell # !/bin/bash # while-count: display a series of numbers count=1 while [[ "$count" -le 5 ]]; do echo "$count" count=$((count + 1)) done echo "Finished." ``` The syntax of a `while` command is as follows: `while *commands*; do *commands*; done`. Like `if`, `while` is a boolean operator that checks the parameter in its container. So long as the parameter's exit value is 1, false, it performs the command inside the loop. ##### Breaking out of a loop Bash has two methods of exiting a look, `break` and `continue`. `break` immediately terminates the loop continues the program beyond it. `continue` causes the remainder of the loop to be skipped and places in the next iteration of the loop. ```shell #!/bin/bash # while-menu2: a menu driven system information program DELAY=3 # Number of seconds to display results while true; do clear cat <<- _EOF_ Please Select: 1. Display System Information 2. Display Disk Space 3. Display Home Space Utilization 0. Quit _EOF_ read -p "Enter selection [0-3] > " if [[ "$REPLY" =~ ^[0-3]$ ]]; then if [[ "$REPLY" == 1 ]]; then echo "Hostname: $HOSTNAME" uptime sleep "$DELAY" continue fi if [[ "$REPLY" == 2 ]]; then df -h sleep "$DELAY" continue fi if [[ "$REPLY" == 3 ]]; then if [[ "$(id -u)" -eq 0 ]]; then echo "Home Space Utilization (All Users)" du -sh /home/* else echo "Home Space Utilization ($USER)" du -sh "$HOME" fi sleep "$DELAY" continue fi if [[ "$REPLY" == 0 ]]; then break fi else echo "Invalid entry." sleep "$DELAY" fi done echo "Program terminated." ``` This version of the script is an *endless loop* since the *true* command will always return the same exit status. An `until` can be used in place of a `while` to achieve the opposite result: the loop will execute **until** it receives a zero exit status. ### Flow Control: Branching with Case A `case` takes a `$REPLY` input and expresses it in a clear to see manner in the script. ```shell #!/bin/bash # case-menu: a menu driven system information program clear echo " Please Select: 1. Display System Information 2. Display Disk Space 3. Display Home Space Utilization 0. Quit " read -p "Enter selection [0-3] > " case "$REPLY" in 0) echo "Program terminated." exit ;; 1) echo "Hostname: $HOSTNAME" uptime ;; 2) df -h ;; 3) if [[ "$(id -u)" -eq 0 ]]; then echo "Home Space Utilization (All Users)" du -sh /home/* else echo "Home Space Utilization ($USER)" du -sh "$HOME" fi ;; *) echo "Invalid entry" >&2 exit 1 ;; esac ``` The command looks at the input, the *word*, and attempts to match it against one of the specified *patterns*. When a match is found, the command associated with it is executed and no further matches are attempted. #### Patterns | Pattern | Description | | ---------------- | -------------------------------------------------- | | a) | Matches if *word* equals a | | \[\[:alpha:\]\]) | Matches if *word* is a single alphabetic character | | ???) | Matches if *word* is exactly three characters long | | \*.txt) | Matches if *word* ends with the characters *.txt* | | \*) | Matches any value of *word*. It is good practice to include this as the last pattern in a case command to catch any values of *word* that did not match a previous pattern, that is, to catch any possible invalid values | It's also possible to combine multiple patterns using the vertical bar character as a separator `|`, creating an "or" conditional operator. It's useful for things such as including upper and lowercase. `case` operations also allow for adding multiple correct answers to a prompt by adding a `;;&` at the end of each function. ```shell #!/bin/bash # case4-2: test a character read -n 1 -p "Type a character > " echo case "$REPLY" in [[:upper:]]) echo "'$REPLY' is upper case." ;;& [[:lower:]]) echo "'$REPLY' is lower case." ;;& [[:alpha:]]) echo "'$REPLY' is alphabetic." ;;& [[:digit:]]) echo "'$REPLY' is a digit." ;;& [[:graph:]]) echo "'$REPLY' is a visible character." ;;& [[:punct:]]) echo "'$REPLY' is a punctuation symbol." ;;& [[:space:]]) echo "'$REPLY' is a whitespace character." ;;& [[:xdigit:]]) echo "'$REPLY' is a hexadecimal digit." ;;& esac ``` ### Positional Parameters *Positional parameters* are the individual words typed on the command line, separated by a space. The variables range from 0 through 9 and can be called by typing `$0`, the number being the variable in question. For numbers greater than 9, surround the number in braces, `${69}`. Even when no arguments are provided, `$0` will always contain the first item appearing on the command line, i.e. the program you called to execute. ^c68c16 There are also special parameters that can be used to call all parameters in a reading. | Parameter | Description | | --------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $\* | Expands into the list of positional parameters, starting with 1. When surrounded by double quotes, it expands into a double-quoted string containing all of the positional parameters, each separated by the first character of the IFS shell variable (by default a space character) | | $@ | Expands into the list of positional parameters, starting with 1. When surrounded by double quotes, it expands each positional parameter into a separated word as if it was surrounded by double quotes | ```shell #!/bin/bash # posit-params3: script to demonstrate $* and $@ print_params () { echo "\$1 = $1" echo "\$2 = $2" echo "\$3 = $3" echo "\$4 = $4" } pass_params () { echo -e "\n" '$* :'; print_params $* echo -e "\n" '"$*" :'; print_params "$*" echo -e "\n" '$@ :'; print_params $@ echo -e "\n" '"$@" :'; print_params "$@" } pass_params "word" "words with spaces" ``` ```shell [jed_hed@archlinux ~]$ posit-param3 $*: $1 = word $2 = words $3 = with $4 = spaces "$*" : $1 = word words with spaces $2 = $3 = $4 = $@ : $1 = word $2 = words $3 = with $4 = spaces "$@" : $1 = word $2 = words with spaces $3 = $4 = ``` ### Flow Control: Looping with For There are two methods to express `for` loops: the **traditional** way and the **arithmetic** way. The traditional way is as follow: ```shell for *variable* [in *words*]; do *commands* done ``` The *variable* is the name of a variable that will increment during the execution of the loop, *word* is an optional list of items that will be sequentially assigned to *variable*, *commands* are the commands that are to be executed on each iteration of the loop. ```shell [me@linuxbox ~]$ for i in A B C D; do echo $i; done A B C D ``` The arithmetic method stems from [[C programming language]] and is directed towards more algebraic and mathematical reasoning: ```shell for (( expression1; expression2; expression3 )); do *commands* done ``` This should look strikingly familiar because it resembles the `for` method found in JavaScript. This method is basically the same as running: ```shell ((expression1)) while ((expression2)); do commands ((expression3)) done ``` `expression1` is initiated, `expression2` is put in the `while` loop, and it triggers `expression3` until the loop is satisfied. ```shell for ((i=o; i<5; i=i+1)); do echo $1 done ``` ### Strings and Numbers General variables can be expressed by using a simple `$` in front of it, but in instances where you would want to call a variable while appended with additional text, the `{}` come in handy to express a real variable. ```shell a="foo" echo "$a_file" # Nothing is echoed because "a_file" does not exist. [Jed_Hed@archlinux] echo "${a}_file" foo_file # This does work because now the terminal can process that we are substuting the "foo" in place of a define variable ``` This is also the reason why multi digit numerals have to be expressed with curly brackets ![[The Linux Command Line#^c68c16]] #### Expansions to Manage Empty Variables | Method | Result | | ---------------------- | ----------------------------------------------------------------------------------------------------- | | ${*parameter*:-*word*} | If *parameter* is empty, *word* will serve as substitute, but *word* will not persist is called again | | ${*parameter*:=*word*} | If *parameter* is empty, *word* will serve as substitute and *word* **will** persist if called again | | ${*parameter*:?*word*} | If *parameter* is empty, the contents of *word* are sent to standard error | | ${*parameter*:+*word*} | If *parameter* is empty, nothing happens. But if *parameter* is filled, *word* is substituted for *parameter* | #### String Operations On top of providing substitutes, parameters carry ways of expressing and using their information without calling other functions or scripts | **Parameters** | **Description** | | :--------------: | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | $# | It parameter represents the total number of arguments passed to the script. | | $0 | This parameter represents the script name. | | $n | This parameter represents the arguments corresponding to a script when a script is invoked such $1 $2…etc. $1, $2…etc are called positional parameters. | | $\* | This parameter describes the positional parameters to be distinct by space. For example, if there are two arguments passed to the script, this parameter will describe them as $1 $2. | | \$\$ | This parameter represents the process ID of a shell in which the execution is taking place. | | $! | This parameter represents the process number of the background that was executed last. | | $@ | This parameter is similar to the parameter $\*. | | $? | This parameter represents exit status of the last command that was executed. Here 0 represents success and 1 represents failure. | | $\_ | This parameter represents the command which is being executed previously. | | $- | This parameter will print the current options flags where the set command can be used to modify the options flags. | For example, a string's length can be expressed by using the `#` command, `${#parameter}`. ```shell [me@linuxbox ~]$ foo="This string is long." [me@linuxbox ~]$ echo "'$foo' is ${#foo} characters long." 'This string is long.' is 20 characters long. ``` ##### Extract portion of parameter If you're looking for a certain portion of a parameter, you can specify its **character count** using `${parameter:offset:length}`. An extraction begins at *offset* characters from the beginning of the string until the end of the string, unless *length* is specified. If the value of *offset* is negative, it's taken at the start of the end instead of the beginning. ```shell [me@linuxbox ~]$ foo="This string is long." [me@linuxbox ~]$ echo ${foo:5} string is long. [me@linuxbox ~]$ echo ${foo:5:6} string ``` ##### Remove leading/ending portion You can remove a portion of string before a *pattern* by calling `${parameter#pattern}` or `${parameter##pattern}`. The difference is that one `#` removes the shortest match while `##` removes the longest. ```shell [me@linuxbox ~]$ foo=file.txt.zip [me@linuxbox ~]$ echo ${foo#*.} txt.zip [me@linuxbox ~]$ echo ${foo##*.} zip ``` The inverse can occur (removing text from the end of the string) by using `%` in place of `#`. ```shell [me@linuxbox ~]$ foo=file.txt.zip [me@linuxbox ~]$ echo ${foo%.*} file.txt [me@linuxbox ~]$ echo ${foo%%.*} file ``` ##### Search and replace As if there wasn't enough substitution, its built into the parameters themselves. The general substitutions works the same as most others, but you can include most all of the text selection mean shown above. ```shell [me@linuxbox ~]$ foo=JPG.JPG [me@linuxbox ~]$ echo ${foo/JPG/jpg} jpg.JPG [me@linuxbox ~]$ echo ${foo//JPG/jpg} jpg.jpg [me@linuxbox ~]$ echo ${foo/#JPG/jpg} jpg.JPG [me@linuxbox ~]$ echo ${foo/%JPG/jpg} JPG.jpg ``` >Expansions can improve the efficiency of the scripts by eliminating the use of external programs. #### Case Conversion There are two methods to achieve case conversion, shifting all/select characters of a string to upper/lower. ##### Declare The `declare` command can be used to normalize strings to either of the cases. To do so, `declare -u variable` or `declare -l variable`. Now a string allocated to one of the declared variables will be converted to that respective casing. ```shell #!/bin/bash # ul-declare: demonstrate case conversion via declare declare -u upper declare -l lower x=aBc if [[ $x ]]; then upper="$x" lower="$x" echo "$upper" # ABC echo "$lower" # abc fi ``` ##### Parameter | Format | Result | | ------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | ${*parameter*,,*pattern*} | Expand the value of *parameter* into all lowercase. *pattern* is an optional shell pattern that will limit which characters (for example, \[A-F\] will be converted.) | | ${*parameter*,*pattern*} | Expand the value of *parameter*, changing only the first character to lowercase | | ${*parameter*^^*pattern*} | Expand the value of *parameter* into all uppercase letters | | ${*parameter*^*pattern*} | Expand the value of *parameter*, changing only the first character to uppercase (capitalization) | Under each of the above parameter methods, *pattern* is purely optional. #### Assignment Operators | Notation | Description | | ---------------------- | -------------------------------------------------------- | | *parameter*=*value* | Simple Assignment. Assigns *value* to *parameter* | | *parameter*\=\=*value* | Check equivalency of between the *value* and *parameter* | | *parameter*+=*value* | Addition; *parameter* = *parameter* + *value* | | *parameter*-=*value* | Subtraction | | *parameter*\*=*value* | Multiplication | | *parameter*/=*value* | Integer division | | *parameter*%=*value* | Modulo; find remainder | | *parameter*++ | Variable post-increment | | *parameter*-- | Variable post-decrement | | ++*parameter* | Variable pre-increment | | --*parameter* | Variable pre-decrement | The increment (++) and decrement (--) are special. The placement of the operators either before or after the parameter in question determine the **when** the parameter is returned. ```shell [me@linuxbox ~]$ foo=1 [me@linuxbox ~]$ echo $((foo++)) 1 [me@linuxbox ~]$ echo $foo 2 ___ [me@linuxbox ~]$ foo=1 [me@linuxbox ~]$ echo $((++foo)) 2 [me@linuxbox ~]$ echo $foo 2 ``` #### Logic | Operator | Description | | ----------------------- | ------------------------ | | <= | Less than or equal to | | >= | Greater than or equal to | | < | less than | | > | greater than | | == | Equal to | | != | Not equal to | | && | Logical AND | | \|\| | Logical OR | | *expr1*?*expr2*:*expr3* | Comparison (ternary) operator. If expression *expr1* evaluates to be nonzero (arithmetic true), then *expr2*; else *expr3* | ![[Quick Javascript Summary#If Else Statements]] ***Note: performing assignment within the expression is not straightforward*** ```shell [me@linuxbox ~]$ a=0 [me@linuxbox ~]$ ((a<1?a+=1:a-=1)) bash: ((: a<1?a+=1:a-=1: attempted assignment to non-variable (error token is "-=1") ___ [me@linuxbox ~]$ ((a<1?(a+=1):(a-=1))) # bracketing arithmatic separate from the logic will allow mathematics to occur on the same line ``` ##### Minecraft Beacon Calculator ```shell #!/bin/zsh sum=0 for ((i=0; i<=9; i=++i)); do var=$((i*i)) sum=$(($var+$sum)) done echo $sum ``` ### Arrays **Scalar Variables**- variables that containa single value **Arrays**- variabless that hold more than one value at a time An array works similar to a spreadsheet: it has cells called *elements* and each of these elements contain data. A single elemenet is accessed by using an address called an *index* or *subscript*. While most programming languages can support *multidimensional arrays*, arrays with two or more dimensions, bash is limited to single dimension. #### Creating an Array ```shell declare -a a # a has been declared an array variable declare -a var a[0]="variable" # the first element of the array is now "variable" var=(value1 value2 value3) # var is defined as an array with specified variables ``` When using the first method shown, the integer in the brackets is the *subscript*, or the location along the array it is. Like JavaScript, an array's first element is subscript zero. #### Array Operations ##### Outputting the Entire Contents of an Array The subscripts `*` and `@` can be used to access every element in an array. ```shell [Jed_Hed@archlinux]$ animals=("a dog" "a cat" "a fish") [Jed_Hed@archlinux]$ for i in ${animals[*]}; do echo $i; done a dog a cat a fish [Jed_Hed@archlinux]$ for i in ${animals[@]}; do echo $i; done a dog a cat a fish [Jed_Hed@archlinux]$ for i in "${animals[*]}"; do echo $1; done a dog a cat a fish [Jed_Hed@archlinux]$ for i in "${animals[@]}"; do echo $1; done a dog a cat a fish ``` ##### Determining the Number of Array Elements ```shell [Jed_Hed@archlinux ~]$ a[100]=foo [Jed_Hed@archlinux ~]$ echo ${#a[@]} # number of array elements 1 [Jed_Hed@archlinux ~]$ echo ${#a[100]} # length of element 100 3 ``` >It's interesting to note that while we assigned the variable to element 100, bash reports only one element in the array. Unlike other languages, arrays elements in bash only exist if they have been assigned a value regardless of their subscript ##### Finding the Subscripts Used by an Array As mentioned above, bash allows for *gaps* to occur when assigning subscripts, so it can be useful to find vacant or already occpied elements. This can be done by using `"${!array[*]}"` (keep in mind that `*` and `@` are interchangable). ```shell [Jed_Hed@archlinux ~]$ foo=([2]=a [4]=b [6]=c) [Jed_Hed@archlinux ~]$ for i in "${foo[@]}"; do echo $i; done a b c [Jed_Hed@archlinux ~]$ for i in "${!foo[@]}"; do echo $1; done 2 4 6 ``` ##### Adding Elements to the End of an Array ```shell [Jed_Hed@archlinux ~]$ foo=(a b c) [Jed_Hed@archlinux ~]$ echo ${foo[@]} a b c [Jed_Hed@archlinux ~]$ foo+=(d e f) [Jed_Hed@archlinux ~]$ echo ${foo[@]} a b c d e f ``` ##### Sorting an Array ```shell a=(f e d c b a) echo "Original array: ${a[@]}" a_sorted=($(for i in "${a[@]}"; do echo $i; done | sort)) echo "Sorted array: ${a[@]}" ___ [Jed_Hed@archlinux ~]$ array-sort Original array: f e d c b a Sorted array: a b c d e f ``` ##### Deleting an Array ```shell # to delete a whole array [Jed_Hed@archlinux ~]$ foo=(a b c d e f) [Jed_Hed@archlinux ~]$ echo ${foo[@]} a b c d e f [Jed_Hed@archlinux ~]$ unset foo [Jed_Hed@archlinux ~]$ echo ${foo[@]} [Jed_Hed@archlinux ~]$ ``` ```shell # to delete a single element [Jed_Hed@archlinux ~]$ foo=(a b c d e f) [Jed_Hed@archlinux ~]$ echo ${foo[@]} a b c d e f [Jed_Hed@archlinux ~]$ unset 'foo[2]' [Jed_Hed@archlinux ~]$ echo ${foo[@]} a b d e f ``` ==Any reference to an array variable without a subscript refers to element zero of the array== ```shell [Jed_Hed@archlinux ~]$ foo=(a b c d e f) [Jed_Hed@archlinux ~]$ echo ${foo[@]} a b c d e f [Jed_Hed@archlinux ~]$ foo=A [Jed_Hed@archlinux ~]$ echo ${foo[@]} A b c d e f ``` #### Associative Arrays *Associative Arrays* use strings rather than integers as array indexes. To call an associative array, use the `delcare -A` comand long with the name of the array to be generated. ```shell declare -A colors colors["red"]="#ff0000" colors["green"]="#00ff00" colors["blue"]="#0000ff" echo ${colors["blue"]} ``` ### Exotica #### Group Commands and Subshells Bash allows commands to be grouped together. A *Group Command* is written as `{ command1; command2; [command3; ...] }` and *Subshells* are written as `(command1; command2; [command3; ...])`. **Note that the the space between the `{` the start of the command is necessary for group commands**. This method proves especially useful for grouping the results of functions and collectively piping them into another function. The key difference between the two methods lie directly in the names: group commands execute all of their commands in the current shell while subshells (as the name suggests) executes their commands in a child copy of the current shell. This means that variables generated and assigned within the subshell environment are lost upon resolution. On top of that, group commands are both faster and require less memory. ==It's for that reason group commands are generally favored.== Subshell environment are default though. That's why piping a function into `read` doesn't work when we try calling it's `$REPLY`. The `read` command is executed in a subshell and its copy of `$REPLY` is destroyed upon resolution. This is where *Process Substitution* steps in. ##### Process Substitution Process substitution can be applied in two ways: `<(list)` or `>(list)`, the first for processes that take standard output; the second, for standard input. To solve the previously mentioned problem with `read`: ```shell read < <(echo "foo") echo $REPLY ``` #### Traps In longer, more complicated scripts, it may be necessary to include a fail switch in case user's computer turns off or they log off prior to resolution. Under these circumstances, most programs include a *Trap*. A trap is set up by calling `trap *argument* *signal* [*signal*...]`. `argument` is a string that will be read and treated as a command. `signal` is the specification of a signal that will trigger the execution of the interpreted command. ```shell # trap-demo: simple signal handing demo exit_on_signal_SIGINT () { echo "Script interrupted." 2>&1 exit 0 } exit_on_signal_SIGTERM () { echo "Script terminated." 2>&1 exit 0 } trap exit_on_signal_SIGINT SIGINT trap exit_on_signal_SIGTERM SIGTERM for i in {1..5}; do echo "Iteration $i of 5" sleep 5 done ``` Under this set up, using the `CTRL+C` will echo the `SIGINT` result and `exit` the program. Without the `exit`, it would print the trap function but it wouldn't end the command. #### Asynchronous Execution with wait Modern operating systems perform multitasking in order to manage and perform several simultaneous actions. *Asynchronous Execution*, created by [[Leslie Lamport - The Man Who Revolutionized Computer Science With Math|Leslie Lamport]], is a methodology for performing several different actions without allocating all of the computer's memory to a particular task. Bash has built in asynchronous functionality through the `wait` command. This command causes the parent script to pause until a specified process (the child script) finishes execution. ## Other Programs of Note ### tr `tr` translates, deletes, and squeezes characters from the standard input and writes the result to the standard output. This can be used to remove repeated characters, replace basic characters, convert upper to lowercase, or vise versa. Typically its used along with piping. ```shell [jed_hed@archlinux ~]$ echo 'linuxize' | tr 'lin' 'red' reduxeze ``` ### bc `bc` is used for command line calculator. It is similar to basic calculator by using which we can do basic mathematical calculations. Along with `bc`, Linux and Unix systems include `expr` for doing arithmetic calculations. ```shell [jed_hed@archlinux ~]$ [-hlwsqv] [long-options] [file ... ] ``` | Command | Long Form | Function | | ------- | ------------- | ---------------------------------------- | | -h | --help | Print the usage and exit | | -i | --interactive | Force interactive mode | | -l | --mathlib | Define the standard math library | | -w | --warn | Give warnings for extensions to POSIX bc | | -s | --standard | Process exactly the POSIX bc language | | -q | --quiet | Do not print the normal GNU bc welcome | | -v | --version | Print the version number and copyright and quit | ### xargs `xargs` takes standard input from a piped function and exports it as if it were a command. ```shell [jed_hed@archlinux ~]$ printf "1\n2\n3\n" 1 2 3 [jed_hed@archlinux ~]$ printf "1\n2\n3\n" | xargs touch [jed_hed@archlinux ~]$ ls 1 2 3 [jed_hed@archlinux ~]$ ls | xargs rm [jed_hed@archlinux ~]$ ls *Nothing here* [jed_hed@archlinux ~]$ printf "1\n2\n3\n" | xargs -i touch {}.txt [jed_hed@archlinux ~]$ ls 1.txt 2.txt 3.txt ``` | Command | Long Form | Function | | ------- | ------------------- | ------------------------------------------------------------------------------------------ | | -d | --delimiter=*delim* | Assign the *delim* as the separator of items assuming you do not want single space or `\n` | | -i | --replace | Replace characters of the output. Refer to the example above | | -a | --arg-file | Read items from a *file* instead of standard input | ## Coding Challenges of Note