Bash <- StackOverflow top 100

1: How to concatenate string variables in Bash (score 3365295 in 2018)

Question

In PHP, strings are concatenated together as follows:

Here, $foo becomes “Hello World”.

How is this accomplished in Bash?

Answer accepted (score 3477)

In general to concatenate two variables you can just write them one after another:

Answer 2 (score 1067)

Bash also supports a += operator as shown in this code:

Answer 3 (score 917)

Bash first

As this question stand specifically for Bash, my first part of the answer would present different ways of doing this properly:

+=: Append to variable

The syntax += may be used in different ways:

Append to string var+=...

(Because I am frugal, I will only use two variables foo and a and then re-use the same in the whole answer. ;-)

Using the Stack Overflow question syntax,

works fine!

Append to an integer ((var+=...))

variable a is a string, but also an integer

Append to an array var+=(...)

Our a is also an array of only one element.

Note that between parentheses, there is a space separated array. If you want to store a string containing spaces in your array, you have to enclose them:

Hmm.. this is not a bug, but a feature… To prevent bash to try to develop !", you could:

printf: Re-construct variable using the builtin command

The printf builtin command gives a powerful way of drawing string format. As this is a Bash builtin, there is a option for sending formatted string to a variable instead of printing on stdout:

There are seven strings in this array. So we could build a formatted string containing exactly seven positional arguments:

Or we could use one argument format string which will be repeated as many argument submitted…

Note that our a is still an array! Only first element is changed!

Under bash, when you access a variable name without specifying index, you always address first element only!

So to retrieve our seven field array, we only need to re-set 1st element:

One argument format string with many argument passed to:

Using the Stack Overflow question syntax:

Nota: The use of double-quotes may be useful for manipulating strings that contain spaces, tabulations and/or newlines

Shell now

Under POSIX shell, you could not use bashisms, so there is no builtin printf.

Basically

But you could simply do:

Formatted, using forked printf

If you want to use more sophisticated constructions you have to use a fork (new child process that make the job and return the result via stdout):

Historically, you could use backticks for retrieving result of a fork:

But this is not easy for nesting:

with backticks, you have to escape inner forks with backslashes:

2: How do I tell if a regular file does not exist in Bash? (score 2390567 in 2017)

Question

I’ve used the following script to see if a file exists:

What’s the correct syntax to use if I only want to check if the file does not exist?

Answer accepted (score 4255)

The test command ([ here) has a “not” logical operator which is the exclamation point (similar to many other languages). Try this:

Answer 2 (score 607)

Bash File Testing

-b filename - Block special file
-c filename - Special character file
-d directoryname - Check for directory Existence
-e filename - Check for file existence, regardless of type (node, directory, socket, etc.)
-f filename - Check for regular file existence not a directory
-G filename - Check if file exists and is owned by effective group ID
-G filename set-group-id - True if file exists and is set-group-id
-k filename - Sticky bit
-L filename - Symbolic link
-O filename - True if file exists and is owned by the effective user id
-r filename - Check if file is a readable
-S filename - Check if file is socket
-s filename - Check if file is nonzero size
-u filename - Check if file set-user-id bit is set
-w filename - Check if file is writable
-x filename - Check if file is executable

How to use:

A test expression can be negated by using the ! operator

Answer 3 (score 280)

You can negate an expression with “!”:

The relevant man page is man test or, equivalently, man [ – or help test or help [ for the built-in bash command.

3: How do I split a string on a delimiter in Bash? (score 1998122 in 2018)

Question

I have this string stored in a variable:

Now I would like to split the strings by ; delimiter so that I have:

I don’t necessarily need the ADDR1 and ADDR2 variables. If they are elements of an array that’s even better.

After suggestions from the answers below, I ended up with the following which is what I was after:

Output:

There was a solution involving setting Internal_field_separator (IFS) to ;. I am not sure what happened with that answer, how do you reset IFS back to default?

RE: IFS solution, I tried this and it works, I keep the old IFS and then restore it:

BTW, when I tried

I only got the first string when printing it in loop, without brackets around $IN it works.

Answer accepted (score 1126)

You can set the internal field separator (IFS) variable, and then let it parse into an array. When this happens in a command, then the assignment to IFS only takes place to that single command’s environment (to read ). It then parses the input according to the IFS variable value into an array, which we can then iterate over.

It will parse one line of items separated by ;, pushing it into an array. Stuff for processing whole of $IN, each time one line of input separated by ;:

Answer 2 (score 886)

Taken from Bash shell script split array:

Explanation:

This construction replaces all occurrences of ';' (the initial // means global replace) in the string IN with ' ' (a single space), then interprets the space-delimited string as an array (that’s what the surrounding parentheses do).

The syntax used inside of the curly braces to replace each ';' character with a ' ' character is called Parameter Expansion.

There are some common gotchas:

  1. If the original string has spaces, you will need to use IFS:

    • IFS=':'; arrIN=($IN); unset IFS;

  2. If the original string has spaces and the delimiter is a new line, you can set IFS with:

    • IFS=$'\n'; arrIN=($IN); unset IFS;

Answer 3 (score 227)

If you don’t mind processing them immediately, I like to do this:

You could use this kind of loop to initialize an array, but there’s probably an easier way to do it. Hope this helps, though.

4: Echo newline in Bash prints literal (score 1869228 in 2016)

Question

In Bash, tried this:

But it doesn’t print a newline, only \n. How can I make it print the newline?

I’m using Ubuntu 11.04.

Answer accepted (score 2592)

You could use printf instead:

printf has more consistent behavior than echo. The behavior of echo varies greatly between different versions.

Answer 2 (score 1562)

Are you sure you are in bash? Works for me, all three ways:

Answer 3 (score 538)

prints

$'' strings use ANSI C Quoting:

Words of the form $'<i>string</i>' are treated specially. The word expands to string, with backslash-escaped characters replaced as specified by the ANSI C standard.

5: How to check if a string contains a substring in Bash (score 1843858 in 2018)

Question

I have a string in Bash:

How can I test if it contains another string?

Where ?? is my unknown operator. Do I use echo and grep?

That looks a bit clumsy.

Answer accepted (score 3248)

You can use Marcus’s answer (* wildcards) outside a case statement, too, if you use double brackets:

Note that spaces in the needle string need to be placed between double quotes, and the * wildcards should be outside.

Answer 2 (score 514)

If you prefer the regex approach:

Answer 3 (score 316)

I am not sure about using an if statement, but you can get a similar effect with a case statement:

6: How do I set a variable to the output of a command in Bash? (score 1657843 in 2019)

Question

I have a pretty simple script that is something like the following:

When I run this script from the command line and pass it the arguments, I am not getting any output. However, when I run the commands contained within the $MOREF variable, I am able to get output.

How can one take the results of a command that needs to be run within a script, save it to a variable, and then output that variable on the screen?

Answer accepted (score 2159)

In addition to backticks &#96;command&#96; you can use $(command) or "$(command)" which I find easier to read, and allow for nesting.

Quoting (") does matter to preserve multi-line values.

Answer 2 (score 259)

The right way is

If you’re going to use an apostrophe, you need ``, not’`. This character is called “backticks” (or “grave accent”).

Like this:

Answer 3 (score 70)

As they have already indicated to you, you should use ‘backticks’.

The alternative proposed $(command) works as well, and it also easier to read, but note that it is valid only with Bash or KornShell (and shells derived from those), so if your scripts have to be really portable on various Unix systems, you should prefer the old backticks notation.

7: How to create a file in Linux from terminal window? (score 1564036 in 2018)

Question

What’s the easiest way to create a file in Linux terminal?

Answer accepted (score 548)

Depending on what you want the file to contain:

  • touch /path/to/file for an empty file

  • somecommand &gt; /path/to/file for a file containing the output of some command.

    sh eg: grep --help > randomtext.txt echo "This is some text" > randomtext.txt

  • nano /path/to/file or vi /path/to/file (or any other editor emacs,gedit etc)
    It either opens the existing one for editing or creates & opens the empty file to enter, if it doesn’t exist

Answer 2 (score 118)

Use touch

Answer 3 (score 85)

Create the file using cat

$ cat &gt; myfile.txt

Now, just type whatever you want in the file:

Hello World!

CTRL-D to save and exit

8: Get the source directory of a Bash script from within the script itself (score 1552214 in 2019)

Question

How do I get the path of the directory in which a Bash script is located, inside that script?

For instance, let’s say I want to use a Bash script as a launcher for another application. I want to change the working directory to the one where the Bash script is located, so I can operate on the files in that directory, like so:

$ ./application

Answer accepted (score 6107)

is a useful one-liner which will give you the full directory name of the script no matter where it is being called from.

It will work as long as the last component of the path used to find the script is not a symlink (directory links are OK). If you also want to resolve any links to the script itself, you need a multi-line solution:

This last one will work with any combination of aliases, source, bash -c, symlinks, etc.

Beware: if you cd to a different directory before running this snippet, the result may be incorrect!

Also, watch out for $CDPATH gotchas, and stderr output side effects if the user has smartly overridden cd to redirect output to stderr instead (including escape sequences, such as when calling update_terminal_cwd &gt;&amp;2 on Mac). Adding &gt;/dev/null 2&gt;&amp;1 at the end of your cd command will take care of both possibilities.

To understand how it works, try running this more verbose form:

And it will print something like:

Answer 2 (score 808)

Use dirname "$0":

using pwd alone will not work if you are not running the script from the directory it is contained in.

Answer 3 (score 455)

The dirname command is the most basic, simply parsing the path up to the filename off of the $0 (script name) variable:

But, as matt b pointed out, the path returned is different depending on how the script is called. pwd doesn’t do the job because that only tells you what the current directory is, not what directory the script resides in. Additionally, if a symbolic link to a script is executed, you’re going to get a (probably relative) path to where the link resides, not the actual script.

Some others have mentioned the readlink command, but at its simplest, you can use:

readlink will resolve the script path to an absolute path from the root of the filesystem. So, any paths containing single or double dots, tildes and/or symbolic links will be resolved to a full path.

Here’s a script demonstrating each of these, whatdir.sh:

Running this script in my home dir, using a relative path:

Again, but using the full path to the script:

Now changing directories:

And finally using a symbolic link to execute the script:

9: Read a file line by line assigning the value to a variable (score 1529494 in 2019)

Question

I have the following .txt file:

I want to read it line-by-line, and for each line I want to assign a .txt line value to a variable. Supposing my variable is $name, the flow is:

  • Read first line from file
  • Assign $name = “Marco”
  • Do some tasks with $name
  • Read second line from file
  • Assign $name = “Paolo”

Answer accepted (score 1240)

The following reads a file passed as an argument line by line:

This is the standard form for reading lines from a file in a loop. Explanation:

  • IFS= (or IFS='') prevents leading/trailing whitespace from being trimmed.
  • -r prevents backslash escapes from being interpreted.

If the above is saved to a script with filename readfile, it can be run as follows:

If the file isn’t a standard POSIX text file (= not terminated by a newline character), the loop can be modified to handle trailing partial lines:

Here, || [[ -n $line ]] prevents the last line from being ignored if it doesn’t end with a \n (since read returns a non-zero exit code when it encounters EOF).

If the commands inside the loop also read from standard input, the file descriptor used by read can be chanced to something else (avoid the standard file descriptors), e.g.:

(Non-Bash shells might not know read -u3; use read &lt;&amp;3 instead.)

Answer 2 (score 299)

I encourage you to use the -r flag for read which stands for:

I am citing from man 1 read.

Another thing is to take a filename as an argument.

Here is updated code:

Answer 3 (score 126)

Using the following Bash template should allow you to read one value at a time from a file and process it.

10: How to permanently set $PATH on Linux/Unix? (score 1467227 in 2018)

Question

I’m trying to add a directory to my path so it will always be in my Linux path. I’ve tried:

This works, however each time I exit the terminal and start a new terminal instance, this path is lost, and I need to run the export command again.

How can I do it so this will be set permanently?

Answer accepted (score 998)

You need to add it to your ~/.profile or ~/.bashrc file. 

Depending on what you’re doing, you also may want to symlink to binaries:

Note that this will not automatically update your path for the remainder of the session. To do this, you should run:

Answer 2 (score 237)

In Ubuntu, edit /etc/environment. Its sole purpose is to store Environment Variables. Originally the $PATH variable is defined here. This is a paste from my /etc/environment file:

So you can just open up this file as root and add whatever you want.

For Immediate results, Run (try as normal user and root):

UPDATE:

If you use zsh (a.k.a Z Shell), add this line right after the comments in /etc/zsh/zshenv:

I encountered this little quirk on Ubuntu 15.10, but if your zsh is not getting the correct PATH, this could be why

Answer 3 (score 152)

There are multiple ways to do it. The actual solution depends on the purpose.

The variable values are usually stored in either a list of assignments or a shell script that is run at the start of the system or user session. In case of the shell script you must use a specific shell syntax.

System wide
  1. /etc/environment List of unique assignments. Perfect for adding system-wide directories like /usr/local/something/bin to PATH variable or defining JAVA_HOME.
  2. /etc/xprofile Shell script executed while starting X Window System session. This is run for every user that logs into X Window System. It is a good choice for PATH entries that are valid for every user like /usr/local/something/bin. The file is included by other script so use POSIX shell syntax not the syntax of your user shell.
  3. /etc/profile and /etc/profile.d/* Shell script. This is a good choice for shell-only systems. Those files are read only by shells.
  4. /etc/&lt;shell&gt;.&lt;shell&gt;rc. Shell script. This is a poor choice because it is single shell specific.
User session
  1. ~/.pam_environment. List of unique assignments. Loaded by PAM at the start of every user session irrelevant if it is an X Window System session or shell. You cannot reference other variable including HOME or PATH so it has limited use.
  2. ~/.xprofile Shell script. This is executed when the user logs into X Window System system. The variables defined here are visible to every X application. Perfect choice for extending PATH with values such as ~/bin or ~/go/bin or defining user specific GOPATH or NPM_HOME. The file is included by other script so use POSIX shell syntax not the syntax of your user shell. Your graphical text editor or IDE started by shortcut will see those values.
  3. ~/.profile Shell script. It will be visible only for programs started from terminal or terminal emulator. It is a good choice for shell-only systems.
  4. ~/.&lt;shell&gt;rc. Shell script. This is a poor choice because it is single shell specific.
Distribution specific documentation

11: Looping through the content of a file in Bash (score 1415142 in 2018)

Question

How do I iterate through each line of a text file with Bash?

With this script:

I get this output on the screen:

(Later I want to do something more complicated with $p than just output to the screen.)

The environment variable SHELL is (from env):

/bin/bash --version output:

cat /proc/version output:

The file peptides.txt contains:

Answer accepted (score 1891)

One way to do it is:

As pointed out in the comments, this has the side effects of trimming leading whitespace, interpretting backslash sequences, and skipping the trailing line if it’s missing a terminating linefeed. If these are concerns, you can do:

Exceptionally, if the loop body may read from standard input, you can open the file using a different file descriptor:

Here, 10 is just an arbitrary number (different from 0, 1, 2).

Answer 2 (score 374)

and the one-liner variant:

Answer 3 (score 137)

Option 1a: While loop: Single line at a time: Input redirection

Option 1b: While loop: Single line at a time:
Open the file, read from a file descriptor (in this case file descriptor #4).

12: Extract filename and extension in Bash (score 1405400 in 2018)

Question

I want to get the filename (without extension) and the extension separately.

The best solution I found so far is:

This is wrong because it doesn’t work if the file name contains multiple . characters. If, let’s say, I have a.b.js, it will consider a and b.js, instead of a.b and js.

It can be easily done in Python with

but I’d prefer not to fire up a Python interpreter just for this, if possible.

Any better ideas?

Answer accepted (score 3271)

First, get file name without the path:

Alternatively, you can focus on the last ‘/’ of the path instead of the ‘.’ which should work even if you have unpredictable file extensions:

You may want to check the documentation :

Answer 2 (score 602)

For more details, see shell parameter expansion in the Bash manual.

Answer 3 (score 362)

Usually you already know the extension, so you might wish to use:

for example:

and we get

13: How do I parse command line arguments in Bash? (score 1334093 in 2019)

Question

Say, I have a script that gets called with this line:

or this one:

What’s the accepted way of parsing this such that in each case (or some combination of the two) $v, $f, and $d will all be set to true and $outFile will be equal to /fizz/someOtherFile ?

Answer accepted (score 2416)

Update: It’s been more than 5 years since I started this answer. Thank you for LOTS of great edits/comments/suggestions. In order save maintenance time, I’ve modified the code block to be 100% copy-paste ready. Please do not post comments like “What if you changed X to Y…”. Instead, copy-paste the code block, see the output, make the change, rerun the script, and comment “I changed X to Y and…” I don’t have time to test your ideas and tell you if they work.

Method #1: Using bash without getopt[s]

Two common ways to pass key-value-pair arguments are:

Bash Space-Separated (e.g., --option argument) (without getopt[s])

Usage demo-space-separated.sh -e conf -s /etc -l /usr/lib /etc/hosts

output from copy-pasting the block above:

Bash Equals-Separated (e.g., --option=argument) (without getopt[s])

Usage demo-equals-separated.sh -e=conf -s=/etc -l=/usr/lib /etc/hosts

output from copy-pasting the block above:

To better understand ${i#*=} search for “Substring Removal” in this guide. It is functionally equivalent to sed 's/[^=]*=//' &lt;&lt;&lt; "$i" which calls a needless subprocess or echo "$i" | sed 's/[^=]*=//' which calls two needless subprocesses.

Method #2: Using bash with getopt[s]

from: http://mywiki.wooledge.org/BashFAQ/035#getopts

getopt(1) limitations (older, relatively-recent getopt versions):

  • can’t handle arguments that are empty strings
  • can’t handle arguments with embedded whitespace

More recent getopt versions don’t have these limitations.

Additionally, the POSIX shell (and others) offer getopts which doesn’t have these limitations. I’ve included a simplistic getopts example.

Usage demo-getopts.sh -vf /etc/hosts foo bar

output from copy-pasting the block above:

The advantages of getopts are:

  1. It’s more portable, and will work in other shells like dash.
  2. It can handle multiple single options like -vf filename in the typical Unix way, automatically.

The disadvantage of getopts is that it can only handle short options (-h, not --help) without additional code.

There is a getopts tutorial which explains what all of the syntax and variables mean. In bash, there is also help getopts, which might be informative.

Answer 2 (score 490)

No answer mentions enhanced getopt. And the top-voted answer is misleading: It either ignores -⁠vfd style short options (requested by the OP) or options after positional arguments (also requested by the OP); and it ignores parsing-errors. Instead:

  • Use enhanced getopt from util-linux or formerly GNU glibc.1
  • It works with getopt_long() the C function of GNU glibc.

  • Has all useful distinguishing features (the others don’t have them):

    • handles spaces, quoting characters and even binary in arguments2 (non-enhanced getopt can’t do this)
    • it can handle options at the end: script.sh -o outFile file1 file2 -v (getopts doesn’t do this)
    • allows =-style long options: script.sh --outfile=fileOut --infile fileIn (allowing both is lengthy if self parsing)
    • allows combined short options, e.g. -vfd (real work if self parsing)
    • allows touching option-arguments, e.g. -oOutfile or -vfdoOutfile
  • Is so old already3 that no GNU system is missing this (e.g. any Linux has it).
  • You can test for its existence with: getopt --test → return value 4.
  • Other getopt or shell-builtin getopts are of limited use.

The following calls

all return

with the following myscript

1 enhanced getopt is available on most “bash-systems”, including Cygwin; on OS X try brew install gnu-getopt or sudo port install getopt
2 the POSIX exec() conventions have no reliable way to pass binary NULL in command line arguments; those bytes prematurely end the argument
3 first version released in 1997 or before (I only tracked it back to 1997)

Answer 3 (score 123)

from : digitalpeer.com with minor modifications

Usage myscript.sh -p=my_prefix -s=dirname -l=libname

To better understand ${i#*=} search for “Substring Removal” in this guide. It is functionally equivalent to sed 's/[^=]*=//' &lt;&lt;&lt; "$i" which calls a needless subprocess or echo "$i" | sed 's/[^=]*=//' which calls two needless subprocesses.

14: YYYY-MM-DD format date in shell script (score 1310931 in 2018)

Question

I tried using $(date) in my bash shell script, however I want the date in YYYY-MM-DD format. How do I get this?

Answer accepted (score 1725)

In bash (>=4.2) it is preferable to use printf’s built-in date formatter (part of bash) rather than the external date (usually GNU date).

As such:

In bash (<4.2):

Other available date formats can be viewed from the date man pages (for external non-bash specific command):

Answer 2 (score 313)

Try: $(date +%F)

Answer 3 (score 80)

You can do something like this:

15: How to count lines in a document? (score 1222035 in 2015)

Question

I have lines like these, and I want to know how many lines I actually have…

Is there a way to count them all using linux commands?

Answer accepted (score 1840)

Use wc:

This will output the number of lines in &lt;filename&gt;:

Or, to omit the &lt;filename&gt; from the result use wc -l &lt; &lt;filename&gt;:

You can also pipe data to wc as well:

Answer 2 (score 133)

To count all lines use:

To filter and count only lines with pattern use:

Or use -v to invert match:

See the grep man page to take a look at the -e,-i and -x args…

Answer 3 (score 70)

Or

16: How to trim whitespace from a Bash variable? (score 1215318 in 2018)

Question

I have a shell script with this code:

But the conditional code always executes, because hg st always prints at least one newline character.

  • Is there a simple way to strip whitespace from $var (like trim() in PHP)?

or

  • Is there a standard way of dealing with this issue?

I could use sed or AWK, but I’d like to think there is a more elegant solution to this problem.

Answer 2 (score 961)

Let’s define a variable containing leading, trailing, and intermediate whitespace:

How to remove all whitespace (denoted by [:space:] in tr):

How to remove leading whitespace only:

How to remove trailing whitespace only:

How to remove both leading and trailing spaces–chain the seds:

Alternatively, if your bash supports it, you can replace echo -e "${FOO}" | sed ... with sed ... &lt;&lt;&lt;${FOO}, like so (for trailing whitespace):

Answer 3 (score 858)

A simple answer is:

Xargs will do the trimming for you. It’s one command/program, no parameters, returns the trimmed string, easy as that!

Note: this doesn’t remove the internal spaces so "foo bar" stays the same. It does NOT become "foobar".

17: How to check if a variable is set in Bash? (score 1161315 in 2016)

Question

How do I know if a variable is set in Bash?

For example, how do I check if the user gave the first parameter to a function?

Answer accepted (score 2067)

(Usually) The right way

where ${var+x} is a parameter expansion which evaluates to nothing if var is unset, and substitutes the string x otherwise.

Quotes Digression

Quotes can be omitted (so we can say ${var+x} instead of "${var+x}") because this syntax & usage guarantees this will only expand to something that does not require quotes (since it either expands to x (which contains no word breaks so it needs no quotes), or to nothing (which results in [ -z ], which conveniently evaluates to the same value (true) that [ -z "" ] does as well)).

However, while quotes can be safely omitted, and it was not immediately obvious to all (it wasn’t even apparent to the first author of this quotes explanation who is also a major Bash coder), it would sometimes be better to write the solution with quotes as [ -z "${var+x}" ], at the very small possible cost of an O(1) speed penalty. The first author also added this as a comment next to the code using this solution giving the URL to this answer, which now also includes the explanation for why the quotes can be safely omitted.

(Often) The wrong way

This is often wrong because it doesn’t distinguish between a variable that is unset and a variable that is set to the empty string. That is to say, if var='', then the above solution will output “var is blank”.

The distinction between unset and “set to the empty string” is essential in situations where the user has to specify an extension, or additional list of properties, and that not specifying them defaults to a non-empty value, whereas specifying the empty string should make the script use an empty extension or list of additional properties.

The distinction may not be essential in every scenario though. In those cases [ -z "$var" ] will be just fine.

Answer 2 (score 816)

To check for non-null/non-zero string variable, i.e. if set, use

It’s the opposite of -z. I find myself using -n more than -z.

You would use it like:

Answer 3 (score 440)

Here’s how to test whether a parameter is unset, or empty (“Null”) or set with a value:

Source: POSIX: Parameter Expansion:

In all cases shown with “substitute”, the expression is replaced with the value shown. In all cases shown with “assign”, parameter is assigned that value, which also replaces the expression.

18: How to compare strings in Bash (score 1126175 in 2018)

Question

How do I compare a variable to a string (and do something if they match)?

Answer accepted (score 1265)

Using variables in if statements

If you want to do something when they don’t match, replace = with !=. You can read more about string operations and arithmetic operations in their respective documentation.

Why do we use quotes around $x?

You want the quotes around $x, because if it is empty, your bash script encounters a syntax error as seen below:

Non-standard use of == operator

Note that bash allows == to be used for equality with [, but this is not standard.

Use either the first case wherein the quotes around $x are optional:

or use the second case:

Answer 2 (score 132)

Or, if you don’t need else clause:

Answer 3 (score 70)

Notes:

  1. Spaces between if and [ and ] are important
  2. &gt; and &lt; are redirection operators so escape it with \&gt; and \&lt; respectively for strings.

19: Extract substring in Bash (score 1122542 in 2017)

Question

Given a filename in the form someletters_12345_moreleters.ext, I want to extract the 5 digits and put them into a variable.

So to emphasize the point, I have a filename with x number of characters then a five digit sequence surrounded by a single underscore on either side then another set of x number of characters. I want to take the 5 digit number and put that into a variable.

I am very interested in the number of different ways that this can be accomplished.

Answer accepted (score 638)

Use cut:

More generic:

Answer 2 (score 1012)

If x is constant, the following parameter expansion performs substring extraction:

where 12 is the offset (zero-based) and 5 is the length

If the underscores around the digits are the only ones in the input, you can strip off the prefix and suffix (respectively) in two steps:

If there are other underscores, it’s probably feasible anyway, albeit more tricky. If anyone knows how to perform both expansions in a single expression, I’d like to know too.

Both solutions presented are pure bash, with no process spawning involved, hence very fast.

Answer 3 (score 91)

Generic solution where the number can be anywhere in the filename, using the first of such sequences:

Another solution to extract exactly a part of a variable:

If your filename always have the format stuff_digits_... you can use awk:

Yet another solution to remove everything except digits, use

20: Check existence of input argument in a Bash shell script (score 1105101 in 2018)

Question

I need to check the existence of an input argument. I have the following script

I get

How do I check the input argument1 first to see if it exists?

Answer accepted (score 2060)

It is:

The $# variable will tell you the number of input arguments the script was passed.

Or you can check if an argument is an empty string or not like:

The -z switch will test if the expansion of “$1” is a null string or not. If it is a null string then the body is executed.

Answer 2 (score 305)

It is better to demonstrate this way

You normally need to exit if you have too few arguments.

Answer 3 (score 92)

In some cases you need to check whether the user passed an argument to the script and if not, fall back to a default value. Like in the script below:

Here if the user hasn’t passed scale as a 2nd parameter, I launch Android emulator with -scale 1 by default. ${varname:-word} is an expansion operator. There are other expansion operators as well:

  • ${varname:=word} which sets the undefined varname instead of returning the word value;
  • ${varname:?message} which either returns varname if it’s defined and is not null or prints the message and aborts the script (like the first example);
  • ${varname:+word} which returns word only if varname is defined and is not null; returns null otherwise.

21: Loop through an array of strings in Bash? (score 1090850 in 2019)

Question

I want to write a script that loops through 15 strings (array possibly?) Is that possible?

Something like:

Answer accepted (score 2144)

You can use it like this:

Also works for multi-line array declaration

Answer 2 (score 709)

That is possible, of course.

See Bash Loops for, while and until for details.

Answer 3 (score 179)

None of those answers include a counter…

Output:

22: In the shell, what does " 2>&1 " mean? (score 1016179 in 2018)

Question

In a Unix shell, if I want to combine stderr and stdout into the stdout stream for further manipulation, I can append the following on the end of my command:

So, if I want to use head on the output from g++, I can do something like this:

so I can see only the first few errors.

I always have trouble remembering this, and I constantly have to go look it up, and it is mainly because I don’t fully understand the syntax of this particular trick.

Can someone break this up and explain character by character what 2&gt;&amp;1 means?

Answer accepted (score 2329)

File descriptor 1 is the standard output (stdout).
File descriptor 2 is the standard error (stderr).

Here is one way to remember this construct (although it is not entirely accurate): at first, 2&gt;1 may look like a good way to redirect stderr to stdout. However, it will actually be interpreted as “redirect stderr to a file named 1”. &amp; indicates that what follows is a file descriptor and not a filename. So the construct becomes: 2&gt;&amp;1.

Answer 2 (score 577)

redirects stdout to afile.txt. This is the same as doing

To redirect stderr, you do:

&gt;&amp; is the syntax to redirect a stream to another file descriptor - 0 is stdin, 1 is stdout, and 2 is stderr.

You can redirect stdout to stderr by doing:

Or vice versa:

So, in short… 2&gt; redirects stderr to an (unspecified) file, appending &amp;1 redirects stderr to stdout.

Answer 3 (score 304)

Some tricks about redirection

Some syntax particularity about this may have important behaviours. There is some little samples about redirections, STDERR, STDOUT, and arguments ordering.

1 - Overwriting or appending?

Symbol &gt; mean redirection.

  • &gt; mean send to as a whole completed file, overwriting target if exist (see noclobber bash feature at #3 later).
  • &gt;&gt; mean send in addition to would append to target if exist.

In any case, the file would be created if they not exist.

2 - The shell command line is order dependent!!

For testing this, we need a simple command which will send something on both outputs:

(Expecting you don’t have a directory named /tnt, of course ;). Well, we have it!!

So, let’s see:

The last command line dumps STDERR to the console, and it seem not to be the expected behaviour… But…

If you want to make some post filtering about one output, the other or both:

Notice that the last command line in this paragraph is exactly same as in previous paragraph, where I wrote seem not to be the expected behaviour (so, this could even be an expected behaviour).

Well, there is a little tricks about redirections, for doing different operation on both outputs:

Nota: &amp;9 descriptor would occur spontaneously because of ) 9&gt;&amp;2.

Addendum: nota! With the new version of (&gt;4.0) there is a new feature and more sexy syntax for doing this kind of things:

And finally for such a cascading output formatting:

Addendum: nota! Same new syntax, in both ways:

Where STDOUT go through a specific filter, STDERR to another and finally both outputs merged go through a third command filter.

3 - A word about noclobber option and &gt;| syntax

That’s about overwriting:

While set -o noclobber instruct bash to not overwrite any existing file, the &gt;| syntax let you pass through this limitation:

The file is overwritten each time, well now:

Pass through with &gt;|:

Unsetting this option and/or inquiring if already set.

4 - Last trick and more…

For redirecting both output from a given command, we see that a right syntax could be:

for this special case, there is a shortcut syntax: &amp;&gt; … or &gt;&amp;

Nota: if 2&gt;&amp;1 exist, 1&gt;&amp;2 is a correct syntax too:

4b- Now, I will let you think about:
4c- If you’re interested in more information

You could read the fine manual by hitting:

in a console ;-)

23: Redirect all output to file (score 1007650 in 2015)

Question

I know that in Linux, to redirect output from the screen to a file, I can either use the &gt; or tee. However, I’m not sure why part of the output is still output to the screen and not written to the file.

Is there a way to redirect all output to file?

Answer accepted (score 1122)

That part is written to stderr, use 2&gt; to redirect it. For example:

or if you want in same file:

Note: this works in (ba)sh, check your shell for proper syntax

Answer 2 (score 141)

All POSIX operating systems have 3 streams: stdin, stdout, and stderr. stdin is the input, which can accept the stdout or stderr. stdout is the primary output, which is redirected with &gt;, &gt;&gt;, or |. stderr is the error output, which is handled separately so that any exceptions do not get passed to a command or written to a file that it might break; normally, this is sent to a log of some kind, or dumped directly, even when the stdout is redirected. To redirect both to the same place, use:

<em>command</em> &amp;> /some/file

EDIT: thanks to Zack for pointing out that the above solution is not portable–use instead:

If you want to silence the error, do:

Answer 3 (score 84)

To get the output on the console AND in a file file.txt for example.

Note: &amp; (in 2&gt;&amp;1) specifies that 1 is not a file name but a file descriptor.

24: How do I compare two string variables in an ‘if’ statement in Bash? (score 1001718 in 2018)

Question

I’m trying to get an if statement to work in Bash (using Ubuntu):

I’ve tried various forms of the if statement, using [["$s1" == "$s2"]], with and without quotes, using =, == and -eq, but I still get the following error:

[hi: command not found

I’ve looked at various sites and tutorials and copied those, but it doesn’t work - what am I doing wrong?

Eventually, I want to say if $s1 contains $s2, so how can I do that?

I did just work out the spaces bit.. :/ How do I say contains?

I tried

but it didn’t work.

Answer accepted (score 789)

For string comparison, use:

For the a contains b, use:

(and make sure to add spaces between the symbols):

bad:

good:

Answer 2 (score 156)

You need spaces:

Answer 3 (score 139)

You should be careful to leave a space between the sign of ‘[’ and double quotes where the variable contains this:

The ^s show the blank spaces you need to leave.

25: How do I iterate over a range of numbers defined by variables in Bash? (score 997426 in 2018)

Question

How do I iterate over a range of numbers in Bash when the range is given by a variable?

I know I can do this (called “sequence expression” in the Bash documentation):

Which gives:

1
2
3
4
5

Yet, how can I replace either of the range endpoints with a variable? This doesn’t work:

Which prints:

{1..5}

Answer accepted (score 1536)

sh for i in $(seq 1 $END); do echo $i; done

edit: I prefer seq over the other methods because I can actually remember it ;)

Answer 2 (score 407)

The seq method is the simplest, but Bash has built-in arithmetic evaluation.

The for ((expr1;expr2;expr3)); construct works just like for (expr1;expr2;expr3) in C and similar languages, and like other ((expr)) cases, Bash treats them as arithmetic.

Answer 3 (score 178)

discussion

Using seq is fine, as Jiaaro suggested. Pax Diablo suggested a Bash loop to avoid calling a subprocess, with the additional advantage of being more memory friendly if $END is too large. Zathrus spotted a typical bug in the loop implementation, and also hinted that since i is a text variable, continuous conversions to-and-fro numbers are performed with an associated slow-down.

integer arithmetic

This is an improved version of the Bash loop:

If the only thing that we want is the echo, then we could write echo $((i++)).

ephemient taught me something: Bash allows for ((expr;expr;expr)) constructs. Since I’ve never read the whole man page for Bash (like I’ve done with the Korn shell (ksh) man page, and that was a long time ago), I missed that.

So,

seems to be the most memory-efficient way (it won’t be necessary to allocate memory to consume seq’s output, which could be a problem if END is very large), although probably not the “fastest”.

the initial question

eschercycle noted that the {a..b} Bash notation works only with literals; true, accordingly to the Bash manual. One can overcome this obstacle with a single (internal) fork() without an exec() (as is the case with calling seq, which being another image requires a fork+exec):

Both eval and echo are Bash builtins, but a fork() is required for the command substitution (the $(…) construct).

26: Passing parameters to a Bash function (score 960488 in 2017)

Question

I am trying to search how to pass parameters in a Bash function, but what comes up is always how to pass parameter from the command line.

I would like to pass parameters within my script. I tried:

But the syntax is not correct, how to pass a parameter to my function?

Answer accepted (score 1473)

There are two typical ways of declaring a function. I prefer the second approach.

or

To call a function with arguments:

The function refers to passed arguments by their position (not by name), that is $1, $2, and so forth. $0 is the name of the script itself.

Example:

Also, you need to call your function after it is declared.

Output:

Reference: Advanced Bash-Scripting Guide.

Answer 2 (score 58)

Knowledge of high level programming languages (C/C++/Java/PHP/Python/Perl …) would suggest to the layman that bash functions should work like they do in those other languages. Instead, bash functions work like shell commands and expect arguments to be passed to them in the same way one might pass an option to a shell command (ls -l). In effect, function arguments in bash are treated as positional parameters ($1, $2..$9, ${10}, ${11}, and so on). This is no surprise considering how getopts works. Parentheses are not required to call a function in bash.

(Note: I happen to be working on Open Solaris at the moment.)

Answer 3 (score 42)

If you prefer named parameters, it’s possible (with a few tricks) to actually pass named parameters to functions (also makes it possible to pass arrays and references).

The method I developed allows you to define named parameters passed to a function like this:

You can also annotate arguments as @required or @readonly, create …rest arguments, create arrays from sequential arguments (using e.g. string[4]) and optionally list the arguments in multiple lines:

In other words, not only you can call your parameters by their names (which makes up for a more readable core), you can actually pass arrays (and references to variables - this feature works only in bash 4.3 though)! Plus, the mapped variables are all in the local scope, just as $1 (and others).

The code that makes this work is pretty light and works both in bash 3 and bash 4 (these are the only versions I’ve tested it with). If you’re interested in more tricks like this that make developing with bash much nicer and easier, you can take a look at my Bash Infinity Framework, the code below is available as one of its functionalities.

shopt -s expand_aliases

function assignTrap {
  local evalString
  local -i paramIndex=${__paramIndex-0}
  local initialCommand="${1-}"

  if [[ "$initialCommand" != ":" ]]
  then
    echo "trap - DEBUG; eval \"${__previousTrap}\"; unset __previousTrap; unset __paramIndex;"
    return
  fi

  while [[ "${1-}" == "," || "${1-}" == "${initialCommand}" ]] || [[ "${#@}" -gt 0 && "$paramIndex" -eq 0 ]]
  do
    shift # first colon ":" or next parameter's comma ","
    paramIndex+=1
    local -a decorators=()
    while [[ "${1-}" == "@"* ]]
    do
      decorators+=( "$1" )
      shift
    done

    local declaration=
    local wrapLeft='"'
    local wrapRight='"'
    local nextType="$1"
    local length=1

    case ${nextType} in
      string | boolean) declaration="local " ;;
      integer) declaration="local -i" ;;
      reference) declaration="local -n" ;;
      arrayDeclaration) declaration="local -a"; wrapLeft= ; wrapRight= ;;
      assocDeclaration) declaration="local -A"; wrapLeft= ; wrapRight= ;;
      "string["*"]") declaration="local -a"; length="${nextType//[a-z\[\]]}" ;;
      "integer["*"]") declaration="local -ai"; length="${nextType//[a-z\[\]]}" ;;
    esac

    if [[ "${declaration}" != "" ]]
    then
      shift
      local nextName="$1"

      for decorator in "${decorators[@]}"
      do
        case ${decorator} in
          @readonly) declaration+="r" ;;
          @required) evalString+="[[ ! -z \$${paramIndex} ]] || echo \"Parameter '$nextName' ($nextType) is marked as required by '${FUNCNAME[1]}' function.\"; " >&2 ;;
          @global) declaration+="g" ;;
        esac
      done

      local paramRange="$paramIndex"

      if [[ -z "$length" ]]
      then
        # ...rest
        paramRange="{@:$paramIndex}"
        # trim leading ...
        nextName="${nextName//\./}"
        if [[ "${#@}" -gt 1 ]]
        then
          echo "Unexpected arguments after a rest array ($nextName) in '${FUNCNAME[1]}' function." >&2
        fi
      elif [[ "$length" -gt 1 ]]
      then
        paramRange="{@:$paramIndex:$length}"
        paramIndex+=$((length - 1))
      fi

      evalString+="${declaration} ${nextName}=${wrapLeft}\$${paramRange}${wrapRight}; "

      # continue to the next param:
      shift
    fi
  done
  echo "${evalString} local -i __paramIndex=${paramIndex};"
}

alias args='local __previousTrap=$(trap -p DEBUG); trap "eval \"\$(assignTrap \$BASH_COMMAND)\";" DEBUG;'

27: How to specify the private SSH-key to use when executing shell command on Git? (score 927267 in 2018)

Question

A rather unusual situation perhaps, but I want to specify a private SSH-key to use when executing a shell (git) command from the local computer.

Basically like this:

Or even better (in Ruby):

I have seen examples of connecting to a remote server with Net::SSH that uses a specified private key, but this is a local command. Is it possible?

Answer accepted (score 690)

Something like this should work (suggested by orip):

if you prefer subshells, you could try the following (though it is more fragile):

Git will invoke SSH which will find its agent by environment variable; this will, in turn, have the key loaded.

Alternatively, setting HOME may also do the trick, provided you are willing to setup a directory that contains only a .ssh directory as HOME; this may either contain an identity.pub, or a config file setting IdentityFile.

Answer 2 (score 1075)

None of these solutions worked for me.

Instead, I elaborate on @Martin v. Löwis ’s mention of setting a config file for SSH.

SSH will look for the user’s ~/.ssh/config file. I have mine setup as:

And I add a remote git repository:

And then git commands work normally for me.

NOTES

  • The IdentitiesOnly yes is required to prevent the SSH default behavior of sending the identity file matching the default filename for each protocol. If you have a file named ~/.ssh/id_rsa that will get tried BEFORE your ~/.ssh/id_rsa.github without this option.

References

Answer 3 (score 390)

Other people’s suggestions about ~/.ssh/config are extra complicated. It can be as simple as:

28: How do I prompt for Yes/No/Cancel input in a Linux shell script? (score 926403 in 2016)

Question

I want to pause input in a shell script, and prompt the user for choices. The standard ‘Yes, No, or Cancel’ type question. How do I accomplish this in a typical bash prompt?

Answer accepted (score 1490)

The simplest and most widely available method to get user input at a shell prompt is the read command. The best way to illustrate its use is a simple demonstration:

Another method, pointed out by Steven Huwig, is Bash’s select command. Here is the same example using select:

With select you don’t need to sanitize the input – it displays the available choices, and you type a number corresponding to your choice. It also loops automatically, so there’s no need for a while true loop to retry if they give invalid input.

Also, Léa Gris demonstrated a way to make the request language agnostic in her answer. Adapting my first example to better serve multiple languages might look like this:

Obviously other communication strings remain untranslated here (Install, Answer) which would need to be addressed in a more fully completed translation, but even a partial translation would be helpful in many cases.

Finally, please check out the excellent answer by F. Hauri.

Answer 2 (score 483)

At least five answers for one generic question.

Depending on

  • compliant: could work on poor systems with generic environments
  • specific: using so called bashisms

and if you want

  • simple ``in line’’ question / answer (generic solutions)
  • pretty formatted interfaces, like or more graphical using libgtk or libqt…
  • use powerful readline history capability
  1. POSIX generic solutions

You could use the read command, followed by if ... then ... else:

(Thanks to Adam Katz’s comment: Replaced the test above with one that is more portable and avoids one fork:)

POSIX, but single key feature

But if you don’t want the user to have to hit Return, you could write:

(Edited: As @JonathanLeffler rightly suggest, saving stty’s configuration could be better than simply force them to sane.)

Note: This was tested under , , , and !

Same, but waiting explicitly for y or n:

Using dedicated tools

There are many tools which were built using libncurses, libgtk, libqt or other graphical libraries. For example, using whiptail:

Depending on your system, you may need to replace whiptail with another similiar tool:

where 20 is height of dialog box in number of lines and 60 is width of the dialog box. These tools all have near same syntax.

  1. Bash specific solutions
Basic in line method

I prefer to use case so I could even test for yes | ja | si | oui if needed…

in line with single key feature

Under bash, we can specify the length of intended input for for the read command:

Under bash, read command accepts a timeout parameter, which could be useful.

Some tricks for dedicated tools

More sophisticated dialog boxes, beyond simple yes - no purposes:

Progress bar:

Little demo:

More sample? Have a look at Using whiptail for choosing USB device and USB removable storage selector: USBKeyChooser

  1. Using readline’s history

Example:

This will create a file .myscript.history in your $HOME directory, than you could use readline’s history commands, like Up, Down, Ctrl+r and others.

Answer 3 (score 346)

29: How to do a logical OR operation in shell scripting (score 896876 in 2019)

Question

I am trying to do a simple condition check, but it doesn’t seem to work.

If $# is equal to 0 or is greater than 1 then say hello.

I have tried the following syntax with no success:

Answer accepted (score 959)

This should work:

I’m not sure if this is different in other shells but if you wish to use <, >, you need to put them inside double parenthesis like so:

Answer 2 (score 55)

This code works for me:

I don’t think sh supports “==”. Use “=” to compare strings and -eq to compare ints.

for more details.

Answer 3 (score 33)

If you are using the bash exit code status $? as variable, it’s better to do this:

Because if you do:

The left part of the OR alters the ? < /strong > variable, sothe < strong > rightpartoftheOR < /strong > doesn′thavetheoriginal < strong>? value.

30: How to change the output color of echo in Linux (score 896232 in 2017)

Question

I am trying to print a text in the terminal using echo command.

I want to print the text in a red color. How can I do that?

Answer 2 (score 2055)

You can use these ANSI escape codes:

And then use them like this in your script:

which prints love in red.

From @james-lim’s comment, if you are using the echo command, be sure to use the -e flag to allow backslash escapes.

(don’t add "\n" when using echo unless you want to add additional empty line)

Answer 3 (score 868)

You can use the awesome tput command (suggested in Ignacio’s answer) to produce terminal control codes for all kinds of things.

Usage

Specific tput sub-commands are discussed later.

Direct

Call tput as part of a sequence of commands:

Use ; instead of &amp;&amp; so if tput errors the text still shows.

Shell variables

Another option is to use shell variables:

tput produces character sequences that are interpreted by the terminal as having a special meaning. They will not be shown themselves. Note that they can still be saved into files or processed as input by programs other than the terminal.

Command substitution

It may be more convenient to insert tput’s output directly into your echo strings using command substitution:

Example

The above command produces this on Ubuntu:

Screenshot of colour terminal text

Foreground & background colour commands

Colours are as follows:

There are also non-ANSI versions of the colour setting functions (setb instead of setab, and setf instead of setaf) which use different numbers, not given here.

Text mode commands
Cursor movement commands
Clear and insert commands
Other commands

With compiz wobbly windows, the bel command makes the terminal wobble for a second to draw the user’s attention.

Scripts

tput accepts scripts containing one command per line, which are executed in order before tput exits.

Avoid temporary files by echoing a multiline string and piping it:

See also
  • See man 1 tput
  • See man 5 terminfo for the complete list of commands and more details on these options. (The corresponding tput command is listed in the Cap-name column of the huge table that starts at line 81.)

31: How to call shell script from another shell script? (score 876027 in 2016)

Question

I have two shell scripts, a.sh and b.sh.

How can I call b.sh from within the shell script a.sh?

Answer 2 (score 856)

There are a couple of different ways you can do this:

  1. Make the other script executable, add the #!/bin/bash line at the top, and the path where the file is to the $PATH environment variable. Then you can call it as a normal command;

  2. Or call it with the source command (alias is .) like this: source /path/to/script;

  3. Or use the bash command to execute it: /bin/bash /path/to/script;

The first and third methods execute the script as another process, so variables and functions in the other script will not be accessible.
The second method executes the script in the first script’s process, and pulls in variables and functions from the other script so they are usable from the calling script.

In the second method, if you are using exit in second script, it will exit the first script as well. Which will not happen in first and third methods.

Answer 3 (score 195)

Check this out.

32: How do I reload .bashrc without logging out and back in? (score 855839 in 2018)

Question

If I make changes to .bashrc, how do I reload it without logging out and back in?

Answer accepted (score 2385)

You just have to enter the command:

or you can use the shorter version of the command:

Answer 2 (score 246)

or you could use;

does the same thing. (and easier to remember, at least for me)

exec command replaces the shell with given program, in our example, it replaces our shell with bash (with the updated configuration files)

Answer 3 (score 111)

To complement and contrast the two most popular answers, . ~/.bashrc and exec bash:

Both solutions effectively reload ~/.bashrc, but there are differences:

  • . ~/.bashrc or source ~/.bashrc will preserve your current shell:

    • Except for the modifications that reloading ~/.bashrc into the current shell (sourcing) makes, the current shell and its state are preserved, which includes environment variables, shell variables, shell options, shell functions, and command history.

  • exec bash, or, more robustly, exec "$BASH"[1], will replace your current shell with a new instance, and therefore only preserve your current shell’s environment variables (including ones you’ve defined ad-hoc).

    • In other words: Any ad-hoc changes to the current shell in terms of shell variables, shell functions, shell options, command history are lost.

Depending on your needs, one or the other approach may be preferred.

[1] exec bash could in theory execute a different bash executable than the one that started the current shell, if it happens to exist in a directory listed earlier in the $PATH. Since special variable $BASH always contains the full path of the executable that started the current shell, exec "$BASH" is guaranteed to use the same executable.
A note re "..." around $BASH: double-quoting ensures that the variable value is used as-is, without interpretation by Bash; if the value has no embedded spaces or other shell metacharacters (which is not likely in this case), you don’t strictly need double quotes, but using them is a good habit to form.

33: Parsing JSON with Unix tools (score 847637 in 2018)

Question

I’m trying to parse JSON returned from a curl request, like so:

The above splits the JSON into fields, for example:

How do I print a specific field (denoted by the -v k=text)?

Answer accepted (score 943)

There are a number of tools specifically designed for the purpose of manipulating JSON from the command line, and will be a lot easier and more reliable than doing it with Awk, such as jq:

You can also do this with tools that are likely already installed on your system, like Python using the json module, and so avoid any extra dependencies, while still having the benefit of a proper JSON parser. The following assume you want to use UTF-8, which the original JSON should be encoded in and is what most modern terminals use as well:

Python 2:

Python 3:

Historical notes

This answer originally recommended jsawk, which should still work, but is a little more cumbersome to use than jq, and depends on a standalone JavaScript interpreter being installed which is less common than a Python interpreter, so the above answers are probably preferable:

This answer also originally used the Twitter API from the question, but that API no longer works, making it hard to copy the examples to test out, and the new Twitter API requires API keys, so I’ve switched to using the GitHub API which can be used easily without API keys. The first answer for the original question would be:

Answer 2 (score 266)

To quickly extract the values for a particular key, I personally like to use “grep -o”, which only returns the regex’s match. For example, to get the “text” field from tweets, something like:

This regex is more robust than you might think; for example, it deals fine with strings having embedded commas and escaped quotes inside them. I think with a little more work you could make one that is actually guaranteed to extract the value, if it’s atomic. (If it has nesting, then a regex can’t do it of course.)

And to further clean (albeit keeping the string’s original escaping) you can use something like: | perl -pe 's/"text"://; s/^"//; s/",$//'. (I did this for this analysis.)

To all the haters who insist you should use a real JSON parser – yes, that is essential for correctness, but

  1. To do a really quick analysis, like counting values to check on data cleaning bugs or get a general feel for the data, banging out something on the command line is faster. Opening an editor to write a script is distracting.
  2. grep -o is orders of magnitude faster than the Python standard json library, at least when doing this for tweets (which are ~2 KB each). I’m not sure if this is just because json is slow (I should compare to yajl sometime); but in principle, a regex should be faster since it’s finite state and much more optimizable, instead of a parser that has to support recursion, and in this case, spends lots of CPU building trees for structures you don’t care about. (If someone wrote a finite state transducer that did proper (depth-limited) JSON parsing, that would be fantastic! In the meantime we have “grep -o”.)

To write maintainable code, I always use a real parsing library. I haven’t tried jsawk, but if it works well, that would address point #1.

One last, wackier, solution: I wrote a script that uses Python json and extracts the keys you want, into tab-separated columns; then I pipe through a wrapper around awk that allows named access to columns. In here: the json2tsv and tsvawk scripts. So for this example it would be:

This approach doesn’t address #2, is more inefficient than a single Python script, and it’s a little brittle: it forces normalization of newlines and tabs in string values, to play nice with awk’s field/record-delimited view of the world. But it does let you stay on the command line, with more correctness than grep -o.

Answer 3 (score 163)

On the basis that some of the recommendations here (esp in the comments) suggested the use of Python, I was disappointed not to find an example.

So, here’s a one liner to get a single value from some JSON data. It assumes that you are piping the data in (from somewhere) and so should be useful in a scripting context.

34: How can I add numbers in a bash script (score 844322 in 2019)

Question

I have this bash script and I had a problem in line 16. How can I take the previous result of line 15 and add it to the variable in line 16?

Answer 2 (score 899)

For integers:

  • Use arithmetic expansion: $((EXPR))

    sh num=$((num1 + num2)) num=$(($num1 + $num2)) # also works num=$((num1 + 2 + 3)) # ... num=$[num1+num2] # old, deprecated arithmetic expression syntax

  • Using the external expr utility. Note that this is only needed for really old systems.

    sh num=`expr $num1 + $num2` # whitespace for expr is important

For floating point:

Bash doesn’t directly support this, but there’s a couple of external tools you can use:

You can also use scientific notation (e.g.: 2.5e+2)

Common pitfalls:

  • When setting a variable, you cannot have whitespace on either side of =, otherwise it will force the shell to interpret the first word as the name of the application to run (eg: num= or num)

    num= 1 num =2

  • bc and expr expect each number and operator as a separate argument, so whitespace is important. They cannot process arguments like 3+ +4.

    num=expr $num1+ $num2``

Answer 3 (score 89)

Use the $(( )) arithmetic expansion.

See http://tldp.org/LDP/abs/html/arithexp.html for more information.

35: How to run a shell script on a Unix console or Mac terminal? (score 838291 in 2017)

Question

I know it, forget it and relearn it again. Time to write it down.

Answer accepted (score 918)

To run a non-executable sh script, use:

To run a non-executable bash script, use:

To start an executable (which is any file with executable permission); you just specify it by its path:

To make a script executable, give it the necessary permission:

When a file is executable, the kernel is responsible for figuring out how to execte it. For non-binaries, this is done by looking at the first line of the file. It should contain a hashbang:

The hashbang tells the kernel what program to run (in this case the command /usr/bin/env is ran with the argument bash). Then, the script is passed to the program (as second argument) along with all the arguments you gave the script as subsequent arguments.

That means every script that is executable should have a hashbang. If it doesn’t, you’re not telling the kernel what it is, and therefore the kernel doesn’t know what program to use to interprete it. It could be bash, perl, python, sh, or something else. (In reality, the kernel will often use the user’s default shell to interprete the file, which is very dangerous because it might not be the right interpreter at all or it might be able to parse some of it but with subtle behavioural differences such as is the case between sh and bash).

A note on /usr/bin/env

Most commonly, you’ll see hash bangs like so:

The result is that the kernel will run the program /bin/bash to interpret the script. Unfortunately, bash is not always shipped by default, and it is not always available in /bin. While on Linux machines it usually is, there are a range of other POSIX machines where bash ships in various locations, such as /usr/xpg/bin/bash or /usr/local/bin/bash.

To write a portable bash script, we can therefore not rely on hard-coding the location of the bash program. POSIX already has a mechanism for dealing with that: PATH. The idea is that you install your programs in one of the directories that are in PATH and the system should be able to find your program when you want to run it by name.

Sadly, you cannot just do this:

The kernel won’t (some might) do a PATH search for you. There is a program that can do a PATH search for you, though, it’s called env. Luckily, nearly all systems have an env program installed in /usr/bin. So we start env using a hardcoded path, which then does a PATH search for bash and runs it so that it can interpret your script:

This approach has one downside: According to POSIX, the hashbang can have one argument. In this case, we use bash as the argument to the env program. That means we have no space left to pass arguments to bash. So there’s no way to convert something like #!/bin/bash -exu to this scheme. You’ll have to put set -exu after the hashbang instead.

This approach also has another advantage: Some systems may ship with a /bin/bash, but the user may not like it, may find it’s buggy or outdated, and may have installed his own bash somewhere else. This is often the case on OS X (Macs) where Apple ships an outdated /bin/bash and users install an up-to-date /usr/local/bin/bash using something like Homebrew. When you use the env approach which does a PATH search, you take the user’s preference into account and use his preferred bash over the one his system shipped with.

Answer 2 (score 80)

To start the shell-script ‘file.sh’:

Another option is set executable permission using chmod command:

Now run .sh file as follows:

Answer 3 (score 15)

For the bourne shell:

For bash:

36: How to declare and use boolean variables in shell script? (score 818063 in 2014)

Question

I tried to declare a boolean variable in a shell script using the following syntax:

Is this correct? Also, if I wanted to update that variable would I use the same syntax? Finally, is the following syntax for using boolean variables as expressions correct:

Answer accepted (score 1064)

Revised Answer (Feb 12, 2014)

Original Answer

Caveats: https://stackoverflow.com/a/21210966/89391

From: Using boolean variables in Bash

The reason the original answer is included here is because the comments before the revision on Feb 12, 2014 pertain only to the original answer, and many of the comments are wrong when associated with the revised answer. For example, Dennis Williamson’s comment about bash builtin true on Jun 2, 2010 only applies to the original answer, not the revised.

Answer 2 (score 719)

TL;DR
Issues with Miku’s (original) answer

I do not recommend the accepted answer1. Its syntax is pretty but it has some flaws.

Say we have the following condition.

In the following cases2, this condition will evaluate to true and execute the nested command.

Typically you only want your condition to evaluate to true when your “boolean” variable, var in this example, is explicitly set to true. All the others cases are dangerously misleading!

The last case (#5) is especially naughty because it will execute the command contained in the variable (which is why the condition evaluates to true for valid commands3, 4).

Here is a harmless example:

Quoting your variables is safer, e.g. if "$var"; then. In the above cases, you should get a warning that the command is not found. But we can still do better (see my recommendations at the bottom).

Also see Mike Holt’s explanation of Miku’s original answer.

Issues with Hbar’s answer

This approach also has unexpected behaviour.

You would expect the above condition to evaluate to false, thus never executing the nested statement. Surprise!

Quoting the value ("false"), quoting the variable ("$var"), or using test or [[ instead of [, do not make a difference.

What I DO recommend:

Here are ways I recommend you check your “booleans”. They work as expected.

They’re all pretty much equivalent. You’ll have to type a few more keystrokes than the approaches in the other answers5 but your code will be more defensive.

Footnotes
  1. Miku’s answer has since been edited and no longer contains (known) flaws.
  2. Not an exhaustive list.
  3. A valid command in this context means a command that exists. It doesn’t matter if the command is used correctly or incorrectly. E.g. man woman would still be considered a valid command, even if no such man page exists.
  4. For invalid (non-existent) commands, Bash will simply complain that the command wasn’t found.
  5. If you care about length, the first recommendation is the shortest.

Answer 3 (score 156)

There seems to be some misunderstanding here about the Bash builtin true, and more specifically, about how Bash expands and interprets expressions inside brackets.

The code in miku’s answer has absolutely nothing to do with the Bash builtin true, nor /bin/true, nor any other flavor of the true command. In this case, true is nothing more than a simple character string, and no call to the true command/builtin is ever made, neither by the variable assignment, nor by the evaluation of the conditional expression.

The following code is functionally identical to the code in the miku’s answer:

The only difference here is that the four characters being compared are ‘y’, ‘e’, ‘a’, and ‘h’ instead of ‘t’, ‘r’, ‘u’, and ‘e’. That’s it. There’s no attempt made to call a command or builtin named yeah, nor is there (in miku’s example) any sort of special handling going on when Bash parses the token true. It’s just a string, and a completely arbitrary one at that.

Update (2014-02-19): After following the link in miku’s answer, now I see where some of the confusion is coming from. Miku’s answer uses single brackets, but the code snippet he links to does not use brackets. It’s just:

Both code snippets will behave the same way, but the brackets completely change what’s going on under the hood.

Here’s what Bash is doing in each case:

No brackets:

  1. Expand the variable $the_world_is_flat to the string "true".
  2. Attempt to parse the string "true" as a command.
  3. Find and run the true command (either a builtin or /bin/true, depending on the Bash version).
  4. Compare the exit code of the true command (which is always 0) with 0. Recall that in most shells, an exit code of 0 indicates success and anything else indicates failure.
  5. Since the exit code was 0 (success), execute the if statement’s then clause

Brackets:

  1. Expand the variable $the_world_is_flat to the string "true".
  2. Parse the now-fully-expanded conditional expression, which is of the form string1 = string2. The = operator is bash’s string comparison operator. So…
  3. Do a string comparison on "true" and "true".
  4. Yep, the two strings were the same, so the value of the conditional is true.
  5. Execute the if statement’s then clause.

The no-brackets code works, because the true command returns an exit code of 0, which indicates success. The bracketed code works, because the value of $the_world_is_flat is identical to the string literal true on the right side of the =.

Just to drive the point home, consider the following two snippets of code:

This code (if run with root privileges) will reboot your computer:

This code just prints “Nice try.” The reboot command is not called.

Update (2014-04-14) To answer the question in the comments regarding the difference between = and ==: AFAIK, there is no difference. The == operator is a Bash-specific synonym for =, and as far as I’ve seen, they work exactly the same in all contexts.

Note, however, that I’m specifically talking about the = and == string comparison operators used in either [ ] or [[ ]] tests. I’m not suggesting that = and == are interchangeable everywhere in bash.

For example, you obviously can’t do variable assignment with ==, such as var=="foo" (well technically you can do this, but the value of var will be "=foo", because Bash isn’t seeing an == operator here, it’s seeing an = (assignment) operator, followed by the literal value ="foo", which just becomes "=foo").

Also, although = and == are interchangeable, you should keep in mind that how those tests work does depend on whether you’re using it inside [ ] or [[ ]], and also on whether or not the operands are quoted. You can read more about that in Advanced Bash Scripting Guide: 7.3 Other Comparison Operators (scroll down to the discussion of = and ==).

37: How to convert a string to lower case in Bash? (score 812250 in 2018)

Question

Is there a way in to convert a string into a lower case string?

For example, if I have:

I want to convert it to:

Answer accepted (score 1988)

The are various ways:

POSIX standard
tr
AWK
Non-POSIX

You may run into portability issues with the following examples:

Bash 4.0
sed
Perl
Bash

Note: YMMV on this one. Doesn’t work for me (GNU bash version 4.2.46 and 4.0.33 (and same behaviour 2.05b.0 but nocasematch is not implemented)) even with using shopt -u nocasematch;. Unsetting that nocasematch causes [[ “fooBaR” == “FOObar” ]] to match OK BUT inside case weirdly [b-z] are incorrectly matched by [A-Z]. Bash is confused by the double-negative (“unsetting nocasematch”)! :-)

Answer 2 (score 411)

In Bash 4:

To lowercase

To uppercase

Toggle (undocumented, but optionally configurable at compile time)

Capitalize (undocumented, but optionally configurable at compile time)

Title case:

To turn off a declare attribute, use +. For example, declare +c string. This affects subsequent assignments and not the current value.

The declare options change the attribute of the variable, but not the contents. The reassignments in my examples update the contents to show the changes.

Edit:

Added “toggle first character by word” (${var~}) as suggested by ghostdog74.

Edit: Corrected tilde behavior to match Bash 4.3.

Answer 3 (score 120)

38: How does “cat << EOF” work in bash? (score 797520 in 2017)

Question

I needed to write a script to enter multi-line input to a program (psql).

After a bit of googling, I found the following syntax works:

This correctly constructs the multi-line string (from BEGIN; to END;, inclusive) and pipes it as an input to psql.

But I have no idea how/why it works, can some one please explain?

I’m referring mainly to cat &lt;&lt; EOF, I know &gt; outputs to a file, &gt;&gt; appends to a file, &lt; reads input from file.

What does &lt;&lt; exactly do?

And is there a man page for it?

Answer accepted (score 466)

This is called heredoc format to provide a string into stdin. See https://en.wikipedia.org/wiki/Here_document#Unix_shells for more details.

From man bash:

Here Documents

This type of redirection instructs the shell to read input from the current source until a line containing only word (with no trailing blanks) is seen.

All of the lines read up to that point are then used as the standard input for a command.

The format of here-documents is:

No parameter expansion, command substitution, arithmetic expansion, or pathname expansion is performed on word. If any characters in word are quoted, the delimiter is the result of quote removal on word, and the lines in the here-document are not expanded. If word is unquoted, all lines of the here-document are subjected to parameter expansion, command substitution, and arithmetic expansion. In the latter case, the character sequence \&lt;newline&gt; is ignored, and \ must be used to quote the characters \, $, and ```.

If the redirection operator is &lt;&lt;-, then all leading tab characters are stripped from input lines and the line containing delimiter. This allows here-documents within shell scripts to be indented in a natural fashion.

Answer 2 (score 441)

The cat &lt;&lt;EOF syntax is very useful when working with multi-line text in Bash, eg. when assigning multi-line string to a shell variable, file or a pipe.

Examples of cat &lt;&lt;EOF syntax usage in Bash:
  1. Assign multi-line string to a shell variable

The $sql variable now holds the new-line characters too. You can verify with echo -e "$sql".

  1. Pass multi-line string to a file in Bash

The print.sh file now contains:

  1. Pass multi-line string to a pipe in Bash

The b.txt file contains bar and baz lines. The same output is printed to stdout.

Answer 3 (score 215)

In your case, “EOF” is known as a “Here Tag”. Basically &lt;&lt;Here tells the shell that you are going to enter a multiline string until the “tag” Here. You can name this tag as you want, it’s often EOF or STOP.

Some rules about the Here tags:

  1. The tag can be any string, uppercase or lowercase, though most people use uppercase by convention.
  2. The tag will not be considered as a Here tag if there are other words in that line. In this case, it will merely be considered part of the string. The tag should be by itself on a separate line, to be considered a tag.
  3. The tag should have no leading or trailing spaces in that line to be considered a tag. Otherwise it will be considered as part of the string.

example:

39: How do I pause my shell script for a second before continuing? (score 794111 in 2017)

Question

I have only found how to wait for user input. However, I only want to pause so that my while true doesn’t crash my computer.

I tried pause(1), but it says -bash: syntax error near unexpected token '1'. How can it be done?

Answer 2 (score 1092)

Use the sleep command.

Example:

One can also employ decimals when specifying a time unit; e.g. sleep 1.5s

Answer 3 (score 66)

And what about:

40: Split string into an array in Bash (score 769103 in 2018)

Question

In a Bash script I would like to split a line into pieces and store them in an array.

The line:

I would like to have them in an array like this:

I would like to use simple code, the command’s speed doesn’t matter. How can I do it?

Answer accepted (score 984)

Note that the characters in $IFS are treated individually as separators so that in this case fields may be separated by either a comma or a space rather than the sequence of the two characters. Interestingly though, empty fields aren’t created when comma-space appears in the input because the space is treated specially.

To access an individual element:

To iterate over the elements:

To get both the index and the value:

The last example is useful because Bash arrays are sparse. In other words, you can delete an element or add an element and then the indices are not contiguous.

To get the number of elements in an array:

As mentioned above, arrays can be sparse so you shouldn’t use the length to get the last element. Here’s how you can in Bash 4.2 and later:

in any version of Bash (from somewhere after 2.05b):

Larger negative offsets select farther from the end of the array. Note the space before the minus sign in the older form. It is required.

Answer 2 (score 261)

All of the answers to this question are wrong in one way or another.

Wrong answer #1

1: This is a misuse of $IFS. The value of the $IFS variable is not taken as a single variable-length string separator, rather it is taken as a set of single-character string separators, where each field that read splits off from the input line can be terminated by any character in the set (comma or space, in this example).

Actually, for the real sticklers out there, the full meaning of $IFS is slightly more involved. From the bash manual:

The shell treats each character of IFS as a delimiter, and splits the results of the other expansions into words using these characters as field terminators. If IFS is unset, or its value is exactly <space><tab><newline>, the default, then sequences of <space>, <tab>, and <newline> at the beginning and end of the results of the previous expansions are ignored, and any sequence of IFS characters not at the beginning or end serves to delimit words. If IFS has a value other than the default, then sequences of the whitespace characters <space>, <tab>, and <newline> are ignored at the beginning and end of the word, as long as the whitespace character is in the value of IFS (an IFS whitespace character). Any character in IFS that is not IFS whitespace, along with any adjacent IFS whitespace characters, delimits a field. A sequence of IFS whitespace characters is also treated as a delimiter. If the value of IFS is null, no word splitting occurs.

Basically, for non-default non-null values of $IFS, fields can be separated with either (1) a sequence of one or more characters that are all from the set of “IFS whitespace characters” (that is, whichever of <space>, <tab>, and <newline> (“newline” meaning line feed (LF)) are present anywhere in $IFS), or (2) any non-“IFS whitespace character” that’s present in $IFS along with whatever “IFS whitespace characters” surround it in the input line.

For the OP, it’s possible that the second separation mode I described in the previous paragraph is exactly what he wants for his input string, but we can be pretty confident that the first separation mode I described is not correct at all. For example, what if his input string was 'Los Angeles, United States, North America'?

2: Even if you were to use this solution with a single-character separator (such as a comma by itself, that is, with no following space or other baggage), if the value of the $string variable happens to contain any LFs, then read will stop processing once it encounters the first LF. The read builtin only processes one line per invocation. This is true even if you are piping or redirecting input only to the read statement, as we are doing in this example with the here-string mechanism, and thus unprocessed input is guaranteed to be lost. The code that powers the read builtin has no knowledge of the data flow within its containing command structure.

You could argue that this is unlikely to cause a problem, but still, it’s a subtle hazard that should be avoided if possible. It is caused by the fact that the read builtin actually does two levels of input splitting: first into lines, then into fields. Since the OP only wants one level of splitting, this usage of the read builtin is not appropriate, and we should avoid it.

3: A non-obvious potential issue with this solution is that read always drops the trailing field if it is empty, although it preserves empty fields otherwise. Here’s a demo:

Maybe the OP wouldn’t care about this, but it’s still a limitation worth knowing about. It reduces the robustness and generality of the solution.

This problem can be solved by appending a dummy trailing delimiter to the input string just prior to feeding it to read, as I will demonstrate later.

Wrong answer #2

Similar idea:

(Note: I added the missing parentheses around the command substitution which the answerer seems to have omitted.)

Similar idea:

These solutions leverage word splitting in an array assignment to split the string into fields. Funnily enough, just like read, general word splitting also uses the $IFS special variable, although in this case it is implied that it is set to its default value of <space><tab><newline>, and therefore any sequence of one or more IFS characters (which are all whitespace characters now) is considered to be a field delimiter.

This solves the problem of two levels of splitting committed by read, since word splitting by itself constitutes only one level of splitting. But just as before, the problem here is that the individual fields in the input string can already contain $IFS characters, and thus they would be improperly split during the word splitting operation. This happens to not be the case for any of the sample input strings provided by these answerers (how convenient…), but of course that doesn’t change the fact that any code base that used this idiom would then run the risk of blowing up if this assumption were ever violated at some point down the line. Once again, consider my counterexample of 'Los Angeles, United States, North America' (or 'Los Angeles:United States:North America').

Also, word splitting is normally followed by filename expansion (aka pathname expansion aka globbing), which, if done, would potentially corrupt words containing the characters *, ?, or [ followed by ] (and, if extglob is set, parenthesized fragments preceded by ?, *, +, @, or !) by matching them against file system objects and expanding the words (“globs”) accordingly. The first of these three answerers has cleverly undercut this problem by running set -f beforehand to disable globbing. Technically this works (although you should probably add set +f afterward to reenable globbing for subsequent code which may depend on it), but it’s undesirable to have to mess with global shell settings in order to hack a basic string-to-array parsing operation in local code.

Another issue with this answer is that all empty fields will be lost. This may or may not be a problem, depending on the application.

Note: If you’re going to use this solution, it’s better to use the ${string//:/ } “pattern substitution” form of parameter expansion, rather than going to the trouble of invoking a command substitution (which forks the shell), starting up a pipeline, and running an external executable (tr or sed), since parameter expansion is purely a shell-internal operation. (Also, for the tr and sed solutions, the input variable should be double-quoted inside the command substitution; otherwise word splitting would take effect in the echo command and potentially mess with the field values. Also, the $(...) form of command substitution is preferable to the old ... form since it simplifies nesting of command substitutions and allows for better syntax highlighting by text editors.)

Wrong answer #3

This answer is almost the same as #2. The difference is that the answerer has made the assumption that the fields are delimited by two characters, one of which being represented in the default $IFS, and the other not. He has solved this rather specific case by removing the non-IFS-represented character using a pattern substitution expansion and then using word splitting to split the fields on the surviving IFS-represented delimiter character.

This is not a very generic solution. Furthermore, it can be argued that the comma is really the “primary” delimiter character here, and that stripping it and then depending on the space character for field splitting is simply wrong. Once again, consider my counterexample: 'Los Angeles, United States, North America'.

Also, again, filename expansion could corrupt the expanded words, but this can be prevented by temporarily disabling globbing for the assignment with set -f and then set +f.

Also, again, all empty fields will be lost, which may or may not be a problem depending on the application.

Wrong answer #4

This is similar to #2 and #3 in that it uses word splitting to get the job done, only now the code explicitly sets $IFS to contain only the single-character field delimiter present in the input string. It should be repeated that this cannot work for multicharacter field delimiters such as the OP’s comma-space delimiter. But for a single-character delimiter like the LF used in this example, it actually comes close to being perfect. The fields cannot be unintentionally split in the middle as we saw with previous wrong answers, and there is only one level of splitting, as required.

One problem is that filename expansion will corrupt affected words as described earlier, although once again this can be solved by wrapping the critical statement in set -f and set +f.

Another potential problem is that, since LF qualifies as an “IFS whitespace character” as defined earlier, all empty fields will be lost, just as in #2 and #3. This would of course not be a problem if the delimiter happens to be a non-“IFS whitespace character”, and depending on the application it may not matter anyway, but it does vitiate the generality of the solution.

So, to sum up, assuming you have a one-character delimiter, and it is either a non-“IFS whitespace character” or you don’t care about empty fields, and you wrap the critical statement in set -f and set +f, then this solution works, but otherwise not.

(Also, for information’s sake, assigning a LF to a variable in bash can be done more easily with the $'...' syntax, e.g. IFS=$'\n';.)

Wrong answer #5

Similar idea:

This solution is effectively a cross between #1 (in that it sets $IFS to comma-space) and #2-4 (in that it uses word splitting to split the string into fields). Because of this, it suffers from most of the problems that afflict all of the above wrong answers, sort of like the worst of all worlds.

Also, regarding the second variant, it may seem like the eval call is completely unnecessary, since its argument is a single-quoted string literal, and therefore is statically known. But there’s actually a very non-obvious benefit to using eval in this way. Normally, when you run a simple command which consists of a variable assignment only, meaning without an actual command word following it, the assignment takes effect in the shell environment:

This is true even if the simple command involves multiple variable assignments; again, as long as there’s no command word, all variable assignments affect the shell environment:

But, if the variable assignment is attached to a command name (I like to call this a “prefix assignment”) then it does not affect the shell environment, and instead only affects the environment of the executed command, regardless whether it is a builtin or external:

Relevant quote from the bash manual:

If no command name results, the variable assignments affect the current shell environment. Otherwise, the variables are added to the environment of the executed command and do not affect the current shell environment.

It is possible to exploit this feature of variable assignment to change $IFS only temporarily, which allows us to avoid the whole save-and-restore gambit like that which is being done with the $OIFS variable in the first variant. But the challenge we face here is that the command we need to run is itself a mere variable assignment, and hence it would not involve a command word to make the $IFS assignment temporary. You might think to yourself, well why not just add a no-op command word to the statement like the : builtin to make the $IFS assignment temporary? This does not work because it would then make the $array assignment temporary as well:

So, we’re effectively at an impasse, a bit of a catch-22. But, when eval runs its code, it runs it in the shell environment, as if it was normal, static source code, and therefore we can run the $array assignment inside the eval argument to have it take effect in the shell environment, while the $IFS prefix assignment that is prefixed to the eval command will not outlive the eval command. This is exactly the trick that is being used in the second variant of this solution:

So, as you can see, it’s actually quite a clever trick, and accomplishes exactly what is required (at least with respect to assignment effectation) in a rather non-obvious way. I’m actually not against this trick in general, despite the involvement of eval; just be careful to single-quote the argument string to guard against security threats.

But again, because of the “worst of all worlds” agglomeration of problems, this is still a wrong answer to the OP’s requirement.

Wrong answer #6

Um… what? The OP has a string variable that needs to be parsed into an array. This “answer” starts with the verbatim contents of the input string pasted into an array literal. I guess that’s one way to do it.

It looks like the answerer may have assumed that the $IFS variable affects all bash parsing in all contexts, which is not true. From the bash manual:

IFS    The Internal Field Separator that is used for word splitting after expansion and to split lines into words with the read builtin command. The default value is <space><tab><newline>.

So the $IFS special variable is actually only used in two contexts: (1) word splitting that is performed after expansion (meaning not when parsing bash source code) and (2) for splitting input lines into words by the read builtin.

Let me try to make this clearer. I think it might be good to draw a distinction between parsing and execution. Bash must first parse the source code, which obviously is a parsing event, and then later it executes the code, which is when expansion comes into the picture. Expansion is really an execution event. Furthermore, I take issue with the description of the $IFS variable that I just quoted above; rather than saying that word splitting is performed after expansion, I would say that word splitting is performed during expansion, or, perhaps even more precisely, word splitting is part of the expansion process. The phrase “word splitting” refers only to this step of expansion; it should never be used to refer to the parsing of bash source code, although unfortunately the docs do seem to throw around the words “split” and “words” a lot. Here’s a relevant excerpt from the linux.die.net version of the bash manual:

Expansion is performed on the command line after it has been split into words. There are seven kinds of expansion performed: brace expansion, tilde expansion, parameter and variable expansion, command substitution, arithmetic expansion, word splitting, and pathname expansion.

The order of expansions is: brace expansion; tilde expansion, parameter and variable expansion, arithmetic expansion, and command substitution (done in a left-to-right fashion); word splitting; and pathname expansion.

You could argue the GNU version of the manual does slightly better, since it opts for the word “tokens” instead of “words” in the first sentence of the Expansion section:

Expansion is performed on the command line after it has been split into tokens.

The important point is, $IFS does not change the way bash parses source code. Parsing of bash source code is actually a very complex process that involves recognition of the various elements of shell grammar, such as command sequences, command lists, pipelines, parameter expansions, arithmetic substitutions, and command substitutions. For the most part, the bash parsing process cannot be altered by user-level actions like variable assignments (actually, there are some minor exceptions to this rule; for example, see the various compatxx shell settings, which can change certain aspects of parsing behavior on-the-fly). The upstream “words”/“tokens” that result from this complex parsing process are then expanded according to the general process of “expansion” as broken down in the above documentation excerpts, where word splitting of the expanded (expanding?) text into downstream words is simply one step of that process. Word splitting only touches text that has been spit out of a preceding expansion step; it does not affect literal text that was parsed right off the source bytestream.

Wrong answer #7

This is one of the best solutions. Notice that we’re back to using read. Didn’t I say earlier that read is inappropriate because it performs two levels of splitting, when we only need one? The trick here is that you can call read in such a way that it effectively only does one level of splitting, specifically by splitting off only one field per invocation, which necessitates the cost of having to call it repeatedly in a loop. It’s a bit of a sleight of hand, but it works.

But there are problems. First: When you provide at least one NAME argument to read, it automatically ignores leading and trailing whitespace in each field that is split off from the input string. This occurs whether $IFS is set to its default value or not, as described earlier in this post. Now, the OP may not care about this for his specific use-case, and in fact, it may be a desirable feature of the parsing behavior. But not everyone who wants to parse a string into fields will want this. There is a solution, however: A somewhat non-obvious usage of read is to pass zero NAME arguments. In this case, read will store the entire input line that it gets from the input stream in a variable named $REPLY, and, as a bonus, it does not strip leading and trailing whitespace from the value. This is a very robust usage of read which I’ve exploited frequently in my shell programming career. Here’s a demonstration of the difference in behavior:

The second issue with this solution is that it does not actually address the case of a custom field separator, such as the OP’s comma-space. As before, multicharacter separators are not supported, which is an unfortunate limitation of this solution. We could try to at least split on comma by specifying the separator to the -d option, but look what happens:

Predictably, the unaccounted surrounding whitespace got pulled into the field values, and hence this would have to be corrected subsequently through trimming operations (this could also be done directly in the while-loop). But there’s another obvious error: Europe is missing! What happened to it? The answer is that read returns a failing return code if it hits end-of-file (in this case we can call it end-of-string) without encountering a final field terminator on the final field. This causes the while-loop to break prematurely and we lose the final field.

Technically this same error afflicted the previous examples as well; the difference there is that the field separator was taken to be LF, which is the default when you don’t specify the -d option, and the &lt;&lt;&lt; (“here-string”) mechanism automatically appends a LF to the string just before it feeds it as input to the command. Hence, in those cases, we sort of accidentally solved the problem of a dropped final field by unwittingly appending an additional dummy terminator to the input. Let’s call this solution the “dummy-terminator” solution. We can apply the dummy-terminator solution manually for any custom delimiter by concatenating it against the input string ourselves when instantiating it in the here-string:

There, problem solved. Another solution is to only break the while-loop if both (1) read returned failure and (2) $REPLY is empty, meaning read was not able to read any characters prior to hitting end-of-file. Demo:

This approach also reveals the secretive LF that automatically gets appended to the here-string by the &lt;&lt;&lt; redirection operator. It could of course be stripped off separately through an explicit trimming operation as described a moment ago, but obviously the manual dummy-terminator approach solves it directly, so we could just go with that. The manual dummy-terminator solution is actually quite convenient in that it solves both of these two problems (the dropped-final-field problem and the appended-LF problem) in one go.

So, overall, this is quite a powerful solution. It’s only remaining weakness is a lack of support for multicharacter delimiters, which I will address later.

Wrong answer #8

(This is actually from the same post as #7; the answerer provided two solutions in the same post.)

The readarray builtin, which is a synonym for mapfile, is ideal. It’s a builtin command which parses a bytestream into an array variable in one shot; no messing with loops, conditionals, substitutions, or anything else. And it doesn’t surreptitiously strip any whitespace from the input string. And (if -O is not given) it conveniently clears the target array before assigning to it. But it’s still not perfect, hence my criticism of it as a “wrong answer”.

First, just to get this out of the way, note that, just like the behavior of read when doing field-parsing, readarray drops the trailing field if it is empty. Again, this is probably not a concern for the OP, but it could be for some use-cases. I’ll come back to this in a moment.

Second, as before, it does not support multicharacter delimiters. I’ll give a fix for this in a moment as well.

Third, the solution as written does not parse the OP’s input string, and in fact, it cannot be used as-is to parse it. I’ll expand on this momentarily as well.

For the above reasons, I still consider this to be a “wrong answer” to the OP’s question. Below I’ll give what I consider to be the right answer.

Right answer

Here’s a naïve attempt to make #8 work by just specifying the -d option:

We see the result is identical to the result we got from the double-conditional approach of the looping read solution discussed in #7. We can almost solve this with the manual dummy-terminator trick:

The problem here is that readarray preserved the trailing field, since the &lt;&lt;&lt; redirection operator appended the LF to the input string, and therefore the trailing field was not empty (otherwise it would’ve been dropped). We can take care of this by explicitly unsetting the final array element after-the-fact:

The only two problems that remain, which are actually related, are (1) the extraneous whitespace that needs to be trimmed, and (2) the lack of support for multicharacter delimiters.

The whitespace could of course be trimmed afterward (for example, see How to trim whitespace from a Bash variable?). But if we can hack a multicharacter delimiter, then that would solve both problems in one shot.

Unfortunately, there’s no direct way to get a multicharacter delimiter to work. The best solution I’ve thought of is to preprocess the input string to replace the multicharacter delimiter with a single-character delimiter that will be guaranteed not to collide with the contents of the input string. The only character that has this guarantee is the NUL byte. This is because, in bash (though not in zsh, incidentally), variables cannot contain the NUL byte. This preprocessing step can be done inline in a process substitution. Here’s how to do it using awk:

There, finally! This solution will not erroneously split fields in the middle, will not cut out prematurely, will not drop empty fields, will not corrupt itself on filename expansions, will not automatically strip leading and trailing whitespace, will not leave a stowaway LF on the end, does not require loops, and does not settle for a single-character delimiter.

Trimming solution

Lastly, I wanted to demonstrate my own fairly intricate trimming solution using the obscure -C callback option of readarray. Unfortunately, I’ve run out of room against Stack Overflow’s draconian 30,000 character post limit, so I won’t be able to explain it. I’ll leave that as an exercise for the reader.

Answer 3 (score 211)

Here is a way without setting IFS:

The idea is using string replacement:

to replace all matches of $substring with white space and then using the substituted string to initialize a array:

Note: this answer makes use of the split+glob operator. Thus, to prevent expansion of some characters (such as *) it is a good idea to pause globbing for this script.

41: Comparing numbers in Bash (score 765855 in 2019)

Question

I’m starting to learn about writing scripts for the bash terminal, but I can’t work out how to get the comparisons to work properly. The script I’m using is:

The problem is that it compares the number from the first digit on, i.e. 9 is bigger than 10, but 1 is greater than 09.

How can I convert the numbers into a type to do a true comparison?

Answer accepted (score 759)

In bash, you should do your check in arithmetic context:

For POSIX shells that don’t support (()), you can use -lt and -gt.

You can get a full list of comparison operators with help test.

Answer 2 (score 145)

Plain and simple

You can check out this cheatsheet if you want more number comparsions in the wonderful world of Bash Scripting.

Shortly, integers can only be compared with:

Answer 3 (score 36)

There is also one nice thing some people might not know about:

This code will print the smallest number out of a and b

42: Listing only directories using ls in bash: An examination (score 761868 in 2015)

Question

This command lists directories in the current path: ls -d */

What exactly does the pattern */ do?

And how can we give the absolute path in the above command (e.g. ls -d /home/alice/Documents) for listing only directories in that path?

Answer accepted (score 914)

*/ is a pattern that matches all of the subdirectories in the current directory (* would match all files and subdirectories; the / restricts it to directories). Similarly, to list all subdirectories under /home/alice/Documents, use ls -d /home/alice/Documents/*/

Answer 2 (score 456)

Four ways to get this done, each with a different output format
  1. Using echo

Example: echo */, echo */*/
Here is what I got:

  1. Using ls only

Example: ls -d */
Here is exactly what I got:

Or as list (with detail info): ls -dl */

  1. Using ls and grep

Example: ls -l | grep "^d" Here is what I got:

  1. Bash Script (Not recommended for filename containing spaces)

Example: for i in $(ls -d */); do echo ${i%%/}; done
Here is what I got:

If you like to have ‘/’ as ending character, the command will be: for i in $(ls -d */); do echo ${i}; done

Answer 3 (score 90)

I use:

This creates a single column with no trailing slash - useful in scripts.

My two cents.

43: How to represent multiple conditions in a shell if statement? (score 751876 in 2017)

Question

I want to represent multiple conditions like this:

but when I execute the script, it shows

where line 15 is the one showing if ….

What is wrong with this condition? I guess something is wrong with the ().

Answer accepted (score 351)

Classic technique (escape metacharacters):

I’ve enclosed the references to $g in double quotes; that’s good practice, in general. Strictly, the parentheses aren’t needed because the precedence of -a and -o makes it correct even without them.

Note that the -a and -o operators are part of the POSIX specification for test, aka [, mainly for backwards compatibility (since they were a part of test in 7th Edition UNIX, for example), but they are explicitly marked as ‘obsolescent’ by POSIX. Bash (see conditional expressions) seems to preempt the classic and POSIX meanings for -a and -o with its own alternative operators that take arguments.

With some care, you can use the more modern [[ operator, but be aware that the versions in Bash and Korn Shell (for example) need not be identical.

Example run, using Bash 3.2.57 on Mac OS X:

You don’t need to quote the variables in [[ as you do with [ because it is not a separate command in the same way that [ is.

Isn’t it a classic question?

I would have thought so. However, there is another alternative, namely:

Indeed, if you read the ‘portable shell’ guidelines for the autoconf tool or related packages, this notation — using ‘||’ and ‘&amp;&amp;’ — is what they recommend. I suppose you could even go so far as:

Where the actions are as trivial as echoing, this isn’t bad. When the action block to be repeated is multiple lines, the repetition is too painful and one of the earlier versions is preferable — or you need to wrap the actions into a function that is invoked in the different then blocks.

Answer 2 (score 174)

In Bash:

Answer 3 (score 30)

Using /bin/bash the following will work:

44: How to count all the lines of code in a directory recursively? (score 718292 in 2012)

Question

We’ve got a PHP application and want to count all the lines of code under a specific directory and its subdirectories. We don’t need to ignore comments, as we’re just trying to get a rough idea.

That command works great within a given directory, but ignores subdirectories. I was thinking this might work, but it is returning 74, which is definitely not the case…

What’s the correct syntax to feed in all the files?

Answer accepted (score 2494)

Try:

The SLOCCount tool may help as well.

It’ll give an accurate source lines of code count for whatever hierarchy you point it at, as well as some additional stats.

Sorted output: find . -name '*.php' | xargs wc -l | sort -nr

Answer 2 (score 450)

For another one-liner:

works on names with spaces, only outputs one number.

Answer 3 (score 383)

If using a decently recent version of Bash (or ZSH), it’s much simpler:

In the Bash shell this requires the globstar option to be set, otherwise the ** glob-operator is not recursive. To enable this setting, issue

To make this permanent, add it to one of the initialization files (~/.bashrc, ~/.bash_profile etc.).

45: Replace one substring for another string in shell script (score 712335 in 2015)

Question

I have “I love Suzi and Marry” and I want to change “Suzi” to “Sara”.

The result must be like this:

Answer accepted (score 1112)

To replace the first occurrence of a pattern with a given string, use ${<em>parameter</em>/<em>pattern</em>/<em>string</em>}:

To replace all occurrences, use ${<em>parameter</em>//<em>pattern</em>/<em>string</em>}:

(This is documented in the Bash Reference Manual, §3.5.3 “Shell Parameter Expansion”.)

Note that this feature is not specified by POSIX — it’s a Bash extension — so not all Unix shells implement it. For the relevant POSIX documentation, see The Open Group Technical Standard Base Specifications, Issue 7, the Shell & Utilities volume, §2.6.2 “Parameter Expansion”.

Answer 2 (score 176)

This can be done entirely with bash string manipulation:

That will replace only the first occurrence; to replace them all, double the first slash:

Answer 3 (score 60)

try this:

46: Setting environment variables on OS X (score 696397 in 2018)

Question

What is the proper way to modify environment variables like PATH in OS X?

I’ve looked on Google a little bit and found three different files to edit:

  • /etc/paths
  • ~/.profile
  • ~/.tcshrc

I don’t even have some of these files, and I’m pretty sure that .tcshrc is wrong, since OS X uses bash now. Where are these variables, especially PATH, defined?

I’m running OS X v10.5 (Leopard).

Answer 2 (score 644)

Bruno is right on track. I’ve done extensive research and if you want to set variables that are available in all GUI applications, your only option is /etc/launchd.conf.

Please note that environment.plist does not work for applications launched via Spotlight. This is documented by Steve Sexton here.

  1. Open a terminal prompt

  2. Type sudo vi /etc/launchd.conf (note: this file might not yet exist)

  3. Put contents like the following into the file

    ```sh # Set environment variables here so they are available globally to all apps # (and Terminal), including those launched via Spotlight. # # After editing this file run the following command from the terminal to update # environment variables globally without needing to reboot. # NOTE: You will still need to restart the relevant application (including # Terminal) to pick up the changes! # grep -E “^setenv” /etc/launchd.conf | xargs -t -L 1 launchctl # # See http://www.digitaledgesw.com/node/31 # and http://stackoverflow.com/questions/135688/setting-environment-variables-in-os-x/ # # Note that you must hardcode the paths below, don’t use environment variables. # You also need to surround multiple values in quotes, see MAVEN_OPTS example below. # setenv JAVA_VERSION 1.6 setenv JAVA_HOME /System/Library/Frameworks/JavaVM.framework/Versions/1.6/Home setenv GROOVY_HOME /Applications/Dev/groovy setenv GRAILS_HOME /Applications/Dev/grails setenv NEXUS_HOME /Applications/Dev/nexus/nexus-webapp setenv JRUBY_HOME /Applications/Dev/jruby

    setenv ANT_HOME /Applications/Dev/apache-ant setenv ANT_OPTS -Xmx512M

    setenv MAVEN_OPTS “-Xmx1024M -XX:MaxPermSize=512m” setenv M2_HOME /Applications/Dev/apache-maven

    setenv JMETER_HOME /Applications/Dev/jakarta-jmeter ``</li> <li><p>Save your changes in vi and reboot your Mac. Or use the <a href="http://linux.die.net/man/1/grep" rel="noreferrer">grep</a>/<a href="https://linux.die.net/man/1/xargs" rel="noreferrer">xargs</a> command which is shown in the code comment above.</p></li> <li><p>Prove that your variables are working by opening a Terminal window and typingexport` and you should see your new variables. These will also be available in IntelliJ IDEA and other GUI applications you launch via Spotlight.

Answer 3 (score 256)

How to set the environment for new processes started by Spotlight (without needing to reboot)

You can set the environment used by launchd (and, by extension, anything started from Spotlight) with launchctl setenv. For example to set the path:

Or if you want to set up your path in .bashrc or similar, then have it mirrored in launchd:

There’s no need to reboot though you will need to restart an app if you want it to pick up the changed environment.

This includes any shells already running under Terminal.app, although if you’re there you can set the environment more directly, e.g. with export PATH=/opt/local/bin:/opt/local/sbin:$PATH for bash or zsh.

How to keeping changes after a reboot

To keep changes after a reboot you can set the environment variables from /etc/launchd.conf, like so:

launchd.conf is executed automatically when you reboot.

If you want these changes to take effect now, you should use this command to reprocess launchctl.conf (thanks @mklement for the tip!)

You can find out more about launchctl and how it loads launchd.conf with the command man launchctl.

47: Check number of arguments passed to a Bash script (score 696353 in 2019)

Question

I would like my Bash script to print an error message if the required argument count is not met.

I tried the following code:

For some unknown reason I’ve got the following error:

What am I doing wrong?

Answer accepted (score 973)

Just like any other simple command, [ ... ] or test requires spaces between its arguments.

Or

Suggestions

When in Bash, prefer using [[ ]] instead as it doesn’t do word splitting and pathname expansion to its variables that quoting may not be necessary unless it’s part of an expression.

It also has some other features like unquoted condition grouping, pattern matching (extended pattern matching with extglob) and regex matching.

The following example checks if arguments are valid. It allows a single argument or two.

For pure arithmetic expressions, using (( )) to some may still be better, but they are still possible in [[ ]] with its arithmetic operators like -eq, -ne, -lt, -le, -gt, or -ge by placing the expression as a single string argument:

That should be helpful if you would need to combine it with other features of [[ ]] as well.

Exiting the script

It’s also logical to make the script exit when invalid parameters are passed to it. This has already been suggested in the comments by ekangas but someone edited this answer to have it with -1 as the returned value, so I might as well do it right.

-1 though accepted by Bash as an argument to exit is not explicitly documented and is not right to be used as a common suggestion. 64 is also the most formal value since it’s defined in sysexits.h with #define EX_USAGE 64 /* command line usage error */. Most tools like ls also return 2 on invalid arguments. I also used to return 2 in my scripts but lately I no longer really cared, and simply used 1 in all errors. But let’s just place 2 here since it’s most common and probably not OS-specific.

References

Answer 2 (score 66)

It might be a good idea to use arithmetic expressions if you’re dealing with numbers.

Answer 3 (score 38)

On []: !=, =, == … are string comparison operators and -eq, -gt … are arithmetic binary ones.

I would use:

Or:

48: How can I redirect and append both stdout and stderr to a file with Bash? (score 692374 in 2015)

Question

To redirect stdout to a truncated file in Bash, I know to use:

To redirect stdout in Bash, appending to a file, I know to use:

To redirect both stdout and stderr to a truncated file, I know to use:

How do I redirect both stdout and stderr appending to a file? cmd &amp;&gt;&gt; file.txt did not work for me.

Answer accepted (score 1853)

Bash executes the redirects from left to right as follows:

  1. &gt;&gt;file.txt: Open file.txt in append mode and redirect stdout there.
  2. 2&gt;&amp;1: Redirect stderr to “where stdout is currently going”. In this case, that is a file opened in append mode. In other words, the &amp;1 reuses the file descriptor which stdout currently uses.

Answer 2 (score 344)

There are two ways to do this, depending on your Bash version.

The classic and portable (Bash pre-4) way is:

A nonportable way, starting with Bash 4 is

(analog to &amp;&gt; outfile)

For good coding style, you should

  • decide if portability is a concern (then use classic way)
  • decide if portability even to Bash pre-4 is a concern (then use classic way)
  • no matter which syntax you use, not change it within the same script (confusion!)

If your script already starts with #!/bin/sh (no matter if intended or not), then the Bash 4 solution, and in general any Bash-specific code, is not the way to go.

Also remember that Bash 4 &amp;&gt;&gt; is just shorter syntax — it does not introduce any new functionality or anything like that.

The syntax is (beside other redirection syntax) described here: http://bash-hackers.org/wiki/doku.php/syntax/redirection#appending_redirected_output_and_error_output

Answer 3 (score 84)

In Bash you can also explicitly specify your redirects to different files:

Appending would be:

49: How to reload .bash_profile from the command line? (score 682357 in 2014)

Question

How can I reload .bash_profile from the command line? I can get the shell to recognize changes to .bash_profile by exiting and logging back in but I would like to be able to do it on demand.

Answer accepted (score 1712)

Simply type source ~/.bash_profile

Alternatively, if you like saving keystrokes you can type . ~/.bash_profile

Answer 2 (score 98)

Just make sure you don’t have any dependencies on the current state in there.

Answer 3 (score 30)

Simply type:

However, if you want to source it to run automatically when terminal starts instead of running it every time you open terminal, you might add . ~/.bash_profile to ~/.bashrc file.

Note:

When you open a terminal, the terminal starts bash in (non-login) interactive mode, which means it will source ~/.bashrc.

~/.bash_profile is only sourced by bash when started in interactive login mode. That is typically only when you login at the console (Ctrl+Alt+F1..F6), or connecting via ssh.

50: How to loop over files in directory and change path and add suffix to filename (score 670034 in 2019)

Question

I need to write a script that starts my program with different arguments, but I’m new to Bash. I start my program with:

./MyProgram.exe Data/data1.txt [Logs/data1_Log.txt].

Here is the pseudocode for what I want to do:

So I’m really puzzled how to create second argument from the first one, so it looks like dataABCD_Log1.txt and start my program.

Answer accepted (score 660)

A couple of notes first: when you use Data/data1.txt as an argument, should it really be /Data/data1.txt (with a leading slash)? Also, should the outer loop scan only for .txt files, or all files in /Data? Here’s an answer, assuming /Data/data1.txt and .txt files only:

Notes:

  • /Data/*.txt expands to the paths of the text files in /Data (including the /Data/ part)
  • $( ... ) runs a shell command and inserts its output at that point in the command line
  • basename somepath .txt outputs the base part of somepath, with .txt removed from the end (e.g. /Data/file.txt -> file)

If you needed to run MyProgram with Data/file.txt instead of /Data/file.txt, use "${filename#/}" to remove the leading slash. On the other hand, if it’s really Data not /Data you want to scan, just use for filename in Data/*.txt.

Answer 2 (score 295)

Sorry for necromancing the thread, but whenever you iterate over files by globbing, it’s good practice to avoid the corner case where the glob does not match (which makes the loop variable expand to the (un-matching) glob pattern string itself).

For example:

Reference: Bash Pitfalls

Answer 3 (score 71)

The name=${file##*/} substitution (shell parameter expansion) removes the leading pathname up to the last /.

The base=${name%.txt} substitution removes the trailing .txt. It’s a bit trickier if the extensions can vary.

51: Redirect stderr and stdout in Bash (score 652646 in 2017)

Question

I want to redirect both stdout and stderr of a process to a single file. How do I do that in Bash?

Answer accepted (score 717)

Take a look here. Should be:

(redirects both stdout and stderr to filename).

Answer 2 (score 437)

This is going to redirect stderr to stdout and stdout to some_file and print it to stdout.

Answer 3 (score 234)

You can redirect stderr to stdout and the stdout into a file:

See http://tldp.org/LDP/abs/html/io-redirection.html

This format is preferred than the most popular &> format that only work in bash. In Bourne shell it could be interpreted as running the command in background. Also the format is more readable 2 (is STDERR) redirected to 1 (STDOUT).

EDIT: changed the order as pointed out in the comments

52: How to echo shell commands as they are executed (score 644659 in 2019)

Question

In a shell script, how do I echo all shell commands called and expand any variable names?

For example, given the following line:

I would like the script to run the command and display the following

The purpose is to save a log of all shell commands called and their arguments. Is there perhaps a better way of generating such a log?

Answer accepted (score 894)

set -x or set -o xtrace expands variables and prints a little + sign before the line.

set -v or set -o verbose does not expand the variables before printing.

Use set +x and set +v to turn off the above settings.

On the first line of the script, one can put #!/bin/sh -x (or -v) to have the same effect as set -x (or -v) later in the script.

The above also works with /bin/sh.

See the bash-hackers’ wiki on set attributes, and on debugging.

Answer 2 (score 288)

set -x will give you what you want.

Here is an example shell script to demonstrate:

This expands all variables and prints the full commands before output of the command.

output:

Answer 3 (score 83)

You can also toggle this for select lines in your script by wrapping them in set -x and set +x e.g.

53: Get current directory name (without full path) in a Bash script (score 626208 in 2018)

Question

How would I get just the current working directory name in a bash script, or even better, just a terminal command.

pwd gives the full path of the current working directory, e.g. /opt/local/bin but I only want bin

Answer accepted (score 1017)

No need for basename, and especially no need for a subshell running pwd (which adds an extra, and expensive, fork operation); the shell can do this internally using parameter expansion:

Note that if you’re applying this technique in other circumstances (not PWD, but some other variable holding a directory name), you might need to trim any trailing slashes. The below uses bash’s extglob support to work even with multiple trailing slashes:

Alternatively, without extglob:

Answer 2 (score 280)

Use the basename program. For your case:

Answer 3 (score 132)

​​​​​

54: if, elif, else statement issues in Bash (score 625671 in 2018)

Question

I can’t seem to work out what the issue with the following if statement is in regards to the elif and then. Keep in mind the printf is still under development I just haven’t been able to test it yet in the statement so is more than likely wrong.

The error I’m getting is:

And the statement is like so.

Answer accepted (score 407)

There is a space missing between elif and [:

should be

As I see this question is getting a lot of views, it is important to indicate that the syntax to follow is:

meaning that spaces are needed around the brackets. Otherwise, it won’t work. This is because [ itself is a command.

The reason why you are not seeing something like elif[: command not found (or similar) is that after seeing if and then, the shell is looking for either elif, else, or fi. However it finds another then (after the mis-formatted elif[). Only after having parsed the statement it would be executed (and an error message like elif[: command not found would be output).

Answer 2 (score 266)

You have some syntax issues with your script. Here is a fixed version:

Answer 3 (score 23)

[ is a command (or a builtin in some shells). It must be separated by whitespace from the preceding statement:

55: Find and Replace Inside a Text File from a Bash Command (score 614510 in 2017)

Question

What’s the simplest way to do a find and replace for a given input string, say abc, and replace with another string, say XYZ in file /tmp/file.txt?

I am writting an app and using IronPython to execute commands through SSH — but I don’t know Unix that well and don’t know what to look for.

I have heard that Bash, apart from being a command line interface, can be a very powerful scripting language. So, if this is true, I assume you can perform actions like these.

Can I do it with bash, and what’s the simplest (one line) script to achieve my goal?

Answer accepted (score 864)

The easiest way is to use sed (or perl):

Which will invoke sed to do an in-place edit due to the -i option. This can be called from bash.

If you really really want to use just bash, then the following can work:

This loops over each line, doing a substitution, and writing to a temporary file (don’t want to clobber the input). The move at the end just moves temporary to the original name.

Answer 2 (score 157)

File manipulation isn’t normally done by Bash, but by programs invoked by Bash, e.g.:

The -i flag tells it to do an in-place replacement.

See man perlrun for more details, including how to take a backup of the original file.

Answer 3 (score 66)

I was surprised because i stumbled over this…

There is a “replace” command which ships with the package “mysql-server”, so if you have installed it try it out:

See man replace for more on this…

56: How to check if a program exists from a Bash script? (score 603726 in 2018)

Question

How would I validate that a program exists, in a way that will either return an error and exit, or continue with the script?

It seems like it should be easy, but it’s been stumping me.

Answer accepted (score 2766)

Answer

POSIX compatible:

For bash specific environments:

Explanation

Avoid which. Not only is it an external process you’re launching for doing very little (meaning builtins like hash, type or command are way cheaper), you can also rely on the builtins to actually do what you want, while the effects of external commands can easily vary from system to system.

Why care?

  • Many operating systems have a which that doesn’t even set an exit status, meaning the if which foo won’t even work there and will always report that foo exists, even if it doesn’t (note that some POSIX shells appear to do this for hash too).
  • Many operating systems make which do custom and evil stuff like change the output or even hook into the package manager.

So, don’t use which. Instead use one of these:

(Minor side-note: some will suggest 2&gt;&amp;- is the same 2&gt;/dev/null but shorter – this is untrue. 2&gt;&amp;- closes FD 2 which causes an error in the program when it tries to write to stderr, which is very different from successfully writing to it and discarding the output (and dangerous!))

If your hash bang is /bin/sh then you should care about what POSIX says. type and hash’s exit codes aren’t terribly well defined by POSIX, and hash is seen to exit successfully when the command doesn’t exist (haven’t seen this with type yet). command’s exit status is well defined by POSIX, so that one is probably the safest to use.

If your script uses bash though, POSIX rules don’t really matter anymore and both type and hash become perfectly safe to use. type now has a -P to search just the PATH and hash has the side-effect that the command’s location will be hashed (for faster lookup next time you use it), which is usually a good thing since you probably check for its existence in order to actually use it.

As a simple example, here’s a function that runs gdate if it exists, otherwise date:

Answer 2 (score 459)

The following is a portable way to check whether a command exists in $PATH and is executable:

Example:

The executable check is needed because bash returns a non-executable file if no executable file with that name is found in $PATH.

Also note that if a non-executable file with the same name as the executable exists earlier in $PATH, dash returns the former, even though the latter would be executed. This is a bug and is in violation of the POSIX standard. [Bug report] [Standard]

In addition, this will fail if the command you are looking for has been defined as an alias.

Answer 3 (score 199)

I agree with lhunath to discourage use of which, and his solution is perfectly valid for BASH users. However, to be more portable, command -v shall be used instead:

Command command is POSIX compliant, see here for its specification: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/command.html

Note: type is POSIX compliant, but type -P is not.

57: Syntax for a single-line Bash infinite while loop (score 601934 in 2017)

Question

I am having trouble coming up with the right combination of semicolons and/or braces. I’d like to do this, but as a one-liner from the command line:

Answer accepted (score 1121)

By the way, if you type it as a multiline (as you are showing) at the command prompt and then call the history with arrow up, you will get it on a single line, correctly punctuated.

Answer 2 (score 159)

It’s also possible to use sleep command in while’s condition. Making one-liner looking more clean imho.

Answer 3 (score 67)

Colon is always “true”:

58: Ubuntu says “bash: ./program Permission denied” (score 599338 in 2017)

Question

I am running Ubuntu on computer 1 and computer 2. I compiled a C++ program on computer 1, and I can execute it from the terminal using ./program_name. It runs fine.

However, when I try to do this on computer 2, it says: bash: ./program_name: permission denied

What’s wrong and what can I do about it?

Answer accepted (score 349)

chmod u+x program_name. Then execute it.

If that does not work, copy the program from the USB device to a native volume on the system. Then chmod u+x program_name on the local copy and execute that.

Unix and Unix-like systems generally will not execute a program unless it is marked with permission to execute. The way you copied the file from one system to another (or mounted an external volume) may have turned off execute permission (as a safety feature). The command chmod u+x name adds permission for the user that owns the file to execute it.

That command only changes the permissions associated with the file; it does not change the security controls associated with the entire volume. If it is security controls on the volume that are interfering with execution (for example, a noexec option may be specified for a volume in the Unix fstab file, which says not to allow execute permission for files on the volume), then you can remount the volume with options to allow execution. However, copying the file to a local volume may be a quicker and easier solution.

Answer 2 (score 28)

Try this:

Answer 3 (score 10)

Sounds like you don’t have the execute flag set on the file permissions, try:

59: How do I test if a variable is a number in Bash? (score 596633 in 2017)

Question

I just can’t figure out how do I make sure an argument passed to my script is a number or not.

All I want to do is something like this:

Any help?

Answer accepted (score 720)

One approach is to use a regular expression, like so:

If the value is not necessarily an integer, consider amending the regex appropriately; for instance:

…or, to handle numbers with a sign:

Answer 2 (score 257)

Without bashisms (works even in the System V sh),

This rejects empty strings and strings containing non-digits, accepting everything else.

Negative or floating-point numbers need some additional work. An idea is to exclude - / . in the first “bad” pattern and add more “bad” patterns containing the inappropriate uses of them (?*-* / *.*.*)

Answer 3 (score 170)

The following solution can also be used in basic shells such as Bourne without the need for regular expressions. Basically any numeric value evaluation operations using non-numbers will result in an error which will be implicitly considered as false in shell:

as in:

You can can also test for $? the return code of the operation which is more explicit:

Redirection of standard error is there to hide the “integer expression expected” message that bash prints out in case we do not have a number.

CAVEATS (thanks to the comments below):

  • Numbers with decimal points are not identified as valid “numbers”
  • Using [[ ]] instead of [ ] will always evaluate to true
  • Most non-Bash shells will always evaluate this expression as true
  • The behavior in Bash is undocumented and may therefore change without warning
  • If the value includes spaces after the number (e.g. “1 a”) produces error, like bash: [[: 1 a: syntax error in expression (error token is "a")
  • If the value is the same as var-name (e.g. i=“i”), produces error, like bash: [[: i: expression recursion level exceeded (error token is "i")

60: Open and write data to text file using Bash? (score 586748 in 2019)

Question

How can I write data to a text file automatically by shell scripting in Linux?

I was able to open the file. However, I don’t know how to write data to it.

Answer accepted (score 338)

Answer 2 (score 126)

Answer 3 (score 52)

You can redirect the output of a command to a file:

or append to it

If you want to write directly the command is echo 'text'

61: How to run the sftp command with a password from Bash script? (score 581214 in 2017)

Question

I need to transfer a log file to a remote host using sftp from a Linux host. I have been provided credentials for the same from my operations group. However, since I don’t have control over other host, I cannot generate and share RSA keys with the other host.

So is there a way to run the sftp command (with the username/password provided) from inside the Bash script through a cron job?

I found a similar Stack Overflow question, Specify password to sftp in a Bash script, but there was no satisfactory answer to my problem.

Answer accepted (score 167)

You have a few options other than using public key authentication:

  1. Use keychain
  2. Use sshpass (less secured but probably that meets your requirement)
  3. Use expect (least secured and more coding needed)

If you decide to give sshpass a chance here is a working script snippet to do so:

Answer 2 (score 89)

Another way would be to use lftp:

The disadvantage of this method is that other users on the computer can read the password from tools like ps and that the password can become part of your shell history.

A more secure alternative which is available since LFTP 4.5.0 is setting the LFTP_PASSWORDenvironment variable and executing lftp with --env-password. Here’s a full example:

LFTP also includes a cool mirroring feature (can include delete after confirmed transfer ‘–Remove-source-files’):

Answer 3 (score 47)

Expect is a great program to use.

On Ubuntu install it with:

On a CentOS Machine install it with:

Lets say you want to make a connection to a sftp server and then upload a local file from your local machine to the remote sftp server

This opens a sftp connection with your password to the server.

Then it goes to the directory where you want to upload your file, in this case “logdirectory”

This uploads a log file from the local directory found at /var/log/ with the files name being file.log to the “logdirectory” on the remote server

62: In a Bash script, how can I exit the entire script if a certain condition occurs? (score 580639 in 2017)

Question

I’m writing a script in Bash to test some code. However, it seems silly to run the tests if compiling the code fails in the first place, in which case I’ll just abort the tests.

Is there a way I can do this without wrapping the entire script inside of a while loop and using breaks? Something like a dun dun dun goto?

Answer accepted (score 702)

Try this statement:

Replace 1 with appropriate error codes. See also Exit Codes With Special Meanings.

Answer 2 (score 651)

Use set -e

The script will terminate after the first line that fails (returns nonzero exit code). In this case, command-that-fails2 will not run.

If you were to check the return status of every single command, your script would look like this:

With set -e it would look like:

Any command that fails will cause the entire script to fail and return an exit status you can check with $?. If your script is very long or you’re building a lot of stuff it’s going to get pretty ugly if you add return status checks everywhere.

Answer 3 (score 206)

A bad-arse SysOps guy once taught me the Three-Fingered Claw technique:

These functions are *NIX OS and shell flavor-robust. Put them at the beginning of your script (bash or otherwise), try() your statement and code on.

Explanation

(based on flying sheep comment).

  • yell: print the script name and all arguments to stderr:

    • $0 is the path to the script ;
    • $* are all arguments.
    • &gt;&amp;2 means &gt; redirect stdout to & pipe 2. pipe 1 would be stdout itself.
  • die does the same as yell, but exits with a non-0 exit status, which means “fail”.

  • try uses the || (boolean OR), which only evaluates the right side if the left one failed.

63: How to iterate over arguments in a Bash script (score 574001 in 2017)

Question

I have a complex command that I’d like to make a shell/bash script of. I can write it in terms of $1 easily:

I want to be able to pass multiple input names to the script. What’s the right way to do it?

And, of course, I want to handle filenames with spaces in them.

Answer accepted (score 1351)

Use "$@" to represent all the arguments:

This will iterate over each argument and print it out on a separate line. $@ behaves like $* except that when quoted the arguments are broken up properly if there are spaces in them:

Answer 2 (score 227)

Rewrite of a now-deleted answer by VonC.

Robert Gamble’s succinct answer deals directly with the question. This one amplifies on some issues with filenames containing spaces.

See also: ${1:+"$@"} in /bin/sh

Basic thesis: "$@" is correct, and $* (unquoted) is almost always wrong. This is because "$@" works fine when arguments contain spaces, and works the same as $* when they don’t. In some circumstances, "$*" is OK too, but "$@" usually (but not always) works in the same places. Unquoted, $@ and $* are equivalent (and almost always wrong).

So, what is the difference between $*, $@, "$*", and "$@"? They are all related to ‘all the arguments to the shell’, but they do different things. When unquoted, $* and $@ do the same thing. They treat each ‘word’ (sequence of non-whitespace) as a separate argument. The quoted forms are quite different, though: "$*" treats the argument list as a single space-separated string, whereas "$@" treats the arguments almost exactly as they were when specified on the command line. "$@" expands to nothing at all when there are no positional arguments; "$*" expands to an empty string — and yes, there’s a difference, though it can be hard to perceive it. See more information below, after the introduction of the (non-standard) command al.

Secondary thesis: if you need to process arguments with spaces and then pass them on to other commands, then you sometimes need non-standard tools to assist. (Or you should use arrays, carefully: "${array[@]}" behaves analogously to "$@".)

Example:

Why doesn’t that work? It doesn’t work because the shell processes quotes before it expands variables. So, to get the shell to pay attention to the quotes embedded in $var, you have to use eval:

This gets really tricky when you have file names such as “He said,”Don’t do this!“” (with quotes and double quotes and spaces).

The shells (all of them) do not make it particularly easy to handle such stuff, so (funnily enough) many Unix programs do not do a good job of handling them. On Unix, a filename (single component) can contain any characters except slash and NUL '\0'. However, the shells strongly encourage no spaces or newlines or tabs anywhere in a path names. It is also why standard Unix file names do not contain spaces, etc.

When dealing with file names that may contain spaces and other troublesome characters, you have to be extremely careful, and I found long ago that I needed a program that is not standard on Unix. I call it escape (version 1.1 was dated 1989-08-23T16:01:45Z).

Here is an example of escape in use - with the SCCS control system. It is a cover script that does both a delta (think check-in) and a get (think check-out). Various arguments, especially -y (the reason why you made the change) would contain blanks and newlines. Note that the script dates from 1992, so it uses back-ticks instead of $(cmd ...) notation and does not use #!/bin/sh on the first line.

(I would probably not use escape quite so thoroughly these days - it is not needed with the -e argument, for example - but overall, this is one of my simpler scripts using escape.)

The escape program simply outputs its arguments, rather like echo does, but it ensures that the arguments are protected for use with eval (one level of eval; I do have a program which did remote shell execution, and that needed to escape the output of escape).

I have another program called al that lists its arguments one per line (and it is even more ancient: version 1.1 dated 1987-01-27T14:35:49). It is most useful when debugging scripts, as it can be plugged into a command line to see what arguments are actually passed to the command.

[Added: And now to show the difference between the various "$@" notations, here is one more example:

Notice that nothing preserves the original blanks between the * and */* on the command line. Also, note that you can change the ‘command line arguments’ in the shell by using:

This sets 4 options, ‘-new’, ‘-opt’, ‘and’, and ‘arg with space’.
]

Hmm, that’s quite a long answer - perhaps exegesis is the better term. Source code for escape available on request (email to firstname dot lastname at gmail dot com). The source code for al is incredibly simple:

That’s all. It is equivalent to the test.sh script that Robert Gamble showed, and could be written as a shell function (but shell functions didn’t exist in the local version of Bourne shell when I first wrote al).

Also note that you can write al as a simple shell script:

The conditional is needed so that it produces no output when passed no arguments. The printf command will produce a blank line with only the format string argument, but the C program produces nothing.

Answer 3 (score 124)

Note that Robert’s answer is correct, and it works in sh as well. You can (portably) simplify it even further:

is equivalent to:

I.e., you don’t need anything!

Testing ($ is command prompt):

I first read about this in Unix Programming Environment by Kernighan and Pike.

In bash, help for documents this:

for NAME [in WORDS ... ;] do COMMANDS; done

If 'in WORDS ...;' is not present, then 'in "$@"' is assumed.

64: Why do you need to put #!/bin/bash at the beginning of a script file? (score 573690 in 2018)

Question

I have made Bash scripts before and they all ran fine without this at the beginning. What’s the point of putting it in? Would things be any different?

Also, how do you pronounce #? I know that ! is pronounced as “bang.”

How is #! pronounced?

Answer accepted (score 398)

It’s a convention so the *nix shell knows what kind of interpreter to run.

For example, older flavors of ATT defaulted to sh (the Bourne shell), while older versions of BSD defaulted to csh (the C shell).

Even today (where most systems run bash, the “Bourne Again Shell”), scripts can be in bash, python, perl, ruby, PHP, etc, etc. For example, you might see #!/bin/perl or #!/bin/perl5.

PS: The exclamation mark (!) is affectionately called “bang”. The shell comment symbol (#) is sometimes called “hash”.

PPS: Remember - under *nix, associating a suffix with a file type is merely a convention, not a “rule”. An executable can be a binary program, any one of a million script types and other things as well. Hence the need for #!/bin/bash.

Answer 2 (score 124)

To be more precise the shebang #!, when it is the first two bytes of an executable (x mode) file, is interpreted by the execve(2) system call (which execute programs). But POSIX specification for execve don’t mention the shebang.

It must be followed by a file path of an interpreter executable (which BTW could even be relative, but most often is absolute).

A nice trick (or perhaps not so nice one) to find an interpreter (e.g. python) in the user’s $PATH is to use the env program (always at /usr/bin/env on all Linux) like e.g.

Any ELF executable can be an interpreter. You could even use #!/bin/cat or #!/bin/true if you wanted to! (but that would be often useless)

Answer 3 (score 47)

It’s called a shebang. In unix-speak, # is called sharp (like in music) or hash (like hashtags on twitter), and ! is called bang. (You can actually reference your previous shell command with !!, called bang-bang). So when put together, you get haSH-BANG, or shebang.

The part after the #! tells Unix what program to use to run it. If it isn’t specified, it will try with bash (or sh, or zsh, or whatever your $SHELL variable is) but if it’s there it will use that program. Plus, # is a comment in most languages, so the line gets ignored in the subsequent execution.

65: How to obtain the number of CPUs/cores in Linux from the command line? (score 571310 in 2015)

Question

I have this script, but I do not know how to get the last element in the printout:

The last element should be the number of CPUs, minus 1.

Answer accepted (score 595)

or simply

which will count the number of lines starting with “processor” in /proc/cpuinfo

For systems with hyper-threading, you can use

which should return (for example) 8 (whereas the command above would return 16)

Answer 2 (score 596)

Processing the contents of /proc/cpuinfo is needlessly baroque. Use nproc which is part of coreutils, so it should be available on most Linux installs.

Command nproc prints the number of processing units available to the current process, which may be less than the number of online processors.

To find the number of all installed cores/processors use nproc --all

On my 8-core machine:

Answer 3 (score 243)

The most portable solution I have found is the getconf command:

This works on both Linux and Mac OS X. Another benefit of this over some of the other approaches is that getconf has been around for a long time. Some of the older Linux machines I have to do development on don’t have the nproc or lscpu commands available, but they have getconf.

Editor’s note: While the getconf utility is POSIX-mandated, the specific _NPROCESSORS_ONLN and _NPROCESSORS_CONF values are not. That said, as stated, they work on Linux platforms as well as on macOS; on FreeBSD/PC-BSD, you must omit the leading _.

66: How to redirect output to a file and stdout (score 560328 in 2014)

Question

In bash, calling foo would display any output from that command on the stdout.

Calling foo &gt; output would redirect any output from that command to the file specified (in this case ‘output’).

Is there a way to redirect output to a file and have it display on stdout?

Answer accepted (score 1200)

The command you want is named tee:

For example, if you only care about stdout:

If you want to include stderr, do:

2&gt;&amp;1 redirects channel 2 (stderr/standard error) into channel 1 (stdout/standard output), such that both is written as stdout. It is also directed to the given output file as of the tee command.

Furthermore, if you want to append to the log file, use tee -a as:

Answer 2 (score 481)

2&gt;&amp;1 dumps the stderr and stdout streams. tee outfile takes the stream it gets and writes it to the screen and to the file “outfile”.

This is probably what most people are looking for. The likely situation is some program or script is working hard for a long time and producing a lot of output. The user wants to check it periodically for progress, but also wants the output written to a file.

The problem (especially when mixing stdout and stderr streams) is that there is reliance on the streams being flushed by the program. If, for example, all the writes to stdout are not flushed, but all the writes to stderr are flushed, then they’ll end up out of chronological order in the output file and on the screen.

It’s also bad if the program only outputs 1 or 2 lines every few minutes to report progress. In such a case, if the output was not flushed by the program, the user wouldn’t even see any output on the screen for hours, because none of it would get pushed through the pipe for hours.

Update: The program unbuffer, part of the expect package, will solve the buffering problem. This will cause stdout and stderr to write to the screen and file immediately and keep them in sync when being combined and redirected to tee. E.g.:

Answer 3 (score 117)

Another way that works for me is,

as shown in gnu bash manual

Example:

If ‘|&’ is used, command1’s standard error, in addition to its standard output, is connected to command2’s standard input through the pipe; it is shorthand for 2>&1 |. This implicit redirection of the standard error to the standard output is performed after any redirections specified by the command.

For more information, refer redirection

67: In bash, how can I check if a string begins with some value? (score 554528 in 2016)

Question

I would like to check if a string begins with “node” e.g. “node001”. Something like

How can I do it correctly？

I further need to combine expressions to check if HOST is either “user1” or begins with “node”

How to do it correctly?

Answer accepted (score 941)

This snippet on the Advanced Bash Scripting Guide says:

So you had it nearly correct; you needed double brackets, not single brackets.

With regards to your second question, you can write it this way:

Which will echo

Bash’s if syntax is hard to get used to (IMO).

Answer 2 (score 183)

If you’re using a recent bash (v3+), I suggest bash regex comparison operator =~, i.e.

To match this or that in a regex use |, i.e.

Note - this is ‘proper’ regular expression syntax.

  • user* means use and zero-or-more occurrences of r, so use and userrrr will match.
  • user.* means user and zero-or-more occurrences of any character, so user1, userX will match.
  • ^user.* means match the pattern user.* at the begin of $HOST.

If you’re not familiar with regular expression syntax, try referring to this resource.

Note - it’s better if you ask each new question as a new question, it makes stackoverflow tidier and more useful. You can always include a link back to a previous question for reference.

Answer 3 (score 118)

I always try to stick with POSIX sh instead of using bash extensions, since one of the major points of scripting is portability. (Besides connecting programs, not replacing them)

In sh, there is an easy way to check for an “is-prefix” condition.

Given how old, arcane and crufty sh is (and bash is not the cure: It’s more complicated, less consistent and less portable), I’d like to point out a very nice functional aspect: While some syntax elements like case are built-in, the resulting constructs are no different than any other job. They can be composed in the same way:

Or even shorter

Or even shorter (just to present ! as a language element – but this is bad style now)

If you like being explicit, build your own language element:

Isn’t this actually quite nice?

And since sh is basically only jobs and string-lists (and internally processes, out of which jobs are composed), we can now even do some light functional programming:

This is elegant. Not that I’d advocate using sh for anything serious – it breaks all too quickly on real world requirements (no lambdas, so must use strings. But nesting function calls with strings is not possible, pipes are not possible…)

68: ls command: how can I get a recursive full-path listing, one line per file? (score 549377 in 2013)

Question

How can I get ls to spit out a flat list of recursive one-per-line paths?

For example, I just want a flat listing of files with their full paths:

ls -a1 almost does what I need, but I do not want path fragments, I want full paths.

Answer accepted (score 362)

If you really want to use ls, then format its output using awk:

Answer 2 (score 527)

Use find:

If you want files only (omit directories, devices, etc):

Answer 3 (score 71)

ls -ld $(find .)

if you want to sort your output by modification time:

ls -ltd $(find .)

69: How do I run a shell script without using “sh” or “bash” commands? (score 530846 in 2013)

Question

I have a shell script which I want to run without using the “sh” or “bash” commands. For example:

Instead of: sh script.sh

I want to use: script.sh

How can I do this?

P.S. (i) I don’t use shell script much and I tried reading about aliases, but I did not understand how to use them.

  1. I also read about linking the script with another file in the PATH variables. I am using my university server and I don’t have permissions to create a file in those locations.

Answer accepted (score 414)

Add a “shebang” at the top of your file:

And make your file executable (chmod +x script.sh).

Finally, modify your path to add the directory where your script is located:

(typically, you want $HOME/bin for storing your own scripts)

Answer 2 (score 66)

These are some of the prerequisites of directly using the script name:

  1. Add the she-bang (#!/bin/bash) line at the very top.
  2. Using chmod u+x scriptname make the script executable. (where scriptname is the name of your script)
  3. Place the script under /usr/local/bin folder.
  4. Run the script using just the name of the script.

Note: The reason I suggested to place it under /usr/local/bin folder is because most likely that will be path already added to your PATH variable.

Update:

If you don’t have access to the /usr/local/bin folder then do the following:

  1. Create a folder in your home directory and let’s call it myscripts.
  2. Do ls -lart on your home directory, to identify the start-up script your shell is using. It should either be .profile or .bashrc.
  3. Once you have identified the start up script, add the following line in your script - export set PATH=$PATH:~/myscript.
  4. Once added, source your start-up script or log out and log back in.
  5. Execute your script using scriptname.

Answer 3 (score 20)

Just make sure it is executable, using chmod +x. By default, the current directory is not on your PATH, so you will need to execute it as ./script.sh - or otherwise reference it by a qualified path. Alternatively, if you truly need just script.sh, you would need to add it to your PATH. (You may not have access to modify the system path, but you can almost certainly modify the PATH of your own current environment.) This also assumes that your script starts with something like #!/bin/sh.

You could also still use an alias, which is not really related to shell scripting but just the shell, and is simple as:

Which would allow you to use just simply script.sh (literally - this won’t work for any other *.sh file) instead of sh script.sh.

70: How to test if string exists in file with Bash? (score 526448 in 2018)

Question

I have a file that contains directory names:

my_list.txt :

I’d like to check in Bash before I’ll add a directory name if that name already exists in the file.

Answer accepted (score 584)

The exit status is 0 (true) if the name was found, 1 (false) if not, so:

Explanation

Here are the relevant sections of the man page for grep:

-F, --fixed-strings

        Interpret PATTERN as a list of fixed strings, separated by newlines, any of which is to be matched.

-x, --line-regexp

        Select only those matches that exactly match the whole line.

-q, --quiet, --silent

        Quiet; do not write anything to standard output. Exit immediately with zero status if any match is found, even if an error was detected. Also see the -s or --no-messages option.
Error handling

As rightfully pointed out in the comments, the above approach silently treats error cases as if the string was found. If you want to handle errors in a different way, you’ll have to omit the -q option, and detect errors based on the exit status:

Normally, the exit status is 0 if selected lines are found and 1 otherwise. But the exit status is 2 if an error occurred, unless the -q or --quiet or --silent option is used and a selected line is found. Note, however, that POSIX only mandates, for programs such as grep, cmp, and diff, that the exit status in case of error be greater than 1; it is therefore advisable, for the sake of portability, to use logic that tests for this general condition instead of strict equality with 2.

To suppress the normal output from grep, you can redirect it to /dev/null. Note that standard error remains undirected, so any error messages that grep might print will end up on the console as you’d probably want.

To handle the three cases, we can use a case statement:

Answer 2 (score 86)

Regarding the following solution:

In case you are wondering (as I did) what -Fxq means in plain English:

  • F: Affects how PATTERN is interpreted (fixed string instead of a regex)
  • x: Match whole line
  • q: Shhhhh… minimal printing

From the man file:

Answer 3 (score 38)

Three methods in my mind:

  1. Short test for a name in a path (I’m not sure this might be your case)


2) Short test for a string in a file


3) Longer bash script using regex:

This should be quicker if you have to test multiple string on a file content using a loop for example changing the regex at any cicle.

71: Add a new element to an array without specifying the index in Bash (score 525121 in 2017)

Question

Is there a way to do something like PHPs $array[] = 'foo'; in bash vs doing:

Answer accepted (score 1405)

Yes there is:

Bash Reference Manual:

In the context where an assignment statement is assigning a value to a shell variable or array index (see Arrays), the ‘+=’ operator can be used to append to or add to the variable’s previous value.

Answer 2 (score 72)

As Dumb Guy points out, it’s important to note whether the array starts at zero and is sequential. Since you can make assignments to and unset non-contiguous indices ${#array[@]} is not always the next item at the end of the array.

Here’s how to get the last index:

That illustrates how to get the last element of an array. You’ll often see this:

As you can see, because we’re dealing with a sparse array, this isn’t the last element. This works on both sparse and contiguous arrays, though:

Answer 3 (score 47)

72: Create timestamp variable in bash script (score 514810 in 2013)

Question

I am trying to create a timestamp variable in a shell script to make the logging a little easier. I want to create the variable at the beginning of the script and have it print out the current time whenever I issue echo $timestamp. It proving to be more difficult then I thought. Here are some things I’ve tried:

timestamp="(date +"%T")" echo prints out (date +"%T")

timestamp="$(date +"%T")" echo prints the time when the variable was initialized.

Other things I’ve tried are just slight variations that didn’t work any better. Does anyone know how to accomplish what I’m trying to do?

Answer accepted (score 267)

In order to get the current timestamp and not the time of when a fixed variable is defined, the trick is to use a function and not a variable:

If you don’t like the format given by the %T specifier you can combine the other time conversion specifiers accepted by date. For GNU date, you can find the complete list of these specifiers in the official documentation here: https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html#Time-conversion-specifiers

Answer 2 (score 511)

If you want to get unix timestamp, then you need to use:

%T will give you just the time; same as %H:%M:%S (via http://www.cyberciti.biz/faq/linux-unix-formatting-dates-for-display/)

Answer 3 (score 52)

73: How to run .sh on Windows Command Prompt? (score 510971 in )

Question

How can I run .sh on Windows 7 Command Prompt? I always get this error when I try to run this line in it,

error,

or,

error,

Any ideas what have I missed?

Here the screen grab, enter image description here

Answer accepted (score 51)

The error message indicates that you have not installed bash, or it is not in your PATH.

The top Google hit is http://win-bash.sourceforge.net/ but you also need to understand that most Bash scripts expect a Unix-like environment; so just installing Bash is probably unlikely to allow you to run a script you found on the net, unless it was specifically designed for this particular usage scenario. The usual solution to that is https://www.cygwin.com/ but there are many possible alternatives, depending on what exactly it is that you want to accomplish.

If Windows is not central to your usage scenario, installing a free OS (perhaps virtualized) might be the simplest way forward.

The second error message is due to the fact that Windows nominally accepts forward slash as a directory separator, but in this context, it is being interpreted as a switch separator. In other words, Windows parses your command line as app /build /build.sh (or, to paraphrase with Unix option conventions, app --build --build.sh). You could try app\build\build.sh but it is unlikely to work, because of the circumstances outlined above.

Answer 2 (score 111)

Install GIT. During installation of GIT, add GIT Bash to windows context menu by selecting its option. After installation right click in your folder select GIT Bash Here (see attached pic) and use your sh command like for example:

enter image description here

Answer 3 (score 47)

The most common way to run a .sh file is using the sh command:

other good option is installing CygWin

in Windows the home is located in:

for example i execute my my-script-test.sh file using the bash command as:

74: Changing default shell in Linux (score 504666 in 2019)

Question

How is it possible to change the default shell? The env command currently says:

and I want to change that to Bash.

Answer accepted (score 391)

Try linux command chsh.

The detailed command is chsh -s /bin/bash. It will prompt you to enter your password. Your default login shell is /bin/bash now. You must log out and log back in to see this change.

The following is quoted from man page:

The chsh command changes the user login shell. This determines the name of the users initial login command. A normal user may only change the login shell for her own account, the superuser may change the login shell for any account

This command will change the default login shell permanently.

Note: If your user account is remote such as on Kerberos authentication (e.g. Enterprise RHEL) then you will not be able to use chsh.

Answer 2 (score 142)

You can change the passwd file directly for the particular user or use the below command

Then log out and log in

Answer 3 (score 16)

You should have a ‘skeleton’ somewhere in /etc, probably /etc/skeleton, or check the default settings, probably /etc/default or something. Those are scripts that define standard environment variables getting set during a login.

If it is just for your own account: check the (hidden) file ~/.profile and ~/.login. Or generate them, if they don’t exist. These are also evaluated by the login process.

75: echo that outputs to stderr (score 493483 in 2014)

Question

Is there a standard Bash tool that acts like echo but outputs to stderr rather than stdout?

I know I can do echo foo 1&gt;&amp;2 but it’s kinda ugly and, I suspect, error prone (e.g. more likely to get edited wrong when things change).

Answer accepted (score 1283)

You could do this, which facilitates reading:

&gt;&amp;2 copies file descriptor #2 to file descriptor #1. Therefore, after this redirection is performed, both file descriptors will refer to the same file: the one file descriptor #2 was originally referring to. For more information see the Bash Hackers Illustrated Redirection Tutorial.

Answer 2 (score 398)

You could define a function:

This would be faster than a script and have no dependencies.

Camilo Martin’s bash specific suggestion uses a “here string” and will print anything you pass to it, including arguments (-n) that echo would normally swallow:

Glenn Jackman’s solution also avoids the argument swallowing problem:

Answer 3 (score 230)

Since 1 is the standard output, you do not have to explicitly name it in front of an output redirection like &gt; but instead can simply type:

echo This message goes to stderr >&2

Since you seem to be worried that 1&gt;&amp;2 will be difficult for you to reliably type, the elimination of the redundant 1 might be a slight encouragement to you!

76: Bash if statement with multiple conditions throws an error (score 493173 in 2017)

Question

I’m trying to write a script that will check two error flags, and in case one flag (or both) are changed it’ll echo– error happened. My script:

Basically, it should be, something along:

The error I get is:

Are my brackets messed up?

Answer accepted (score 211)

Use -a (for and) and -o (for or) operations.

tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html

Update

Actually you could still use &amp;&amp; and || with the -eq operation. So your script would be like this:

Although in your case you can discard the last two expressions and just stick with one or operation like this:

Answer 2 (score 58)

You can use either [[ or (( keyword. When you use [[ keyword, you have to use string operators such as -eq, -lt. I think, (( is most preferred for arithmetic, because you can directly use operators such as ==, &lt; and &gt;.

Using [[ operator

Using (( operator

Do not use -a or -o operators Since it is not Portable.

Answer 3 (score 7)

Please try following

77: How to use “:” as awk field separator? (score 491168 in 2010)

Question

Given following command:

why does awk output:

Answer accepted (score 348)

“-F” is a command line argument not awk syntax, try:

Answer 2 (score 59)

If you want to do it programatically, you can use the FS variable:

Note that if you change it in the main loop rather than the BEGIN loop, it takes affect for the next line read in, since the current line has already been split.

Answer 3 (score 27)

You have multiple ways to set : as separator:

All of them are equivalent and for an will return 1 for a sample input “1:2:3”:

78: How can I remove the first line of a text file using bash/sed script? (score 486346 in 2011)

Question

I need to repeatedly remove the first line from a huge text file using a bash script.

Right now I am using sed -i -e "1d" $FILE - but it takes around a minute to do the deletion.

Is there a more efficient way to accomplish this?

Answer accepted (score 948)

Try tail:

-n x: Just print the last x lines. tail -n 5 would give you the last 5 lines of the input. The + sign kind of inverts the argument and make tail print anything but the first x-1 lines. tail -n +1 would print the whole file, tail -n +2 everything but the first line, etc.

GNU tail is much faster than sed. tail is also available on BSD and the -n +2 flag is consistent across both tools. Check the FreeBSD or OS X man pages for more.

The BSD version can be much slower than sed, though. I wonder how they managed that; tail should just read a file line by line while sed does pretty complex operations involving interpreting a script, applying regular expressions and the like.

Note: You may be tempted to use

but this will give you an empty file. The reason is that the redirection (&gt;) happens before tail is invoked by the shell:

  1. Shell truncates file $FILE
  2. Shell creates a new process for tail
  3. Shell redirects stdout of the tail process to $FILE
  4. tail reads from the now empty $FILE

If you want to remove the first line inside the file, you should use:

The &amp;&amp; will make sure that the file doesn’t get overwritten when there is a problem.

Answer 2 (score 152)

You can use -i to update the file without using ‘>’ operator. The following command will delete the first line from the file and save it to the file.

Answer 3 (score 71)

For those who are on SunOS which is non-GNU, the following code will help:

79: ./configure : /bin/sh^M : bad interpreter (score 480455 in 2010)

Question

I’ve been trying to install lpng142 on my fed 12 system. Seems like a problem to me. I get this error

How do I fix this? The /etc/fstab file:

Answer accepted (score 717)

To fix, open your script with vi or vim and enter in vi command mode (key Esc), then type this:

Finally save it

:x! or :wq!

Answer 2 (score 360)

Looks like you have a dos line ending file. The clue is the ^M.

You need to re-save the file using Unix line endings.

You might have a dos2unix command line utility that will also do this for you.

Answer 3 (score 94)

Or if you want to do this with a script:

80: How to do a recursive find/replace of a string with awk or sed? (score 478857 in 2018)

Question

How do I find and replace every occurrence of:

with

in every text file under the /home/www/ directory tree recursively?

Answer 2 (score 809)

From man find:

-print0 (GNU find only) tells find to use the null character (\0) instead of whitespace as the output delimiter between pathnames found. This is a safer option if your files can contain blanks or other special characters. It is recommended to use the -print0 argument to find if you use -exec &lt;command&gt; or xargs (the -0 argument is needed in xargs.)

Answer 3 (score 238)

Note: Do not run this command on a folder including a git repo - changes to .git could corrupt your git index.

Compared to other answers here, this is simpler than most and uses sed instead of perl, which is what the original question asked for.

81: How to get the primary IP address of the local machine on Linux and OS X? (score 477129 in 2019)

Question

I am looking for a command line solution that would return me the primary (first) IP address of the localhost, other than 127.0.0.1

The solution should work at least for Linux (Debian and RedHat) and OS X 10.7+

I am aware that ifconfig is available on both but its output is not so consistent between these platforms.

Answer accepted (score 447)

Use grep to filter IP address from ifconfig:

ifconfig | grep -Eo 'inet (addr:)?([0-9]*\.){3}[0-9]*' | grep -Eo '([0-9]*\.){3}[0-9]*' | grep -v '127.0.0.1'

Or with sed:

ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'

If you are only interested in certain interfaces, wlan0, eth0, etc. then:

ifconfig wlan0 | ...

You can alias the command in your .bashrc to create your own command called myip for instance.

alias myip="ifconfig | sed -En 's/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p'"

A much simpler way is hostname -I (hostname -i for older versions of hostname but see comments). However, this is on Linux only.

Answer 2 (score 214)

For linux machines (not OS X) :

Answer 3 (score 210)

The following will work on Linux but not OSX.

This doesn’t rely on DNS at all, and it works even if /etc/hosts is not set correctly (1 is shorthand for 1.0.0.0):

or avoiding awk and using Google’s public DNS at 8.8.8.8 for obviousness:

A less reliable way: (see comment below)

82: How to append output to the end of a text file (score 475123 in 2018)

Question

How do I append the output of a command to the end of a text file?

Answer 2 (score 582)

Use &gt;&gt; instead of &gt; when directing output to a file:

If file_to_append_to does not exist, it will be created.

Example:

Answer 3 (score 74)

To append a file use &gt;&gt;

then the output should be

To overwrite a file use &gt;

then the out put is

hello tom

83: Given two directory trees, how can I find out which files differ? (score 474513 in )

Question

If I want find the differences between two directory trees, I usually just execute:

This outputs exactly what the differences are between corresponding files. I’m interested in just getting a list of corresponding files whose content differs. I assumed that this would simply be a matter of passing a command line option to diff, but I couldn’t find anything on the man page.

Any suggestions?

Answer accepted (score 1052)

You said Linux, so you luck out (at least it should be available, not sure when it was added):

Should do what you need.

If you also want to see differences for files that may not exist in either directory:

Answer 2 (score 284)

The command I use is:

It is exactly the same as Mark’s :) But his answer bothered me as it uses different types of flags, and it made me look twice. Using Mark’s more verbose flags it would be:

I apologise for posting when the other answer is perfectly acceptable. Could not stop myself… working on being less pedantic.

Answer 3 (score 96)

I like to use git diff --no-index dir1/ dir2/, because it can show the differences in color (if you have that option set in your git config) and because it shows all of the differences in a long paged output using “less”.

84: How can I exclude all “permission denied” messages from “find”? (score 470044 in 2015)

Question

I need to hide all permission denied messages from:

I am experimenting when such message arises. I need to gather all folders and files, to which it does not arise.

Is it possible to direct the permission levels to the files_and_folders file?

How can I hide the errors at the same time?

Answer accepted (score 246)

Note:
* This answer probably goes deeper than the use case warrants, and find 2&gt;/dev/null may be good enough in many situations. It may still be of interest for a cross-platform perspective and for its discussion of some advanced shell techniques in the interest of finding a solution that is as robust as possible, even though the cases guarded against may be largely hypothetical.
* If your system is configured to show localized error messages, prefix the find calls below with LC_ALL=C (LC_ALL=C find ...) to ensure that English messages are reported, so that grep -v 'Permission denied' works as intended. Invariably, however, any error messages that do get displayed will then be in English as well.

If your shell is bash or zsh, there’s a solution that is robust while being reasonably simple, using only POSIX-compliant find features; while bash itself is not part of POSIX, most modern Unix platforms come with it, making this solution widely portable:

Note: There’s a small chance that some of grep’s output may arrive after find completes, because the overall command doesn’t wait for the command inside &gt;(...) to finish. In bash, you can prevent this by appending | cat to the command.

  • &gt;(...) is a (rarely used) output process substitution that allows redirecting output (in this case, stderr output (2&gt;) to the stdin of the command inside &gt;(...).
    In addition to bash and zsh, ksh supports them as well in principle, but trying to combine them with redirection from stderr, as is done here (2&gt; &gt;(...)), appears to be silently ignored (in ksh 93u+).

    • grep -v 'Permission denied' filters out (-v) all lines (from the find command’s stderr stream) that contain the phrase Permission denied and outputs the remaining lines to stderr (&gt;&amp;2).

This approach is:

  • robust: grep is only applied to error messages (and not to a combination of file paths and error messages, potentially leading to false positives), and error messages other than permission-denied ones are passed through, to stderr.

  • side-effect free: find’s exit code is preserved: the inability to access at least one of the filesystem items encountered results in exit code 1 (although that won’t tell you whether errors other than permission-denied ones occurred (too)).

POSIX-compliant solutions:

Fully POSIX-compliant solutions either have limitations or require additional work.

If find’s output is to be captured in a file anyway (or suppressed altogether), then the pipeline-based solution from Jonathan Leffler’s answer is simple, robust, and POSIX-compliant:

Note that the order of the redirections matters: 2&gt;&amp;1 must come first.

Capturing stdout output in a file up front allows 2&gt;&amp;1 to send only error messages through the pipeline, which grep can then unambiguously operate on.

The only downside is that the overall exit code will be the grep command’s, not find’s, which in this case means: if there are no errors at all or only permission-denied errors, the exit code will be 1 (signaling failure), otherwise (errors other than permission-denied ones) 0 - which is the opposite of the intent.
That said, find’s exit code is rarely used anyway, as it often conveys little information beyond fundamental failure such as passing a non-existent path.
However, the specific case of even only some of the input paths being inaccessible due to lack of permissions is reflected in find’s exit code (in both GNU and BSD find): if a permissions-denied error occurs for any of the files processed, the exit code is set to 1.

The following variation addresses that:

Now, the exit code indicates whether any errors other than Permission denied occurred: 1 if so, 0 otherwise.
In other words: the exit code now reflects the true intent of the command: success (0) is reported, if no errors at all or only permission-denied errors occurred.
This is arguably even better than just passing find’s exit code through, as in the solution at the top.

gniourf_gniourf in the comments proposes a (still POSIX-compliant) generalization of this solution using sophisticated redirections, which works even with the default behavior of printing the file paths to stdout:

In short: Custom file descriptor 3 is used to temporarily swap stdout (1) and stderr (2), so that error messages alone can be piped to grep via stdout.

Without these redirections, both data (file paths) and error messages would be piped to grep via stdout, and grep would then not be able to distinguish between error message Permission denied and a (hypothetical) file whose name happens to contain the phrase Permission denied.

As in the first solution, however, the the exit code reported will be grep’s, not find’s, but the same fix as above can be applied.

Notes on the existing answers:

  • There are several points to note about Michael Brux’s answer, find . ! -readable -prune -o -print:

    • It requires GNU find; notably, it won’t work on macOS. Of course, if you only ever need the command to work with GNU find, this won’t be a problem for you.

    • Some Permission denied errors may still surface: find ! -readable -prune reports such errors for the child items of directories for which the current user does have r permission, but lacks x (executable) permission. The reason is that because the directory itself is readable, -prune is not executed, and the attempt to descend into that directory then triggers the error messages. That said, the typical case is for the r permission to be missing.

    • Note: The following point is a matter of philosophy and/or specific use case, and you may decide it is not relevant to you and that the command fits your needs well, especially if simply printing the paths is all you do:

      • If you conceptualize the filtering of the permission-denied error messages a separate task that you want to be able to apply to any find command, then the opposite approach of proactively preventing permission-denied errors requires introducing “noise” into the find command, which also introduces complexity and logical pitfalls.
      • For instance, the most up-voted comment on Michael’s answer (as of this writing) attempts to show how to extend the command by including a -name filter, as follows:
        find . ! -readable -prune -o -name '*.txt'
        This, however, does not work as intended, because the trailing -print action is required (an explanation can be found in this answer). Such subtleties can introduce bugs.

  • The first solution in Jonathan Leffler’s answer, find . 2&gt;/dev/null &gt; files_and_folders, as he himself states, blindly silences all error messages (and the workaround is cumbersome and not fully robust, as he also explains). Pragmatically speaking, however, it is the simplest solution, as you may be content to assume that any and all errors would be permission-related.

  • mist’s answer, sudo find . &gt; files_and_folders, is concise and pragmatic, but ill-advised for anything other than merely printing filenames, for security reasons: because you’re running as the root user, “you risk having your whole system being messed up by a bug in find or a malicious version, or an incorrect invocation which writes something unexpectedly, which could not happen if you ran this with normal privileges” (from a comment on mist’s answer by tripleee).

  • The 2nd solution in viraptor’s answer, find . 2&gt;&amp;1 | grep -v 'Permission denied' &gt; some_file runs the risk of false positives (due to sending a mix of stdout and stderr through the pipeline), and, potentially, instead of reporting non-permission-denied errors via stderr, captures them alongside the output paths in the output file.

Answer 2 (score 533)

Use:

This hides not just the Permission denied errors, of course, but all error messages.

If you really want to keep other possible errors, such as too many hops on a symlink, but not the permission denied ones, then you’d probably have to take a flying guess that you don’t have many files called ‘permission denied’ and try:

If you strictly want to filter just standard error, you can use the more elaborate construction:

The I/O redirection on the find command is: 2&gt;&amp;1 &gt; files_and_folders |. The pipe redirects standard output to the grep command and is applied first. The 2&gt;&amp;1 sends standard error to the same place as standard output (the pipe). The &gt; files_and_folders sends standard output (but not standard error) to a file. The net result is that messages written to standard error are sent down the pipe and the regular output of find is written to the file. The grep filters the standard output (you can decide how selective you want it to be, and may have to change the spelling depending on locale and O/S) and the final &gt;&amp;2 means that the surviving error messages (written to standard output) go to standard error once more. The final redirection could be regarded as optional at the terminal, but would be a very good idea to use it in a script so that error messages appear on standard error.

There are endless variations on this theme, depending on what you want to do. This will work on any variant of Unix with any Bourne shell derivative (Bash, Korn, …) and any POSIX-compliant version of find.

If you wish to adapt to the specific version of find you have on your system, there may be alternative options available. GNU find in particular has a myriad options not available in other versions — see the currently accepted answer for one such set of options.

Answer 3 (score 277)

Use:

or more generally

  • to avoid “Permission denied”
  • AND do NOT suppress (other) error messages
  • AND get exit status 0 (“all files are processed successfully”)

Works with: find (GNU findutils) 4.4.2. Background:

  • The -readable test matches readable files. The ! operator returns true, when test is false. And ! -readable matches not readable directories (&files).
  • The -prune action does not descend into directory.
  • ! -readable -prune can be translated to: if directory is not readable, do not descend into it.
  • The -readable test takes into account access control lists and other permissions artefacts which the -perm test ignores.

See also find(1) manpage for many more details.

85: How to wait in bash for several subprocesses to finish and return exit code !=0 when any subprocess ends with code !=0? (score 469941 in 2018)

Question

How to wait in a bash script for several subprocesses spawned from that script to finish and return exit code !=0 when any of the subprocesses ends with code !=0 ?

Simple script:

The above script will wait for all 10 spawned subprocesses, but it will always give exit status 0 (see help wait). How can I modify this script so it will discover exit statuses of spawned subprocesses and return exit code 1 when any of subprocesses ends with code !=0?

Is there any better solution for that than collecting PIDs of the subprocesses, wait for them in order and sum exit statuses?

Answer accepted (score 460)

wait also (optionally) takes the PID of the process to wait for, and with $! you get the PID of the last command launched in background. Modify the loop to store the PID of each spawned sub-process into an array, and then loop again waiting on each PID.

Answer 2 (score 273)

http://jeremy.zawodny.com/blog/archives/010717.html :

Answer 3 (score 46)

If you have GNU Parallel installed you can do:

GNU Parallel will give you exit code:

  • 0 - All jobs ran without error.

  • 1-253 - Some of the jobs failed. The exit status gives the number of failed jobs

  • 254 - More than 253 jobs failed.

  • 255 - Other error.

Watch the intro videos to learn more: http://pi.dk/1

86: Extract file basename without path and extension in bash (score 465033 in 2017)

Question

Given file names like these:

I hope to get:

Why this doesn’t work?

What’s the right way to do it?

Answer accepted (score 587)

You don’t have to call the external basename command. Instead, you could use the following commands:

Note that this solution should work in all recent (post 2004) POSIX compliant shells, (e.g. bash, dash, ksh, etc.).

Source: Shell Command Language 2.6.2 Parameter Expansion

More on bash String Manipulations: http://tldp.org/LDP/LG/issue18/bash.html

Answer 2 (score 267)

The basename command has two different invocations; in one, you specify just the path, in which case it gives you the last component, while in the other you also give a suffix that it will remove. So, you can simplify your example code by using the second invocation of basename. Also, be careful to correctly quote things:

fbname=$(basename "$1" .txt)
echo "$fbname"

Answer 3 (score 57)

A combination of basename and cut works fine, even in case of double ending like .tar.gz:

Would be interesting if this solution needs less arithmetic power than Bash Parameter Expansion.

87: Make a Bash alias that takes a parameter? (score 463475 in 2017)

Question

I used to use CShell (), which lets you make an alias that takes a parameter. The notation was something like

In Bash, this does not seem to work. Given that Bash has a multitude of useful features, I would assume that this one has been implemented but I am wondering how.

Answer accepted (score 1928)

Bash alias does not directly accept parameters. You will have to create a function.

alias does not accept parameters but a function can be called just like an alias. For example:

By the way, Bash functions defined in your .bashrc and other files are available as commands within your shell. So for instance you can call the earlier function like this

Answer 2 (score 193)

Refining the answer above, you can get 1-line syntax like you can for aliases, which is more convenient for ad-hoc definitions in a shell or .bashrc files:

Don’t forget the semi-colon before the closing right-bracket. Similarly, for the actual question:

Or:

Answer 3 (score 105)

The question is simply asked wrong. You don’t make an alias that takes parameters because alias just adds a second name for something that already exists. The functionality the OP wants is the function command to create a new function. You do not need to alias the function as the function already has a name.

I think you want something like this :

That’s it! You can use parameters $1, $2, $3, etc, or just stuff them all with $@

88: Find all storage devices attached to a Linux machine (score 460866 in 2013)

Question

I have a need to find all of the writable storage devices attached to a given machine, whether or not they are mounted.

The dopey way to do this would be to try every entry in /dev that corresponds to a writable devices (hd* and sd*)……

Is there a better solution, or should I stick with this one?

Answer accepted (score 78)

/proc/partitions will list all the block devices and partitions that the system recognizes. You can then try using file -s &lt;device&gt; to determine what kind of filesystem is present on the partition, if any.

Answer 2 (score 71)

You can always do fdisk -l which seems to work pretty well, even on strange setups such as EC2 xvda devices.

Here is a dump for a m1.large instance:

While mount says:

And /proc/partitions says:

Side Note

How fdisk -l works is something I would love to know myself.

Answer 3 (score 35)

you can also try lsblk … is in util-linux … but i have a question too

no result

fdisk is good but not that good … seems like it cannot “see” everything

in my particular example i have a stick that have also a card reader build in it and i can see only the stick using fdisk:

but not the card (card being /dev/sdl)

also, file -s is inefficient …

that’s nice … BUT

to see information about a disk that cannot be accesed by fdisk, you can use parted:

89: Length of string in bash (score 460216 in 2017)

Question

How do you get the length of a string stored in a variable and assign that to another variable?

How do you set another variable to the output 11?

Answer accepted (score 234)

UTF-8 string length

In addition to fedorqui’s correct answer, I would like to show the difference between string length and byte length:

will render:

you could even have a look at stored chars:

will answer:

Nota: According to Isabell Cowan’s comment, I’ve added setting to $LC_ALL along with $LANG.

Length of an argument

Argument work same as regular variables

will work as

Useful printf correction tool:

If you:

Not really pretty… For this, there is a little function:

Then now:

But there left some strange UTF-8 behaviour, like double-spaced chars, zero spaced chars, reverse deplacement and other that could not be as simple… Have a look at diffU8test.sh or diffU8test.sh.txt for more limitations.

Answer 2 (score 442)

To get the length of a string stored in a variable, say:

To confirm it was properly saved, echo it:

Answer 3 (score 22)

You can use:

  • wc -c or wc --bytes for byte counts = Unicode characters are counted with 2, 3 or more bytes.
  • wc -m or wc --chars for character counts = Unicode characters are counted single until they use more bytes.

90: Get current time in seconds since the Epoch on Linux, Bash (score 459374 in 2018)

Question

I need something simple like date, but in seconds since 1970 instead of the current date, hours, minutes, and seconds.

date doesn’t seem to offer that option. Is there an easy way?

Answer accepted (score 1026)

This should work:

Answer 2 (score 119)

Just to add.

Get the seconds since epoch(Jan 1 1970) for any given date(e.g Oct 21 1973).


Convert the number of seconds back to date


The command date is pretty versatile. Another cool thing you can do with date(shamelessly copied from date --help). Show the local time for 9AM next Friday on the west coast of the US

Better yet, take some time to read the man page http://man7.org/linux/man-pages/man1/date.1.html

Answer 3 (score 36)

So far, all the answers use the external program date.

Since Bash 4.2, printf has a new modifier %(dateformat)T that, when used with argument -1 outputs the current date with format given by dateformat, handled by strftime(3) (man 3 strftime for informations about the formats).

So, for a pure Bash solution:

or if you need to store the result in a variable var:

No external programs and no subshells!

Since Bash 4.3, it’s even possible to not specify the -1:

(but it might be wiser to always give the argument -1 nonetheless).

If you use -2 as argument instead of -1, Bash will use the time the shell was started instead of the current date (but why would you want this?).

91: How do I edit $PATH (.bash_profile) on OSX? (score 458023 in 2019)

Question

I am trying to edit an entry to PATH, as I did something wrong.

I am using Mac OS X 10.10.3

I have tried:

But the file editor opens with nothing inside.

My problem:

I am trying to install ANDROID_HOME to my PATH

I misspelled it, but when I closed the terminal and went back it was gone, so I tried again:

This time, I typed the command correctly but, when I closed the terminal, my settings disappeared again.

How do I execute my desired settings?

If I was to edit bash.profile, how would I enter the above code?

Thanks!

Answer accepted (score 300)

You have to open that file with a text editor and then save it.

It will open the file with TextEdit, paste your things and then save it. If you open it again you’ll find your edits.

You can use other editors:

But if you don’t know how to use them, it’s easier to use the open approach.

Alternatively, you can rely on pbpaste. Copy

in the system clipboard and then in a shell run

Or alternatively you can also use cat

(now cat waits for input: paste the two export definitions and then hit ctrl-D).

Answer 2 (score 64)

A bit more detailed for beginners:

First, make sure the .bash_profile file is existing? Remember that the .bash_profile file isn’t there by default. You have to create it on your own.

Go into your user folder in finder. The .bash_profile file should be findable there. -> HD/Users/[USERNAME]

Remember: Files with a point at the beginning ‘.’ are hidden by default.

To show hidden files in Mac OS Finder:

If it’s not existing, you have to create .bash_profile on your own.

Open terminal app and switch into user folder with simple command:

If it’s not existing, use this command to create the file:

Second, if you can’t memorise the nerdy commands for save and close in vim, nano etc (the way recommended above) the easiest way to edit is to open .bash_profile file in your favored code editor (Sublime etc.).

Finder -> User folder. Right click -> open with : Sublime Text (or other code editor). Or drag it on app in dock.

… and there you can edit it, pass export commands in new lines.

Answer 3 (score 8)

For beginners: To create your .bash_profile file in your home directory on MacOS, run:

nano ~/.bash_profile

Then you can paste in the following:

https://gist.github.com/mocon/0baf15e62163a07cb957888559d1b054

As you can see, it includes some example aliases and an environment variable at the bottom.

One you’re done making your changes, follow the instructions at the bottom of the Nano editor window to WriteOut (Ctrl-O) and Exit (Ctrl-X). Then quit your Terminal and reopen it, and you will be able to use your newly defined aliases and environment variables.

92: Bash script and /bin/bash^M: bad interpreter: No such file or directory (score 456363 in 2019)

Question

I’m learning through this tutorial to learn bash scripts to automate a few tasks for me. I’m connecting to a server using putty.

The script, located in .../Documents/LOG, is:

And I executed the following for read/write/execute permissions

Then, when I enter ./my_script, I’m getting the error given in the title.

Some similar questions wanted to see these, so I think they might help:

/bin/bash

and

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/bin/mh

I tried adding current directory to PATH, but that doesn’t work..

Answer accepted (score 501)

I have seen this issue when creating scripts in Windows env and then porting over to run on a Unix environment.

Try running dos2unix on the script:

http://dos2unix.sourceforge.net/

Or just rewrite the script in your Unix env using vi and test.

Unix uses different line endings so can’t read the file you created on Windows. Hence it is seeing ^M as an illegal character.

If you want to write a file on Windows and then port over, make sure your editor is set to create files in UNIX format.

In notepad++ in the bottom right of the screen, it tells you the document format. By default, it will say Dos\Windows. To change it go to

  • settings->preferences
  • new document / default directory tab
  • select the format as unix and close
  • create a new document

Answer 2 (score 430)

Run following command in terminal

Then try

It should work.

Answer 3 (score 116)

If you use Sublime Text on Windows or Mac to edit your scripts:

Click on View &gt; Line Endings &gt; Unix and save the file again.

enter image description here

93: Check if a Bash array contains a value (score 456169 in 2017)

Question

In Bash, what is the simplest way to test if an array contains a certain value?

Edit: With help from the answers and the comments, after some testing, I came up with this:

I’m not sure if it’s the best solution, but it seems to work.

Answer accepted (score 7)

There is sample code that shows how to replace a substring from an array. You can make a copy of the array and try to remove the target value from the copy. If the copy and original are then different, then the target value exists in the original string.

The straightforward (but potentially more time-consuming) solution is to simply iterate through the entire array and check each item individually. This is what I typically do because it is easy to implement and you can wrap it in a function (see this info on passing an array to a function).

Answer 2 (score 368)

This approach has the advantage of not needing to loop over all the elements (at least not explicitly). But since array_to_string_internal() in array.c still loops over array elements and concatenates them into a string, it’s probably not more efficient than the looping solutions proposed, but it’s more readable.

Note that in cases where the value you are searching for is one of the words in an array element with spaces, it will give false positives. For example

The regex will see Jack as being in the array even though it isn’t. So you’ll have to change IFS and the separator characters on your regex if you want still to use this solution, like this

Answer 3 (score 366)

Below is a small function for achieving this. The search string is the first argument and the rest are the array elements:

A test run of that function could look like:

94: Find and kill a process in one line using bash and regex (score 456080 in 2010)

Question

I often need to kill a process during programming.

The way I do it now is:

How can I extract the process id automatically and kill it in the same line?

Like this:

Answer accepted (score 1287)

In bash, you should be able to do:

Details on its workings are as follows:

  • The ps gives you the list of all the processes.
  • The grep filters that based on your search string, [p] is a trick to stop you picking up the actual grep process itself.
  • The awk just gives you the second field of each line, which is the PID.
  • The $(x) construct means to execute x then take its output and put it on the command line. The output of that ps pipeline inside that construct above is the list of process IDs so you end up with a command like kill 1234 1122 7654.

Here’s a transcript showing it in action:

and you can see it terminating all the sleepers.

Explaining the grep '[p]ython csp_build.py' bit in a bit more detail:

When you do sleep 3600 &amp; followed by ps -ef | grep sleep, you tend to get two processes with sleep in it, the sleep 3600 and the grep sleep (because they both have sleep in them, that’s not rocket science).

However, ps -ef | grep '[s]leep' won’t create a process with sleep in it, it instead creates grep '[s]leep' and here’s the tricky bit: the grep doesn’t find it because it’s looking for the regular expression "any character from the character class [s] (which is s) followed by leep.

In other words, it’s looking for sleep but the grep process is grep '[s]leep' which doesn’t have sleep in it.

When I was shown this (by someone here on SO), I immediately started using it because

  • it’s one less process than adding | grep -v grep; and
  • it’s elegant and sneaky, a rare combination :-)

Answer 2 (score 125)

if you have pkill,

If you only want to grep against the process name (instead of the full argument list) then leave off -f.

Answer 3 (score 82)

One liner:

ps aux | grep -i csp_build | awk '{print $2}' | xargs sudo kill -9

  • Print out column 2: awk '{print $2}'
  • sudo is optional
  • Run kill -9 5124, kill -9 5373 etc (kill -15 is more graceful but slightly slower)
Bonus:

I also have 2 shortcut functions defined in my .bash_profile (~/.bash_profile is for osx, you have to see what works for your *nix machine).

  1. p keyword

    • lists out all Processes containing keyword
    • usage e.g: p csp_build , p python etc

bash_profile code:

  1. ka keyword

    • Kills All processes that have this keyword
    • usage e.g: ka csp_build , ka python etc
    • optional kill level e.g: ka csp_build 15, ka python 9

bash_profile code:

95: How to determine the current shell I’m working on? (score 455839 in 2013)

Question

How can I determine the current shell I am working on?

Would the output of the ps command alone be sufficient?

How can this be done in different flavors of UNIX?

Answer accepted (score 738)

  • There are 3 approaches to finding the name of the current shell’s executable:

    Please note that all 3 approaches can be fooled if the executable of the shell is /bin/sh but it’s really a renamed bash, for example (which frequently happens).

    Thus your second question of whether ps output will do is answered with “not always”.

    1. echo $0 - will print the program name… which in the case of shell is the actual shell

    2. ps -ef | grep $$ | grep -v grep - This will look for the current process ID in the list of running processes. Since the current process is shell, it will be included.

      This is not 100% reliable, as you might have OTHER processes whose ps listing includes the same number as shell’s process ID, especially if that ID is a small # (e.g. if shell’s PID is “5”, you may find processes called “java5” or “perl5” in the same grep output!). This is the second problem with the “ps” approach, on top of not being able to rely on the shell name.

    3. echo $SHELL - The path to the current shell is stored as the SHELL variable for any shell. The caveat for this one is that if you launch a shell explicitly as a subprocess (e.g. it’s not your login shell), you will get your login shell’s value instead. If that’s a possibility, use the ps or $0 approach.

  • If, however, the executable doesn’t match your actual shell (e.g. /bin/sh is actually bash or ksh), you need heuristics. Here are some environmental variables specific to various shells:

    • $version is set on tcsh

    • $BASH is set on bash

    • $shell (lowercase) is set to actual shell name in csh or tcsh

    • $ZSH_NAME is set on zsh

    • ksh has $PS3 and $PS4 set, whereas normal Bourne shell (sh) only has $PS1 and $PS2 set. This generally seems like the hardest to distinguish - the ONLY difference in entire set of envionmental variables between sh and ksh we have installed on Solaris boxen is $ERRNO, $FCEDIT, $LINENO, $PPID, $PS3, $PS4, $RANDOM, $SECONDS, $TMOUT.

Answer 2 (score 89)

ps -p $$

should work anywhere that the solutions involving ps -ef and grep do (on any Unix variant which supports POSIX options for ps) and will not suffer from the false positives introduced by grepping for a sequence of digits which may appear elsewhere.

Answer 3 (score 38)

Try

or

96: Running Bash commands in Python (score 452753 in 2019)

Question

On my local machine, I run a python script which contains this line

This works fine.

Then I run the same code on a server and I get the following error message

So what I did then is I inserted a print bashCommand which prints me than the command in the terminal before it runs it with os.system().

Of course, I get again the error (caused by os.system(bashCommand)) but before that error it prints the command in the terminal. Then I just copied that output and did a copy paste into the terminal and hit enter and it works…

Does anyone have a clue what’s going on?

Answer 2 (score 276)

Don’t use os.system. It has been deprecated in favor of subprocess. From the docs: “This module intends to replace several older modules and functions: os.system, os.spawn”.

Like in your case:

Answer 3 (score 101)

To somewhat expand on the earlier answers here, there are a number of details which are commonly overlooked.

  • Prefer subprocess.run() over subprocess.check_call() and friends over subprocess.call() over subprocess.Popen() over os.system() over os.popen()
  • Understand and probably use text=True, aka universal_newlines=True.
  • Understand the meaning of shell=True or shell=False and how it changes quoting and the availability of shell conveniences.
  • Understand differences between sh and Bash
  • Understand how a subprocess is separate from its parent, and generally cannot change the parent.
  • Avoid running the Python interpreter as a subprocess of Python.

These topics are covered in some more detail below.

Prefer subprocess.run() or subprocess.check_call()

The subprocess.Popen() function is a low-level workhorse but it is tricky to use correctly and you end up copy/pasting multiple lines of code … which conveniently already exist in the standard library as a set of higher-level wrapper functions for various purposes, which are presented in more detail in the following.

Here’s a paragraph from the documentation:

The recommended approach to invoking subprocesses is to use the run() function for all use cases it can handle. For more advanced use cases, the underlying Popen interface can be used directly.

Unfortunately, the availability of these wrapper functions differs between Python versions.

  • subprocess.run() was officially introduced in Python 3.5. It is meant to replace all of the following.
  • subprocess.check_output() was introduced in Python 2.7 / 3.1. It is basically equivalent to subprocess.run(..., check=True, stdout=subprocess.PIPE).stdout
  • subprocess.check_call() was introduced in Python 2.5. It is basically equivalent to subprocess.run(..., check=True)
  • subprocess.call() was introduced in Python 2.4 in the original subprocess module (PEP-324). It is basically equivalent to subprocess.run(...).returncode
High-level API vs subprocess.Popen()

The refactored and extended subprocess.run() is more logical and more versatile than the older legacy functions it replaces. It returns a CompletedProcess object which has various methods which allow you to retrieve the exit status, the standard output, and a few other results and status indicators from the finished subprocess.

subprocess.run() is the way to go if you simply need a program to run and return control to Python. For more involved scenarios (background processes, perhaps with interactive I/O with the Python parent program) you still need to use subprocess.Popen() and take care of all the plumbing yourself. This requires a fairly intricate understanding of all the moving parts and should not be undertaken lightly. The simpler Popen object represents the (possibly still-running) process which needs to be managed from your code for the remainder of the lifetime of the subprocess.

It should perhaps be emphasized that just subprocess.Popen() merely creates a process. If you leave it at that, you have a subprocess running concurrently alongside with Python, so a “background” process. If it doesn’t need to do input or output or otherwise coordinate with you, it can do useful work in parallel with your Python program.

Avoid os.system() and os.popen()

Since time eternal (well, since Python 2.5) the os module documentation has contained the recommendation to prefer subprocess over os.system():

The subprocess module provides more powerful facilities for spawning new processes and retrieving their results; using that module is preferable to using this function.

The problems with system() are that it’s obviously system-dependent and doesn’t offer ways to interact with the subprocess. It simply runs, with standard output and standard error outside of Python’s reach. The only information Python receives back is the exit status of the command (zero means success, though the meaning of non-zero values is also somewhat system-dependent).

PEP-324 (which was already mentioned above) contains a more detailed rationale for why os.system is problematic and how subprocess attempts to solve those issues.

os.popen() used to be even more strongly discouraged:

Deprecated since version 2.6: This function is obsolete. Use the subprocess module.

However, since sometime in Python 3, it has been reimplemented to simply use subprocess, and redirects to the subprocess.Popen() documentation for details.

Understand and usually use check=True

You’ll also notice that subprocess.call() has many of the same limitations as os.system(). In regular use, you should generally check whether the process finished successfully, which subprocess.check_call() and subprocess.check_output() do (where the latter also returns the standard output of the finished subprocess). Similarly, you should usually use check=True with subprocess.run() unless you specifically need to allow the subprocess to return an error status.

In practice, with check=True or subprocess.check_*, Python will throw a CalledProcessError exception if the subprocess returns a nonzero exit status.

A common error with subprocess.run() is to omit check=True and be surprised when downstream code fails if the subprocess failed.

On the other hand, a common problem with check_call() and check_output() was that users who blindly used these functions were surprised when the exception was raised e.g. when grep did not find a match. (You should probably replace grep with native Python code anyway, as outlined below.)

All things counted, you need to understand how shell commands return an exit code, and under what conditions they will return a non-zero (error) exit code, and make a conscious decision how exactly it should be handled.

Understand and probably use text=True aka universal_newlines=True

Since Python 3, strings internal to Python are Unicode strings. But there is no guarantee that a subprocess generates Unicode output, or strings at all.

(If the differences are not immediately obvious, Ned Batchelder’s Pragmatic Unicode is recommended, if not outright obligatory, reading. There is a 36-minute video presentation behind the link if you prefer, though reading the page yourself will probably take significantly less time.)

Deep down, Python has to fetch a bytes buffer and interpret it somehow. If it contains a blob of binary data, it shouldn’t be decoded into a Unicode string, because that’s error-prone and bug-inducing behavior - precisely the sort of pesky behavior which riddled many Python 2 scripts, before there was a way to properly distinguish between encoded text and binary data.

With text=True, you tell Python that you, in fact, expect back textual data in the system’s default encoding, and that it should be decoded into a Python (Unicode) string to the best of Python’s ability (usually UTF-8 on any moderately up to date system, except perhaps Windows?)

If that’s not what you request back, Python will just give you bytes strings in the stdout and stderr strings. Maybe at some later point you do know that they were text strings after all, and you know their encoding. Then, you can decode them.

Python 3.7 introduced the shorter and more descriptive and understandable alias text for the keyword argument which was previously somewhat misleadingly called universal_newlines.

Understand shell=True vs shell=False

With shell=True you pass a single string to your shell, and the shell takes it from there.

With shell=False you pass a list of arguments to the OS, bypassing the shell.

When you don’t have a shell, you save a process and get rid of a fairly substantial amount of hidden complexity, which may or may not harbor bugs or even security problems.

On the other hand, when you don’t have a shell, you don’t have redirection, wildcard expansion, job control, and a large number of other shell features.

A common mistake is to use shell=True and then still pass Python a list of tokens, or vice versa. This happens to work in some cases, but is really ill-defined and could break in interesting ways.

The common retort “but it works for me” is not a useful rebuttal unless you understand exactly under what circumstances it could stop working.

Refactoring Example

Very often, the features of the shell can be replaced with native Python code. Simple Awk or sed scripts should probably simply be translated to Python instead.

To partially illustrate this, here is a typical but slightly silly example which involves many shell features.

Some things to note here:

  • With shell=False you don’t need the quoting that the shell requires around strings. Putting quotes anyway is probably an error.
  • It often makes sense to run as little code as possible in a subprocess. This gives you more control over execution from within your Python code.
  • Having said that, complex shell pipelines are tedious and sometimes challenging to reimplement in Python.

The refactored code also illustrates just how much the shell really does for you with a very terse syntax – for better or for worse. Python says explicit is better than implicit but the Python code is rather verbose and arguably looks more complex than this really is. On the other hand, it offers a number of points where you can grab control in the middle of something else, as trivially exemplified by the enhancement that we can easily include the host name along with the shell command output. (This is by no means challenging to do in the shell, either, but at the expense of yet another diversion and perhaps another process.)

Common Shell Constructs

For completeness, here are brief explanations of some of these shell features, and some notes on how they can perhaps be replaced with native Python facilities.

  • Globbing aka wildcard expansion can be replaced with glob.glob() or very often with simple Python string comparisons like for file in os.listdir('.'): if not file.endswith('.png'): continue. Bash has various other expansion facilities like .{png,jpg} brace expansion and {1..100} as well as tilde expansion (~ expands to your home directory, and more generally ~account to the home directory of another user)
  • Shell variables like $SHELL or $my_exported_var can sometimes simply be replaced with Python variables. Exported shell variables are available as e.g. os.environ['SHELL'] (the meaning of export is to make the variable available to subprocesses – a variable which is not available to subprocesses will obviously not be available to Python running as a subprocess of the shell, or vice versa. The env= keyword argument to subprocess methods allows you to define the environment of the subprocess as a dictionary, so that’s one way to make a Python variable visible to a subprocess). With shell=False you will need to understand how to remove any quotes; for example, cd "$HOME" is equivalent to os.chdir(os.environ['HOME']) without quotes around the directory name. (Very often cd is not useful or necessary anyway, and many beginners omit the double quotes around the variable and get away with it until one day …)
  • Redirection allows you to read from a file as your standard input, and write your standard output to a file. grep 'foo' &lt;inputfile &gt;outputfile opens outputfile for writing and inputfile for reading, and passes its contents as standard input to grep, whose standard output then lands in outputfile. This is not generally hard to replace with native Python code.
  • Pipelines are a form of redirection. echo foo | nl runs two subprocesses, where the standard output of echo is the standard input of nl (on the OS level, in Unix-like systems, this is a single file handle). If you cannot replace one or both ends of the pipeline with native Python code, perhaps think about using a shell after all, especially if the pipeline has more than two or three processes (though look at the pipes module in the Python standard library or a number of more modern and versatile third-party competitors).
  • Job control lets you interrupt jobs, run them in the background, return them to the foreground, etc. The basic Unix signals to stop and continue a process are of course available from Python, too. But jobs are a higher-level abstraction in the shell which involve process groups etc which you have to understand if you want to do something like this from Python.
Understand differences between sh and Bash

subprocess runs your shell commands with /bin/sh unless you specifically request otherwise (except of course on Windows, where it uses the value of the COMSPEC variable). This means that various Bash-only features like arrays, [[ etc are not available.

If you need to use Bash-only syntax, you can pass in the path to the shell as executable='/bin/bash' (where of course if your Bash is installed somewhere else, you need to adjust the path).

A subprocess is separate from its parent, and cannot change it

A somewhat common mistake is doing something like

which aside from the lack of elegance also betrays a fundamental lack of understanding of the “sub” part of the name “subprocess”.

A child process runs completely separate from Python, and when it finishes, Python has no idea what it did (apart from the vague indicators that it can infer from the exit status and output from the child process). A child generally cannot change the parent’s environment; it cannot set a variable, change the working directory, or, in so many words, communicate with its parent without cooperation from the parent.

The immediate fix in this particular case is to run both commands in a single subprocess;

though obviously this particular use case doesn’t require the shell at all. Remember, you can manipulate the environment of the current process (and thus also its children) via

or pass an environment setting to a child process with

(not to mention the obvious refactoring subprocess.run(['echo', 'bar']); but echo is a poor example of something to run in a subprocess in the first place, of course).

Don’t run Python from Python

This is slightly dubious advice; there are certainly situations where it does make sense or is even an absolute requirement to run the Python interpreter as a subprocess from a Python script. But very frequently, the correct approach is simply to import the other Python module into your calling script and call its functions directly.

If the other Python script is under your control, and it isn’t a module, consider turning it into one. (This answer is too long already so I will not delve into details here.)

If you need parallelism, you can run Python functions in subprocesses with the multiprocessing module. There is also threading which runs multiple tasks in a single process (which is more lightweight and gives you more control, but also more constrained in that threads within a process are tightly coupled, and bound to a single GIL.)

97: Pipe to/from the clipboard in Bash script (score 452594 in 2018)

Question

Is it possible to pipe to/from the clipboard in Bash?

Whether it is piping to/from a device handle or using an auxiliary application, I can’t find anything.

For example, if /dev/clip was a device linking to the clipboard we could do:

Answer accepted (score 793)

There’s a wealth of clipboards you could be dealing with. I expect you’re probably a Linux user who wants to put stuff in the X Windows primary clipboard. Usually, the clipboard you want to talk to has a utility that lets you talk to it.

In the case of X, there’s xclip (and others). xclip -selection c will send data to the clipboard that works with Ctrl + C, Ctrl + V in most applications.

If you’re on Mac OS X, there’s pbcopy.

If you’re in Linux terminal mode (no X) then look into gpm or screen which has a clipboard. Try the screen command readreg.

Under Windows 10+ or cygwin, use /dev/clipboard or clip.

Answer 2 (score 257)

Make sure you are using alias xclip="xclip -selection c" otherwise you can’t just use to Ctrl+v to paste it back in a different place.

Ctrl+v === test

Answer 3 (score 165)

Install

If you do not have access to apt-get nor pacman, nor dnf, the sources are available on sourceforge.

Set-up
Bash

In ~/.bash_aliases, add:

Do not forget to load your new configuration using . ~/.bash_aliases or by restarting your profile.

Fish

In ~/.config/fish/config.fish, add:

Do not forget to restart your fish instance by restarting your terminal for changes to apply.

Usage

You can now use setclip and getclip, e.g:

98: How do I use shell variables in an awk script? (score 452441 in 2018)

Question

I found some ways to pass external shell variables to an awk script, but I’m confused about ' and ".

First, I tried with a shell script:

Then tried awk:

Why is the difference?

Lastly I tried this:

I’m confused about this.

Answer accepted (score 437)

Getting shell variables into awk
may be done in several ways. Some are better than others. This should cover most of them. If you have a comment, please leave below.
Using -v (The best way, most portable)

Use the -v option: (P.S. use a space after -v or it will be less portable. E.g., awk -v var= not awk -vvar=)

This should be compatible with most awk, and the variable is available in the BEGIN block as well:

If you have multiple variables:

Warning. As Ed Morton writes, escape sequences will be interpreted so \t becomes a real tab and not \t if that is what you search for. Can be solved by using ENVIRON[] or access it via ARGV[]

PS If you like three vertical bar as separator |||, it can’t be escaped, so use -F"[|][|][|]"

Example on getting data from a program/function inn to awk (here date is used)
Variable after code block

Here we get the variable after the awk code. This will work fine as long as you do not need the variable in the BEGIN block:

This also works with multiple variables awk '{print a,b,$0}' a="$var1" b="$var2" file

Using variable this way does not work in BEGIN block:

Here-string

Variable can also be added to awk using a here-string from shells that support them (including Bash):

This is the same as:

P.S. this treats the variable as a file input.

ENVIRON input

As TrueY writes, you can use the ENVIRON to print Environment Variables. Setting a variable before running AWK, you can print it out like this:

ARGV input

As Steven Penny writes, you can use ARGV to get the data into awk:

To get the data into the code itself, not just the BEGIN:

Variable within the code: USE WITH CAUTION

You can use a variable within the awk code, but it’s messy and hard to read, and as Charles Duffy points out, this version may also be a victim of code injection. If someone adds bad stuff to the variable, it will be executed as part of the awk code.

This works by extracting the variable within the code, so it becomes a part of it.

If you want to make an awk that changes dynamically with use of variables, you can do it this way, but DO NOT use it for normal variables.

Here is an example of code injection:

You can add lots of commands to awk this way. Even make it crash with non valid commands.

Extra info:
Use of double quote

It’s always good to double quote variable "$variable"
If not, multiple lines will be added as a long single line.

Example:

Other errors you can get without double quote:

And with single quote, it does not expand the value of the variable:

More info about AWK and variables

Read this faq.

Answer 2 (score 26)

It seems that the good-old ENVIRON built-in hash is not mentioned at all. An example of its usage:

Answer 3 (score 9)

Use either of these depending how you want backslashes in the shell variables handled (avar is an awk variable, svar is a shell variable):

See http://cfajohnson.com/shell/cus-faq-2.html#Q24 for details and other options. The first method above is almost always your best option and has the most obvious semantics.

99: How to make “if not true condition”? (score 449960 in 2018)

Question

I would like to have the echo command executed when cat /etc/passwd | grep "sysa" is not true.

What am I doing wrong?

Answer accepted (score 411)

try

grep returns true if it finds the search target, and false if it doesn’t.

So NOT false == true.

if evaluation in shells are designed to be very flexible, and many times doesn’t require chains of commands (as you have written).

Also, looking at your code as is, your use of the $( ... ) form of cmd-substitution is to be commended, but think about what is coming out of the process. Try echo $(cat /etc/passwd | grep "sysa") to see what I mean. You can take that further by using the -c (count) option to grep and then do if ! [ $(grep -c "sysa" /etc/passwd) -eq 0 ] ; then which works but is rather old school.

BUT, you could use the newest shell features (arithmetic evaluation) like

which also gives you the benefit of using the c-lang based comparison operators, ==,&lt;,&gt;,&gt;=,&lt;=,% and maybe a few others.

In this case, per a comment by Orwellophile, the arithmetic evaluation can be pared down even further, like

OR

Finally, there is an award called the Useless Use of Cat (UUOC). :-) Some people will jump up and down and cry gothca! I’ll just say that grep can take a file name on its cmd-line, so why invoke extra processes and pipe constructions when you don’t have to? ;-)

I hope this helps.

Answer 2 (score 30)

I think it can be simplified into:

or in a single command line

$ grep sysa /etc/passwd || { echo "ERROR - The user sysa could not be looked up"; exit 2; }

Answer 3 (score 6)

What am I doing wrong?

$(...) holds the value, not the exit status, that is why this approach is wrong. However, in this specific case, it does indeed work because sysa will be printed which makes the test statement come true. However, if ! [ $(true) ]; then echo false; fi would always print false because the true command does not write anything to stdout (even though the exit code is 0). That is why it needs to be rephrased to if ! grep ...; then.

An alternative would be cat /etc/passwd | grep "sysa" || echo error. Edit: As Alex pointed out, cat is useless here: grep "sysa" /etc/passwd || echo error.

Found the other answers rather confusing, hope this helps someone.

100: What’s the difference between .bashrc, .bash_profile, and .environment? (score 448999 in )

Question

I’ve used a number of different *nix-based systems of the years, and it seems like every flavor of Bash I use has a different algorithm for deciding which startup scripts to run. For the purposes of tasks like setting up environment variables and aliases and printing startup messages (e.g. MOTDs), which startup script is the appropriate place to do these?

What’s the difference between putting things in .bashrc, .bash_profile, and .environment? I’ve also seen other files such as .login, .bash_login, and .profile; are these ever relevant? What are the differences in which ones get run when logging in physically, logging in remotely via ssh, and opening a new terminal window? Are there any significant differences across platforms (including Mac OS X (and its Terminal.app) and Cygwin Bash)?

Answer accepted (score 73)

The main difference with shell config files is that some are only read by “login” shells (eg. when you login from another host, or login at the text console of a local unix machine). these are the ones called, say, .login or .profile or .zlogin (depending on which shell you’re using).

Then you have config files that are read by “interactive” shells (as in, ones connected to a terminal (or pseudo-terminal in the case of, say, a terminal emulator running under a windowing system). these are the ones with names like .bashrc, .tcshrc, .zshrc, etc.

bash complicates this in that .bashrc is only read by a shell that’s both interactive and non-login, so you’ll find most people end up telling their .bash_profile to also read .bashrc with something like

[[ -r ~/.bashrc ]] &amp;&amp; . ~/.bashrc

Other shells behave differently - eg with zsh, .zshrc is always read for an interactive shell, whether it’s a login one or not.

The manual page for bash explains the circumstances under which each file is read. Yes, behaviour is generally consistent between machines.

.profile is simply the login script filename originally used by /bin/sh. bash, being generally backwards-compatible with /bin/sh, will read .profile if one exists.

Answer 2 (score 48)

That’s simple. It’s explained in man bash:

Login shells are the ones that are read one you login (so, they are not executed when merely starting up xterm, for example). There are other ways to login. For example using an X display manager. Those have other ways to read and export environment variables at login time.

Also read the INVOCATION chapter in the manual. It says “The following paragraphs describe how bash executes its startup files.”, i think that’s a spot-on :) It explains what an “interactive” shell is too.

Bash does not know about .environment. I suspect that’s a file of your distribution, to set environment variables independent of the shell that you drive.

Answer 3 (score 9)

Classically, ~/.profile is used by Bourne Shell, and is probably supported by Bash as a legacy measure. Again, ~/.login and ~/.cshrc were used by C Shell - I’m not sure that Bash uses them at all.

The ~/.bash_profile would be used once, at login. The ~/.bashrc script is read every time a shell is started. This is analogous to /.cshrc for C Shell.

One consequence is that stuff in ~/.bashrc should be as lightweight (minimal) as possible to reduce the overhead when starting a non-login shell.

I believe the ~/.environment file is a compatibility file for Korn Shell.