<h2 id="introduction">Introduction</h2>
In Bash, parameter expansion is a feature that allows you to manipulate the value of a variable or to extract part of its value using a special syntax. It is often used to modify the value of a variable or to extract information from it.
Parameter expansion uses a syntax similar to variable substitution, but with additional operators and options that allow you to manipulate the value of the variable in more advanced ways. For example, you can use parameter expansion to extract a substring from a variable, replace a part of a variable with a different value, or add a prefix or suffix to a variable.
Parameter expansion is a powerful feature that is widely used in Bash scripts. It allows you to manipulate and extract information from variables in a flexible and efficient way and is an important tool for working with variables in the Bash shell.
<blockquote>
In this guide, we'll go over parameter expansion in Bash. We'll go over what it is, why you should consider using it, how to use it, and, finally, give you several tips on how to use parameter expansion in Bash more efficiently.
</blockquote>
<h2 id="whyisparameterexpansionuseful">Why is Parameter Expansion Useful?</h2>
As we already indicated, there are many scenarios where parameter expansion comes in handy, here are several most common:
<ul>
<li>
Modifying the value of a variable - parameter expansion allows you to modify the value of a variable by extracting a substring, replacing a part of the value, or adding a prefix or suffix. This can be useful for adapting the value of a variable to different situations or for extracting information from it.
</li>
<li>
Improving the readability and maintainability of scripts - by using parameter expansion, you can avoid using complex and error-prone string manipulation techniques, such as using multiple &quot;sed&quot; or &quot;awk&quot; commands. This can help to make your scripts more readable and easier to maintain.
</li>
<li>
Increasing the efficiency of scripts - parameter expansion can often be faster and more efficient than using external commands or utilities to manipulate strings, especially when working with large amounts of data. This can be especially important in scripts that are designed to run efficiently and quickly.
</li>
</ul>
As you can see, parameter expansion is a useful and powerful feature in Bash that allows you to manipulate and extract information from variables in a flexible and efficient way. It is an important tool for working with variables in the Bash shell.
<h2 id="parameterexpansionsyntax">Parameter Expansion Syntax</h2>
In this section, we'll finally explore the various syntax options available for parameter expansion in Bash. We will cover the basic <code>${variable}</code> syntax for extracting the value of a variable, as well as more advanced techniques such as extracting substrings, replacing parts of a variable, adding prefixes and suffixes, and providing default values. By the end of this section, you will have a solid understanding of the different ways you can use parameter expansion in Bash.
<h3 id="basicsyntaxvariable">Basic syntax: ${variable}</h3>
The <code>${variable}</code> syntax is a basic form of parameter expansion in Bash that allows you to access the value of a variable. It is similar to variable substitution, which allows you to insert the value of a variable into a command or script.
Here is a quick example of using the <code>${variable}</code> syntax in a Bash script:
<pre><code class="hljs">name=&quot;John&quot;
echo &quot;Hello, ${name}!&quot;
</code></pre>
This will output:
<pre><code class="hljs">Hello, John!
</code></pre>
You can also use the <code>${variable}</code> syntax to assign a new value to a variable:
<pre><code class="hljs">name=&quot;John&quot;
echo &quot;Original value: ${name}&quot;
name=&quot;Jane&quot;
echo &quot;New value: ${name}&quot;
</code></pre>
This will give us:
<pre><code class="hljs">Original value: John
New value: Jane
</code></pre>
The <code>${variable}</code> syntax is a simple and straightforward way to access and manipulate the value of a variable in Bash. It is a fundamental feature of the Bash shell and is widely used in scripts and on the command line.
<h3 id="extractingasubstringvariableoffsetlength">Extracting a substring: ${variable:offset:length}</h3>
To extract a substring from a variable using parameter expansion in Bash, you can use the following syntax:
<pre><code class="hljs">substring=${variable:offset:length}
</code></pre>
Here, <code>variable</code> is the name of the variable that contains the string you want to extract a substring from, <code>offset</code> is the starting position of the substring (measured in characters from the beginning of the string), and <code>length</code> is the number of characters in the substring.
For example, you can extract the first four characters of a string stored in the <code>filename</code> variable:
<pre><code class="hljs">substring=${filename:0:4}
</code></pre>
Let's take a look at an example of using this syntax on an actual variable:
<pre><code class="hljs">filename=&quot;file.txt&quot;
substring=${filename:0:4}
echo &quot;Substring: ${substring}&quot;
</code></pre>
This will extract the first 4 characters of the <code>filename</code> variable:
<pre><code class="hljs">Substring: file
</code></pre>
You can also use negative values for the <code>offset</code> parameter to specify a position relative to the end of the string. For example, take a look at how you'd extract the last four characters of a string:
<pre><code class="hljs">substring=${filename:-4:4}
</code></pre>
This will extract the last four characters of the <code>filename</code> variable, regardless of its length.
Overall, the <code>${variable:offset:length}</code> syntax is a powerful and flexible way to extract substrings from variables in Bash. It allows you to easily extract specific parts of a string and use them in your scripts and commands.
<h3 id="replacingapartofavariablevariablepatternreplacement">Replacing a part of a variable: ${variable/pattern/replacement}</h3>
To replace a part of a variable using parameter expansion in Bash, you can use the following syntax:
<pre><code class="hljs">result=${variable/pattern/replacement}
</code></pre>
Here, <code>variable</code> is the name of the variable that contains the string you want to modify, <code>pattern</code> is a regular expression that specifies the part of the string you want to replace, and <code>replacement</code> is the string that will replace the matched pattern.
For example, to replace the first occurrence of the string <code>&quot;txt&quot;</code> with the string <code>&quot;doc&quot;</code> in a variable named <code>filename</code>, you can use the following syntax:
<pre><code class="hljs">result=${filename/txt/doc}
</code></pre>
Here is an example of using this syntax in a Bash script:
<pre><code class="hljs">filename=&quot;file.txt&quot;
result=${filename/txt/doc}
echo &quot;Result: ${result}&quot;
</code></pre>
This script will output &quot;Result: file.doc&quot;.
You can also use the <code>g</code> flag to replace all occurrences of the pattern, rather than just the first one. For example:
<pre><code class="hljs">result=${filename/txt/doc/g}
</code></pre>
This syntax will replace all occurrences of <code>&quot;txt&quot;</code> with <code>&quot;doc&quot;</code> in the <code>filename</code> variable.
Overall, the <code>&quot;${variable/pattern/replacement}&quot;</code> syntax is a powerful and flexible way to replace parts of a string stored in a variable in Bash. It allows you to easily modify strings and adapt them to different situations.
<h3 id="addingaprefixorsuffixprefixvariablesuffix">Adding a prefix or suffix: ${prefix}${variable}${suffix}</h3>
To add a prefix or suffix to a variable using parameter expansion in Bash, you can use the following syntax:
<pre><code class="hljs">result=&quot;${prefix}${variable}${suffix}&quot;
</code></pre>
Here, <code>prefix</code> and <code>suffix</code> are the strings you want to add to the beginning and end of the <code>variable</code> respectively.
For example, to add the prefix <code>prefix_</code> and the suffix <code>_suffix</code> to a variable named <code>filename</code>, you can use the following syntax:
<pre><code class="hljs">result=&quot;${prefix}${filename}${suffix}&quot;
</code></pre>
Here is an example of using this syntax in a Bash script:
<pre><code class="hljs">prefix=&quot;prefix_&quot;
suffix=&quot;_suffix&quot;
filename=&quot;file.txt&quot;
result=&quot;${prefix}${filename}${suffix}&quot;
echo &quot;Result: ${result}&quot;
</code></pre>
This script will output:
<pre><code class="hljs">Result: prefix_file.txt_suffix
</code></pre>
You can also use this syntax to add a prefix or suffix to the value of a variable that is stored in another variable. For example:
<pre><code class="hljs">result=&quot;${prefix}${variable_name}${suffix}&quot;
</code></pre>
This syntax will add the prefix and suffix to the value of the <code>variable_name</code> variable.
Overall, the <code>&quot;${prefix}${variable}${suffix}&quot;</code> syntax is a simple and flexible way to add a prefix or suffix to a variable in Bash. It allows you to easily modify strings and adapt them to different situations.
<h2 id="4protipsforparameterexpansioninbash">4 Pro Tips for Parameter Expansion in Bash</h2>
<h3 id="usequotestopreventwordsplitting">Use Quotes to Prevent Word Splitting</h3>
Word splitting is a process that occurs when the Bash shell separates a string of text into separate words based on spaces and other delimiters. This can cause problems when you are trying to pass multiple words or phrases as arguments to a command.
For example, consider the following command:
<pre><code class="hljs">ls $files
</code></pre>
If the value of the <code>$files</code> variable is <code>file1 file2 file3</code>, the Bash shell will split the string into three separate words, and the <code>ls</code> command will be executed with each word as a separate argument. This will result in an error because the <code>ls</code> command does not recognize <code>file1</code>, <code>file2</code>, and <code>file3</code> as valid arguments.
To prevent this problem, you can use quotes around the variable expansion to tell the Bash shell to treat the entire string as a single argument. Here's the same example, but with quotes added:
<pre><code class="hljs">ls &quot;$files&quot;
</code></pre>
Now, the Bash shell will treat the entire value of the <code>$files</code> variable as a single argument, and the <code>ls</code> command will be executed correctly.
Using quotes in this way can be particularly helpful when working with variables that contain multiple words or phrases, or when you want to pass multiple arguments to a command. By using quotes to prevent word splitting, you can avoid common errors and ensure that your commands are executed correctly.
<h3 id="usecurlybracestopreventambiguity">Use Curly Braces to Prevent Ambiguity</h3>
Another helpful tip when using parameter expansion in the Bash shell is to use curly braces to prevent ambiguity.
Ambiguity can occur when you are working with variables that have multiple parts or that are nested within one another. For example, consider the following code:
<pre><code class="hljs">$ fruit=apple
$ echo $fruit${color}
</code></pre>
Without the curly braces, the Bash shell will interpret this code as an expansion of the <code>$fruit</code> variable followed by an expansion of the <code>$color</code> variable. However, if the <code>$color</code> variable is not defined, the expansion will be empty, and the output will be simply <code>&quot;apple&quot;</code>.
To prevent this ambiguity, you can use curly braces to explicitly specify which parts of the expansion should be evaluated. Here's the same example, but with curly braces added:
<pre><code class="hljs">$ fruit=apple
$ echo ${fruit}${color}
</code></pre>
Now, the Bash shell will interpret the code as an expansion of the <code>$fruit</code> variable followed by an expansion of the <code>$color</code> variable, and the output will be <code>&quot;apple&quot;</code> followed by the value of the <code>$color</code> variable (if it is defined).
Using curly braces in this way can be particularly helpful when working with complex expansions or when you want to ensure that your expansions are evaluated correctly. By using curly braces to prevent ambiguity, you can avoid common errors and ensure that your code works as intended.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Using curly braces everywhere, not just in ambiguous cases, is considered a good programming practice!

 </div>
 </div>
 </div>
 <h3 id="useandtoremovethelongestmatchingpattern">Use &quot;##&quot; and &quot;%%&quot; to Remove the Longest Matching Pattern</h3>
The <code>##</code> and <code>%%</code> operators are used in the Bash shell to remove the longest matching pattern from the beginning or end of a variable, respectively.
For example, consider the following code:
<pre><code class="hljs">$ var=abc123
$ echo ${var##*b}
</code></pre>
In this case, the <code>##</code> operator will remove the longest matching pattern from the beginning of the <code>$var</code> variable. The pattern <code>*b</code> matches the string <code>&quot;abc&quot;</code>, so the output will be <code>123</code>.
Similarly, the <code>%%</code> operator can be used to remove the longest matching pattern from the end of a variable:
<pre><code class="hljs">$ var=abc123
$ echo ${var%%3*}
</code></pre>
In this case, the <code>%%</code> operator will remove the longest matching pattern from the end of the <code>$var</code> variable. The pattern <code>3*</code> matches the string <code>&quot;123&quot;</code>, so the output will be <code>&quot;abc&quot;</code>.
These operators can be particularly useful when you want to remove a specific prefix or suffix from a variable, or when you want to extract a specific part of a string. By using the <code>##</code> and <code>%%</code> operators, you can easily manipulate the contents of variables and extract the information that you need.
<h3 id="usetoprovideadefaultvalueifthevariableisunsetorempty">Use :- to provide a default value if the variable is unset or empty</h3>
You can use the <code>:-</code> operator in parameter expansion in Bash to provide a default value for a variable if it is unset or empty. This can be useful when you want to ensure that a variable has a valid value, even if it has not been set or if it has been set to an empty string.
To use the <code>:-</code> operator, you can use the following syntax:
<pre><code class="hljs">result=${variable:-default}
</code></pre>
Here, <code>variable</code> is the name of the variable you want to check, and <code>default</code> is the value you want to use if the variable is unset or empty.
For example, to set a default value for the <code>filename</code> variable if it is unset or empty, you can use the following syntax:
<pre><code class="hljs">filename=${filename:-default.txt}
</code></pre>
This syntax will set the <code>filename</code> variable to <code>default.txt</code> if it is unset or empty, and will leave it unchanged if it already has a value.
You can also use this syntax to assign a default value to a variable that is stored in another variable:
<pre><code class="hljs">result=${variable_name:-default}
</code></pre>
This syntax will assign the value of <code>default</code> to the <code>result</code> variable if the <code>variable_name</code> variable is unset or empty, and will assign the value of <code>variable_name</code> to <code>result</code> if it is set and non-empty.
Overall, the <code>:-</code> operator is a useful feature of parameter expansion in Bash that allows you to provide a default value for a variable if it is unset or empty. It can be helpful for ensuring that your scripts and commands always have access to a valid value for a variable.
<h2 id="conclusion">Conclusion</h2>
All-in-all, parameter expansion is a pretty useful concept to understand when working with variables in Bash - both in terms of efficiency, readability, flexibility, and maintainability of the scripts you are writing.
In this guide, we took a look at the concept of parameter expansion in Bash and explained what parameter expansion is, why it is important to understand it, gave you some practical use-case examples, and, finally, gave you some pro tips you should definitely consider when using parameter expansion in Bash.

David Landup

Dimitrije Stamenic

In this in-depth guide, learn everything you need to know about parameter expansion in Bash, with practical code samples, best practices and lots of examples.

Guide to Parameter Expansion in Bash

<h2 id="introduction">Introduction</h2>
Repetitive tasks are ripe for automation. It is common for developers and system administrators to automate routine tasks like health checks and file backups with shell scripts. However, as those tasks become more complex, shell scripts may become harder to maintain.
<blockquote>
Fortunately, we can use Python instead of shell scripts for automation. Python provides methods to run shell commands, giving us the same functionality of those shells scripts, with additional support from Python's ecosystem. Learning how to run shell commands in Python opens the door for us to automate computer tasks in a structured and scalable way.
</blockquote>
In this article, we will look at the various ways to execute shell commands in Python, and the ideal situation to use each method.
<h2 id="usingossystemtorunacommand">Using os.system to Run a Command</h2>
Python allows us to immediately execute a shell command that's stored in a string using the <code>os.system()</code> function. Let's start by creating a new Python file called <code>echo_adelle.py</code> and enter the following:
<pre><code class="hljs">import os
os.system(&quot;echo Hello from the other side!&quot;)
</code></pre>
The first thing we do in our Python file is import the <code>os</code> module, which contains the <code>system</code> function that can execute shell commands. The next line does exactly that, runs the <code>echo</code> command in our shell through Python.
In your Terminal, run this file with using the following command, and you should see the corresponding output:
<pre><code class="hljs">$ python3 echo_adelle.py
Hello from the other side!
</code></pre>
As the <code>echo</code> commands prints to our <code>stdout</code>, <code>os.system()</code> also displays the output on our <code>stdout</code> stream. While not visible in the console, the <code>os.system()</code> command returns the exit code of the shell command. An exit code of 0 means it ran without any problems and any other number means an error. This way, you ca control the flow of your commands, check the results and either retry or return an error if any call fails:
<pre><code class="hljs"># cd_return_codes.py
import os

home_dir = os.system(&quot;cd ~&quot;)
print(&quot;`cd ~` ran with exit code %d&quot; % home_dir)
unknown_dir = os.system(&quot;cd doesnotexist&quot;)
print(&quot;`cd doesnotexis` ran with exit code %d&quot; % unknown_dir)
</code></pre>
In this script, we create two variables that store the result of executing commands that change the directory to the home folder, and to a folder that does not exist. Running this file, we will see:
<pre><code class="hljs">$ python3 cd_return_codes.py
`cd ~` ran with exit code 0
sh: line 0: cd: doesnotexist: No such file or directory
`cd doesnotexist` ran with exit code 256
</code></pre>
The first command, which changes the directory to the home directory, executes successfully. Therefore, <code>os.system()</code> returns its exit code, zero, which is stored in <code>home_dir</code>. On the other hand, <code>unknown_dir</code> stores the exit code of the failed bash command to change the directory to a folder that does not exist.
The <code>os.system()</code> function executes a command, prints any output of the command to the console, and returns the exit code of the command. If we would like more fine grained control of a shell command's input and output in Python, we should use the <code>subprocess</code> module.
<h2 id="runningacommandwithsubprocess">Running a Command with subprocess</h2>
The <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/subprocess.html">subprocess module</a> is Python's recommended way to executing shell commands. It gives us the flexibility to suppress the output of shell commands or chain inputs and outputs of various commands together, while still providing a similar experience to <code>os.system()</code> for basic use cases.
In a new file called <code>list_subprocess.py</code>, let's run a command with the <code>subprocess</code> module:
<pre><code class="hljs">import subprocess

list_files = subprocess.run([&quot;ls&quot;, &quot;-l&quot;])
print(&quot;The exit code was: %d&quot; % list_files.returncode)
</code></pre>
In the first line, we import the <code>subprocess</code> module, which is part of the Python standard library. We then use the <code>subprocess.run()</code> function to execute the command. Like <code>os.system()</code>, the <code>subprocess.run()</code> command returns the exit code of what was executed.
Unlike <code>os.system()</code>, note how <code>subprocess.run()</code> requires a list of strings as input instead of a single string. The first item of the list is the name of the command. The remaining items of the list are the flags and the arguments of the command.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: As a rule of thumb, you need to separate the arguments based on space, for example <code>ls -alh</code> would be <code>[&quot;ls&quot;, &quot;-alh&quot;]</code>, while <code>ls -a -l -h</code>, would be <code>[&quot;ls&quot;, &quot;-a&quot;, -&quot;l&quot;, &quot;-h&quot;]</code>. As another example, <code>echo hello world</code> would be <code>[&quot;echo&quot;, &quot;hello&quot;, &quot;world&quot;]</code>, whereas <code>echo &quot;hello world&quot;</code> or <code>echo hello\ world</code> would be <code>[&quot;echo&quot;, &quot;hello world&quot;]</code>.

 </div>
 </div>
 </div>
 Run this file and your console's output would be similar to:
<pre><code class="hljs">$ python3 list_subprocess.py
total 80
-rw-r--r--@ 1 stackabuse staff 216 Dec 6 10:29 cd_return_codes.py
-rw-r--r--@ 1 stackabuse staff 56 Dec 6 10:11 echo_adelle.py
-rw-r--r--@ 1 stackabuse staff 116 Dec 6 11:20 list_subprocess.py
The exit code was: 0
</code></pre>
Now let's try to use one of the more advanced features of <code>subprocess.run()</code>, namely ignore output to <code>stdout</code>. In the same <code>list_subprocess.py</code> file, change:
<pre><code class="hljs">list_files = subprocess.run([&quot;ls&quot;, &quot;-l&quot;])
</code></pre>
To this:
<pre><code class="hljs">list_files = subprocess.run([&quot;ls&quot;, &quot;-l&quot;], stdout=subprocess.DEVNULL)
</code></pre>
The standard output of the command now pipes to the special <code>/dev/null</code> device, which means the output would not appear on our consoles. Execute the file in your shell to see the following output:
<pre><code class="hljs">$ python3 list_subprocess.py
The exit code was: 0
</code></pre>
What if we wanted to provide input to a command? The <code>subprocess.run()</code> facilitates this by its <code>input</code> argument:
<pre><code class="hljs">import subprocess

useless_cat_call = subprocess.run([&quot;cat&quot;], 
 stdout=subprocess.PIPE, 
 text=True, 
 input=&quot;Hello from the other side&quot;)

print(useless_cat_call.stdout) # Hello from the other side
</code></pre>
We use <code>subprocess.run()</code> with quite a few commands, let's go through them:
<ul>
<li><code>stdout=subprocess.PIPE</code> tells Python to redirect the output of the command to an object so it can be manually read later</li>
<li><code>text=True</code> returns <code>stdout</code> and <code>stderr</code> as strings. The default return type is bytes.</li>
<li><code>input=&quot;Hello from the other side&quot;</code> tells Python to add the string as input to the <code>cat</code> command.</li>
</ul>
Running this produces the following output:
<pre><code class="hljs">Hello from the other side
</code></pre>
We can also raise an <code>Exception</code> without manually checking the return value. In a new file, <code>false_subprocess.py</code>, add the code below:
<pre><code class="hljs">import subprocess

failed_command = subprocess.run([&quot;false&quot;], check=True)
print(&quot;The exit code was: %d&quot; % failed_command.returncode)
</code></pre>
In your Terminal, run this file. You will see the following error:
<pre><code class="hljs">$ python3 false_subprocess.py
Traceback (most recent call last):
 File &quot;false_subprocess.py&quot;, line 4, in &lt;module&gt;
 failed_command = subprocess.run([&quot;false&quot;], check=True)
 File &quot;/usr/local/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py&quot;, line 512, in run
 output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command &#x27;[&#x27;false&#x27;]&#x27; returned non-zero exit status 1.
</code></pre>
By using <code>check=True</code>, we tell Python to raise any exceptions if an error is encountered. Since we did encounter an error, the <code>print</code> statement on the final line was not executed.
The <code>subprocess.run()</code> function gives us immense flexibility that <code>os.system()</code> doesn't when executing shell commands. This function is a simplified abstraction of the <code>subprocess.Popen</code> class, which provides additional functionality we can explore.
<h2 id="runningacommandwithpopen">Running a Command with Popen</h2>
The <code>subprocess.Popen</code> class exposes more options to the developer when interacting with the shell. However, we need to be more explicit about retrieving results and errors.
By default, <code>subprocess.Popen</code> does not stop processing of a Python program if its command has not finished executing. In a new file called <code>list_popen.py</code>, type the following:
<pre><code class="hljs">import subprocess

list_dir = subprocess.Popen([&quot;ls&quot;, &quot;-l&quot;])
list_dir.wait()
</code></pre>
This code is equivalent to that of <code>list_subprocess.py</code>. It runs a command using <code>subprocess.Popen</code>, and waits for it to complete before executing the rest of the Python script.
Let's say we do not want to wait for our shell command to complete execution so the program can work on other things. How would it know when the shell command has finished execution?
The <code>poll()</code> method returns the exit code if a command has finished running, or <code>None</code> if it's still executing. For example, if we wanted to check if <code>list_dir</code> was complete instead of wait for it, we would have the following line of code:
<pre><code class="hljs">list_dir.poll()
</code></pre>
To manage input and output with <code>subprocess.Popen</code>, we need to use the <code>communicate()</code> method.
In a new file called <code>cat_popen.py</code>, add the following code snippet:
<pre><code class="hljs">import subprocess

useless_cat_call = subprocess.Popen([&quot;cat&quot;], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
output, errors = useless_cat_call.communicate(input=&quot;Hello from the other side!&quot;)
useless_cat_call.wait()
print(output)
print(errors)
</code></pre>
The <code>communicate()</code> method takes an <code>input</code> argument that's used to pass input to the shell command. The <code>communicate</code> method also returns both the <code>stdout</code> and <code>stderr</code> when they are set.
Having seen the core ideas behind <code>subprocess.Popen</code>, we have now covered three ways to run shell commands in Python. Let's re-examine their characteristics so we'll know which method is best suited for a project's requirements.
<h2 id="bestapproachtorunshellcommandswithpython">Best Approach to Run Shell Commands with Python?</h2>
If you need to run one or a few simple commands and do not mind if their output goes to the console, you can use the <code>os.system()</code> command. If you want to manage the input and output of a shell command, use <code>subprocess.run()</code>. If you want to run a command and continue doing other work while it's being executed, use <code>subprocess.Popen</code>.
Here is a table with some usability differences that you can also use to inform your decision:
<table class="table table-striped" style="font-size:15px;overflow-x:auto;">
 <thead>
 <tr>
 <th></th>
 <th>os.system</th>
 <th>subprocess.run</th>
 <th>subprocess.Popen</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <th>Requires parsed arguments</th>
 <th>no</th>
 <th>yes</th>
 <th>yes</th>
 </tr>
 <tr>
 <th>Waits for the command</th>
 <th>yes</th>
 <th>yes</th>
 <th>no</th>
 </tr>
 <tr>
 <th>Communicates with stdin and stdout</th>
 <th>no</th>
 <th>yes</th>
 <th>yes</th>
 </tr>
 <tr>
 <th>Returns</th>
 <th>return value</th>
 <th>object</th>
 <th>object</th>
 </tr>
 </tbody>
</table>
<h2 id="conclusion">Conclusion</h2>
Python allows you to execute shell commands, which you can use to start other programs or better manage shell scripts that you use for automation. Depending on our use case, we can use <code>os.system()</code>, <code>subprocess.run()</code> or <code>subprocess.Popen</code> to run bash commands.
Using these techniques, what external task would you run via Python?

Marcus Sanatan

Sajjad Heydari

Instead of shell scripts, which can get complex and tedious, we can use Python to automate shell commands. In this tutorial, we'll learn how to run them for the sake of scalability and maintainability of Python projects.

Executing Shell Commands with Python

N Kaushik

In this tutorial, we'll be converting Strings to datetime in Python, dealing with Timezones. We'll also use dateutil, Maya and Arrow to convert Strings to datetime with automatic format recognition.

Converting Strings to datetime in Python

<h2 id="quickanswermergebranchintomaster">Quick Answer - Merge Branch into Master</h2>
If you're looking for a quick answer, to merge a branch into the <code>master</code> branch - you <code>checkout master</code> and <code>merge some_branch</code>:
<pre><code class="hljs">$ git checkout new-branch
# ...develop some code...

$ git add .
$ git commit –m &quot;Some commit message&quot;
$ git checkout master
Switched to branch &#x27;master&#x27;
$ git merge new-branch
</code></pre>
You always <code>checkout</code> the branch you're merging into, and <code>merge</code> a branch that already has changes.
<h2 id="understandingbranchmergingwithgit">Understanding Branch Merging with Git</h2>
If you're newer to Git, though, it's worth taking a few minutes to understand these commands and how they work. It's surprisingly simple and will take you a long way.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: In 2020, Git (alongside major repository hosting platforms like GitHub, GitLab, etc.) adopted a change in official terminology, and the default branch name was changed to <code>main</code>, due to the negative connotations the word <code>master</code> may entail. Many projects haven't migrated or renamed their main branches, so for the foreseeable future - the terms <code>master</code> and <code>main</code> will likely be used interchangeably.

 </div>
 </div>
 </div>
 One of Git's most powerful features is the ability to easily create and merge branches and changes to the codebase.
Git's distributed nature encourages users to create new branches often and to merge them regularly as a part of the development process - and certain Git workflows exploit this extensively.
<blockquote>
This fundamentally improves the development workflow for most projects by encouraging smaller, more focused, granular commits, subject to rigorous peer review.
</blockquote>
In legacy Version Control Systems (like CVS) the difficulty of merging restricted it to advanced users. Other modern but centralized version control systems like Subversion require commits to be made to a central repository, so a nimble workflow with local branching and merging is atypical.
A commonly used branching workflow in Git is to create a new code branch for each new feature, bug fix, or enhancement.
<blockquote>
These are called Feature Branches.
</blockquote>
Each branch compartmentalizes the commits related to a particular feature. Once the new feature is complete – i.e. a set of changes has been committed on the feature branch – it is ready to be merged back into the master branch (or other main code line branch depending on the workflow in use).

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Feature branching isn't the only branching workflow you can go with, but it's a widely adopted one, and is enabled by the simplicity of Git's merging functionalities.

 </div>
 </div>
 </div>
 <h2 id="mergebranchintoanotherwithgit">Merge Branch into Another with Git</h2>
The <code>git branch</code> command is used to list all existing branches in a repository. An asterisk will appear next to the currently active branch:
<pre><code class="hljs">$ git branch
* master
</code></pre>
To create a new branch, we can use the <code>git branch new-branch</code> command. This will create a new branch mirroring the commits on the currently active branch:
<pre><code class="hljs">$ git branch new-branch
$ git branch
* master
new-branch
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Behind the scenes, Git does not actually create a new set of commits to represent the new branch. A branch is like a tag, and the commits are shared. You're branching out a new set of changes from the main branch. Once a feature branch is finished and merged into the main branch, the changes in it become the main branch, until you merge a new feature branch into the main branch.

 </div>
 </div>
 </div>
 At this point we have created a new branch, but are still located on the source branch. To start working on the new branch we first need to run the command <code>git checkout new-branch</code>. This will change the active branch to the new branch:
<pre><code class="hljs">$ git checkout new-branch
Switched to branch ‘new-branch&#x27;
$ git branch
master
* new-branch
</code></pre>
At this point, commits can be made on the new branch to implement the new feature. Once the feature is complete, the branch can be merged back into the main code branch.
First we run <code>git checkout master</code> to change the active branch back to the <code>master</code> branch. Then we run the command <code>git merge new-branch</code> to merge the new feature into the master branch.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: <code>git merge</code> merges the specified branch into the currently active branch. So we need to be on the branch that we are merging into.

 </div>
 </div>
 </div>
 If you're merging a new feature into the main branch, you first want to switch to the main branch and then merge into it:
<pre><code class="hljs"># ...develop some code...

$ git add .
$ git commit –m &quot;Some commit message&quot;
$ git checkout master
Switched to branch &#x27;master&#x27;
$ git merge new-branch
</code></pre>
If all goes well then our job is done. The new feature commits now appear in the main branch. However, it is possible that Git won't be able to complete the merge due to a conflict change in the source branch. This is called a merge conflict.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice: If you'd like to read more about Merge Conflicts, how they can arise and how to resolve them - read our <a target="_blank" href="https://stackabuse.com/git-guide-to-solving-merge-conflicts/">&quot;Git: Guide to Resolving Merge Conflicts&quot;!</a>

 </div>
 </div>
 </div>
 To summarize, here are the commands to create a new branch, make some commits, and merge it back into master:
<pre><code class="hljs">$ git checkout master
$ git branch new-branch
$ git checkout new-branch

# ...develop some code...

$ git add .
$ git commit –m &quot;Some commit message&quot;
$ git checkout master
$ git merge new-branch
</code></pre>
<h3 id="abouttheauthor">About the Author</h3>
This article was written by Jacob Stopak, a software consultant and developer with passion for helping others improve their lives through code. Jacob is the creator of <a target="_blank" href="https://initialcommit.com">Initial Commit</a> - a site dedicated to helping curious developers learn how their favorite programs are coded. Its featured project helps people <a rel="nofollow" target="_blank" href="https://initialcommit.com/blog/Learn-Git-Code-Definitive-Guide">learn Git</a> at the code level.

Jacob Stopak

Git: Merge Branch into Master

<h2 id="introduction">Introduction</h2>
A common task in programming is opening a file and parsing its contents. What do you do when the file you are trying to process is quite large, like several GB of data or larger? The answer to this problem is to read in chunks of a file at a time, process it, then free it from memory so you can process another chunk until the whole massive file has been processed. While it's up to you to determine a suitable size for the chunks of data you're processing, for many applications, it's suitable to process a file one line at a time.
Throughout this article, we'll be covering a number of code examples that demonstrate how to read files line by line. In case you want to try out some of these examples by yourself, the code used in this article can be found at the following <a rel="nofollow noopener" target="_blank" href="https://github.com/amcquistan/Python-read-file-line-by-line">GitHub repo</a>.
<ul>
<li><a href="#basicfileioinpython">Basic File IO in Python</a></li>
<li><a href="#readafilelinebylineinpythonwithreadline">Read a File Line-by-Line in Python with <code>readline()</code></a></li>
<li><a href="#readafilelinebylineinpythonwithreadlines">Read a File Line-by-Line in Python with <code>readlines()</code></a></li>
<li><a href="#readafilelinebylinewithaforloopmostpythonicapproach">Read a File Line-by-Line with a <code>for</code> Loop</a> - Best Approach!</li>
<li><a href="#applicationsofreadingfileslinebyline">Applications of Reading Files Line-by-Line</a></li>
</ul>
<h2 id="basicfileioinpython">Basic File IO in Python</h2>
Python is a great general-purpose programming language, and it has a number of very useful file IO functionality in its standard library of built-in functions and modules.
The built-in <code>open()</code> function is what you use to open a <a rel="nofollow noopener" target="_blank" href="https://docs.python.org/3/library/functions.html#open">file object</a> for either reading or writing purposes. Here's how you can use it to open a file:
<pre><code class="hljs">fp = open(&#x27;path/to/file.txt&#x27;, &#x27;r&#x27;)
</code></pre>
As demonstrated above, the <code>open()</code> function takes in multiple arguments. We will be focusing on two arguments, with the first being a positional string parameter representing the path to the file you want to open. The second (optional) parameter is also a string, and it specifies the mode of interaction you intend to be used on the file object being returned by the function call. The most common modes are listed in the table below, with the default being 'r' for reading:
<table class="table table-striped">
 <thead>
 <tr>
 <th>
 Mode
 </th>
 <th>
 Description
 </th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td><code>r</code></td>
 <td>Open for reading plain text</td>
 </tr>
 <tr>
 <td><code>w</code></td>
 <td>Open for writing plain text</td>
 </tr>
 <tr>
 <td><code>a</code></td>
 <td>Open an existing file for appending plain text</td>
 </tr>
 <tr>
 <td><code>rb</code></td>
 <td>Open for reading binary data</td>
 </tr>
 <tr>
 <td><code>wb</code></td>
 <td>Open for writing binary data</td>
 </tr>
 </tbody>
</table>
Once you have written or read all of the desired data in a file object, you need to close the file so that resources can be reallocated on the operating system that the code is running on.
<pre><code class="hljs">fp.close()
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: It's always good practice to close a file object resource, but it's a task that's easy to forget.

 </div>
 </div>
 </div>
 While you can always remember to call <code>close()</code> on a file object, there's an alternate and more elegant way to open a file object and ensure that the Python interpreter cleans up after its use:
<pre><code class="hljs">with open(&#x27;path/to/file.txt&#x27;) as fp:
 # Do stuff with fp
</code></pre>
By simply using the <a target="_blank" href="https://stackabuse.com/python-context-managers/"><code>with</code> keyword</a> (introduced in Python 2.5) to the code we use to open a file object, Python will do something similar to the following code. This ensures that no matter what the file object is closed after use:
<pre><code class="hljs">try:
 fp = open(&#x27;path/to/file.txt&#x27;)
 # Do stuff with fp
finally:
 fp.close()
</code></pre>
Either of these two methods is suitable, with the first example being more Pythonic.
The file object returned from the <code>open()</code> function has three common explicit methods (<code>read()</code>, <code>readline()</code>, and <code>readlines()</code>) to read in data. The <code>read()</code> method reads all the data into a single string. This is useful for smaller files where you would like to do text manipulation on the entire file. Then there is <code>readline()</code>, which is a useful way to only read in individual lines, in incremental amounts at a time, and return them as strings. The last explicit method, <code>readlines()</code>, will read all the lines of a file and return them as a list of strings.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: For the remainder of this article we will be working with the text of the book The &quot;Iliad of Homer&quot;, which can be found at <a rel="nofollow noopener" target="_blank" href="http://www.gutenberg.org">gutenberg.org</a>, as well as in the GitHub repo where the code is for this article.

 </div>
 </div>
 </div>
 <h2 id="readingafilelinebylineinpythonwithreadline">Reading a File Line-by-Line in Python with readline()</h2>
Let's start off with the <code>readline()</code> method, which reads a single line, which will require us to use a counter and increment it:
<pre><code class="hljs">filepath = &#x27;Iliad.txt&#x27;
with open(filepath) as fp:
 line = fp.readline()
 cnt = 1
 while line:
 print(&quot;Line {}: {}&quot;.format(cnt, line.strip()))
 line = fp.readline()
 cnt += 1
</code></pre>
This code snippet opens a file object whose reference is stored in <code>fp</code>, then reads in a line one at a time by calling <code>readline()</code> on that file object iteratively in a <code>while</code> loop. It then simply prints the line to the console.
Running this code, you should see something like the following:
<pre><code class="hljs">...
Line 567: exceedingly trifling. We have no remaining inscription earlier than the
Line 568: fortieth Olympiad, and the early inscriptions are rude and unskilfully
Line 569: executed; nor can we even assure ourselves whether Archilochus, Simonides
Line 570: of Amorgus, Kallinus, Tyrtaeus, Xanthus, and the other early elegiac and
Line 571: lyric poets, committed their compositions to writing, or at what time the
Line 572: practice of doing so became familiar. The first positive ground which
Line 573: authorizes us to presume the existence of a manuscript of Homer, is in the
Line 574: famous ordinance of Solon, with regard to the rhapsodies at the
Line 575: Panathenaea: but for what length of time previously manuscripts had
Line 576: existed, we are unable to say.
...
</code></pre>
Though, this approach is crude and explicit. Most certainly not very Pythonic. We can utilize the <code>readlines()</code> method to make this code much more succinct.
<h2 id="readafilelinebylinewithreadlines">Read a File Line-by-Line with readlines()</h2>
The <code>readlines()</code> method reads all the lines and stores them into a <code>List</code>. We can then iterate over that list and using <code>enumerate()</code>, make an index for each line for our convenience:
<pre><code class="hljs">file = open(&#x27;Iliad.txt&#x27;, &#x27;r&#x27;)
lines = file.readlines()

for index, line in enumerate(lines):
 print(&quot;Line {}: {}&quot;.format(index, line.strip()))
 
file.close()
</code></pre>
This results in:
<pre><code class="hljs">...
Line 160: INTRODUCTION.
Line 161:
Line 162:
Line 163: Scepticism is as much the result of knowledge, as knowledge is of
Line 164: scepticism. To be content with what we at present know, is, for the most
Line 165: part, to shut our ears against conviction; since, from the very gradual
Line 166: character of our education, we must continually forget, and emancipate
Line 167: ourselves from, knowledge previously acquired; we must set aside old
Line 168: notions and embrace fresh ones; and, as we learn, we must be daily
Line 169: unlearning something which it has cost us no small labour and anxiety to
Line 170: acquire.
...
</code></pre>
Now, although much better, we don't even need to call the <code>readlines()</code> method to achieve this same functionality. This is the traditional way of reading a file line-by-line, but there's a more modern, shorter one.
<h2 id="readafilelinebylinewithaforloopmostpythonicapproach">Read a File Line-by-Line with a for Loop - Most Pythonic Approach</h2>
The returned <code>File</code> itself is an iterable. We don't need to extract the lines via <code>readlines()</code> at all - we can iterate the returned object itself. This also makes it easy to <code>enumerate()</code> it so we can write the line number in each <code>print()</code> statement.
This is the shortest, most Pythonic approach to solving the problem, and the approach favored by most:
<pre><code class="hljs">with open(&#x27;Iliad.txt&#x27;) as f:
 for index, line in enumerate(f):
 print(&quot;Line {}: {}&quot;.format(index, line.strip()))
</code></pre>
This results in:
<pre><code class="hljs">...
Line 277: Mentes, from Leucadia, the modern Santa Maura, who evinced a knowledge and
Line 278: intelligence rarely found in those times, persuaded Melesigenes to close
Line 279: his school, and accompany him on his travels. He promised not only to pay
Line 280: his expenses, but to furnish him with a further stipend, urging, that,
Line 281: &quot;While he was yet young, it was fitting that he should see with his own
Line 282: eyes the countries and cities which might hereafter be the subjects of his
Line 283: discourses.&quot; Melesigenes consented, and set out with his patron,
Line 284: &quot;examining all the curiosities of the countries they visited, and
...
</code></pre>
Here, we're taking advantage of the built-in functionalities of Python that allow us to effortlessly iterate over an iterable object, simply using a <code>for</code> loop. If you'd like to read more about Python's built-in functionalities on iterating objects, we've got you covered:
<ul>
<li><a href="https://stackabuse.com/pythons-itertools-count-cycle-and-chain">Python's itertools – count(), cycle() and chain()</a></li>
<li><a href="https://stackabuse.com/pythons-iteration-tools-filter-islice-map-and-zip">Python's itertools: filter(), islice(), map() and zip()</a></li>
</ul>
<h2 id="applicationsofreadingfileslinebyline">Applications of Reading Files Line-by-Line</h2>
How can you use this practically? Most NLP applications deal with large corpora of data. Most of the time, it won't be wise to read the entire corpora into memory. While rudimentary, you can write a from-scratch solution to count the frequency of certain words, without using any external libraries. Let's write a simple script that loads in a file, reads it line-by-line, and counts the frequency of words, printing the 10 most frequent words and the number of their occurrences:
<pre><code class="hljs">import sys
import os

def main():
 filepath = sys.argv[1]
 if not os.path.isfile(filepath):
 print(&quot;File path {} does not exist. Exiting...&quot;.format(filepath))
 sys.exit()
 
 bag_of_words = {}
 with open(filepath) as fp:
 for line in fp:
 record_word_cnt(line.strip().split(&#x27; &#x27;), bag_of_words)
 sorted_words = order_bag_of_words(bag_of_words, desc=True)
 print(&quot;Most frequent 10 words {}&quot;.format(sorted_words[:10]))
 
def order_bag_of_words(bag_of_words, desc=False):
 words = [(word, cnt) for word, cnt in bag_of_words.items()]
 return sorted(words, key=lambda x: x[1], reverse=desc)

def record_word_cnt(words, bag_of_words):
 for word in words:
 if word != &#x27;&#x27;:
 if word.lower() in bag_of_words:
 bag_of_words[word.lower()] += 1
 else:
 bag_of_words[word.lower()] = 1

if __name__ == &#x27;__main__&#x27;:
 main()
</code></pre>
The script uses the <code>os</code> module to make sure that the file we're attempting to read actually exists. If so, its read line-by-line and each line is passed on into the <code>record_word_cnt()</code> function. It delimits the spaces between words and adds the word to the dictionary - <code>bag_of_words</code>. Once all the lines are recorded into the dictionary, we order it via <code>order_bag_of_words()</code> which returns a list of tuples in the <code>(word, word_count)</code> format, sorted by the word count.
Finally, we print the top ten most common words.
Typically, for this, you'd create a <a target="_blank" href="https://stackabuse.com/python-for-nlp-creating-bag-of-words-model-from-scratch">Bag of Words Model</a>, using libraries like NLTK, though, this implementation will suffice. Let's run the script and provide our <code>Iliad.txt</code> to it:
<pre><code class="hljs">$ python app.py Iliad.txt
</code></pre>
This results in:
<pre><code class="hljs">Most frequent 10 words [(&#x27;the&#x27;, 15633), (&#x27;and&#x27;, 6959), (&#x27;of&#x27;, 5237), (&#x27;to&#x27;, 4449), (&#x27;his&#x27;, 3440), (&#x27;in&#x27;, 3158), (&#x27;with&#x27;, 2445), (&#x27;a&#x27;, 2297), (&#x27;he&#x27;, 1635), (&#x27;from&#x27;, 1418)]
</code></pre>

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice: If you'd like to read more about NLP, we've got a series of guides on various tasks: <a target="_blank" href="https://stackabuse.com/tag/nlp">Natural Language Processing in Python</a>.

 </div>
 </div>
 </div>
 <h2 id="conclusion">Conclusion</h2>
In this article, we've explored multiple ways to read a file line-by-line in Python, as well as created a rudimentary Bag of Words model to calculate the frequency of words in a given file.

Scott Robinson

Adam McQuistan

In this tutorial, we'll be reading a file line by line in Python with the readline() and readlines() functions as well as a for loop - through hands-on examples.

Read a File Line-by-Line in Python

<h2 id="introduction">Introduction</h2>
Python comes with a variety of useful objects that can be used out of the box. Date objects are examples of such objects. Date types are difficult to manipulate from scratch, due to the complexity of dates and times. However, Python date objects make it extremely easy to convert dates into the desirable string formats.
Date formatting is a common task that you will face as a programmer, and communicating dates properly to end users, if your application exposes them, is an important part of UX.
<blockquote>
Different regions around the world have different ways of representing dates/times, therefore your goal as a programmer is to present the date values in a way that is readable to the users.
</blockquote>
For example, you may need to represent a date value numerically like &quot;02-23-2023&quot;. On the flip side, you may need to write the same date value in a longer textual format like &quot;Feb 23, 2023&quot;. In another scenario, you may want to extract the month in string format from a numerically formated date value.
In this article, we will study different types of date objects along with their functionalities.
<h2 id="thedatetimemodule">The datetime Module</h2>
Python's <code>datetime</code> module contains methods that can be used to work with date and time values. To use this module, we first import it via the <code>import</code> statement as follows:
<pre><code class="hljs">import datetime
</code></pre>
We can represent time values using the <code>time</code> class. The attributes for the <code>time</code> class include the hour, minute, second and microsecond.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: The arguments for the <code>time</code> class are optional. Although if you don't specify any argument you will get back a time of 0, which is unlikely to be what you need most of the time.

 </div>
 </div>
 </div>
 For example, you can initialize a time object with a value of 1 hour, 10 minutes, 20 seconds and 13 microseconds using:
<pre><code class="hljs">t = datetime.time(1, 10, 20, 13)
</code></pre>
To see the time, let's use the <code>print</code> function:
<pre><code class="hljs">print(t)
# 01:10:20.000013
</code></pre>
To extract fields, such as the hour, minute, second or microsecond - you can access each field respectively:
<pre><code class="hljs">print(&#x27;hour:&#x27;, t.hour)
# hour: 1
</code></pre>
The minutes, seconds and microseconds for the above time can be retrieved as follows:
<pre><code class="hljs">print(&#x27;Minutes:&#x27;, t.minute) # Minutes: 10
print(&#x27;Seconds:&#x27;, t.second) # Seconds: 20
print(&#x27;Microsecond:&#x27;, t.microsecond) # Microseconds: 13
</code></pre>
The values for the calendar date can be represented via the <code>date</code> class. The instances will have attributes for year, month, and day.
Let us call the <code>today()</code> method to see today's date:
<pre><code class="hljs">import datetime

today = datetime.date.today()
print(today) # 2022-09-15
</code></pre>
The code will return the date for today, therefore the output you see will depend on the day you run the above script.
Now let's call the <code>ctime</code> method to print the date in another format:
<pre><code class="hljs">print(&#x27;ctime:&#x27;, today.ctime())
# ctime: Sat Sep 15 00:00:00 2022
</code></pre>
The <code>ctime</code> method uses a longer date-time format than the examples we saw before. This method is primarily used for converting Unix-time (the number of seconds since Jan. 1st, 1970) to a string format.
And here is how we can display the year, the month, and the day using the <code>date</code> class:
<pre><code class="hljs">print(&#x27;Year:&#x27;, today.year)
print(&#x27;Month:&#x27;, today.month)
print(&#x27;Day :&#x27;, today.day)
</code></pre>
This results in:
<pre><code class="hljs">Year: 2022
Month: 9
Day : 15
</code></pre>
<h2 id="convertingdatestostringswithstrftime">Converting Dates to Strings with strftime()</h2>
Now that you know how to create Date and Time objects, let us learn how to format them into more readable strings.
To achieve this, we will be using the <code>strftime()</code> method. This method helps us convert date objects into readable strings. It takes two parameters, as shown in the following syntax:
<pre><code class="hljs">time.strftime(format, t)
</code></pre>
The first parameter is the format string, while the second parameter is the time to be formatted, which is optional.
This method can also be used on a <code>datetime</code> object directly, as shown in the following example:
<pre><code class="hljs">import datetime

x = datetime.datetime(2022, 9, 15)
print(x.strftime(&quot;%b %d %Y %H:%M:%S&quot;))
# Sep 15 2022 00:00:00
</code></pre>
We have used the following character strings to format the date:
<ul>
<li><code>%b</code>: Returns the first three characters of the month name. In our example, it returned &quot;Sep&quot;.</li>
<li><code>%d</code>: Returns day of the month, from 1 to 31. In our example, it returned &quot;15&quot;.</li>
<li><code>%Y</code>: Returns the year in four-digit format. In our example, it returned &quot;2022&quot;.</li>
<li><code>%H</code>: Returns the hour. In our example, it returned &quot;00&quot;.</li>
<li><code>%M</code>: Returns the minute, from 00 to 59. In our example, it returned &quot;00&quot;.</li>
<li><code>%S</code>: Returns the second, from 00 to 59. In our example, it returned &quot;00&quot;.</li>
</ul>
We did not pass a time, hence the values for time are all &quot;00&quot;. The following example shows how the time can be formatted as well:
<pre><code class="hljs">import datetime

x = datetime.datetime(2022, 9, 15, 12, 45, 35)
print(x.strftime(&quot;%b %d %Y %H:%M:%S&quot;))
# Sep 15 2022 12:45:35
</code></pre>
<h3 id="thecompletecharactercodelist">The Complete Character Code List</h3>
Other than the character strings given above, the <code>strftime</code> method takes several other directives for formatting date values:
<ul>
<li><code>%a</code>: Returns the first three characters of the weekday, e.g. Wed.</li>
<li><code>%A</code>: Returns the full name of the weekday, e.g. Wednesday.</li>
<li><code>%B</code>: Returns the full name of the month, e.g. September.</li>
<li><code>%w</code>: Returns the weekday as a number, from 0 to 6, with Sunday being 0.</li>
<li><code>%m</code>: Returns the month as a number, from 01 to 12.</li>
<li><code>%p</code>: Returns AM/PM for time.</li>
<li><code>%y</code>: Returns the year in two-digit format, that is, without the century. For example, &quot;18&quot; instead of &quot;2018&quot;.</li>
<li><code>%f</code>: Returns microsecond from 000000 to 999999.</li>
<li><code>%Z</code>: Returns the timezone.</li>
<li><code>%z</code>: Returns UTC offset.</li>
<li><code>%j</code>: Returns the number of the day in the year, from 001 to 366.</li>
<li><code>%W</code>: Returns the week number of the year, from 00 to 53, with Monday being counted as the first day of the week.</li>
<li><code>%U</code>: Returns the week number of the year, from 00 to 53, with Sunday counted as the first day of each week.</li>
<li><code>%c</code>: Returns the local date and time version.</li>
<li><code>%x</code>: Returns the local version of date.</li>
<li><code>%X</code>: Returns the local version of time.</li>
</ul>
Consider the following example:
<pre><code class="hljs">import datetime

x = datetime.datetime(2022, 9, 15)
print(x.strftime(&#x27;%b/%d/%Y&#x27;))
# Sep/15/2022
</code></pre>
And here is how you can get the month only:
<pre><code class="hljs">print(x.strftime(&#x27;%B&#x27;))
# September
</code></pre>
Let us display the year:
<pre><code class="hljs">print(x.strftime(&#x27;%Y&#x27;))
# 2022
</code></pre>
In this example we have used the format code <code>%Y</code>. Notice that the <code>Y</code> is in uppercase. Now write it in lowercase:
<pre><code class="hljs">print(x.strftime(&#x27;%y&#x27;))
# 22
</code></pre>
As you can see, with these formatting codes you can represent the date-time in just about any form that you'd like.
<h2 id="convertingstringstodateswithstrptime">Converting Strings to Dates with strptime</h2>
The <code>strftime()</code> method helped us convert date objects into more readable strings. The <code>strptime()</code> method does the opposite, that is, it takes strings and converts them into date objects that Python can understand.
<blockquote>
This is useful when you receive a string-formatted date, and wish to convert it to a different string-formatted date. By converting to an intermediary datetime object, you gain access to a parsed structure that can be reformatted to any other format.
</blockquote>
Here is the syntax for the method:
<pre><code class="hljs">datetime.strptime(string, format)
</code></pre>
The <code>string</code> parameter is the value in string format that we want to convert into date format. The <code>format</code> parameter is the directive specifying the format to be taken by the date after the conversion.
For example, let's say we need to convert the string &quot;9/15/22&quot; into a <code>datetime</code> object.
Let's first import the <code>datetime</code> module:
<pre><code class="hljs">from datetime import datetime
</code></pre>
We can then define the date in the form of a string:
<pre><code class="hljs">str = &#x27;9/15/22&#x27;
</code></pre>
Python will not be able to understand the above string as a datetime until we convert it to an actual <code>datetime</code> object. We can successfully do so by calling the <code>strptime</code> method.
Execute the following command to convert the string:
<pre><code class="hljs">date_object = datetime.strptime(str, &#x27;%m/%d/%y&#x27;)
</code></pre>
Let's now call the <code>print</code> function to display the string in <code>datetime</code> format:
<pre><code class="hljs">print(date_object)
# 2022-09-15 00:00:00
</code></pre>
As you can see, the conversion was successful!
<blockquote>
You can now convert this date object to any other string format.
</blockquote>
Handling different separators is as easy as using them in the format string:
<pre><code class="hljs">from datetime import datetime

str = &#x27;9-15-22&#x27;
date_object = datetime.strptime(str, &#x27;%m-%d-%y&#x27;)
print(date_object)
# 2022-09-15 00:00:00
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this article, we studied how to format dates in Python. We saw how the <code>datetime</code> module in Python can be used for the manipulation of date and time values. The module contains a number of classes that can be used for this purpose. For example, the <code>time</code> class is used to represent time values while the <code>date</code> class is used to represent calendar date values.

Nicholas Samuel

How to Format Dates in Python

<h2 id="introduction">Introduction</h2>
There are many reasons why you might want to extract the filename and extension of a file in Bash:
<ul>
<li>
To manipulate the file name or extension - You may want to extract the filename or extension in order to modify it, such as adding a prefix or suffix to the file name, or changing the file extension.
</li>
<li>
To create a file with a unique name - You may want to extract the filename and extension in order to create a new file with a unique name, such as by adding a timestamp or a random number to the file name.
</li>
<li>
To use the filename or extension in a script or command - You may want to extract the filename or extension in order to use it as an argument or input for a script or command, such as to pass it to a program or to create a file with the same name as a directory.
</li>
<li>
To extract information from the file name or extension - You may want to extract the filename or extension in order to extract information from it, such as the date or the file type.
</li>
</ul>
<blockquote>
In this article, we'll take a look at the three most common ways you can extract filename and file extension in Bash. We'll take a look at each of them and give you the pros and cons, so you can make an educated decision on what approach suits you the best.
</blockquote>
<h2 id="method1usingthebasenamecommand">Method 1: Using the basename Command</h2>
The <code>basename</code> command can be used to extract the filename and extension from a file path:
<pre><code class="hljs">filename=$(basename /path/to/file.txt)
echo $filename
# Output: &quot;file.txt&quot;
</code></pre>
Although this method is quite simple and easy to use, unfortunately, there is no way we can extract only the file name (with no extension) without any post-processing.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 You can also use the <code>dirname</code> command to extract the directory path separately:
<pre><code class="hljs">directory=$(dirname /path/to/file.txt)
echo $directory
# Output: &quot;/path/to&quot;
</code></pre>

 </div>
 </div>
 </div>
 <h3 id="pros">Pros:</h3>
<ul>
<li>Simple to use</li>
<li>Handles filenames with spaces correctly</li>
</ul>
<h3 id="cons">Cons:</h3>
<ul>
<li>Only extracts the filename and cannot extract the extension separately without additional processing</li>
</ul>
<h2 id="method2usingparameterexpansion">Method 2: Using Parameter Expansion</h2>
Bash provides a feature called parameter expansion, which allows you to extract parts of a variable using a special syntax. For example, you can use the following syntax to extract the filename and extension from a file path stored in a variable:
<pre><code class="hljs">filepath=&quot;/path/to/file.txt&quot;
filename=${filepath##*/}
echo $filename
# Output: &quot;file.txt&quot;
</code></pre>
You can also use parameter expansion to extract the extension separately:
<pre><code class="hljs">extension=${filename##*.}
echo $extension
# Output: &quot;txt&quot;
</code></pre>
<h3 id="pros">Pros:</h3>
<ul>
<li>Flexible</li>
<li>Can extract both the filename and extension separately,</li>
<li>Handles filenames with spaces correctly</li>
</ul>
<h3 id="cons">Cons:</h3>
<ul>
<li>Requires a variable to store the file path</li>
</ul>
<h2 id="method3usingtheawkcommand">Method 3: Using the awk command</h2>
The <code>awk</code> command is a powerful text processing tool that can be used to extract parts of a string. For example, you can use the following syntax to extract the filename and extension from a file path:
<pre><code class="hljs">filepath=&quot;/path/to/file.txt&quot;
filename=$(echo $filepath | awk -F/ &#x27;{print $NF}&#x27;)
echo $filename
# Output: &quot;file.txt&quot;
</code></pre>
You can also use <code>awk</code> to extract the extension separately:
<pre><code class="hljs">extension=$(echo $filename | awk -F. &#x27;{print $NF}&#x27;)
echo $extension
# Output: &quot;txt&quot;
</code></pre>
<h3 id="pros">Pros:</h3>
<ul>
<li>Powerful</li>
<li>Can extract both the filename and extension separately</li>
<li>Handles filenames with spaces correctly</li>
</ul>
<h3 id="cons">Cons:</h3>
<ul>
<li>Syntax may be unfamiliar to some users</li>
<li>Requires piping the file path through <code>awk</code></li>
</ul>
<h2 id="conclusion">Conclusion</h2>
Overall, extracting the filename and extension of a file in Bash can be a useful technique for working with files and performing various tasks in the Bash shell.
Each of the mentioned methods has its own advantages and disadvantages, and the best choice will depend on your specific needs and preferences. It is often useful to be familiar with multiple approaches so that you can choose the one that is most suitable for your situation.

In this short tutorial, learn how to extract the filename and extension using Bash.

Extract Filename and Extension in Bash

In Python, strings are sequences of characters, which are effectively stored in memory as an object. Each object can be identified using the <code>id()</code> method, as you can see below. Python tries to re-use objects in memory that have the same value, which also makes comparing objects very fast in Python:
<pre><code class="hljs">$ python
Python 3.9.0 (v3.9.0:9cf6752276, Oct 5 2020, 11:29:23) 
[Clang 6.0 (clang-600.0.57)] on darwin
Type &quot;help&quot;, &quot;copyright&quot;, &quot;credits&quot; or &quot;license&quot; for more information.
&gt;&gt;&gt; a = &quot;abc&quot;
&gt;&gt;&gt; b = &quot;abc&quot;
&gt;&gt;&gt; c = &quot;def&quot;
&gt;&gt;&gt; print (id(a), id(b), id(c))
(139949123041320, 139949123041320, 139949122390576)
&gt;&gt;&gt; quit()
</code></pre>
In order to compare strings, Python offers a few different operators to do so. First, we will explain them in more detail below. Second, we'll go over both the <code>string</code> and the <code>re</code> modules, which contain methods to handle case-insensitive and inexact matches. Third, to deal with multi-line strings the <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/difflib.html"><code>difflib</code> module</a> is quite handy. A number of examples will help you to understand how to use them.
<h2 id="comparestringswiththetheandoperators">Compare Strings with the The == and != Operators</h2>
As a basic comparison operator you'll want to use <code>==</code> and <code>!=</code>. They work in exactly the same way as with integer and float values. The <code>==</code> operator returns <code>True</code> if there is an exact match, otherwise <code>False</code> will be returned. In contrast, the <code>!=</code> operator returns <code>True</code> if there is no match and otherwise returns <code>False</code>. Listing 1 demonstrates this.
In a <code>for</code> loop, a string containing the name of the Swiss city &quot;Lausanne&quot; is compared with an entry from a list of other places, and the comparison result is printed using <code>stdout</code>.
Listing 1:
<pre><code class="hljs"># Define strings
listOfPlaces = [&quot;Berlin&quot;, &quot;Paris&quot;, &quot;Lausanne&quot;]
currentCity = &quot;Lausanne&quot;

for place in listOfPlaces:
 print (f&quot;comparing {place} with {currentCity}: %{place == currentCity}&quot;)
</code></pre>
Running the Python script from above the output is as follows:
<pre><code class="hljs">$ python3 comparing-strings.py
comparing Berlin with Lausanne: False
comparing Paris with Lausanne: False
comparing Lausanne with Lausanne: True
</code></pre>
<h2 id="theandisoperators">The <code>==</code> and <code>is</code> Operators</h2>
Python has the two comparison operators <code>==</code> and <code>is</code>. At first sight they seem to be the same, but actually they are not.
<blockquote>
<code>==</code> compares two variables based on the value they represent. In contrast, the <code>is</code> operator compares two variables based on the object ID in memory.
</blockquote>
John (Doe) and John (Moe) are both called John. If we can reduce them to just their names, they'd be equal in value, but still two different people qualitatively.
The next example demonstrates that for three variables with string values. The two variables <code>a</code> and <code>b</code> have the same value, and Python refers to the same object in order to minimize memory usage.
This is done for simple types and strings by default, but not for other objects:
<pre><code class="hljs">&gt;&gt;&gt; a = &#x27;hello&#x27;
&gt;&gt;&gt; b = &#x27;hello&#x27;
&gt;&gt;&gt; c = &#x27;world&#x27;
&gt;&gt;&gt; a is b
True
&gt;&gt;&gt; a is c
False
&gt;&gt;&gt; id(a)
140666888153840
&gt;&gt;&gt; id(b)
140666888153840
&gt;&gt;&gt; 
</code></pre>
As soon as the value changes Python will re-instantiate the object and assign the variable. In the next code snippet <code>b</code> gets the value of 2, and subsequently <code>b</code> and <code>c</code> refer to the same object:
<pre><code class="hljs">&gt;&gt;&gt; b = &#x27;world&#x27;
&gt;&gt;&gt; id(b)
140666888154416
&gt;&gt;&gt; id(c)
140666888154416
</code></pre>
<h2 id="morecomparisonoperators">More Comparison Operators</h2>
For a comparison regarding a lexicographical order you can use the comparison operators <code>&lt;</code>, <code>&gt;</code>, <code>&lt;=</code>, and <code>&gt;=</code>. The comparison itself is done character by character. The order depends on the order of the characters in the alphabet. This order depends on the character table that is in use on your machine while executing the Python code.
Keep in mind the order is case-sensitive. As an example for the Latin alphabet, &quot;Bus&quot; comes before &quot;bus&quot;. Listing 2 shows how these comparison operators work in practice.
Listing 2:
<pre><code class="hljs"># Define the strings
listOfPlaces = [&quot;Berlin&quot;, &quot;Paris&quot;, &quot;Lausanne&quot;]
currentCity = &quot;Lausanne&quot;

for place in listOfPlaces:
 if place &lt; currentCity:
 print (f&quot;{place} comes before {currentCity}&quot;)
 elif place &gt; currentCity:
 print (f&quot;{place} comes after {currentCity}&quot;)
 else:
 print (f&quot;{place} is equal to {currentCity}&quot;)
</code></pre>
Running the Python script from above the output is as follows:
<pre><code class="hljs">$ python3 comparing-strings-order.py
Berlin comes before Lausanne
Paris comes after Lausanne
Lausanne is equal to Lausanne
</code></pre>
<h2 id="caseinsensitivestringcomparisons">Case-Insensitive String Comparisons</h2>
The previous examples focused on exact matches between strings. To allow case-insensitive comparisons Python offers special string methods such as <code>upper()</code> and <code>lower()</code>. Both of them are directly available as methods of the according string object.
<code>upper()</code> converts the entire string into uppercase letters, and <code>lower()</code> into lowercase letters, respectively. Based on Listing 1 the next listing shows how to use the <code>lower()</code> method.
Listing 3:
<pre><code class="hljs"># using the == operator
listOfPlaces = [&quot;Berlin&quot;, &quot;Paris&quot;, &quot;Lausanne&quot;]
currentCity = &quot;lausANne&quot;

for place in listOfPlaces:
 print (f&quot;comparing {place} with {place.lower() == currentCity.lower()}: {currentCity}&quot;)
</code></pre>
The output is as follows:
<pre><code class="hljs">$ python3 comparing-strings-case-insensitive.py
comparing Berlin with lausANne: False
comparing Paris with lausANne: False
comparing Lausanne with lausANne: True
</code></pre>
<h2 id="comparestringsusingregularexpressionsregex">Compare Strings Using Regular Expressions (RegEx)</h2>
A <a target="_blank" href="/using-regex-for-text-manipulation-in-python/">Regular Expression</a> - or &quot;regex&quot; for short - defines a specific pattern of characters.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice: If you'd like to read more about Regular Expressions, read our <a href="https://stackabuse.com/introduction-to-regular-expressions-in-python/">&quot;Introduction to Regular Expressions in Python&quot;</a>!

 </div>
 </div>
 </div>
 To make use of this mechanism in Python import the <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/re.html"><code>re</code> module</a> first and define a specific pattern, next. Again, the following example is based on Listing 1. The search pattern matches &quot;bay&quot;, and begins with either a lowercase or an uppercase letter. Precisely, the following Python code finds all the strings in which the search pattern occurs no matter at which position of the string - at the beginning, or in the middle, or at the end.
Listing 4:
<pre><code class="hljs"># import the additional module
import re

# define list of places
listOfPlaces = [&quot;Bayswater&quot;, &quot;Table Bay&quot;, &quot;Beijing&quot;, &quot;Bombay&quot;]

# define search string
pattern = re.compile(&quot;[Bb]ay&quot;)

for place in listOfPlaces:
 if pattern.search(place):
 print (f&quot;{place} matches the search pattern&quot;)
</code></pre>
The output is as follows, and matches &quot;Bayswater&quot;, &quot;Table Bay&quot;, and &quot;Bombay&quot; from the list of places:
<pre><code class="hljs">$ python3 comparing-strings-re.py
Bayswater matches the search pattern
Table Bay matches the search pattern
Bombay matches the search pattern
</code></pre>
<h2 id="multilineandlistcomparisons">Multi-Line and List Comparisons</h2>
So far our comparisons have only been on a few words. Using the <code>difflib</code> module Python also offers a way to compare multi-line strings, and entire lists of words. The output can be configured according to various formats of diff tools.
The next example (Listing 5) compares two multi-line strings line by line, and shows deletions as well as additions. After the initialization of the <code>Differ</code> object in line 12 the comparison is made using the <code>compare()</code> method in line 15. The result is printed on the standard output:
<pre><code class="hljs"># Import the additional module
import difflib
 
# Define original text, 
# taken from: https://en.wikipedia.org/wiki/Internet_Information_Services
original = [&quot;About the IIS&quot;, &quot;&quot;, &quot;IIS 8.5 has several improvements related&quot;, &quot;to performance in large-scale scenarios, such&quot;, &quot;as those used by commercial hosting providers and Microsoft&#x27;s&quot;, &quot;own cloud offerings.&quot;]

# Define modified text
edited = [&quot;About the IIS&quot;, &quot;&quot;, &quot;It has several improvements related&quot;, &quot;to performance in large-scale scenarios.&quot;]

# Initiate the Differ object
d = difflib.Differ()
 
# Calculate the difference between the two texts
diff = d.compare(original, edited)
 
# Output the result
print (&#x27;\n&#x27;.join(diff))
</code></pre>
Running the script creates the output as seen below. Lines with deletions are indicated by <code>-</code> signs whereas lines with additions start with a <code>+</code> sign. Furthermore, lines with changes start with a question mark. Changes are indicated using <code>^</code> signs at the according position. Lines without an indicator are still the same:
<pre><code class="hljs">$ python comparing-strings-difflib.py
 About the IIS
 
- IIS 8.5 has several improvements related
? ^^^^^^

+ It has several improvements related
? ^

- to performance in large-scale scenarios, such
? ^^^^^^

+ to performance in large-scale scenarios.
? ^

- as those used by commercial hosting providers and Microsoft&#x27;s
- own cloud offerings.
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this article you have learned various ways to compare strings in Python. We hope that this overview helps you effectively program in your developer life.
<h2 id="acknowledgements">Acknowledgements</h2>
The author would like to thank Mandy Neumeyer for her support while preparing the article.

Frank Hofmann

Comparing Strings using Python

In this tutorial, we're diving into Command Line Arguments in Python. Using the sys module, getopt module and argparse module, we'll parse and read arguments.