Scott Robinson

Uchechukwu Azubuko

Guide to Building Rest APIs with Strapi

<h3 id="introduction">Introduction</h3>
When creating a website or web application, especially if they feature a lot of templated content (such as a grid or list of items belonging to a category) - it's generally a good idea to divide it into pages to reduce the number of items that appear on the screen at once.
<blockquote>
In this article, we will learn how to implement pagination in our web projects using vanilla JavaScript from the ground up.
</blockquote>
For the purpose of this article, we will fetch the content from <a rel="nofollow noopener noreferrer" target="_blank" href="https://jsonplaceholder.typicode.com/posts">this</a> example API response. It contains 100 data points, but we'll only use 30 and display 6 posts on our screen at a time.
<img src="https://s3.stackabuse.com/media/articles/pagination-in-javascript-1.gif" alt="pagination-in-javascript-animation">
<h3 id="gettingstarted">Getting Started</h3>
Before fetching all required posts from the API using the browsers built-in Fetch API, we first need to create a basic HTML markup that will be dynamically filled via our script later:
<pre><code class="hljs">&lt;div class=&quot;container&quot;&gt;
 &lt;div class=&quot;title-section&quot;&gt;
 &lt;h1 class=&quot;title&quot;&gt;Posts&lt;/h1&gt;
 &lt;/div&gt;
 &lt;div class=&quot;posts-section&quot;&gt;&lt;/div&gt;
 &lt;div class=&quot;pagination-section&quot;&gt;
 &lt;ul class=&quot;pagination&quot;&gt;
 &lt;li class=&quot;page-item prev&quot;&gt;&lt;a class=&quot;page-link&quot; href=&quot;#&quot;&gt;Previous&lt;/a&gt;&lt;/li&gt;
 &lt;li class=&quot;page-item next&quot;&gt;&lt;a class=&quot;page-link&quot; href=&quot;#&quot;&gt;Next&lt;/a&gt;&lt;/li&gt;
 &lt;/ul&gt;
 &lt;/div&gt;
&lt;/div&gt;
</code></pre>
<h3 id="fetchingrestapiposts">Fetching REST API Posts</h3>
As mentioned before, pagination is all about splitting up content into smaller chunks. It requires you to fetch the data, decide when and how to split, calculate the number of pages, and then show a portion of it to the user. Thankfully - back-end functionality usually takes care of the first few tasks, and return the relevant page, total number of pages, and the content per page.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Depending on the specific API you're working with - you may or may not be able to lazy load results. Whenever possible - prefer lazy loading results instead of loading them all upfront. Most modern APIs follow practices that let you set a <code>limit</code> or <code>page</code> count, and return the total number of pages that you can display to the user.

 </div>
 </div>
 </div>
 Let's get started by first fetching all our posts and then later we will amend this to query only a few data points per page:
<pre><code class="hljs">const postsSection = document.querySelector(&quot;.posts-section&quot;);

const fetchPosts = async () =&gt; {
 const response = await fetch(
 &quot;https://jsonplaceholder.typicode.com/posts&quot;
 );
 const posts = await response.json();
 postsSection.innerHTML = &quot;&quot;;
 posts.forEach((post) =&gt; {
 postsSection.innerHTML += `
 &lt;div class=&quot;posts-card&quot;&gt;
 &lt;div class=&quot;post-title&quot;&gt;
 &lt;h2 class=&quot;post-title-text&quot;&gt;${post.title}&lt;/h2&gt;
 &lt;/div&gt;
 &lt;div class=&quot;post-body&quot;&gt;
 &lt;p class=&quot;post-body-text&quot;&gt;
 ${post.body}
 &lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt; 
 `;
 });
};

fetchPosts();
</code></pre>
Let's quickly examine the code above. First of all, we began by obtaining the <code>div</code> element where we would be displaying all of our content via the <code>class</code> name we assigned to the <code>div</code>. Finally, we wrote a function to handle the fetch operation.
In the <code>fetchPosts()</code> function, we used the Fetch API to retrieve posts from the JSON Placeholder posts API, then store the JSON data referenced by the <code>posts</code> variable and used the <code>innerHTML</code> property to add each piece of content to the <code>posts-section</code> by looping through them.
At this point, we have successfully fetched all of our content.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: You can also fetch the content using a different method, but make sure that all of your content is loaded on that page before we dive into creating the pagination.

 </div>
 </div>
 </div>
 <h3 id="howtoimplementpaginationinjavascript">How to Implement Pagination in JavaScript</h3>
Let's begin by declaring three variables that are critical for implementing pagination within our web page. The first one is the number of posts we want to declare per page, then the current page number (<code>1</code> by default), and the total number of pages.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: When consuming data from a startdard API and database, the total count of the pages or the data points is usually returned. If we don't receive a total page count, it can be calculated via the total object count and the page size.

 </div>
 </div>
 </div>
 For this guide we will give the total number of pages a fixed number of <code>30</code>:
<pre><code class="hljs">const numberPerPage = 6;
var pageNumber = 1;
var numberOfPages = 30;
</code></pre>
In the previous section, we've displayed all posts on a single page, but we want to show only six at a time. Therefore, we've set the <code>numberPerPage</code> to <code>6</code> which we will now use to adjust the fetch operation so it only displays <code>6</code> posts.
It depends on the specific implementation, but it's wide-spread practice to allow query parameters to be used when fetching results from APIs, that allow you to fetch a certain page of the results. For example the mock REST API that we're using allows for the <code>page</code> and <code>limit</code> parameters that let you only load batches that you'll show at a given time.
<blockquote>
This way, we only load the data we want to show to the user! We can then either pre-fetch the next page for faster loading time, or revel in the computational speedup achieved by only loading the data to be shown.
</blockquote>
We'll be making use of these parameters by ammending our fetch request:
<pre><code class="hljs">const fetchPosts = async (pageNumber) =&gt; {
 const response = await fetch(
 `https://jsonplaceholder.typicode.com/posts?_page=${pageNumber}&amp;_limit=${numberPerPage}`
 );
 const posts = await response.json();
 postsSection.innerHTML = &quot;&quot;;
 posts.forEach((post) =&gt; {
 postsSection.innerHTML += `
 &lt;div class=&quot;posts-card&quot;&gt;
 &lt;div class=&quot;post-title&quot;&gt;
 &lt;h2 class=&quot;post-title-text&quot;&gt;${post.title}&lt;/h2&gt;
 &lt;/div&gt;
 &lt;div class=&quot;post-body&quot;&gt;
 &lt;p class=&quot;post-body-text&quot;&gt;
 ${post.body}
 &lt;/p&gt;
 &lt;/div&gt;
 &lt;/div&gt; 
 `;
 });
};

fetchPosts();
</code></pre>
In the above code, we added the two parameters to the API endpoint which are the <code>pageNumber</code> and the number of posts per page which would help us breeak our posts into several pages and then these posts can now be displayed based on the page number.
Additionally, we also passed in the <code>pageNumber</code> to the <code>fetchPosts()</code> function so we can call this function whenever the page changes:
<img src="https://s3.stackabuse.com/media/articles/pagination-in-javascript-2.png" alt="javascript-pagination-no-navigation">
<blockquote>
Let's now add functionalitry to the page navigation buttons at the bottom of our page and have them display appropriate content based on the page number.
</blockquote>
<h4 id="implementingpaginationlogic">Implementing Pagination Logic</h4>
We will notice that in the markup we had a section that showing the pagination buttons:
<pre><code class="hljs">&lt;div class=&quot;pagination-section&quot;&gt;
 &lt;ul class=&quot;pagination&quot;&gt;
 &lt;li class=&quot;page-item prev&quot;&gt;&lt;a class=&quot;page-link&quot; href=&quot;#&quot;&gt;Previous&lt;/a&gt;&lt;/li&gt;
 &lt;li class=&quot;page-item next&quot;&gt;&lt;a class=&quot;page-link&quot; href=&quot;#&quot;&gt;Next&lt;/a&gt;&lt;/li&gt;
 &lt;/ul&gt;
&lt;/div&gt;
</code></pre>
We are now going to add <code>click</code> events to each button so that when they are clicked, the content intended for that page appears.
<h5 id="implementingthepreviousbutton">Implementing the Previous Button</h5>
The logic here is pretty simple. All we have to do is to retrieve the element representing the previous button, add the <code>click</code> event listener to it and show the appropriate content when the button is clicked:
<pre><code class="hljs">// Add event listeners to the prev button
const prev = document.querySelector(&#x27;.prev&#x27;);
prev.addEventListener(&#x27;click&#x27;, (e) =&gt; {
 e.preventDefault();
 if (pageNumber &gt; 1) {
 pageNumber--;
 fetchPosts(pageNumber);
 }
});
</code></pre>
After adding a <code>click</code> event listener, we've checked if the current page number is greater than <code>1</code> in the call back function. If the number is equal to or less than <code>1</code>, we'll just keep showing the current content. But if the current page number is greater than <code>1</code> we can freely decrement its value and call the <code>fetchPosts(pageNumber)</code> function with the new page number passed in it as an argument, therefore showing the content of the previous page.
<h5 id="implementingthenextbutton">Implementing the Next Button</h5>
The logic here is completely the same as for the previous button, with just a few minor changes. Of course, we'll retrieve the list element with the class <code>next</code> instead of <code>prev</code>. Also, we'll check whether the current page number is less than the number of pages which we set to <code>30</code> manually earlier. In the end, we'll increment the current page number instead of decrementing it:
<pre><code class="hljs">// Add event listeners to the next button
const next = document.querySelector(&quot;.next&quot;);
next.addEventListener(&quot;click&quot;, (e) =&gt; {
 e.preventDefault();
 if (pageNumber &lt; numberOfPages) {
 pageNumber++;
 fetchPosts(pageNumber);
 }
});
</code></pre>
<h3 id="conclusion">Conclusion</h3>
Splitting content into smaller, more manageable chunks that are displayed individually is crucial whenever you try to display a database or any other data source on your web application.
In this article, we took a look at how to implement pagination with JavaScript from scratch, without any external libraries and tools.

Guest Contributor

David Landup

Dimitrije Stamenic

In this comprehensive tutorial, we'll take a look at how to split the content up into individual pages in Vanilla JavaScript. We'll learn how to create pagination and how to navigate through the individual pages without the use of any external libraries.

Pagination in Vanilla JavaScript

<h3 id="introduction">Introduction</h3>
It is important to check for the existence of directories and files in Bash scripts for several reasons. Firstly, it allows the script to handle situations where the expected directories or files do not exist. This can prevent errors and unexpected behavior in the script - if a script expects a certain file to be present in a certain directory, and that file does not exist, the script will fail if it does not check for the file's existence first.
Secondly, checking for the existence of directories and files can help to ensure that the script is running in the correct environment. Some scripts are intended to run on a system with a specific directory structure or set of files. By checking for the existence of these directories and files, the script can ensure that it is running in the correct environment before attempting to execute its tasks.
Finally, checking for the existence of directories and files can help to make Bash scripts more efficient by avoiding unnecessary tasks. If a script does not need to create a new directory if it already exists, the script can check for the directory's existence first and only create it if necessary. This can save time and resources compared to blindly creating the directory every time the script is run.
<blockquote>
In this article, we'll take a look at several most common methods you can use to check if a file or directory exists in Bash.
</blockquote>
<h3 id="howtocheckifadirectoryexistsinbash">How to Check if a Directory Exists in Bash</h3>
There are several methods for checking for the existence of directories in Bash, and all of them are pretty similar to methods for checking for file existence. In this section, we'll take a look at several methods to do so. Each of the covered methods can be easily altered to check for the existence of files in Bash, too. But we'll go over that in the latter sections. For now, let's just focus on checking for the existence of a directory.
<h4 id="method1theifddirectorycommandoperator">Method 1: The if [ -d $directory ] Command ([ Operator)</h4>
The <code>if [ -d $directory ]</code> command can be used to check for the existence of a directory. If the specified directory exists, the command will return an exit status of <code>0</code> (<code>true</code>). If the directory does not exist, the command will return a non-zero exit status (<code>false</code>).
Let's take a look at the general syntax of the <code>[ -d $directory ]</code> command:
<pre><code class="hljs">if [ -d &quot;$directory&quot; ]
then
 # code to execute if $directory exists
else
 # code to execute if $directory does not exist
fi
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: In this example, the <code>$directory</code> represents the name of the directory you want to check for.

 </div>
 </div>
 </div>
 As you probably guessed, if the directory exists, the code within the <code>then</code> block will be executed. If the directory does not exist, the code within the <code>else</code> block will be executed.
Now we can take a look at the practical example of how you might use this command in a Bash script:
<pre><code class="hljs"># Check if the /my-dir directory exists
directory=&quot;/my-dir&quot;
if [ -d &quot;$directory&quot; ]
then
 # The /my-dir directory exists, so print a message
 echo &quot;The /my-dir directory exists.&quot;
else
 # The /my-dir directory does not exist, so print a message and create it
 echo &quot;The /my-dir directory does not exist. Creating it now...&quot;
 mkdir $directory
fi
</code></pre>
The script first checks if the <code>/my-dir</code> directory exists. If it does, a message is printed to the console. If the <code>/my-dir</code> directory does not exist, a message is printed and the <code>/my-dir</code> directory is created using the <code>mkdir</code> command.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: If you need to check if a directory/file does not exist you can just add the not logical operator (<code>!</code>) inside the <code>[</code> operator. For example: <code>if [ ! -d &quot;$directory&quot; ]</code>.

 </div>
 </div>
 </div>
 <h4 id="method2thetestcommand">Method 2: The test Command</h4>
To use the <code>test</code> command to check if a directory exists in Bash, you can use the following syntax:
<pre><code class="hljs">if test -d $directory
then
 # code to execute if $directory exists
else
 # code to execute if $directory does not exist
fi
</code></pre>
As in the previous example, the <code>$directory</code> represents the name of the directory you want to check for. If the directory exists, the code within the <code>then</code> block will be executed. If the directory does not exist, the code within the <code>else</code> block will be executed.
Here is an example of how you might use this command in a Bash script:
<pre><code class="hljs"># Check if the /my-dir directory exists
directory=&quot;/my-dir&quot;
if test -d $directory
then
 # The /my-dir directory exists, so print a message
 echo &quot;The /my-dir directory exists.&quot;
else
 # The /my-dir directory does not exist, so print a message and create it
 echo &quot;The /my-dir directory does not exist. Creating it now...&quot;
 mkdir $directory
fi
</code></pre>
Here, the script first checks if the <code>/my-dir</code> directory exists using the <code>test -d</code> command. If it does, a message is printed to the console. If the <code>/my-dir</code> directory does not exist, a message is printed and the <code>/my-dir</code> directory is created using the <code>mkdir</code> command.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: The <code>[</code> operator is an alias for the <code>test</code> command in Bash, so you could also use the <code>[ -d $directory ]</code> syntax to achieve the same result.

 </div>
 </div>
 </div>
 <h4 id="method3theoperator">Method 3: The [[ Operator</h4>
To use the <code>[[</code> operator to check if a directory exists in Bash, you can use the syntax similar to one seen in previous methods:
<pre><code class="hljs">if [[ -d $directory ]]
then
 # code to execute if $directory exists
else
 # code to execute if $directory does not exist
fi
</code></pre>
You guessed it, the <code>$directory</code> represents the name of the directory you want to check for. As we've seen in previous sections, if the directory exists, the code within the <code>then</code> block will be executed. If the directory does not exist, the code within the <code>else</code> block will be executed.
Here is an example of how you might use <code>[[</code> operator in a Bash script:
<pre><code class="hljs"># Check if the /my-dir directory exists
directory=&quot;/my-dir&quot;
if [[ -d $directory ]]
then
 # The /my-dir directory exists, so print a message
 echo &quot;The /my-dir directory exists.&quot;
else
 # The /my-dir directory does not exist, so print a message and create it
 echo &quot;The /my-dir directory does not exist. Creating it now...&quot;
 mkdir $directory
fi
</code></pre>

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: The <code>[[</code> operator is a more powerful version of the <code>[</code> operator, and is generally preferred in Bash scripts due to its additional features and improved syntax. However, the <code>[</code> and <code>[[</code> operators can generally be used interchangeably. If you are interested in a more in-depth overview of the differences between those two operators, you should read our <a target="_blank" href="https://stackabuse.com/bash-difference-between-and-operators/">&quot;Bash: Difference Between [ and [[ Operators&quot;</a>.

 </div>
 </div>
 </div>
 <h3 id="howtocheckifafileexistsinbash">How to Check if a File Exists in Bash</h3>
All of the above is pretty much applicable in the case you want to check if a file exists - there are only a few trivial changes that you need to perform in order to make previous scripts compatible with the new use case. Therefore, in this section, we won't dig deep into each method, we'll just go over code snippets that show exactly how to check for the file existence in Bash. Let's start with the <code>if [ -f $file ]</code> command.
<h4 id="method1theifffilecommand">Method 1: The if [-f $file] Command</h4>
The <code>if [ -f $file ]</code> command is the equivalent of the <code>if [ -d $directory ]</code> command. They are pretty much the same, apart from the <code>-f</code> and <code>-d</code> arguments:
<pre><code class="hljs"># Check if the &#x27;my-file.txt&#x27; exists
file=&quot;/my-file.txt&quot;
if [ -f &quot;$file&quot; ]
then
 # The &#x27;my-file.txt&#x27; exists, so print a message
 echo &quot;The &#x27;my-file.txt exists.&quot;
else
 # The &#x27;my-file.txt&#x27; does not exist, so print a message and create it
 echo &quot;The &#x27;my-file.txt&#x27; does not exist. Creating it now...&quot;
 mkdir $file
fi
</code></pre>
<h4 id="method2thetestcommand">Method 2: The test Command</h4>
Also, the <code>test</code> command used to check for the existence of directories can be used to verify the (non)existence of a file:
<pre><code class="hljs"># Check if the &#x27;my-file.txt&#x27; directory exists
file=&quot;my-file.txt&quot;
if test -f $file
then
 # The &#x27;my-file.txt&#x27; file exists, so print a message
 echo &quot;The &#x27;my-file.txt&#x27; file exists.&quot;
else
 # The &#x27;my-file.txt&#x27; file does not exist, so print a message and create it
 echo &quot;The &#x27;my-file.txt&#x27; file does not exist. Creating it now...&quot;
 mkdir $file
fi
</code></pre>
<h4 id="method3theoperator">Method 3: The [[ Operator</h4>
And, finally, the <code>[[</code> can also be used for the purpose of checking if a file exists in Bash:
<pre><code class="hljs"># Check if the &#x27;my-file.txt&#x27; file exists
file=&quot;my-file.txt&quot;
if [[ -d $file ]]
then
 # The &#x27;my-file.txt&#x27; file exists, so print a message
 echo &quot;The &#x27;my-file.txt&#x27; file exists.&quot;
else
 # The &#x27;my-file.txt&#x27; file does not exist, so print a message and create it
 echo &quot;The &#x27;my-file.txt&#x27; file does not exist. Creating it now...&quot;
 mkdir $file
fi
</code></pre>
<h3 id="conclusion">Conclusion</h3>
In conclusion, being able to check if a file or directory exists in Bash is a crucial skill for any Linux administrator or developer. The main methods we've discussed in this article are <code>[</code> and <code>[[</code> operators, as well as the <code>test</code> command in Bash. Additionally, the <code>ls</code> command can also be used to check for file or directory existence, although it is less efficient and less versatile. By understanding and utilizing these methods, you will be able to easily check for file or directory existence in Bash and take appropriate actions based on the outcome.

In this tutorial, learn how to check whether a file or directory exists in Bash, with practical examples, best practices and comparisons of methods.

Bash: Check if Directory or File Exists

<h3 id="introduction">Introduction</h3>
In bash, the <code>[</code> and <code>[[</code> operators are used for comparison and testing. The <code>[</code> operator is an alias for the <code>test</code> command, which is used to check the properties of files or to compare strings or integers. The <code>[[</code> operator is a more advanced version of the test command. It allows you to perform more powerful comparisons and tests, such as regular expression matching and string manipulation.
<blockquote>
In this article, we'll take a look at the difference between those very similar operators in Bash. We'll assume you already know the basics of both operators, and you just want to make an educated decision on when to use which one.
</blockquote>
<h3 id="operatorbacktobasics">[ Operator - Back to Basics</h3>
The <code>[</code> operator, also known as the test command, is a built-in command in bash that is used to perform various tests on files and strings. It can be used to check for the existence of files, compare values, and test the properties of strings. The operator must be enclosed in a whitespace or be escaped with a backslash.
Let's quickly take a look at the basic syntax for the <code>[</code> operator:
<pre><code class="hljs">if [ expression ]
then
 # commands to execute if expression is true
else
 # commands to execute if expression is false
fi
</code></pre>
Here, <code>expression</code> is a combination of options and arguments that are used to perform the test. Some common options used with the <code>[</code> operator include:
<ul>
<li><code>-e file</code> to check if a file exists</li>
<li><code>-f file</code> to check if a file is a regular file</li>
<li><code>-d file</code> to check if a file is a directory</li>
<li><code>-z string</code> to check if a string has a length of 0</li>
<li><code>-n string</code> to check if a string has a non-zero length</li>
</ul>
<h3 id="operatorbacktobasics">[[ Operator - Back to Basics</h3>
The <code>[[</code> operator is an extended version of the <code>[</code> operator in Bash. It is used to perform various tests and arithmetic operations, and has a more flexible syntax. It is not as strictly dependent on whitespace or backslashes around it, as the <code>[</code> operator.
The basic syntax for the <code>[[</code> operator is almost identical to the <code>[</code> operator syntax we've discussed above:
<pre><code class="hljs">if [[ expression ]]
then
 # commands to execute if expression is true
else
 # commands to execute if expression is false
fi
</code></pre>
The <code>expression</code> is a combination of options and arguments that are used to perform the test. The basic options are the same as the ones we've listed in the previous section when talking about the <code>[</code> operator.
In addition to these options, the <code>[[</code> operator supports additional string and arithmetic operations such as regular expressions matching, logical and/or operators, and comparison operators without the need for <code>-eq</code>, <code>-lt</code>, etc:
<pre><code class="hljs">if [[ $string =~ &quot;regex&quot; ]]
then
 echo &quot;string matches the regex&quot;
fi
</code></pre>
<pre><code class="hljs">if [[ $x &gt; 5 &amp;&amp; $y &lt; 10 ]]
then
 echo &quot; x is greater than 5 and y is less than 10&quot;
fi
</code></pre>
The <code>[[</code> operator also supports the assignment and arithmetic operations, this makes it more versatile than the <code>[</code> operator:
<pre><code class="hljs">if [[ ( x+=5) -lt 10 ]]
then
 echo &quot;x is now less than 10&quot;
fi
</code></pre>
Overall, the <code>[[</code> operator is a more recent addition to Bash and is more powerful than <code>[</code> operator in terms of functionality and flexibility, it offers improved readability, better performance, and additional functionality to perform various tests and operations.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Be aware that the behavior of <code>[[</code> might be slightly different than <code>[</code> operator, so it's important to check the documentation if you are not sure which to use!

 </div>
 </div>
 </div>
 <h3 id="andoperatorsheadtoheadcomparison">[ and [[ Operators - Head to Head Comparison</h3>
Now, after we recapped the basics of both operators, we can compare them head-to-head. Let's take a look at several use cases of <code>[</code> and <code>[[</code> operators and compare the syntax.
First of all, take a look at the example where we want to check for the existence of the <code>file.txt</code>:
<pre><code class="hljs"># Using the &quot;[&quot; operator
if [ -e &quot;file.txt&quot; ]
then
 echo &quot;file.txt exists&quot;
fi

# Using the &quot;[[&quot; operator
if [[ -e &quot;file.txt&quot; ]]
then
 echo &quot;file.txt exists&quot;
fi
</code></pre>
Pretty much the same! Both of the above code samples check if the file <code>file.txt</code> exists and print a message if it does. However, while the <code>[</code> operator requires whitespace or a backslash around it, the <code>[[</code> operator does not.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: The <code>[</code> operator is the alias for the <code>test</code> built-in function in Bash, and all of the built-in functions accept arguments as an array of whitespace-separated strings. That's why the <code>[</code> operator requires whitespaces around each of its arguments!

 </div>
 </div>
 </div>
 The previous example wasn't sufficient at illustrating the need for whitespaces when using the <code>[</code> operator, so, let's choose another, more illustrative example. We'll compare two strings that are the same, and expect the script to return <code>&quot;Equal&quot;</code>:
<pre><code class="hljs">str1=&quot;My String&quot;

# Using the &quot;[&quot; operator WITHOUT whitespaces
# This will return &quot;Not equal&quot; even though two strings are the same
if [ &quot;$str1&quot;=&quot;$str1&quot; ] 
then 
 echo &quot;Equal&quot; 
else 
 echo &quot;Not equal&quot;
fi

# Using the &quot;[&quot; operator WITH whitespaces
# This will return &quot;Equal&quot;, as expected
if [ &quot;$str1&quot; = &quot;$str1&quot; ] 
then 
 echo &quot;Equal&quot; 
else 
 echo &quot;Not equal&quot;
fi

# Using the &quot;[[&quot; operator WITHOUT whitespaces
# This will return &quot;Equal&quot;, as expected
# Even without whitespaces
if [ &quot;$str1&quot;=&quot;$str1&quot; ] 
then 
 echo &quot;Equal&quot; 
else 
 echo &quot;Not equal&quot;
fi
</code></pre>
Another interesting difference between <code>[</code> and <code>[[</code> is that the <code>[[</code> operator supports more operators than <code>[</code>. For example, it can use the <code>==</code> comparison operator rather than the <code>-eq</code> operator used with <code>[</code>:
<pre><code class="hljs"># Using the &quot;[&quot; operator
if [ $x -eq 5 ]
then
 echo &quot;x is equal to 5&quot;
fi

# Using the &quot;[[&quot; operator
if [[ $x == 5 ]]
then
 echo &quot;x is equal to 5&quot;
fi
</code></pre>
Both of the above code samples check if the variable <code>x</code> is equal to <code>5</code> and print a message if it does. The first code sample uses the <code>-eq</code> operator with the <code>[</code> operator and the second code sample uses the <code>==</code> operator with the <code>[[</code> operator to check for equality.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: The same can be applied to <code>-lt</code> and <code>&lt;</code>, <code>-gt</code> and <code>&gt;</code>, and other operators mentioned in the sections above.

 </div>
 </div>
 </div>
 Another handy feature of the <code>[[</code> operator is that we don't need to wrap strings with quotes when using <code>[[</code>:
<pre><code class="hljs"># Using the &quot;[&quot; operator
if [ -z &quot;$string&quot; ]
then
 echo &quot;string is empty&quot;
fi

# Using the &quot;[[&quot; operator
if [[ -z $string ]]
then
 echo &quot;string is empty&quot;
fi
</code></pre>
Both of the above code samples check if the variable <code>string</code> has a length of <code>0</code> and print a message if it does. However, while the <code>[</code> operator requires that <code>string</code> be enclosed in quotes(<code>&quot;&quot;</code>), the <code>[[</code> operator doesn't.
In general, the <code>[[</code> operator offers more functionality, improved readability, and better performance than the <code>[</code> operator. The <code>[[</code> operator supports additional string and arithmetic operations, regular expressions, logical and/or operators, and comparison operators without the need for <code>-eq</code>, <code>-lt</code>, etc, assignment and arithmetic operation, allowing for more complex and powerful tests in Bash scripts.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: It is important to note that the behavior of <code>[[</code> might be slightly different than <code>[</code> operator and in certain scenarios <code>[</code> is more appropriate to use - most notably when you need to run your scripts on old systems.

 </div>
 </div>
 </div>
 <h4 id="comparisonrecap">Comparison Recap</h4>
Finally, let's recap several key differences in functionality and performance between the <code>[</code> and <code>[[</code> operators in bash.
Functionality
<ul>
<li>
The <code>[</code> operator is a synonym for the <code>test</code> command and is used to perform tests on files and strings. It supports a limited set of options and has a more restrictive syntax.
</li>
<li>
The <code>[[</code> operator is an extended version of the <code>[</code> operator, and is more powerful in terms of functionality. It supports additional string and arithmetic operations, regular expressions, logical and/or operators, and comparison operators without the need for <code>-eq</code>, <code>-lt</code>, etc, assignment and arithmetic operations, allowing for more complex and powerful tests in Bash scripts. It also allows more flexible syntax.
</li>
</ul>
Performance
<ul>
<li>
The <code>[</code> operator is generally considered to be slower than the <code>[[</code> operator as it requires more parsing, and also requires whitespace or backslashes around it, which can lead to additional overhead.
</li>
<li>
The <code>[[</code> operator, on the other hand, is faster and more efficient, as it has a more flexible syntax and does not require additional characters for parsing.
</li>
</ul>
Compatibility
<ul>
<li>
The <code>[[</code> operator is a more recent addition to Bash, and as such, it is not always guaranteed to be available on older systems. It is mostly used in more recent versions of Bash.
</li>
<li>
The <code>[</code> operator, on the other hand, has been around for a longer time and is widely supported across different systems.
</li>
</ul>
<h3 id="conclusion">Conclusion</h3>
All-in-all, after reading this article you'll be able to make an educated decision on what operator suits your needs the best.
Both operators are used to perform tests and make decisions in Bash scripts. In general, it's recommended to use <code>[[</code> operator for improved functionality, readability, and performance. However, in the case of scripts that need to be run on older systems or cross-platform, it is better to stick with the <code>[</code> operator, as it is more widely supported.

In this in-depth guide - learn about the difference between the [ and [[ operators in Bash, and how to use the nuanced differences to your advantage.

Bash: Difference Between [ and [[ Operators 

<h3 id="introduction">Introduction</h3>
In Bash, parameter expansion is a feature that allows you to manipulate the value of a variable or to extract part of its value using a special syntax. It is often used to modify the value of a variable or to extract information from it.
Parameter expansion uses a syntax similar to variable substitution, but with additional operators and options that allow you to manipulate the value of the variable in more advanced ways. For example, you can use parameter expansion to extract a substring from a variable, replace a part of a variable with a different value, or add a prefix or suffix to a variable.
Parameter expansion is a powerful feature that is widely used in Bash scripts. It allows you to manipulate and extract information from variables in a flexible and efficient way and is an important tool for working with variables in the Bash shell.
<blockquote>
In this guide, we'll go over parameter expansion in Bash. We'll go over what is it, why should you consider using it, how to use it, and, finally, give you several tips on how to use parameter expansion in Bash more efficiently.
</blockquote>
<h3 id="whyisparameterexpansionuseful">Why is Parameter Expansion Useful?</h3>
As we already indicated, there are many scenarios where parameter expansion comes in handy, here are several most common:
<ul>
<li>
Modifying the value of a variable - parameter expansion allows you to modify the value of a variable by extracting a substring, replacing a part of the value, or adding a prefix or suffix. This can be useful for adapting the value of a variable to different situations or for extracting information from it.
</li>
<li>
Improving the readability and maintainability of scripts - by using parameter expansion, you can avoid using complex and error-prone string manipulation techniques, such as using multiple &quot;sed&quot; or &quot;awk&quot; commands. This can help to make your scripts more readable and easier to maintain.
</li>
<li>
Increasing the efficiency of scripts - parameter expansion can often be faster and more efficient than using external commands or utilities to manipulate strings, especially when working with large amounts of data. This can be especially important in scripts that are designed to run efficiently and quickly.
</li>
</ul>
As you can see, parameter expansion is a useful and powerful feature in Bash that allows you to manipulate and extract information from variables in a flexible and efficient way. It is an important tool for working with variables in the Bash shell.
<h3 id="parameterexpansionsyntax">Parameter Expansion Syntax</h3>
In this section, we'll finally explore the various syntax options available for parameter expansion in Bash. We will cover the basic <code>${variable}</code> syntax for extracting the value of a variable, as well as more advanced techniques such as extracting substrings, replacing parts of a variable, adding prefixes and suffixes, and providing default values. By the end of this section, you will have a solid understanding of the different ways you can use parameter expansion in Bash.
<h4 id="basicsyntaxvariable">Basic syntax: ${variable}</h4>
The <code>${variable}</code> syntax is a basic form of parameter expansion in Bash that allows you to access the value of a variable. It is similar to variable substitution, which allows you to insert the value of a variable into a command or script.
Here is a quick example of using the <code>${variable}</code> syntax in a Bash script:
<pre><code class="hljs">name=&quot;John&quot;
echo &quot;Hello, ${name}!&quot;
</code></pre>
This will output:
<pre><code class="hljs">Hello, John!
</code></pre>
You can also use the <code>${variable}</code> syntax to assign a new value to a variable:
<pre><code class="hljs">name=&quot;John&quot;
echo &quot;Original value: ${name}&quot;
name=&quot;Jane&quot;
echo &quot;New value: ${name}&quot;
</code></pre>
This will give us:
<pre><code class="hljs">Original value: John
New value: Jane
</code></pre>
The <code>${variable}</code> syntax is a simple and straightforward way to access and manipulate the value of a variable in Bash. It is a fundamental feature of the Bash shell and is widely used in scripts and on the command line.
<h4 id="extractingasubstringvariableoffsetlength">Extracting a substring: ${variable:offset:length}</h4>
To extract a substring from a variable using parameter expansion in Bash, you can use the following syntax:
<pre><code class="hljs">substring=${variable:offset:length}
</code></pre>
Here, <code>variable</code> is the name of the variable that contains the string you want to extract a substring from, <code>offset</code> is the starting position of the substring (measured in characters from the beginning of the string), and <code>length</code> is the number of characters in the substring.
For example, you can extract the first four characters of a string stored in the <code>filename</code> variable:
<pre><code class="hljs">substring=${filename:0:4}
</code></pre>
Let's take a look at an example of using this syntax on an actual variable:
<pre><code class="hljs">filename=&quot;file.txt&quot;
substring=${filename:0:4}
echo &quot;Substring: ${substring}&quot;
</code></pre>
This will extract the first 4 characters of the <code>filename</code> variable:
<pre><code class="hljs">Substring: file
</code></pre>
You can also use negative values for the <code>offset</code> parameter to specify a position relative to the end of the string. For example, take a look at how you'd extract the last four characters of a string:
<pre><code class="hljs">substring=${filename:-4:4}
</code></pre>
This will extract the last four characters of the <code>filename</code> variable, regardless of its length.
Overall, the <code>${variable:offset:length}</code> syntax is a powerful and flexible way to extract substrings from variables in Bash. It allows you to easily extract specific parts of a string and use them in your scripts and commands.
<h4 id="replacingapartofavariablevariablepatternreplacement">Replacing a part of a variable: ${variable/pattern/replacement}</h4>
To replace a part of a variable using parameter expansion in Bash, you can use the following syntax:
<pre><code class="hljs">result=${variable/pattern/replacement}
</code></pre>
Here, <code>variable</code> is the name of the variable that contains the string you want to modify, <code>pattern</code> is a regular expression that specifies the part of the string you want to replace, and <code>replacement</code> is the string that will replace the matched pattern.
For example, to replace the first occurrence of the string <code>&quot;txt&quot;</code> with the string <code>&quot;doc&quot;</code> in a variable named <code>filename</code>, you can use the following syntax:
<pre><code class="hljs">result=${filename/txt/doc}
</code></pre>
Here is an example of using this syntax in a Bash script:
<pre><code class="hljs">filename=&quot;file.txt&quot;
result=${filename/txt/doc}
echo &quot;Result: ${result}&quot;
</code></pre>
This script will output &quot;Result: file.doc&quot;.
You can also use the <code>g</code> flag to replace all occurrences of the pattern, rather than just the first one. For example:
<pre><code class="hljs">result=${filename/txt/doc/g}
</code></pre>
This syntax will replace all occurrences of <code>&quot;txt&quot;</code> with <code>&quot;doc&quot;</code> in the <code>filename</code> variable.
Overall, the <code>&quot;${variable/pattern/replacement}&quot;</code> syntax is a powerful and flexible way to replace parts of a string stored in a variable in Bash. It allows you to easily modify strings and adapt them to different situations.
<h4 id="addingaprefixorsuffixprefixvariablesuffix">Adding a prefix or suffix: ${prefix}${variable}${suffix}</h4>
To add a prefix or suffix to a variable using parameter expansion in Bash, you can use the following syntax:
<pre><code class="hljs">result=&quot;${prefix}${variable}${suffix}&quot;
</code></pre>
Here, <code>prefix</code> and <code>suffix</code> are the strings you want to add to the beginning and end of the <code>variable</code> respectively.
For example, to add the prefix <code>prefix_</code> and the suffix <code>_suffix</code> to a variable named <code>filename</code>, you can use the following syntax:
<pre><code class="hljs">result=&quot;${prefix}${filename}${suffix}&quot;
</code></pre>
Here is an example of using this syntax in a Bash script:
<pre><code class="hljs">prefix=&quot;prefix_&quot;
suffix=&quot;_suffix&quot;
filename=&quot;file.txt&quot;
result=&quot;${prefix}${filename}${suffix}&quot;
echo &quot;Result: ${result}&quot;
</code></pre>
This script will output:
<pre><code class="hljs">Result: prefix_file.txt_suffix
</code></pre>
You can also use this syntax to add a prefix or suffix to the value of a variable that is stored in another variable. For example:
<pre><code class="hljs">result=&quot;${prefix}${variable_name}${suffix}&quot;
</code></pre>
This syntax will add the prefix and suffix to the value of the <code>variable_name</code> variable.
Overall, the <code>&quot;${prefix}${variable}${suffix}&quot;</code> syntax is a simple and flexible way to add a prefix or suffix to a variable in Bash. It allows you to easily modify strings and adapt them to different situations.
<h3 id="4protipsforparameterexpansioninbash">4 Pro Tips for Parameter Expansion in Bash</h3>
<h4 id="usequotestopreventwordsplitting">Use Quotes to Prevent Word Splitting</h4>
Word splitting is a process that occurs when the Bash shell separates a string of text into separate words based on spaces and other delimiters. This can cause problems when you are trying to pass multiple words or phrases as arguments to a command.
For example, consider the following command:
<pre><code class="hljs">ls $files
</code></pre>
If the value of the <code>$files</code> variable is <code>file1 file2 file3</code>, the Bash shell will split the string into three separate words, and the <code>ls</code> command will be executed with each word as a separate argument. This will result in an error because the <code>ls</code> command does not recognize <code>file1</code>, <code>file2</code>, and <code>file3</code> as valid arguments.
To prevent this problem, you can use quotes around the variable expansion to tell the Bash shell to treat the entire string as a single argument. Here's the same example, but with quotes added:
<pre><code class="hljs">ls &quot;$files&quot;
</code></pre>
Now, the Bash shell will treat the entire value of the <code>$files</code> variable as a single argument, and the <code>ls</code> command will be executed correctly.
Using quotes in this way can be particularly helpful when working with variables that contain multiple words or phrases, or when you want to pass multiple arguments to a command. By using quotes to prevent word splitting, you can avoid common errors and ensure that your commands are executed correctly.
<h4 id="usecurlybracestopreventambiguity">Use Curly Braces to Prevent Ambiguity</h4>
Another helpful tip when using parameter expansion in the Bash shell is to use curly braces to prevent ambiguity.
Ambiguity can occur when you are working with variables that have multiple parts or that are nested within one another. For example, consider the following code:
<pre><code class="hljs">$ fruit=apple
$ echo $fruit${color}
</code></pre>
Without the curly braces, the Bash shell will interpret this code as an expansion of the <code>$fruit</code> variable followed by an expansion of the <code>$color</code> variable. However, if the <code>$color</code> variable is not defined, the expansion will be empty, and the output will be simply <code>&quot;apple&quot;</code>.
To prevent this ambiguity, you can use curly braces to explicitly specify which parts of the expansion should be evaluated. Here's the same example, but with curly braces added:
<pre><code class="hljs">$ fruit=apple
$ echo ${fruit}${color}
</code></pre>
Now, the Bash shell will interpret the code as an expansion of the <code>$fruit</code> variable followed by an expansion of the <code>$color</code> variable, and the output will be <code>&quot;apple&quot;</code> followed by the value of the <code>$color</code> variable (if it is defined).
Using curly braces in this way can be particularly helpful when working with complex expansions or when you want to ensure that your expansions are evaluated correctly. By using curly braces to prevent ambiguity, you can avoid common errors and ensure that your code works as intended.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Using curly braces everywhere, not just in ambiguous cases, is considered a good programming practice!

 </div>
 </div>
 </div>
 <h4 id="useandtoremovethelongestmatchingpattern">Use &quot;##&quot; and &quot;%%&quot; to Remove the Longest Matching Pattern</h4>
The <code>##</code> and <code>%%</code> operators are used in the Bash shell to remove the longest matching pattern from the beginning or end of a variable, respectively.
For example, consider the following code:
<pre><code class="hljs">$ var=abc123
$ echo ${var##*b}
</code></pre>
In this case, the <code>##</code> operator will remove the longest matching pattern from the beginning of the <code>$var</code> variable. The pattern <code>*b</code> matches the string <code>&quot;abc&quot;</code>, so the output will be <code>123</code>.
Similarly, the <code>%%</code> operator can be used to remove the longest matching pattern from the end of a variable:
<pre><code class="hljs">$ var=abc123
$ echo ${var%%3*}
</code></pre>
In this case, the <code>%%</code> operator will remove the longest matching pattern from the end of the <code>$var</code> variable. The pattern <code>3*</code> matches the string <code>&quot;123&quot;</code>, so the output will be <code>&quot;abc&quot;</code>.
These operators can be particularly useful when you want to remove a specific prefix or suffix from a variable, or when you want to extract a specific part of a string. By using the <code>##</code> and <code>%%</code> operators, you can easily manipulate the contents of variables and extract the information that you need.
<h4 id="usetoprovideadefaultvalueifthevariableisunsetorempty">Use :- to provide a default value if the variable is unset or empty</h4>
You can use the <code>:-</code> operator in parameter expansion in Bash to provide a default value for a variable if it is unset or empty. This can be useful when you want to ensure that a variable has a valid value, even if it has not been set or if it has been set to an empty string.
To use the <code>:-</code> operator, you can use the following syntax:
<pre><code class="hljs">result=${variable:-default}
</code></pre>
Here, <code>variable</code> is the name of the variable you want to check, and <code>default</code> is the value you want to use if the variable is unset or empty.
For example, to set a default value for the <code>filename</code> variable if it is unset or empty, you can use the following syntax:
<pre><code class="hljs">filename=${filename:-default.txt}
</code></pre>
This syntax will set the <code>filename</code> variable to <code>default.txt</code> if it is unset or empty, and will leave it unchanged if it already has a value.
You can also use this syntax to assign a default value to a variable that is stored in another variable:
<pre><code class="hljs">result=${variable_name:-default}
</code></pre>
This syntax will assign the value of <code>default</code> to the <code>result</code> variable if the <code>variable_name</code> variable is unset or empty, and will assign the value of <code>variable_name</code> to <code>result</code> if it is set and non-empty.
Overall, the <code>:-</code> operator is a useful feature of parameter expansion in Bash that allows you to provide a default value for a variable if it is unset or empty. It can be helpful for ensuring that your scripts and commands always have access to a valid value for a variable.
<h3 id="conclusion">Conclusion</h3>
All-in-all, parameter expansion is a pretty useful concept to understand when working with variables in Bash - both in terms of efficiency, readability, flexibility, and maintainability of the scripts you are writing.
In this guide, we took a look at the concept of parameter expansion in Bash and explained what parameter expansion is, why is it important to understand it, gave you some practical use-case examples, and, finally, gave you some pro tips you should definitely consider when using parameter expansion in Bash.

In this in-depth guide, learn everything you need to know about parameter expansion in Bash, with practical code samples, best practices and lots of examples.

Guide to Parameter Expansion in Bash

<h3 id="introduction">Introduction</h3>
Repetitive tasks are ripe for automation. It is common for developers and system administrators to automate routine tasks like health checks and file backups with shell scripts. However, as those tasks become more complex, shell scripts may become harder to maintain.
<blockquote>
Fortunately, we can use Python instead of shell scripts for automation. Python provides methods to run shell commands, giving us the same functionality of those shells scripts, with additional support from Python's ecosystem. Learning how to run shell commands in Python opens the door for us to automate computer tasks in a structured and scalable way.
</blockquote>
In this article, we will look at the various ways to execute shell commands in Python, and the ideal situation to use each method.
<h3 id="usingossystemtorunacommand">Using os.system to Run a Command</h3>
Python allows us to immediately execute a shell command that's stored in a string using the <code>os.system()</code> function. Let's start by creating a new Python file called <code>echo_adelle.py</code> and enter the following:
<pre><code class="hljs">import os
os.system(&quot;echo Hello from the other side!&quot;)
</code></pre>
The first thing we do in our Python file is import the <code>os</code> module, which contains the <code>system</code> function that can execute shell commands. The next line does exactly that, runs the <code>echo</code> command in our shell through Python.
In your Terminal, run this file with using the following command, and you should see the corresponding output:
<pre><code class="hljs">$ python3 echo_adelle.py
Hello from the other side!
</code></pre>
As the <code>echo</code> commands prints to our <code>stdout</code>, <code>os.system()</code> also displays the output on our <code>stdout</code> stream. While not visible in the console, the <code>os.system()</code> command returns the exit code of the shell command. An exit code of 0 means it ran without any problems and any other number means an error. This way, you ca control the flow of your commands, check the results and either retry or return an error if any call fails:
<pre><code class="hljs"># cd_return_codes.py
import os

home_dir = os.system(&quot;cd ~&quot;)
print(&quot;`cd ~` ran with exit code %d&quot; % home_dir)
unknown_dir = os.system(&quot;cd doesnotexist&quot;)
print(&quot;`cd doesnotexis` ran with exit code %d&quot; % unknown_dir)
</code></pre>
In this script, we create two variables that store the result of executing commands that change the directory to the home folder, and to a folder that does not exist. Running this file, we will see:
<pre><code class="hljs">$ python3 cd_return_codes.py
`cd ~` ran with exit code 0
sh: line 0: cd: doesnotexist: No such file or directory
`cd doesnotexist` ran with exit code 256
</code></pre>
The first command, which changes the directory to the home directory, executes successfully. Therefore, <code>os.system()</code> returns its exit code, zero, which is stored in <code>home_dir</code>. On the other hand, <code>unknown_dir</code> stores the exit code of the failed bash command to change the directory to a folder that does not exist.
The <code>os.system()</code> function executes a command, prints any output of the command to the console, and returns the exit code of the command. If we would like more fine grained control of a shell command's input and output in Python, we should use the <code>subprocess</code> module.
<h3 id="runningacommandwithsubprocess">Running a Command with subprocess</h3>
The <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/subprocess.html">subprocess module</a> is Python's recommended way to executing shell commands. It gives us the flexibility to suppress the output of shell commands or chain inputs and outputs of various commands together, while still providing a similar experience to <code>os.system()</code> for basic use cases.
In a new file called <code>list_subprocess.py</code>, let's run a command with the <code>subprocess</code> module:
<pre><code class="hljs">import subprocess

list_files = subprocess.run([&quot;ls&quot;, &quot;-l&quot;])
print(&quot;The exit code was: %d&quot; % list_files.returncode)
</code></pre>
In the first line, we import the <code>subprocess</code> module, which is part of the Python standard library. We then use the <code>subprocess.run()</code> function to execute the command. Like <code>os.system()</code>, the <code>subprocess.run()</code> command returns the exit code of what was executed.
Unlike <code>os.system()</code>, note how <code>subprocess.run()</code> requires a list of strings as input instead of a single string. The first item of the list is the name of the command. The remaining items of the list are the flags and the arguments of the command.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: As a rule of thumb, you need to separate the arguments based on space, for example <code>ls -alh</code> would be <code>[&quot;ls&quot;, &quot;-alh&quot;]</code>, while <code>ls -a -l -h</code>, would be <code>[&quot;ls&quot;, &quot;-a&quot;, -&quot;l&quot;, &quot;-h&quot;]</code>. As another example, <code>echo hello world</code> would be <code>[&quot;echo&quot;, &quot;hello&quot;, &quot;world&quot;]</code>, whereas <code>echo &quot;hello world&quot;</code> or <code>echo hello\ world</code> would be <code>[&quot;echo&quot;, &quot;hello world&quot;]</code>.

 </div>
 </div>
 </div>
 Run this file and your console's output would be similar to:
<pre><code class="hljs">$ python3 list_subprocess.py
total 80
-rw-r--r--@ 1 stackabuse staff 216 Dec 6 10:29 cd_return_codes.py
-rw-r--r--@ 1 stackabuse staff 56 Dec 6 10:11 echo_adelle.py
-rw-r--r--@ 1 stackabuse staff 116 Dec 6 11:20 list_subprocess.py
The exit code was: 0
</code></pre>
Now let's try to use one of the more advanced features of <code>subprocess.run()</code>, namely ignore output to <code>stdout</code>. In the same <code>list_subprocess.py</code> file, change:
<pre><code class="hljs">list_files = subprocess.run([&quot;ls&quot;, &quot;-l&quot;])
</code></pre>
To this:
<pre><code class="hljs">list_files = subprocess.run([&quot;ls&quot;, &quot;-l&quot;], stdout=subprocess.DEVNULL)
</code></pre>
The standard output of the command now pipes to the special <code>/dev/null</code> device, which means the output would not appear on our consoles. Execute the file in your shell to see the following output:
<pre><code class="hljs">$ python3 list_subprocess.py
The exit code was: 0
</code></pre>
What if we wanted to provide input to a command? The <code>subprocess.run()</code> facilitates this by its <code>input</code> argument:
<pre><code class="hljs">import subprocess

useless_cat_call = subprocess.run([&quot;cat&quot;], 
 stdout=subprocess.PIPE, 
 text=True, 
 input=&quot;Hello from the other side&quot;)

print(useless_cat_call.stdout) # Hello from the other side
</code></pre>
We use <code>subprocess.run()</code> with quite a few commands, let's go through them:
<ul>
<li><code>stdout=subprocess.PIPE</code> tells Python to redirect the output of the command to an object so it can be manually read later</li>
<li><code>text=True</code> returns <code>stdout</code> and <code>stderr</code> as strings. The default return type is bytes.</li>
<li><code>input=&quot;Hello from the other side&quot;</code> tells Python to add the string as input to the <code>cat</code> command.</li>
</ul>
Running this produces the following output:
<pre><code class="hljs">Hello from the other side
</code></pre>
We can also raise an <code>Exception</code> without manually checking the return value. In a new file, <code>false_subprocess.py</code>, add the code below:
<pre><code class="hljs">import subprocess

failed_command = subprocess.run([&quot;false&quot;], check=True)
print(&quot;The exit code was: %d&quot; % failed_command.returncode)
</code></pre>
In your Terminal, run this file. You will see the following error:
<pre><code class="hljs">$ python3 false_subprocess.py
Traceback (most recent call last):
 File &quot;false_subprocess.py&quot;, line 4, in &lt;module&gt;
 failed_command = subprocess.run([&quot;false&quot;], check=True)
 File &quot;/usr/local/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py&quot;, line 512, in run
 output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command &#x27;[&#x27;false&#x27;]&#x27; returned non-zero exit status 1.
</code></pre>
By using <code>check=True</code>, we tell Python to raise any exceptions if an error is encountered. Since we did encounter an error, the <code>print</code> statement on the final line was not executed.
The <code>subprocess.run()</code> function gives us immense flexibility that <code>os.system()</code> doesn't when executing shell commands. This function is a simplified abstraction of the <code>subprocess.Popen</code> class, which provides additional functionality we can explore.
<h3 id="runningacommandwithpopen">Running a Command with Popen</h3>
The <code>subprocess.Popen</code> class exposes more options to the developer when interacting with the shell. However, we need to be more explicit about retrieving results and errors.
By default, <code>subprocess.Popen</code> does not stop processing of a Python program if its command has not finished executing. In a new file called <code>list_popen.py</code>, type the following:
<pre><code class="hljs">import subprocess

list_dir = subprocess.Popen([&quot;ls&quot;, &quot;-l&quot;])
list_dir.wait()
</code></pre>
This code is equivalent to that of <code>list_subprocess.py</code>. It runs a command using <code>subprocess.Popen</code>, and waits for it to complete before executing the rest of the Python script.
Let's say we do not want to wait for our shell command to complete execution so the program can work on other things. How would it know when the shell command has finished execution?
The <code>poll()</code> method returns the exit code if a command has finished running, or <code>None</code> if it's still executing. For example, if we wanted to check if <code>list_dir</code> was complete instead of wait for it, we would have the following line of code:
<pre><code class="hljs">list_dir.poll()
</code></pre>
To manage input and output with <code>subprocess.Popen</code>, we need to use the <code>communicate()</code> method.
In a new file called <code>cat_popen.py</code>, add the following code snippet:
<pre><code class="hljs">import subprocess

useless_cat_call = subprocess.Popen([&quot;cat&quot;], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
output, errors = useless_cat_call.communicate(input=&quot;Hello from the other side!&quot;)
useless_cat_call.wait()
print(output)
print(errors)
</code></pre>
The <code>communicate()</code> method takes an <code>input</code> argument that's used to pass input to the shell command. The <code>communicate</code> method also returns both the <code>stdout</code> and <code>stderr</code> when they are set.
Having seen the core ideas behind <code>subprocess.Popen</code>, we have now covered three ways to run shell commands in Python. Let's re-examine their characteristics so we'll know which method is best suited for a project's requirements.
<h3 id="bestapproachtorunshellcommandswithpython">Best Approach to Run Shell Commands with Python?</h3>
If you need to run one or a few simple commands and do not mind if their output goes to the console, you can use the <code>os.system()</code> command. If you want to manage the input and output of a shell command, use <code>subprocess.run()</code>. If you want to run a command and continue doing other work while it's being executed, use <code>subprocess.Popen</code>.
Here is a table with some usability differences that you can also use to inform your decision:
<table class="table table-striped" style="font-size:15px;overflow-x:auto;">
 <thead>
 <tr>
 <th></th>
 <th>os.system</th>
 <th>subprocess.run</th>
 <th>subprocess.Popen</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <th>Requires parsed arguments</th>
 <th>no</th>
 <th>yes</th>
 <th>yes</th>
 </tr>
 <tr>
 <th>Waits for the command</th>
 <th>yes</th>
 <th>yes</th>
 <th>no</th>
 </tr>
 <tr>
 <th>Communicates with stdin and stdout</th>
 <th>no</th>
 <th>yes</th>
 <th>yes</th>
 </tr>
 <tr>
 <th>Returns</th>
 <th>return value</th>
 <th>object</th>
 <th>object</th>
 </tr>
 </tbody>
</table>
<h3 id="conclusion">Conclusion</h3>
Python allows you to execute shell commands, which you can use to start other programs or better manage shell scripts that you use for automation. Depending on our use case, we can use <code>os.system()</code>, <code>subprocess.run()</code> or <code>subprocess.Popen</code> to run bash commands.
Using these techniques, what external task would you run via Python?

Marcus Sanatan

Sajjad Heydari

Instead of shell scripts, which can get complex and tedious, we can use Python to automate shell commands. In this tutorial, we'll learn how to run them for the sake of scalability and maintainability of Python projects.

Executing Shell Commands with Python

N Kaushik

In this tutorial, we'll be converting Strings to datetime in Python, dealing with Timezones. We'll also use dateutil, Maya and Arrow to convert Strings to datetime with automatic format recognition.

Converting Strings to datetime in Python

<h3 id="quickanswermergebranchintomaster">Quick Answer - Merge Branch into Master</h3>
If you're looking for a quick answer, to merge a branch into the <code>master</code> branch - you <code>checkout master</code> and <code>merge some_branch</code>:
<pre><code class="hljs">$ git checkout new-branch
# ...develop some code...

$ git add .
$ git commit –m &quot;Some commit message&quot;
$ git checkout master
Switched to branch &#x27;master&#x27;
$ git merge new-branch
</code></pre>
You always <code>checkout</code> the branch you're merging into, and <code>merge</code> a branch that already has changes.
<h3 id="understandingbranchmergingwithgit">Understanding Branch Merging with Git</h3>
If you're newer to Git, though, it's worth taking a few minutes to understand these commands and how they work. It's surprisingly simple and will take you a long way.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: In 2020, Git (alongside major repository hosting platforms like GitHub, GitLab, etc.) adopted a change in official terminology, and the default branch name was changed to <code>main</code>, due to the negative conotations the word <code>master</code> may entail. Many projects haven't migrated or renamed their main branches, so for the foreseeable future - the terms <code>master</code> and <code>main</code> will likely be used interchangeably.

 </div>
 </div>
 </div>
 One of Git's most powerful features is the ability to easily create and merge branches and changes to the codebase.
Git's distributed nature encourages users to create new branches often and to merge them regularly as a part of the development process - and certain Git workflows exploit this extensively.
<blockquote>
This fundamentally improves the development workflow for most projects by encouraging smaller, more focused, granular commits, subject to rigorous peer review.
</blockquote>
In legacy Version Control Systems (like CVS) the difficulty of merging restricted it to advanced users. Other modern but centralized version control systems like Subversion require commits to be made to a central repository, so a nimble workflow with local branching and merging is atypical.
A commonly used branching workflow in Git is to create a new code branch for each new feature, bug fix, or enhancement.
<blockquote>
These are called Feature Branches.
</blockquote>
Each branch compartmentalizes the commits related to a particular feature. Once the new feature is complete – i.e. a set of changes has been committed on the feature branch – it is ready to be merged back into the master branch (or other main code line branch depending on the workflow in use).

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Feature branching isn't the only branching workflow you can go with, but it's a widely adopted one, and is enabled by the simplicity of Git's merging functionalities.

 </div>
 </div>
 </div>
 <h3 id="mergebranchintoanotherwithgit">Merge Branch into Another with Git</h3>
The <code>git branch</code> command is used to list all existing branches in a repository. An asterisk will appear next to the currently active branch:
<pre><code class="hljs">$ git branch
* master
</code></pre>
To create a new branch, we can use the <code>git branch new-branch</code> command. This will create a new branch mirroring the commits on the currently active branch:
<pre><code class="hljs">$ git branch new-branch
$ git branch
* master
new-branch
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Behind the scenes, Git does not actually create a new set of commits to represent the new branch. A branch is like a tag, and the commits are shared. You're branching out a new set of changes from the main branch. Once a feature branch is finished and merged into the main branch, the changes in it become the main branch, until you merge a new feature branch into the main branch.

 </div>
 </div>
 </div>
 At this point we have created a new branch, but are still located on the source branch. To start working on the new branch we first need to run the command <code>git checkout new-branch</code>. This will change the active branch to the new branch:
<pre><code class="hljs">$ git checkout new-branch
Switched to branch ‘new-branch&#x27;
$ git branch
master
* new-branch
</code></pre>
At this point, commits can be made on the new branch to implement the new feature. Once the feature is complete, the branch can be merged back into the main code branch.
First we run <code>git checkout master</code> to change the active branch back to the <code>master</code> branch. Then we run the command <code>git merge new-branch</code> to merge the new feature into the master branch.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: <code>git merge</code> merges the specified branch into the currently active branch. So we need to be on the branch that we are merging into.

 </div>
 </div>
 </div>
 If you're merging a new feature into the main branch, you first want to switch to the main branch and then merge into it:
<pre><code class="hljs"># ...develop some code...

$ git add .
$ git commit –m &quot;Some commit message&quot;
$ git checkout master
Switched to branch &#x27;master&#x27;
$ git merge new-branch
</code></pre>
If all goes well then our job is done. The new feature commits now appear in the main branch. However, it is possible that Git won't be able to complete the merge due to a conflict change in the source branch. This is called a merge conflict.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice: If you'd like to read more about Merge Conflicts, how they can arise and how to resolve them - read our <a target="_blank" href="https://stackabuse.com/git-guide-to-solving-merge-conflicts/">&quot;Git: Guide to Resolving Merge Conflicts&quot;!</a>

 </div>
 </div>
 </div>
 To summarize, here are the commands to create a new branch, make some commits, and merge it back into master:
<pre><code class="hljs">$ git checkout master
$ git branch new-branch
$ git checkout new-branch

# ...develop some code...

$ git add .
$ git commit –m &quot;Some commit message&quot;
$ git checkout master
$ git merge new-branch
</code></pre>
<h4 id="abouttheauthor">About the Author</h4>
This article was written by Jacob Stopak, a software consultant and developer with passion for helping others improve their lives through code. Jacob is the creator of <a target="_blank" href="https://initialcommit.com">Initial Commit</a> - a site dedicated to helping curious developers learn how their favorite programs are coded. Its featured project helps people <a rel="nofollow" target="_blank" href="https://initialcommit.com/blog/Learn-Git-Code-Definitive-Guide">learn Git</a> at the code level.

Jacob Stopak

Git: Merge Branch into Master

<h3 id="introduction">Introduction</h3>
A common task in programming is opening a file and parsing its contents. What do you do when the file you are trying to process is quite large, like several GB of data or larger? The answer to this problem is to read in chunks of a file at a time, process it, then free it from memory so you can process another chunk until the whole massive file has been processed. While it's up to you to determine a suitable size for the chunks of data you're processing, for many applications, it's suitable to process a file one line at a time.
Throughout this article, we'll be covering a number of code examples that demonstrate how to read files line by line. In case you want to try out some of these examples by yourself, the code used in this article can be found at the following <a rel="nofollow noopener" target="_blank" href="https://github.com/amcquistan/Python-read-file-line-by-line">GitHub repo</a>.
<ul>
<li><a href="#basicfileioinpython">Basic File IO in Python</a></li>
<li><a href="#readafilelinebylineinpythonwithreadline">Read a File Line-by-Line in Python with <code>readline()</code></a></li>
<li><a href="#readafilelinebylineinpythonwithreadlines">Read a File Line-by-Line in Python with <code>readlines()</code></a></li>
<li><a href="#readafilelinebylinewithaforloopmostpythonicapproach">Read a File Line-by-Line with a <code>for</code> Loop</a> - Best Approach!</li>
<li><a href="#applicationsofreadingfileslinebyline">Applications of Reading Files Line-by-Line</a></li>
</ul>
<h3 id="basicfileioinpython">Basic File IO in Python</h3>
Python is a great general-purpose programming language, and it has a number of very useful file IO functionality in its standard library of built-in functions and modules.
The built-in <code>open()</code> function is what you use to open a <a rel="nofollow noopener" target="_blank" href="https://docs.python.org/3/library/functions.html#open">file object</a> for either reading or writing purposes. Here's how you can use it to open a file:
<pre><code class="hljs">fp = open(&#x27;path/to/file.txt&#x27;, &#x27;r&#x27;)
</code></pre>
As demonstrated above, the <code>open()</code> function takes in multiple arguments. We will be focusing on two arguments, with the first being a positional string parameter representing the path to the file you want to open. The second (optional) parameter is also a string, and it specifies the mode of interaction you intend to be used on the file object being returned by the function call. The most common modes are listed in the table below, with the default being 'r' for reading:
<table class="table table-striped">
 <thead>
 <tr>
 <th>
 Mode
 </th>
 <th>
 Description
 </th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td><code>r</code></td>
 <td>Open for reading plain text</td>
 </tr>
 <tr>
 <td><code>w</code></td>
 <td>Open for writing plain text</td>
 </tr>
 <tr>
 <td><code>a</code></td>
 <td>Open an existing file for appending plain text</td>
 </tr>
 <tr>
 <td><code>rb</code></td>
 <td>Open for reading binary data</td>
 </tr>
 <tr>
 <td><code>wb</code></td>
 <td>Open for writing binary data</td>
 </tr>
 </tbody>
</table>
Once you have written or read all of the desired data in a file object, you need to close the file so that resources can be reallocated on the operating system that the code is running on.
<pre><code class="hljs">fp.close()
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: It's always good practice to close a file object resource, but it's a task that's easy to forget.

 </div>
 </div>
 </div>
 While you can always remember to call <code>close()</code> on a file object, there's an alternate and more elegant way to open a file object and ensure that the Python interpreter cleans up after its use:
<pre><code class="hljs">with open(&#x27;path/to/file.txt&#x27;) as fp:
 # Do stuff with fp
</code></pre>
By simply using the <a target="_blank" href="https://stackabuse.com/python-context-managers/"><code>with</code> keyword</a> (introduced in Python 2.5) to the code we use to open a file object, Python will do something similar to the following code. This ensures that no matter what the file object is closed after use:
<pre><code class="hljs">try:
 fp = open(&#x27;path/to/file.txt&#x27;)
 # Do stuff with fp
finally:
 fp.close()
</code></pre>
Either of these two methods is suitable, with the first example being more Pythonic.
The file object returned from the <code>open()</code> function has three common explicit methods (<code>read()</code>, <code>readline()</code>, and <code>readlines()</code>) to read in data. The <code>read()</code> method reads all the data into a single string. This is useful for smaller files where you would like to do text manipulation on the entire file. Then there is <code>readline()</code>, which is a useful way to only read in individual lines, in incremental amounts at a time, and return them as strings. The last explicit method, <code>readlines()</code>, will read all the lines of a file and return them as a list of strings.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: For the remainder of this article we will be working with the text of the book The &quot;Iliad of Homer&quot;, which can be found at <a rel="nofollow noopener" target="_blank" href="http://www.gutenberg.org">gutenberg.org</a>, as well as in the GitHub repo where the code is for this article.

 </div>
 </div>
 </div>
 <h3 id="readingafilelinebylineinpythonwithreadline">Reading a File Line-by-Line in Python with readline()</h3>
Let's start off with the <code>readline()</code> method, which reads a single line, which will require us to use a counter and increment it:
<pre><code class="hljs">filepath = &#x27;Iliad.txt&#x27;
with open(filepath) as fp:
 line = fp.readline()
 cnt = 1
 while line:
 print(&quot;Line {}: {}&quot;.format(cnt, line.strip()))
 line = fp.readline()
 cnt += 1
</code></pre>
This code snippet opens a file object whose reference is stored in <code>fp</code>, then reads in a line one at a time by calling <code>readline()</code> on that file object iteratively in a <code>while</code> loop. It then simply prints the line to the console.
Running this code, you should see something like the following:
<pre><code class="hljs">...
Line 567: exceedingly trifling. We have no remaining inscription earlier than the
Line 568: fortieth Olympiad, and the early inscriptions are rude and unskilfully
Line 569: executed; nor can we even assure ourselves whether Archilochus, Simonides
Line 570: of Amorgus, Kallinus, Tyrtaeus, Xanthus, and the other early elegiac and
Line 571: lyric poets, committed their compositions to writing, or at what time the
Line 572: practice of doing so became familiar. The first positive ground which
Line 573: authorizes us to presume the existence of a manuscript of Homer, is in the
Line 574: famous ordinance of Solon, with regard to the rhapsodies at the
Line 575: Panathenaea: but for what length of time previously manuscripts had
Line 576: existed, we are unable to say.
...
</code></pre>
Though, this approach is crude and explicit. Most certainly not very Pythonic. We can utilize the <code>readlines()</code> method to make this code much more succinct.
<h3 id="readafilelinebylinewithreadlines">Read a File Line-by-Line with readlines()</h3>
The <code>readlines()</code> method reads all the lines and stores them into a <code>List</code>. We can then iterate over that list and using <code>enumerate()</code>, make an index for each line for our convenience:
<pre><code class="hljs">file = open(&#x27;Iliad.txt&#x27;, &#x27;r&#x27;)
lines = file.readlines()

for index, line in enumerate(lines):
 print(&quot;Line {}: {}&quot;.format(index, line.strip()))
 
file.close()
</code></pre>
This results in:
<pre><code class="hljs">...
Line 160: INTRODUCTION.
Line 161:
Line 162:
Line 163: Scepticism is as much the result of knowledge, as knowledge is of
Line 164: scepticism. To be content with what we at present know, is, for the most
Line 165: part, to shut our ears against conviction; since, from the very gradual
Line 166: character of our education, we must continually forget, and emancipate
Line 167: ourselves from, knowledge previously acquired; we must set aside old
Line 168: notions and embrace fresh ones; and, as we learn, we must be daily
Line 169: unlearning something which it has cost us no small labour and anxiety to
Line 170: acquire.
...
</code></pre>
Now, although much better, we don't even need to call the <code>readlines()</code> method to achieve this same functionality. This is the traditional way of reading a file line-by-line, but there's a more modern, shorter one.
<h3 id="readafilelinebylinewithaforloopmostpythonicapproach">Read a File Line-by-Line with a for Loop - Most Pythonic Approach</h3>
The returned <code>File</code> itself is an iterable. We don't need to extract the lines via <code>readlines()</code> at all - we can iterate the returned object itself. This also makes it easy to <code>enumerate()</code> it so we can write the line number in each <code>print()</code> statement.
This is the shortest, most Pythonic approach to solving the problem, and the approach favored by most:
<pre><code class="hljs">with open(&#x27;Iliad.txt&#x27;) as f:
 for index, line in enumerate(f):
 print(&quot;Line {}: {}&quot;.format(index, line.strip()))
</code></pre>
This results in:
<pre><code class="hljs">...
Line 277: Mentes, from Leucadia, the modern Santa Maura, who evinced a knowledge and
Line 278: intelligence rarely found in those times, persuaded Melesigenes to close
Line 279: his school, and accompany him on his travels. He promised not only to pay
Line 280: his expenses, but to furnish him with a further stipend, urging, that,
Line 281: &quot;While he was yet young, it was fitting that he should see with his own
Line 282: eyes the countries and cities which might hereafter be the subjects of his
Line 283: discourses.&quot; Melesigenes consented, and set out with his patron,
Line 284: &quot;examining all the curiosities of the countries they visited, and
...
</code></pre>
Here, we're taking advantage of the built-in functionalities of Python that allow us to effortlessly iterate over an iterable object, simply using a <code>for</code> loop. If you'd like to read more about Python's built-in functionalities on iterating objects, we've got you covered:
<ul>
<li><a href="https://stackabuse.com/pythons-itertools-count-cycle-and-chain">Python's itertools – count(), cycle() and chain()</a></li>
<li><a href="https://stackabuse.com/pythons-iteration-tools-filter-islice-map-and-zip">Python's itertools: filter(), islice(), map() and zip()</a></li>
</ul>
<h3 id="applicationsofreadingfileslinebyline">Applications of Reading Files Line-by-Line</h3>
How can you use this practically? Most NLP applications deal with large corpora of data. Most of the time, it won't be wise to read the entire corpora into memory. While rudimentary, you can write a from-scratch solution to count the frequency of certain words, without using any external libraries. Let's write a simple script that loads in a file, reads it line-by-line, and counts the frequency of words, printing the 10 most frequent words and the number of their occurrences:
<pre><code class="hljs">import sys
import os

def main():
 filepath = sys.argv[1]
 if not os.path.isfile(filepath):
 print(&quot;File path {} does not exist. Exiting...&quot;.format(filepath))
 sys.exit()
 
 bag_of_words = {}
 with open(filepath) as fp:
 for line in fp:
 record_word_cnt(line.strip().split(&#x27; &#x27;), bag_of_words)
 sorted_words = order_bag_of_words(bag_of_words, desc=True)
 print(&quot;Most frequent 10 words {}&quot;.format(sorted_words[:10]))
 
def order_bag_of_words(bag_of_words, desc=False):
 words = [(word, cnt) for word, cnt in bag_of_words.items()]
 return sorted(words, key=lambda x: x[1], reverse=desc)

def record_word_cnt(words, bag_of_words):
 for word in words:
 if word != &#x27;&#x27;:
 if word.lower() in bag_of_words:
 bag_of_words[word.lower()] += 1
 else:
 bag_of_words[word.lower()] = 1

if __name__ == &#x27;__main__&#x27;:
 main()
</code></pre>
The script uses the <code>os</code> module to make sure that the file we're attempting to read actually exists. If so, its read line-by-line and each line is passed on into the <code>record_word_cnt()</code> function. It delimits the spaces between words and adds the word to the dictionary - <code>bag_of_words</code>. Once all the lines are recorded into the dictionary, we order it via <code>order_bag_of_words()</code> which returns a list of tuples in the <code>(word, word_count)</code> format, sorted by the word count.
Finally, we print the top ten most common words.
Typically, for this, you'd create a <a target="_blank" href="https://stackabuse.com/python-for-nlp-creating-bag-of-words-model-from-scratch">Bag of Words Model</a>, using libraries like NLTK, though, this implementation will suffice. Let's run the script and provide our <code>Iliad.txt</code> to it:
<pre><code class="hljs">$ python app.py Iliad.txt
</code></pre>
This results in:
<pre><code class="hljs">Most frequent 10 words [(&#x27;the&#x27;, 15633), (&#x27;and&#x27;, 6959), (&#x27;of&#x27;, 5237), (&#x27;to&#x27;, 4449), (&#x27;his&#x27;, 3440), (&#x27;in&#x27;, 3158), (&#x27;with&#x27;, 2445), (&#x27;a&#x27;, 2297), (&#x27;he&#x27;, 1635), (&#x27;from&#x27;, 1418)]
</code></pre>

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice: If you'd like to read more about NLP, we've got a series of guides on various tasks: <a target="_blank" href="https://stackabuse.com/tag/nlp">Natural Language Processing in Python</a>.

 </div>
 </div>
 </div>
 <h3 id="conclusion">Conclusion</h3>
In this article, we've explored multiple ways to read a file line-by-line in Python, as well as created a rudimentary Bag of Words model to calculate the frequency of words in a given file.

Adam McQuistan

In this tutorial, we'll be reading a file line by line in Python with the readline() and readlines() functions as well as a for loop - through hands-on examples.