<h2 id="introduction">Introduction</h2>
Python has a set of built-in library objects and functions to help us with this task. In this tutorial, we'll learn how to check if a file or directory is empty in Python.
<h2 id="distinguishbetweenafileandadirectory">Distinguish Between a File and a Directory</h2>
When we'd like to check if a path is empty or not, we'll want to know if it's a file or directory since this affects the approach we'll want to use.
Let's say we have two placeholder variables <code>dirpath</code> and <code>filepath</code> identifying a local directory and file:
<pre><code class="hljs">dirpath = &#x27;/mnt/f/code.books/articles/python&#x27;
filepath = &#x27;/mnt/f/code.books/articles/python/code/file_dir.py&#x27;
</code></pre>
<h3 id="usingospath">Using os.path</h3>
Python provides the <a rel="nofollow noopener" target="_blank" href="https://docs.python.org/3/library/os.html#module-os"><code>os</code></a> module which is a standard Python package of functions, objects, and constants to work with the operating system.
<code>os.path</code> provides us with the <code>isfile()</code> and <code>isdir()</code> functions to easily distinguish between a file and a directory:
<pre><code class="hljs">import os

dirpath = &#x27;/mnt/f/code.books/articles/python&#x27;
filepath = &#x27;/mnt/f/code.books/articles/python/code/file_dir.py&#x27;

os.path.isfile(dirpath) # False
os.path.isdir(dirpath) # True
os.path.isfile(filepath) # True
os.path.isdir(filepath) # False
</code></pre>
Both of these functions return a <code>Boolean</code> value.
<h3 id="usingpathlib">Using pathlib</h3>
Python 3.4 introduced the <a rel="nofollow noopener" target="_blank" href="https://docs.python.org/3/library/pathlib.html"><code>pathlib</code></a> module, that provides an Object-oriented interface to work with the filesystems.
<code>pathlib</code> simplifies working with filesystems as compared to <code>os</code> or <code>os.path</code>.
The <code>Path</code> class of the <code>pathlib</code> module accepts a path as its argument and returns a <code>Path</code> object, that can be easily queried or chained further with methods and attributes:
<pre><code class="hljs">from pathlib import Path

dirpath = &#x27;/mnt/f/code.books/articles/python&#x27;
filepath = &#x27;/mnt/f/code.books/articles/python/code/file_dir.py&#x27;

Path(dirpath).is_file() # False
Path(dirpath).is_dir() # True
Path(filepath).is_file() # True
Path(dirpath).is_file() # False
</code></pre>
Here, we're checking if the <code>Path</code> object is a file or directory instead.
<h2 id="checkifafileisempty">Check if a File is Empty</h2>
An empty file or a zero-byte file is any file that contains no data or content. The file can be any file type. Certain files (such as music files) may have no data but still contain metadata (such as the author). Such files can't be considered as an empty file.
One can create an empty file quickly on Linux and MacOS:
<pre><code class="hljs">$ touch emptyfile
</code></pre>
Or on Windows:
<pre><code class="hljs">$ type nul &gt; emptyfile
</code></pre>
Let's define variables now - <code>emptyfile</code> and <code>nonemptyfile</code> pointing to an empty file having zero bytes and a non-empty file having the size of one byte:
<pre><code class="hljs">emptyfile = &#x27;/mnt/f/code.books/articles/python/emptyfile&#x27;
nonemptyfile = &#x27;/mnt/f/code.books/articles/python/onebytefile&#x27;
</code></pre>
Let's take a look at the type and size of these files:
<pre><code class="hljs">$ ls -l
-rwxrwxrwx 1 root root 0 Sep 10 18:06 emptyfile
-rwxrwxrwx 1 root root 1 Sep 10 18:08 onebytefile
$ file emptyfile
emptyfile: empty
$ file onebytefile
onebytefile: very short file (no magic)
</code></pre>
<h3 id="usingosstat">Using os.stat</h3>
Alternatively, we can use Python's <code>os</code> module to check this information as well. The <code>os.stat()</code> function returns a <code>stat_result</code> object. This object is basically a data structure that is collection of the file's properties:
<pre><code class="hljs">import os

emptyfile = &#x27;/mnt/f/code.books/articles/python/emptyfile&#x27;
nonemptyfile = &#x27;/mnt/f/code.books/articles/python/onebytefile&#x27;

result = os.stat(nonemptyfile)
result.st_size # 1

result = os.stat(emptyfile)
result.st_size # 0
</code></pre>
<h3 id="usingospath">Using os.path</h3>
Python's <code>os.path</code> module makes it very easy to work with file paths. Apart from checking existence of a path or distinguishing their type we can also retrieve the size of a file specified as a string.
<code>os.path.getsize()</code> returns size of a file specified as a <a target="_blank" rel="nofollow noopener" href="https://docs.python.org/3/glossary.html#term-path-like-object">path-like-object</a> and is much easier to use than <code>os.stat()</code>:
<pre><code class="hljs">import os

emptyfile = &#x27;/mnt/f/code.books/articles/python/emptyfile&#x27;
nonemptyfile = &#x27;/mnt/f/code.books/articles/python/onebytefile&#x27;

os.path.getsize(emptyfile) # 0

os.path.getsize(nonemptyfile) # 1
</code></pre>
<h3 id="usingpathlib">Using pathlib</h3>
If we are working on Python 3.4 or above we can use the <code>pathlib</code> module to retrieve size of a file. This basically replaces the <code>os</code> module. <code>Path.stat()</code> returns the <code>stat_result</code> property of a <code>Path</code> object which is equivalent to return value of <code>os.stat()</code>:
<pre><code class="hljs">from pathlib import Path

emptyfile = &#x27;/mnt/f/code.books/articles/python/emptyfile&#x27;
nonemptyfile = &#x27;/mnt/f/code.books/articles/python/onebytefile&#x27;

print(&#x27;File stats: &#x27; + Path(emptyfile).stat())

print(&#x27;File size: &#x27; + Path(emptyfile).stat().st_size + &#x27; byte(s)&#x27;)

print(&#x27;File stats: &#x27; + Path(nonemptyfile).stat())

print(&#x27;File size: &#x27; + Path(nonemptyfile).stat().st_size + &#x27; byte(s)&#x27;)
</code></pre>
This results in:
<pre><code class="hljs">File stats: os.stat_result(st_mode=33279, st_ino=14355223812249048, st_dev=17, st_nlink=1, st_uid=0, st_gid=0, st_size=0, st_atime=1600087010, st_mtime=1600087010, st_ctime=1600087010)
File size: 0 byte(s)

File stats: os.stat_result(st_mode=33279, st_ino=5629499534218713, st_dev=17, st_nlink=1, st_uid=0, st_gid=0, st_size=1, st_atime=1600088120, st_mtime=1600088072, st_ctime=1600088072)
File size: 1 byte(s)
</code></pre>
<h2 id="checkifadirectoryisempty">Check if a Directory is Empty</h2>
A directory that contains no other files or sub-directories is an empty directory. However, every directory (even empty ones) do contain the following 2 entries:
<ul>
<li>. (pronounced dot) references current directory and is useful in operations like finding something inside the current directory</li>
<li>.. (pronounced double dot) references parent directory of the current directory, is required to step back from the current directory</li>
</ul>
Let's define two variables - <code>emptydirectory</code> and <code>nonemptydirectory</code> pointing to an empty and a non-empty directory:
<pre><code class="hljs">emptydirectory = &#x27;/mnt/f/code.books/articles/python/markdown&#x27;
nonemptydirectory = &#x27;/mnt/f/code.books/articles/python/code&#x27;
</code></pre>
The empty directory doesn't have any items in it:
<pre><code class="hljs">$ pwd
/mnt/f/code.books/articles/python/markdown
$ ls -la
total 0
drwxrwxrwx 1 root root 512 Sep 11 11:52 .
drwxrwxrwx 1 root root 512 Sep 10 20:22 ..
</code></pre>
The non-empty directory has a single file:
<pre><code class="hljs">$ pwd
/mnt/f/code.books/articles/python/code
$ ls -la
total 0
drwxrwxrwx 1 root root 512 Sep 14 11:02 .
drwxrwxrwx 1 root root 512 Sep 14 18:22 ..
-rwxrwxrwx 1 root root 425 Sep 14 12:27 file_dir.py
</code></pre>
<h3 id="usingoslistdir">Using os.listdir()</h3>
The <code>os.listdir()</code> returns a sequence that contains the name of all the items found in the directory path passed as the argument. It does not include the <code>.</code> and <code>..</code> entries:
<pre><code class="hljs">import os

os.listdir(emptydirectory) # []
os.listdir(nonemptydirectory) # [&#x27;file_dir.py&#x27;]
</code></pre>
Calculating the length of the returned list easily determines if the directory is empty or not. An empty directory always has a length of zero:
<pre><code class="hljs">import os

print(len(os.listdir(nonemptydirectory))) # 1
print(len(os.listdir(emptydirectory))) # 0
</code></pre>
<h3 id="usingosscandir">Using os.scandir()</h3>
The <code>os.listdir()</code> function is useful when you need a whole bunch of entries name as a list for further processing. However, to check if there's at least a single entry, we don't need a list of all the files inside.
If a directory is huge, the <code>os.listdir()</code> function will take a long time to run, whereas, as long as there's more than <code>0</code> entries, our question is answered.
A function that comes to aid is <code>os.scandir()</code> which returns a lazy iterable or generator.
<a target="_blank" href="/python-generators">Generators</a> return iterators that can be looped over like normal iterables such as a list. But unlike a list, set or dictionary, they do not store a whole bunch of values in memory and instead return a new value on request.
This approach is approximately ~200 times faster on directories of ~1000 files.
So instead of looping over the whole directory structure, we can use <code>os.scandir()</code> to check if there is at least one entry found in the directory path:
<pre><code class="hljs">import os

emptydirectory = &#x27;/mnt/f/code.books/articles/python/markdown&#x27;
nonemptydirectory = &#x27;/mnt/f/code.books/articles/python/code&#x27;

print(next(os.scandir(emptydirectory), None))
print(next(os.scandir(nonemptydirectory), None)) # &lt;DirEntry &#x27;file_dir.py&#x27;&gt;
</code></pre>
We are using <code>next()</code> which is a built-in function to retrieve the next available item from the lazy iterator returned by <code>os.scandir()</code>. Since <code>emptydirectory</code> has no available items - it is returning <code>None</code> whereas for <code>nonemptydirectory</code> it is returning an <code>os.DirEntry</code> object.
<h3 id="usingpathlib">Using pathlib</h3>
A preferred approach to the <code>os</code> module is the <code>pathlib</code> module. We'll use <code>pathlib.Path.iterdir()</code>, which is not only simpler but also much easier to use than <code>os.listdir()</code> or <code>os.scandir()</code>.
It returns back a lazy iterable or generator object much like <code>os.scandir()</code>, that iterates over the files in directory path passed as argument:
<pre><code class="hljs">from pathlib import Path
print(Path(emptydirectory).iterdir()) # &lt;generator object Path.iterdir at 0x7f2cf6f584a0&gt;
</code></pre>
Using <code>next()</code>, we are trying to fetch next available item. With <code>None</code> as the default return item, <code>next()</code> won't raise a <code>StopIteration</code> exception in case there is no item in the collection:
<pre><code class="hljs">print(next(Path(emptydirectory).iterdir(), None)) # None
print(next(Path(nonemptydirectory).iterdir(), None)) # /mnt/f/code.books/articles/python/code/file_dir.py
</code></pre>
Most of the built-in Python functions work with iterables, including the <a target="_blank" href="/any-and-all-in-python-with-examples/">any()</a> function that returns back <code>True</code> if the iterable has at least one element that can be evaluated as <code>True</code>:
<pre><code class="hljs">from pathlib import Path
print(any(Path(emptydirectory).iterdir()) # False
print(any(nonemptydirectory).iterdir()) # True
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this tutorial, we've gone over how to distinguish between files and directories, after which we've checked for their emptiness.
This can be done via the <code>os</code> or <code>pathlib</code> modules and their convenience functions and classes.

Tapan Pandey

In this tutorial, we'll learn how to check if a file or directory is empty in Python with examples. This can be done using the os and pathlib modules.

Python: Check if File or Directory is Empty

<h2 id="introduction">Introduction</h2>
In this article, we'll be going through a few examples of how to check if a variable is a number in Python.
Python is dynamically typed. There is no need to declare a variable type, while instantiating it - the interpreter infers the type at runtime:
<pre><code class="hljs">variable = 4
another_variable = &#x27;hello&#x27;
</code></pre>
Additionally, a variable can be reassigned to a new type at any given time:
<pre><code class="hljs"># Assign a numeric value
variable = 4

# Reassign a string value
variable = &#x27;four&#x27;
</code></pre>
This approach, while having advantages, also introduces us to a few issues. Namely, when we receive a variable, we typically don't know of which type it is. If we're expecting a Number, but receive <code>variable</code>, we'll want to check if it's a number before working with it.
<h3 id="usingthetypefunction">Using the type() Function</h3>
The <code>type()</code> function in Python returns the type of the argument we pass to it, so it's a handy function for this purpose:
<pre><code class="hljs">myNumber = 1
print(type(myNumber))

myFloat = 1.0
print(type(myFloat))

myString = &#x27;s&#x27;
print(type(myString))
</code></pre>
This results in:
<pre><code class="hljs">&lt;class &#x27;int&#x27;&gt;
&lt;class &#x27;float&#x27;&gt;
&lt;class &#x27;str&#x27;&gt;
</code></pre>
Thus, a way to check for the type is:
<pre><code class="hljs">myVariable = input(&#x27;Enter a number&#x27;)
if type(myVariable) == int or type(myVariable) == float:
 # Do something
else:
 print(&#x27;The variable is not a number&#x27;)
</code></pre>
Here, we check if the variable type, entered by the user is an <code>int</code> or a <code>float</code>, proceeding with the program if it is. Otherwise, we notify the user that they've entered a non-Number variable. Please keep in mind that if you're comparing to multiple types, such as <code>int</code> or <code>float</code>, you have to use the <code>type()</code> function both times.
If we just said <code>if type(var) == int or float</code>, which is seemingly fine, an issue would arise:
<pre><code class="hljs">myVariable = &#x27;A string&#x27;
if type(myVariable) == int or float:
 print(&#x27;The variable a number&#x27;)
else:
 print(&#x27;The variable is not a number&#x27;)
</code></pre>
This, regardless of the input, returns:
<pre><code class="hljs">The variable is a number
</code></pre>
This is because Python checks for truth values of the statements. Variables in Python can be evaluated as <code>True</code> except for <code>False</code>, <code>None</code>, <code>0</code> and empty containers like <code>[]</code>, <code>{}</code>, <code>set()</code>, <code>()</code>, <code>''</code> or <code>&quot;&quot;</code>.
Hence when we write <code>or float</code> in our <code>if</code> condition, it is equivalent to writing <code>or True</code> which will always evaluate to <code>True</code>.
<h2 id="numbersnumber">numbers.Number</h2>
A good way to check if a variable is a number is the <code>numbers</code> module. You can check if the variable is an instance the <code>Number</code> class, with the <code>isinstance()</code> function:
<pre><code class="hljs">import numbers

variable = 5
print(isinstance(5, numbers.Number))
</code></pre>
This will result in:
<pre><code class="hljs">True
</code></pre>
Note: This approach can behave unexpectedly with numeric types outside of core Python. Certain frameworks might have non-<code>Number</code> numeric implementation, in which case this approach will falsely return <code>False</code>.
<h3 id="usingatryexceptblock">Using a try-except block</h3>
Another method to check if a variable is a number is using a try-except block. In the try block, we cast the given variable to an <code>int</code> or <code>float</code>. Successful execution of the <code>try</code> block means that a variable is a number i.e. either <code>int</code> or <code>float</code>:
<pre><code class="hljs">myVariable = 1
try:
 tmp = int(myVariable)
 print(&#x27;The variable a number&#x27;)
except:
 print(&#x27;The variable is not a number&#x27;)
</code></pre>
This works for both <code>int</code> and <code>float</code> because you can cast an <code>int</code> to <code>float</code> and a <code>float</code> to an <code>int</code>.
If you specifically only want to check if a variable is one of these, you should use the <code>type()</code> function.
<h2 id="conclusion">Conclusion</h2>
Python is a dynamically typed language, which means that we might receive a data type different than the one we're expecting.
In cases where we'd want to enforce data types, it's worth checking if a variable is of the desired type. In this article, we've covered three ways to check if a variable is a Number in Python.

Hassan Saeed

In this tutorial, we'll go over three examples of how to check if a variable is a number in Python. These are the type() function, the numbers module and the try-catch approach.

Python: Check if Variable is a Number

<h2 id="introduction">Introduction</h2>
A web application often needs to communicate with web servers to get various resources. You might need to fetch data from or post data to an external web server or API.
Using client-side JavaScript, this can be achieved using the fetch API and the <code>window.fetch()</code> function. In NodeJS, several packages/libraries can achieve the same result. One of them is the <a rel="nofollow noopener" target="_blank" href="https://www.npmjs.com/package/node-fetch"><code>node-fetch</code></a> package.
<code>node-fetch</code> is a lightweight module that enables us to use the <code>fetch()</code> function in NodeJS, with very similar functionality as <code>window.fetch()</code> in native JavaScript, but with a <a rel="nofollow noopener" target="_blank" href="https://github.com/node-fetch/node-fetch/blob/master/docs/v3-LIMITS.md">few differences</a>.
<h2 id="gettingstartedwithnodefetch">Getting Started With node-fetch</h2>
To use <code>node-fetch</code> in your project, <code>cd</code> into your project directory, and run:
<pre><code class="hljs">$ npm install node-fetch
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 As of version 3.0, <code>node-fetch</code> is an ESM-only module - you are not able to import it with <code>require()</code>. If you don't use ESM yourself, it's advised to stay on version 2.0 instead of the latest one, in which case you can use the standard <code>require()</code> syntax.

 </div>
 </div>
 </div>
 To use the module in code (for versions prior to version 3.0), use:
<pre><code class="hljs">const fetch = require(&#x27;node-fetch&#x27;);
</code></pre>
If you're using ESM, you'll import the module in a different manner:
<pre><code class="hljs">import fetch from &#x27;node-fetch&#x27;;
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: The API between <code>node-fetch</code> 3.0 and 2.0 is the same, just the import differs.

 </div>
 </div>
 </div>
 To install a specific version of the module, you can use <code>npm</code>:
<pre><code class="hljs">$ npm install node-fetch@2.0
</code></pre>
As previously mentioned, the <code>fetch()</code> function in the <code>node-fetch</code> module behaves very similarly to the native <code>window.fetch()</code> function. Its signature is:
<pre><code class="hljs">fetch(url[, options]);
</code></pre>
The <code>url</code> parameter is simply the direct URL to the resource we wish to fetch. It has to be an absolute URL or the function will throw an error. The optional <code>options</code> parameter is used when we want to use <code>fetch()</code> for anything other than a simple <code>GET</code> request, but we will talk about that more in-depth later.
The function returns a <code>Response</code> object that contains useful functions and information about the HTTP response, such as:
<ul>
<li><code>text()</code> - returns the response body as a string</li>
<li><code>json()</code> - parses the response body into a JSON object, and throws an error if the body can't be parsed</li>
<li><code>status</code> and <code>statusText</code> - contain information about the HTTP status code</li>
<li><code>ok</code> - equals <code>true</code> if <code>status</code> is a 2xx status code (a successful request)</li>
<li><code>headers</code> - an object containing response headers, a specific header can be accessed using the <code>get()</code> function.</li>
</ul>
<h2 id="sendinggetrequestsusingnodefetch">Sending GET Requests Using node-fetch</h2>
There are two common use cases of fetching data from a web server. You might want to retrieve text from the web server, a whole web page, or data from using REST API. The <code>node-fetch</code> package allows you to do all of that.
Create a directory for your project, <code>cd</code> into the directory and initialize a Node project with default settings:
<pre><code class="hljs">$ npm init -y
</code></pre>
This will create a <code>package.json</code> file in the directory. Next, install <code>node-fetch</code> as shown above and add an <code>index.js</code> file.
<h3 id="fetchingtextorwebpages">Fetching Text or Web Pages</h3>
Let's make a simple <code>GET</code> request to Google's home page:
<pre><code class="hljs">fetch(&#x27;https://google.com&#x27;)
 .then(res =&gt; res.text())
 .then(text =&gt; console.log(text));
</code></pre>
In the code above, we're loading the <code>node-fetch</code> module and then fetching the Google home page. The only parameter we've added to the <code>fetch()</code> function is the URL of the server we're making an HTTP request to. Because <code>node-fetch</code> is promise-based, we're chaining a couple of <code>.then()</code> functions to help us manage the response and data from our request.
In this line, we're waiting to receive the response from the Google web server and converting it to text format:
<pre><code class="hljs">.then(res =&gt; res.text());
</code></pre>
Here we're waiting for the result of the previous conversion and printing it to the console:
<pre><code class="hljs">.then(text =&gt; console.log(text));
</code></pre>
If we run the code above from the console:
<pre><code class="hljs">$ node index.js
</code></pre>
We'll get the entire HTML markup of the Google home page logged to the console:
<pre><code class="hljs">&lt;!doctype html&gt;
&lt;html itemscope=&quot;&quot; itemtype=&quot;http://schema.org/WebPage&quot; lang=&quot;en-RS&quot;&gt;
 &lt;head&gt;
 &lt;meta charset=&quot;UTF-8&quot;&gt;
 &lt;meta content=&quot;origin&quot; name=&quot;referrer&quot;&gt;
 &lt;!-- Rest of the page --&gt;
</code></pre>
<h3 id="fetchingjsondatafromrestapi">Fetching JSON Data From REST API</h3>
Another common use case for the <code>node-fetch</code> module is getting data using the REST API.
We will retrieve fake user data from the <a rel="nofollow noopener" target="_blank" href="https://jsonplaceholder.typicode.com/">JSONPlaceholder</a> REST API. As before, the <code>fetch()</code> function takes in the URL of the server and awaits a response.
Let us see how that works:
<pre><code class="hljs">fetch(&#x27;https://jsonplaceholder.typicode.com/users&#x27;)
 .then(res =&gt; res.json())
 .then(json =&gt; {
 console.log(&quot;First user in the array:&quot;);
 console.log(json[0]);
 console.log(&quot;Name of the first user in the array:&quot;);
 console.log(json[0].name);
})
</code></pre>
The body of the HTTP response contains JSON-formatted data, namely an array containing information about users. With this in mind, we used the <code>.json()</code> function, and this allowed us to easily access individual elements and their fields.
Running this program would give us:
<pre><code class="hljs">First element in the array:
{
 id: 1,
 name: &#x27;Leanne Graham&#x27;,
 username: &#x27;Bret&#x27;,
 email: &#x27;Sincere@april.biz&#x27;,
 address: {
 street: &#x27;Kulas Light&#x27;,
 suite: &#x27;Apt. 556&#x27;,
 city: &#x27;Gwenborough&#x27;,
 zipcode: &#x27;92998-3874&#x27;,
 geo: { lat: &#x27;-37.3159&#x27;, lng: &#x27;81.1496&#x27; }
 },
 phone: &#x27;1-770-736-8031 x56442&#x27;,
 website: &#x27;hildegard.org&#x27;,
 company: {
 name: &#x27;Romaguera-Crona&#x27;,
 catchPhrase: &#x27;Multi-layered client-server neural-net&#x27;,
 bs: &#x27;harness real-time e-markets&#x27;
 }
}

Name of the first person in the array:
Leanne Graham
</code></pre>
We also could have printed the entire JSON returned by <code>res.json()</code>.
<h2 id="sendingpostrequestsusingnodefetch">Sending POST Requests Using node-fetch</h2>
We can also use the <code>fetch()</code> function to post data instead of retrieving it. As we mentioned earlier, the <code>fetch()</code> function allows for an additional parameter to be added to make <code>POST</code> requests to a web server. Without this optional parameter, our request is a <code>GET</code> request, by default.
There are many possible options we can set using this parameter, but the only ones we'll use in this article are <code>method</code>, <code>body</code> and <code>headers</code>.
These fields have straight-forward meanings: <code>method</code> sets what type of HTTP request we're using (<code>POST</code> in our case), <code>body</code> contains the body/data of our request, and <code>headers</code> contains all the necessary headers, which in our case is just the <code>Content-Type</code> so there isn't any confusion when parsing our request.
For a full list of options, you can visit the <a rel="nofollow noopener" target="_blank" href="https://www.npmjs.com/package/node-fetch#options">documentation</a>.
We'll demonstrate how this works by adding a new item to <a rel="nofollow noopener" target="_blank" href="https://jsonplaceholder.typicode.com/todos">JSONPlaceholder's todos</a>. Let's add a new item to that list for the user whose <code>id</code> equals <code>123</code>. First, we need to create a <code>todo</code> object, and later convert it to JSON when adding it to the <code>body</code> field:
<pre><code class="hljs">let todo = {
 userId: 123,
 title: &quot;loren impsum doloris&quot;,
 completed: false
};

fetch(&#x27;https://jsonplaceholder.typicode.com/todos&#x27;, {
 method: &#x27;POST&#x27;,
 body: JSON.stringify(todo),
 headers: { &#x27;Content-Type&#x27;: &#x27;application/json&#x27; }
}).then(res =&gt; res.json())
 .then(json =&gt; console.log(json));
</code></pre>
The process is very similar to making a <code>GET</code> request. We called the <code>fetch()</code> function, with the appropriate URL and we set the necessary options using the optional parameter of the <code>fetch()</code> function. We used <code>JSON.stringify()</code> to convert our object to a JSON-formatted string before sending it to the web server. Then, same as with retrieving data - we awaited the response, converted it to JSON, and printed it to the console.
Running the code gives us the output:
<pre><code class="hljs">{
 userId: 123,
 title: &#x27;loren impsum doloris&#x27;,
 completed: false,
 id: 201
}
</code></pre>
<h2 id="handlingexceptionsanderrors">Handling Exceptions and Errors</h2>
Our requests can sometimes fail, for a variety of reasons - an error occurring in the <code>fetch()</code> function, internet issues, internal server errors, and others. We need a way to handle these situations, or in the very least be able to see that they occurred.
We can handle runtime exceptions by adding <code>catch()</code> at the end of the promise-chain. Let's add a simple <code>catch()</code> function to our program above:
<pre><code class="hljs">let todo = {
 userId: 123,
 title: &quot;loren impsum doloris&quot;,
 completed: false
}

fetch(&#x27;https://jsonplaceholder.typicode.com/todos&#x27;, {
 method: &#x27;POST&#x27;,
 body: JSON.stringify(todo),
 headers: { &#x27;Content-Type&#x27;: &#x27;application/json&#x27; }
}).then(res =&gt; res.json())
 .then(json =&gt; console.log(json))
 .catch(err =&gt; console.log(err));
</code></pre>
Ideally, you shouldn't simply ignore and print errors, but instead have a system in place for handling them.
We should keep in mind that if our response has a 3xx/4xx/5xx status code, the request either failed or additional steps need to be taken by the client.
Namely, 3xx HTTP status codes indicate that additional steps need to be taken by the client, 4xx codes indicate an invalid request, and 5xx codes indicate server errors. All these status codes tell us that our request wasn't successful in practical terms.
<code>catch()</code> won't register any of these cases because communication with the server went well, i.e. we made a request and got a response successfully. This means that we need to take additional steps to make sure we cover the situation when the client-server communication was successful, but we didn't receive any of the successful (2xx) HTTP status codes.
A common way to make sure that failed requests throw an error is to create a function that checks the HTTP status of the response from the server. In that function, if the status code doesn't indicate success, we can throw an error and <code>catch()</code> will catch it.
We can use the previously mentioned <code>ok</code> field of <code>Response</code> objects, which equals <code>true</code> if the status code is 2xx.
Let's see how this works:
<pre><code class="hljs">function checkResponseStatus(res) {
 if(res.ok){
 return res
 } else {
 throw new Error(`The HTTP status of the reponse: ${res.status} (${res.statusText})`);
 }
}

fetch(&#x27;https://jsonplaceholder.typicode.com/MissingResource&#x27;)
 .then(checkResponseStatus);
 .then(res =&gt; res.json());
 .then(json =&gt; console.log(json));
 .catch(err =&gt; console.log(err));
</code></pre>
We used the function at the beginning of the promise-chain (before parsing the response body) in order to see whether we encountered an issue. You can also throw a custom error instead.
Again, you should have a strategy in place for handling errors like this instead of just printing to the console.
If everything went as expected, and the status code indicated success, the program will proceed as before.
<h2 id="conclusion">Conclusion</h2>
Making requests to web servers is a common web development task and in this article, we have seen how we can do it effectively using <code>node-fetch</code> - a library that makes the native browser fetch API compatible with NodeJS.
In addition to that, we have also taken a look at how to handle errors that might occur with HTTP requests.

Aniuchi Adaobi

In this tutorial, we'll be sending GET and POST requests in Node.js using NPM's node-fetch package. It's different from the client side window.fetch() function and offers similar functionality.

Making HTTP Requests in Node.js with node-fetch

Leandro Cofre

In this tutorial, we'll be making a demo application that explains how Mongoose works with Node.js. We'll perform object data modeling and implement CRUD functionality.

Mongoose with Node.js - Object Data Modeling

<h2 id="introduction">Introduction</h2>
One of the first things you encounter while learning the basics of programming is the concept of strings. Similar to various programming languages, Python strings are arrays of bytes representing Unicode characters - an array or sequence of characters. Python, unlike many programming languages, doesn’t have a distinct character datatype, and characters are considered strings of length 1.
You can define a string using single or double quotation marks, for example, <code>a = &quot;Hello World&quot;</code> or <code>a = 'Hello World'</code>. To access a specific element of a string, you would use square brackets (<code>[]</code>) with the index of the character you wish to access (indexing starts at 0). Calling <code>a[0]</code>, for example, would return <code>H</code>.
That being said, let's take a look at this code example:
<pre><code class="hljs">a = &#x27;Hello World&#x27;
b = &#x27;Hello World&#x27;
c = &#x27;Hello Worl&#x27;

print(a is b)
print(a == b)
print(a is c+&#x27;d&#x27;)
print(a == c+&#x27;d&#x27;)
</code></pre>
All of the strings we compare hold the value of <code>Hello World</code> (<code>a</code>, <code>b</code>, and <code>c +'d'</code>). It might be intuitive to assume that the output would be <code>True</code> for all these statements.
However, when we run the code, it results in:
<pre><code class="hljs">True
True
False
True
</code></pre>
What's might look unintuitive about this output is that <code>a is c + 'd'</code> returns <code>False</code>, while a very similar statement <code>a is b</code> returns <code>True</code>. With this, we can conclude that <code>a</code> and <code>b</code> are the same object, while <code>c</code> is a different one, even though they hold the same value.
If you are unfamiliar with the <a target="_blank" href="https://stackabuse.com/is-vs-python-object-comparison/">Difference Between <code>==</code> and <code>is</code></a> - <code>is</code> checks whether the variables are referring to the same object in memory, while <code>==</code> checks whether the variables have the same value.
This distinction between <code>a</code>, <code>b</code> and <code>c</code> is the product of String Interning.
Note: The environment in which you run the code affects how string interning works. The previous examples were results of running the code as a script in a non-interactive environment, using the current latest version of Python (version 3.8.5). The behavior will be different when using the console/Jupyter because of different ways in which the code is optimized, or even between different versions of Python.
This is because different environments have different optimization levels.
<h2 id="stringinterning">String Interning</h2>
Strings are immutable objects in Python. This means that once strings are created, we can not change or update them. Even if it seems like a string has been modified, under the hood, a copy with the modified value was created and assigned to the variable, while the original string remained the same.
Let's attempt to modify a string:
<pre><code class="hljs">name = &#x27;Wtack Abuse!&#x27;
name[0] = &#x27;S&#x27;
</code></pre>
As the string <code>name</code> is immutable, this code will fail on the last line:
<pre><code class="hljs">name[0] = &#x27;S&#x27;
TypeError: &#x27;str&#x27; object does not support item assignment
</code></pre>
Note: If you really did want to change a particular character of a string, you could convert the string into a mutable object like a <code>list</code>, and change the desired element:
<pre><code class="hljs">name = &#x27;Wtack Abuse!&#x27;
name = list(name)
name[0] = &#x27;S&#x27;
# Converting back to string
name = &quot;&quot;.join(name) 

print(name)
</code></pre>
Which gives us the desired output:
<pre><code class="hljs">Stack Abuse!
</code></pre>
The reason why we could change the character in the list (and not in the string) is because lists are mutable - meaning that we can change their elements.
<blockquote>
String Interning is a process of storing only one copy of each distinct string value in memory.
</blockquote>
This means that, when we create two strings with the same value - instead of allocating memory for both of them, only one string is actually committed to memory. The other one just points to that same memory location.
Given this information, let's go back to the initial <code>Hello World</code> example:
<pre><code class="hljs">a = &#x27;Hello World&#x27;
b = &#x27;Hello World&#x27;
c = &#x27;Hello Worl&#x27;
</code></pre>
When string <code>a</code> is created, the compiler checks if <code>Hello World</code> is present in interned memory. Since it is the first occurrence of this string value, Python creates an object and caches this string in memory and points <code>a</code> to this reference.
When <code>b</code> is created, <code>Hello World</code> is found by the compiler in the interned memory so instead of creating another string, <code>b</code> simply points to the previously allocated memory.
<img src="https://stackabuse.s3.amazonaws.com/media/guide-to-string-interning-in-python-1.png" alt="python string values in memory">
<code>a is b</code> and <code>a == b</code> in this case.
Finally, when we create the string <code>c = 'Hello Worl'</code>, the compiler instantiates another object in interned memory because it could not find the same object for reference.
When we compare <code>a</code> and <code>c+'d'</code>, the latter is evaluated to <code>Hello World</code>. However, since Python doesn't do interning during runtime, a new object is created instead. Thus, since no interning was done, these two aren't the same object and <code>is</code> returns <code>False</code>.
In contrast to the <code>is</code> operator, the <code>==</code> operator compares the values of the strings after computing runtime expressions - <code>Hello World == Hello World</code>.
At that time, <code>a</code> and <code>c+'d'</code> are the same, value-wise, so this returns <code>True</code>.
<h3 id="verification">Verification</h3>
Let's see the id of string objects we created. The <code>id(object)</code> function in Python returns the ID of <code>object</code>, which is guaranteed to be unique during the lifetime of the said object. If two variables point to the same object, calling <code>id</code> would return the same number:
<pre><code class="hljs">letter_d = &#x27;d&#x27;

a = &#x27;Hello World&#x27;
b = &#x27;Hello World&#x27;
c = &#x27;Hello Worl&#x27; + letter_d
d = &#x27;Hello Worl&#x27; + &#x27;d&#x27;

print(f&quot;The ID of a: {id(a)}&quot;)
print(f&quot;The ID of b: {id(b)}&quot;)
print(f&quot;The ID of c: {id(c)}&quot;)
print(f&quot;The ID of d: {id(d)}&quot;)
</code></pre>
This results in:
<pre><code class="hljs">The ID of a: 16785960
The ID of b: 16785960
The ID of c: 17152424
The ID of d: 16785960
</code></pre>
Only <code>c</code> has a different id. All references now point to the object with the same <code>Hello World</code> value. However, <code>c</code> was not computed at compile-time, but rather at runtime. Even <code>d</code>, which we generated by adding the <code>'d'</code> character now points to the same object that <code>a</code> and <code>b</code> point to.
<h2 id="howstringsareinterned">How Strings Are Interned</h2>
In Python, there are two ways strings can be interned based on programmer-interaction:
<ul>
<li>Implicit interning</li>
<li>Explicit interning</li>
</ul>
<h3 id="implicitinterning">Implicit Interning</h3>
Python automatically interns some strings at the moment of their creation. Whether or not a string is interned depends on several factors:
<ul>
<li>
All empty strings and strings of length 1 are interned.
</li>
<li>
Up until version 3.7, Python used <a rel="nofollow noopener" target="_blank" href="https://github.com/python/cpython/blob/0f21fe6155227d11dc02bd3ef3b061de4ecea445/Python/peephole.c">peephole optimization,</a> and all strings longer than 20 characters were not interned. However, now it uses the <a rel="nofollow noopener" target="_blank" href="https://github.com/python/cpython/blob/3.7/Python/ast_opt.c">AST optimizer</a>, and (most) strings up to 4096 characters are interned.
</li>
<li>
Names of functions, class, variables, arguments, etc. are implicitly interned.
</li>
<li>
The keys of dictionaries used to hold module, class, or instance attributes are interned.
</li>
<li>
Strings are interned only at compile-time, this means that they will not be interned if their value can't be computed at compile-time.
<ul>
<li>These strings will be interned for example:</li>
</ul>
<pre><code class="hljs">a = &#x27;why&#x27;
b = &#x27;why&#x27; * 5
</code></pre>
<ul>
<li>The following expression is computed at runtime thus the string is not interned.</li>
</ul>
<pre><code class="hljs">b = &quot;&quot;.join([&#x27;w&#x27;,&#x27;h&#x27;,&#x27;y&#x27;])
</code></pre>
</li>
<li>
Strings having characters apart from ASCII are most likely not going to be interned.
</li>
</ul>
If you recall, we said that <code>'Hello Worl' + letter_d</code> was computed at runtime, and therefore it will not be interned. Since there is no consistent standard on string interning, a good rule of thumb to use is the compile-time/runtime idea, where you can assume that a string will be interned if it can be computed at compile-time.
<h3 id="explicitinterning">Explicit Interning</h3>
We often come across strings that do not lie under the conditions of implicit interning in Python, but there is a way to intern any string you want. There is a function in the <code>sys</code> module called <code>intern(immutable_object)</code>, this function tells Python to store the <code>immutable_object</code> (string in our case) in the interned memory table.
You can intern any kind of string as follows:
<pre><code class="hljs">import sys
c = sys.intern(&#x27;Hello World&#x27;+&#x27;!&#x27;)
</code></pre>
We can see that this would work in our previous example:
<pre><code class="hljs">import sys

letter_d = &#x27;d&#x27;

a = sys.intern(&#x27;Hello World&#x27;)
b = sys.intern(&#x27;Hello Worl&#x27; + letter_d)

print(f&quot;The ID of a: {id(a)}&quot;)
print(f&quot;The ID of b: {id(b)}&quot;)
print(f&quot;a is b? {a is b}&quot;)
</code></pre>
Would give the output:
<pre><code class="hljs">The ID of a: 26878464
The ID of b: 26878464
a is b? True
</code></pre>
Now that we know how and which strings are interned in Python. One question remains - why was string interning introduced?
<h2 id="advantagesofstringinterning">Advantages of String Interning</h2>
String interning has several advantages:
<ul>
<li>Saving Memory: We never have to save two string objects in memory separately if they are the same. Every new variable with the same content just points to the reference in the interned table literal. If for some reason, you wanted to have a list containing every single word and its appearance in Jane Austen's Pride and Prejudice, without explicit interning, you'd need 4.006.559 bytes, and with explicit interning of each word, you would need only 785.509 bytes of memory.</li>
<li>Fast Comparisons: The comparison of interned strings is much faster than non-interned strings which are useful when your program has lots of comparisons. This happens because in order to compare interned strings, you only need to compare whether their memory addresses are the same, instead of comparing the contents.</li>
<li>Fast Dictionary Lookups: If the lookup keys are interned, the comparison can be done by pointer comparisons instead of string compare, which works on the same principle as the previous point.</li>
</ul>
<h2 id="disadvantagesofstringinterning">Disadvantages of String Interning</h2>
However interning strings does have some drawbacks and things to consider before using:
<ul>
<li>Memory Cost: In case your program has a large number of strings with different values and relatively fewer comparisons overall because the interned table itself consumes memory. Meaning that you want to intern strings if you have relatively few strings and a lot of comparisons between them.</li>
<li>Time Cost: The call to <code>intern()</code> function is expensive as it has to manage the interned table.</li>
<li>Multi-threaded Environments: The interned memory (table) is a global resource in a multi-threaded environment whose synchronization needs to be modified. This check might only be needed when the interned table is accessed i.e. when a new string is created but it can be expensive.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
By using string interning, you ensure that only one object is created even if you define multiple strings with the same content. You should however keep in mind the balance between the advantages and disadvantages of string interning, and only use it when you think your program could benefit.
Always remember to add comments or documentation if you are using string interning so other team members know how to deal with strings in the program.
While the results may vary as per the implementation of your Python interpreter, as well as the environment you run your code in, you should definitely play around with <code>intern()</code> function to make yourself comfortable with it. This concept can help you to the improve design and performance of your code. It might help you in your next job interview as well.

Muhammad Hashir Hassan

In this article, we'll dive into the concept of String Interning in Python. With examples and theory, we'll explain how it works and what benefits it brings to everyday tasks.

Guide to String Interning in Python

<h2 id="introduction">Introduction</h2>
Working with files is a common task with any programming language. File manipulation requires us to know their location. One of the most fundamental ways to interact with files is to list files in a directory.
In this article, we'll be using Node.js and the built-in fs module as well as the directory-tree module from NPM to list all files from a directory.
We'll read a directory, <code>files</code>, located at the same place our <code>app.js</code> file is in:
<pre><code class="hljs">09/10/2020 01:27 PM 332 app.js
09/10/2020 01:24 PM &lt;DIR&gt; files
</code></pre>
The directory contains:
<pre><code class="hljs">files
│ anotherDirectory
│ └── fileInDirectory.txt
└── textFile.txt
└── anotherTextFile.txt
</code></pre>
<h2 id="directorytree">directory-tree</h2>
<code>directory-tree</code> is a handy NPM module that takes care of this task for us and formats the output nicely. First, let's install it:
<pre><code class="hljs">$ npm install directory-tree
</code></pre>
Now, let's import it into our script and supply it with our directory's location:
<pre><code class="hljs">const dirTree = require(&quot;directory-tree&quot;);

const tree = dirTree(&#x27;./files/&#x27;);
console.log(tree);
</code></pre>
The <code>tree</code> constant now contains the information we'd like to access. This code results in:
<pre><code class="hljs">{
 path: &#x27;./files/&#x27;,
 name: &#x27;files&#x27;,
 children: [
 {
 path: &#x27;files\\anotherDirectory&#x27;,
 name: &#x27;anotherDirectory&#x27;,
 children: [Array],
 size: 8,
 type: &#x27;directory&#x27;
 },
 {
 path: &#x27;files\\anotherTextFile.txt&#x27;,
 name: &#x27;anotherTextFile.txt&#x27;,
 size: 2218,
 extension: &#x27;.txt&#x27;,
 type: &#x27;file&#x27;
 },
 {
 path: &#x27;files\\textFile.txt&#x27;,
 name: &#x27;textFile.txt&#x27;,
 size: 7,
 extension: &#x27;.txt&#x27;,
 type: &#x27;file&#x27;
 }
 ],
 size: 2233,
 type: &#x27;directory&#x27;
}
</code></pre>
We can also specify the extensions we'd like to filter based on, using the <code>dirTree()</code> function:
<pre><code class="hljs">const tree = dirTree(&#x27;./files/&#x27;, {extensions:/\.js$/});
console.log(tree);
</code></pre>
This would return an empty result since there are no JS files in the <code>files</code> directory.
<h2 id="fsreaddir">fs.readdir()</h2>
The easiest way to read files from a directory without external modules is with the help of the <code>readdir()</code> function. It's asynchronous and returns an array containing file names in the directory you've specified.
Let's go ahead and list the files from the <code>files</code> directory:
<pre><code class="hljs">const directory = &#x27;./files/&#x27;;
const fs = require(&#x27;fs&#x27;);

fs.readdir(directory, (err, files) =&gt; {
 files.forEach(file =&gt; {
 console.log(file);
 });
});
</code></pre>
Here, we've specified the <code>directory</code> constant, pointing to the <code>files</code> folder, after which, we've imported the <code>fs</code> module.
Then, we've supplied the <code>directory</code> to the <code>readdir()</code> function and logged their name via a callback. This results in:
<pre><code class="hljs">anotherDirectory
anotherTextFile.txt
textFile.txt
textFile.txt - Shortcut.lnk
</code></pre>
Note: The <code>readdir()</code> function also reads directories, but without an indication as to whether it's a directory or a file. A file without an extension looks the same as a directory in this case.
However, we can use the <code>fs.lstatSync()</code> function to help us out with this:
<pre><code class="hljs">const directory = &#x27;./files/&#x27;;
const path = require(&#x27;path&#x27;);
const fs = require(&#x27;fs&#x27;);

fs.readdir(directory, (err, files) =&gt; {
 files.forEach(file =&gt; {
 if (fs.lstatSync(path.resolve(directory, file)).isDirectory()) {
 console.log(&#x27;Directory: &#x27; + file);
 } else {
 console.log(&#x27;File: &#x27; + file);
 }
 });
});
</code></pre>
Using <code>isDirectory()</code>, we've checked if what we're encountering at the given path is a directory or a file. This now results in:
<pre><code class="hljs">Directory: anotherDirectory
File: anotherTextFile.txt
File: file
File: textFile.txt
</code></pre>
<h2 id="fsreaddirsync">fs.readdirSync()</h2>
The <code>readdirSync()</code> function is virtually the same as the <code>readdir()</code> function, but it reads synchronously, instead of asynchronously.
It works in the very same way as the previous approach - it just handles the operation synchronously:
<pre><code class="hljs">const directory = &#x27;./files/&#x27;;
const path = require(&#x27;path&#x27;);
const fs = require(&#x27;fs&#x27;);

fs.readdirSync(directory).forEach(file =&gt; {
 if (fs.lstatSync(path.resolve(directory, file)).isDirectory()) {
 console.log(&#x27;Directory: &#x27; + file);
 } else {
 console.log(&#x27;File: &#x27; + file);
 }
});
</code></pre>
Here, we've done the same thing we've done before. Checking for files and directories, changing the flow slightly based on the results. This code prints out:
<pre><code class="hljs">Directory: anotherDirectory
File: anotherTextFile.txt
File: file
File: textFile.txt
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this article, we've gone over a few ways to list files in a directory in Node.js We've started off with <code>directory-tree</code>, an NPM package built for this purpose, after which we've used the built-in <code>fs</code> module and its <code>readdir()</code> and <code>readdirSync()</code> functions to list the files.

David Landup

In this tutorial, we'll learn how to list files in a directory using Node.js. We'll use the NPM package directory-tree for this, as well as the built-in fs module.

Node: List Files in a Directory

<h2 id="introduction">Introduction</h2>
Redis is an in-memory data store, which can be used as a NoSQL database, cache, or as a typical message broker. It's written in ANSI C, which compiles into significantly efficient machine code and its ability to store data as key-value pairs makes in-memory caching an attractive use-case for Redis, besides also persisting data to a disk.
In this article, we'll use Pipelining to allow a Spring Boot application to send multiple requests to the Redis server, in a non-blocking way.
<h2 id="usecaseofpipelininginredis">Use Case of Pipelining in Redis</h2>
Redis is based on a Client/Server(Request/Response) architecture. In these architectures, a client typically sends a query or request to the server and waits for a response. This is usually done in a blocking way such that a new request can't be sent until the response for the last one was sent:
<pre><code class="hljs">Client: &lt;command 1&gt;
Server: Response for &lt;command 1&gt;
Client: &lt;command 2&gt;
Server: Response for &lt;command 2&gt;
Client: &lt;command 3&gt;
Server: Response for &lt;command 3&gt;
</code></pre>
This can lead to massive inefficiencies, with high latency between receiving commands and processing them.
Pipelining allows us to send multiple commands, as a single operation by the client, without waiting for the response from the server between each command. Then, all the responses are read together instead:
<pre><code class="hljs">Client: &lt;command 1&gt;
Client: &lt;command 2&gt;
Client: &lt;command 3&gt;
Server: Response for &lt;command 1&gt;
Server: Response for &lt;command 2&gt;
Server: Response for &lt;command 3&gt;
</code></pre>
Since the client is not waiting for the server's response before issuing another command, latency is reduced which in turn improves the performance of the application.
Note: The commands here are placed in a queue. This queue should remain of a reasonable size. If you're dealing with tens of thousands of commands, it's better to send and process them in batches, lest the benefits of pipelining become redundant.
<h2 id="implementation">Implementation</h2>
Let's go ahead and make a small Spring Boot application that works with Redis and pipelines multiple commands. This is made easier with the help of the <a target="_blank" rel="nofollow noopener" href="https://spring.io/projects/spring-data-redis">Spring Data Redis</a> project.
<h3 id="springbootsetup">Spring Boot Setup</h3>
The easiest way to start off with a blank Spring Boot app is to use <a rel="nofollow noopener" target="_blank" href="https://start.spring.io">Spring Initializr</a>:
<img src="https://stackabuse.s3.amazonaws.com/media/spring-boot-with-redis-hashoperations-crud-functionality-1.PNG" alt="spring boot initializr">
Alternatively, you can also use the <a rel="nofollow noopener" target="_blank" href="https://docs.spring.io/spring-boot/docs/current/reference/html/spring-boot-cli.html">Spring Boot CLI</a> to bootstrap the application:
<pre><code class="hljs">$ spring init --dependencies=spring-boot-starter-data-redis redis-spring-boot-demo
</code></pre>
We're starting off with the <code>spring-boot-starter-data-redis</code> dependency as it includes <code>spring-data-redis</code>, <code>spring-boot-starter</code> and <code>lettuce-core</code>.
If you already have a Maven/Spring application, add the dependency to your <code>pom.xml</code> file:
<pre><code class="hljs">&lt;dependency&gt;
 &lt;groupId&gt;org.springframework.boot&lt;/groupId&gt;
 &lt;artifactId&gt;spring-boot-starter-data-redis&lt;/artifactId&gt;
 &lt;version&gt;${version}&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>
Or if you're using Gradle:
<pre><code class="hljs">compile group: &#x27;org.springframework.data&#x27;, name: &#x27;spring-data-redis&#x27;, version: &#x27;${version}&#x27;
</code></pre>
We'll also be using Jedis as the connection client, instead of Lettuce:
<pre><code class="hljs">&lt;dependency&gt;
 &lt;groupId&gt;redis.clients&lt;/groupId&gt;
 &lt;artifactId&gt;jedis&lt;/artifactId&gt;
 &lt;version&gt;${version}&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>
<h3 id="redisconfiguration">Redis Configuration</h3>
We will host Redis on <a target="_blank" rel="nofollow noopener" href="https://scalegrid.io/">ScaleGrid</a>, which provides a free trial account, for hosting a Redis server instance. Alternatively, you can <a target="_blank" rel="nofollow noopener" href="https://redis.io/download">download the server</a> and host it on your own computer on Linux and MacOS. Windows requires a bit of hacking and is tricky to set up.
Let's set up the <code>JedisConnectionFactory</code> so that our application can connect to the Redis server instance. In your <code>@Configuration</code> class, annotate the adequate <code>@Bean</code>:
<pre><code class="hljs">@Configuration
public class Config {
 @Bean
 public JedisConnectionFactory redisConnectionFactory() {
 JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory();
 jedisConnectionFactory.setHostName(&quot;&lt;server-hostname-here&gt;&quot;);
 jedisConnectionFactory.setPort(6379);
 jedisConnectionFactory.setPassword(&quot;&lt;server-password-here&gt;&quot;);
 jedisConnectionFactory.afterPropertiesSet();
 return jedisConnectionFactory;
 }
}
</code></pre>
<code>RedisTemplate</code> is an entry-class provided by Spring Data through which we interact with the Redis server.
We'll pass our configured connection factory to it:
<pre><code class="hljs">@Bean
public RedisTemplate&lt;String, String&gt; redisTemplate() {
 RedisTemplate&lt;String, String&gt; redisTemplate = new RedisTemplate&lt;&gt;();
 redisTemplate.setConnectionFactory(redisConnectionFactory());
 redisTemplate.setDefaultSerializer(RedisSerializer.string());
 redisTemplate.afterPropertiesSet();
 return redisTemplate;
}
</code></pre>
We have set the default serializer to store keys and values as <code>String</code>. Alternatively, you can define your own custom serializer.
<h3 id="pipeliningusingredistemplate">Pipelining Using RedisTemplate</h3>
Let's add a few items from a list to the Redis server. We'll do this without pipelining, using the <code>RedisTemplate</code>. Specifically, we'll use the <code>ListOperations</code> interface, acquired from <code>opsForList()</code>:
<pre><code class="hljs">List&lt;String&gt; values = Arrays.asList(&quot;value-1&quot;, &quot;value-2&quot;, &quot;value-3&quot;, &quot;value-4&quot;, &quot;value-5&quot;);
redisTemplate.opsForList().leftPushAll(&quot;pipeline-list&quot;, values);
</code></pre>
Running this code will result in:
<img src="https://stackabuse.s3.amazonaws.com/media/spring-boot-with-redis-pipeline-operations-1.png" alt="redis operation results">
Now, let's remove these. Imagining that this can be an expensive operation, we'll pipeline each <code>rPop()</code> command so that they get sent together and that the results synchronize upon all of the elements being removed. Then, we'll receive these results back. To pipeline commands, we use the <code>executedPipeline()</code> method.
It accepts a <code>RedisCallback</code> or <code>SessionCallback</code> that we provide it with. The <code>executedPipeline()</code> method returns the results that we can then capture and review. If this isn't needed, and you'd just like to execute the commands, you can use the <code>execute()</code> method, and pass <code>true</code> as the <code>pipeline</code> argument instead:
<pre><code class="hljs">List&lt;Object&gt; results = redisTemplate.executePipelined(new RedisCallback&lt;Object&gt;() {
 public Object doInRedis(RedisConnection connection) throws DataAccessException {
 for(int i = 0; i &lt; 5; i++) {
 connection.rPop(&quot;pipeline-list&quot;.getBytes());
 }
 return null;
 }
});
return results;
</code></pre>
The <code>executePipelined()</code> method accepts a <code>new RedisCallback()</code>, in which we use the <code>doInRedis()</code> method to specify what we'd like to do.
Specifically, we've run the <code>rPop()</code> method 5 times, removing the 5 list elements we've inserted beforehand.
Once all five of these commands have been executed, the elements removed from the list and sent back - the results are packed in the <code>results</code> list:
<img src="https://stackabuse.s3.amazonaws.com/media/spring-boot-with-redis-pipeline-operations-2.png" alt="redis pipeline results">
<h2 id="conclusion">Conclusion</h2>
Redis' most popular use-case is as a cache store. However, it can also be used as a database, or as a message broker.
Redis allows us to increase the performance of applications, by minimizing the calls to the database layer. Its support for pipelining allows for multiple commands to be sent to the server in a single write operation, thereby reducing round trip Time to and from the server.
In this article, we've pipelined multiple commands using the <code>RedisTemplate</code> API and verified the results.

Guest Contributor

In this tutorial, we'll perform Redis operation pipelining for Spring Boot Java applications to improve the efficiency and performance.

Spring Boot with Redis: Pipeline Operations

<h2 id="introduction">Introduction</h2>
In this article, we'll convert a JSON array into a Java Array and Java List using Jackson.
Since we're using Jackson, you'll have to add it to your project. If you're using Maven, it's as easy as adding the dependency:
<pre><code class="hljs">&lt;dependency&gt;
 &lt;groupId&gt;com.fasterxml.jackson.core&lt;/groupId&gt;
 &lt;artifactId&gt;jackson-databind&lt;/artifactId&gt;
 &lt;version&gt;2.11.2&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>
Or, if you're using Gradle:
<pre><code class="hljs">compile group: &#x27;com.fasterxml.jackson.core&#x27;, name: &#x27;jackson-databind&#x27;, version: &#x27;2.11.2&#x27;
</code></pre>
Since we're mapping from JSON to our own objects, let's go ahead and define a POJO:
<pre><code class="hljs">public class Language {

 private String name;
 private String description;

 // Getters, setters and toString() method
</code></pre>
<h2 id="readingjsonfromastring">Reading JSON from a String</h2>
Let's start out with reading JSON from a String. The String contains an array of programming languages, with brief descriptions:
<pre><code class="hljs">String json = &quot;[{\&quot;name\&quot;: \&quot;Java\&quot;, \&quot;description\&quot;: \&quot;Java is a class-based, object-oriented programming language.\&quot;},{\&quot;name\&quot;: \&quot;Python\&quot;, \&quot;description\&quot;: \&quot;Python is an interpreted, high-level and general-purpose programming language.\&quot;}, {\&quot;name\&quot;: \&quot;JS\&quot;, \&quot;description\&quot;: \&quot;JS is a programming language that conforms to the ECMAScript specification.\&quot;}]&quot;;
</code></pre>
Using Jackson's <code>ObjectMapper</code> class, it's easy to read values and map them to an object, or an array of objects. We just use the <code>readValue()</code> method, passing the JSON contents and the class we'd like to map to. Since we're mapping to an array of <code>Language</code>, we'll also specify this in the <code>readValue()</code> method:
<pre><code class="hljs">// It&#x27;s advised to use ObjectMapper as a singleton and reuse the instance
final ObjectMapper objectMapper = new ObjectMapper();
Language[] langs = objectMapper.readValue(json, Language[].class);
</code></pre>
Alternatively, you can extract the values directly into a list, using Jackson's <code>TypeReference</code>:
<pre><code class="hljs">List&lt;Language&gt; langList = objectMapper.readValue(json, new TypeReference&lt;List&lt;Language&gt;&gt;(){});
</code></pre>
Without using <code>TypeReference&lt;&gt;</code>, which is advised, you can <a target="_blank" href="/how-to-convert-a-java-array-to-arraylist/">convert the array into a list</a> with any other approach at your disposal, such as:
<pre><code class="hljs">List&lt;Language&gt; langList = new ArrayList(Arrays.asList(langs));
</code></pre>
And then print out the values:
<pre><code class="hljs">langList.forEach(x -&gt; System.out.println(x.toString()));
</code></pre>
This results in:
<pre><code class="hljs">Language{name=&#x27;Java&#x27;, description=&#x27;Java is a class-based, object-oriented programming language.&#x27;}
Language{name=&#x27;Python&#x27;, description=&#x27;Python is an interpreted, high-level and general-purpose programming language.&#x27;}
Language{name=&#x27;JS&#x27;, description=&#x27;JS is a programming language that conforms to the ECMAScript specification.&#x27;}
</code></pre>
<h2 id="readingjsonfromafile">Reading JSON from a File</h2>
We don't always deal with JSON in String format. Oftentimes, the contents come from a <code>File</code>. Thankfully, Jackson makes this task as easy as the last one, we just provide the <code>File</code> to the <code>readValue()</code> method:
<pre><code class="hljs">final ObjectMapper objectMapper = new ObjectMapper();
List&lt;Language&gt; langList = objectMapper.readValue(
 new File(&quot;langs.json&quot;), 
 new TypeReference&lt;List&lt;Language&gt;&gt;(){});

langList.forEach(x -&gt; System.out.println(x.toString()));
</code></pre>
The <code>langs.json</code> file contains:
<pre><code class="hljs">[
 {
 &quot;name&quot;: &quot;Java&quot;,
 &quot;description&quot;: &quot;Java is a class-based, object-oriented programming language that is designed to have as few implementation dependencies as possible.&quot;
 },
 {
 &quot;name&quot;: &quot;Python&quot;,
 &quot;description&quot;: &quot;Python is an interpreted, high-level and general-purpose programming language. Created by Guido van Rossum and first released in 1991.&quot;
 },
 {
 &quot;name&quot;: &quot;JS&quot;,
 &quot;description&quot;: &quot;JS is a programming language that conforms to the ECMAScript specification. JavaScript is high-level, often just-in-time compiled, and multi-paradigm.&quot;
 }
]
</code></pre>
Running this code results in:
<pre><code class="hljs">Language{name=&#x27;Java&#x27;, description=&#x27;Java is a class-based, object-oriented programming language that is designed to have as few implementation dependencies as possible.&#x27;}
Language{name=&#x27;Python&#x27;, description=&#x27;Python is an interpreted, high-level and general-purpose programming language. Created by Guido van Rossum and first released in 1991.&#x27;}
Language{name=&#x27;JS&#x27;, description=&#x27;JS is a programming language that conforms to the ECMAScript specification. JavaScript is high-level, often just-in-time compiled, and multi-paradigm.&#x27;}
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this article, we've used Jackson to parse and map the values from a JSON String and file into a Java array and list.
This is done via the <code>readValue()</code> method, by specifying the JSON contents (String or file) and by specifying the POJO we'd like to map to.

In this tutorial, we'll use Jackson with Java to convert a JSON array into a Java array and a Java List with examples. This can be done from Strings or Files.

Convert JSON Array to a Java Array or List with Jackson

<h2 id="introduction">Introduction</h2>
A common development task is reading data from files. A common file format is the <code>.csv</code> format.
While you can read CSV files using the <code>fs</code> module that comes with Node and get the content of the file, in most cases, parsing and further conversion is much easier with the help of modules made exactly for that purpose.
Multiple modules provide such capabilities like the <code>neat-csv</code> or <code>csv-parser</code> packages. However, in this article, we'll be using <a href="https://csv.js.org/" rel="nofollow noopener" target="_blank"><code>node-csv</code></a> - a suite of CSV packages for generating, parsing, transforming and stringifying CSV data.
<h2 id="installingnodecsv">Installing node-csv</h2>
The module consists of <code>csv-generate</code>, <code>csv-parse</code>, <code>csv-transform</code> and <code>csv-stringify</code> packages.
You can either install the whole suite or each package one by one if you don't need them all. Let's initialize a Node project with default settings:
<pre><code class="hljs">$ npm init -y
</code></pre>
Then, let's install the entire <code>node-csv</code> suite:
<pre><code class="hljs">$ npm install csv
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Although the suite is called <code>node-csv</code> - the relevant NPM package is actually called <code>csv</code>.

 </div>
 </div>
 </div>
 We'll be working with a CSV file with these contents:
<pre><code class="hljs">Account Name,Account Code,Type,Description
Cash,101,Assets,Checking account balance
Wages Payable,220,Liabilities,Amount owed to employee for hours not yet paid
Rent expense,560,Expenses,Cost of occupied rented facilities during accounting period
</code></pre>
<h2 id="readingcsvfileswithcsvparse">Reading CSV Files with csv-parse</h2>
To read CSV files, we’ll be using the <a href="https://csv.js.org/parse/" rel="nofollow noopener" target="_blank"><code>csv-parse</code></a> package from <code>node-csv</code>.
The <code>csv-parse</code> package provides multiple approaches for parsing CSV files - using callbacks, a stream + callback as well as the Sync and Async API. We'll be covering the stream + callback API and the Sync API.
<h3 id="streamcallbackapi">Stream + Callback API</h3>
Let's create a file, called <code>index.js</code> and construct a <code>parser</code>:
<pre><code class="hljs">var fs = require(&#x27;fs&#x27;); 
var parse = require(&#x27;csv-parse&#x27;);
var parser = parse({columns: true}, function (err, records) {
 console.log(records);
});

fs.createReadStream(__dirname+&#x27;/chart-of-accounts.csv&#x27;).pipe(parser);
</code></pre>
First, we import the native file system module (<code>fs</code>) and the <code>csv-parse</code> module. Then, we create a <code>parser</code> which accepts an object literal, containing the options we'd like to set. The second argument is the callback function that's used to access the records - or just print them out, in our case.
The options we can set aren't mandatory. In most cases, you'll be using any of the <code>delimiter</code>, <code>cast</code> or <code>columns</code> options:
<ul>
<li>
The delimiter option defaults to a comma <code>,</code>. If the data from the file you’re trying to parse uses some other delimiter like a semi-colon <code>;</code>, or a pipe <code>|</code>, you can specify that with this option.
</li>
<li>
The cast option defaults to <code>false</code> and is used to indicate whether you want to cast the strings to their native data types. For example, a column that is made up of date fields can be cast into a <code>Date</code>.
</li>
<li>
The columns option is to indicate whether you want to generate the record in the form of object literals. By default, this column is set to <code>false</code> and records are generated by the parser in the form of arrays. If set to <code>true</code>, the parser will infer the column name from the first line.
</li>
</ul>
Finally, we've opened up a read stream using the <code>fs</code> module and started piping it into the parser.
Let's run this file:
<pre><code class="hljs">$ node index.js
</code></pre>
This results in:
<pre><code class="hljs">[
 {
 &#x27;Account Name&#x27;: &#x27;Cash&#x27;,
 &#x27;Account Code&#x27;: &#x27;101&#x27;,
 Type: &#x27;Assets&#x27;,
 Description: &#x27;Checking account balance&#x27;
 },
 {
 &#x27;Account Name&#x27;: &#x27;Wages Payable&#x27;,
 &#x27;Account Code&#x27;: &#x27;220&#x27;,
 Type: &#x27;Liabilities&#x27;,
 Description: &#x27;Amount owed to employee for hours not yet paid&#x27;
 },
 {
 &#x27;Account Name&#x27;: &#x27;Rent expense&#x27;,
 &#x27;Account Code&#x27;: &#x27;560&#x27;,
 Type: &#x27;Expenses&#x27;,
 Description: &#x27;Cost of occupied rented facilities during accounting period&#x27;
 }
]
</code></pre>
Instead of just printing the contents out, you can manipulate this data, construct objects with the information from these fields or save them into a database, for example.
<h3 id="usingsyncapi">Using Sync API</h3>
Let's replicate this functionality using the Sync API:
<pre><code class="hljs">var fs = require(&#x27;fs&#x27;).promises;
var parse = require(&#x27;csv-parse/lib/sync&#x27;);
(async function () {
 const fileContent = await fs.readFile(__dirname+&#x27;/chart-of-accounts.csv&#x27;);
 const records = parse(fileContent, {columns: true});
 console.log(records)
})();
</code></pre>
Again, we're importing the <code>fs</code> module and the Sync API from the <code>csv-parse</code> module.
Then, we're creating an <code>async</code> function, in which we retrieve the contents of the file by <code>await</code>ing the response of the <code>readFile()</code> function.
Then, we can create a <code>parser</code> which takes in the file contents as the first argument and an object literal as the second. This object literal contains options for creating the parser (we've set <code>columns</code> to <code>true</code>). This parser is assigned to a constant variable and we simply print its contents out for brevity's sake:
<pre><code class="hljs">[
 {
 &#x27;Account Name&#x27;: &#x27;Cash&#x27;,
 &#x27;Account Code&#x27;: &#x27;101&#x27;,
 Type: &#x27;Assets&#x27;,
 Description: &#x27;Checking account balance&#x27;
 },
 {
 &#x27;Account Name&#x27;: &#x27;Wages Payable&#x27;,
 &#x27;Account Code&#x27;: &#x27;220&#x27;,
 Type: &#x27;Liabilities&#x27;,
 Description: &#x27;Amount owed to employee for hours not yet paid&#x27;
 },
 {
 &#x27;Account Name&#x27;: &#x27;Rent expense&#x27;,
 &#x27;Account Code&#x27;: &#x27;560&#x27;,
 Type: &#x27;Expenses&#x27;,
 Description: &#x27;Cost of occupied rented facilities during accounting period&#x27;
 }
]
</code></pre>
<h2 id="writingcsvfilesusingcsvstringify">Writing CSV Files using CSV Stringify</h2>
Similar to reading, we'd sometimes like to write data down into a CSV format. For this, we'll use the <a href="https://csv.js.org/stringify/" rel="nofollow noopener" target="_blank"><code>csv-stringify</code></a> package from the <code>node-csv</code> suite. Stringification just means that we'll convert some data (JSON in our example) into a string. This string is then written to a file, in CSV format.
Let's assume you've got some JSON contents that you'd like to write down as a CSV file:
<pre><code class="hljs">var someData = [
 {
 &quot;Country&quot;: &quot;Nigeria&quot;,
 &quot;Population&quot;: &quot;200m&quot;,
 &quot;Continent&quot;: &quot;Africa&quot;,
 &quot;Official Language(s)&quot;: &quot;English&quot;
 },
 {
 &quot;Country&quot;: &quot;India&quot;,
 &quot;Population&quot;: &quot;1b&quot;,
 &quot;Continent&quot;: &quot;Asia&quot;,
 &quot;Official Language(s)&quot;: &quot;Hindi, English&quot;
 },
 {
 &quot;Country&quot;: &quot;United States of America&quot;,
 &quot;Population&quot;: &quot;328m&quot;,
 &quot;Continent&quot;: &quot;North America&quot;,
 &quot;Official Language&quot;: &quot;English&quot;
 },
 {
 &quot;Country&quot;: &quot;United Kingdom&quot;,
 &quot;Population&quot;: &quot;66m&quot;,
 &quot;Continent&quot;: &quot;Europe&quot;,
 &quot;Official Language&quot;: &quot;English&quot;
 },
 {
 &quot;Country&quot;: &quot;Brazil&quot;,
 &quot;Population&quot;: &quot;209m&quot;,
 &quot;Continent&quot;: &quot;South America&quot;,
 &quot;Official Language&quot;: &quot;Portuguese&quot;
 }
]
</code></pre>
The <code>csv-stringify</code> package also has a couple of API options, though, the Callback API offers a really simple way to stringify data, without the need to handle events like with the Stream API.
Let's go ahead and stringify the data above, before writing it to a file:
<pre><code class="hljs">var fs = require(&#x27;fs&#x27;);
var stringify = require(&#x27;csv-stringify&#x27;);
 
stringify(someData, {
 header: true
}, function (err, output) {
 fs.writeFile(__dirname+&#x27;/someData.csv&#x27;, output);
})
</code></pre>
Here, we're importing the <code>fs</code> and <code>csv-stringify</code> modules. Then, using the <code>stringify()</code> function, we supply the data we'd like to convert to a string. We've also supplied an object literal containing the <code>header</code> option. Finally, there's also a callback function that's used to write the contents down into a file.
Other options like <code>cast</code>, <code>columns</code> and <code>delimiter</code> are also available. In our case, we’re setting the <code>header</code> option to <code>true</code> to tell the stringifier to generate the column names in the first record.
Running this code generates a file with the proper contents:
<img src="https://stackabuse.s3.amazonaws.com/media/reading-and-writing-csv-files-in-nodejs-with-node-csv-1.png" alt="writing csv to a file in nodejs">
<h2 id="conclusion">Conclusion</h2>
The <code>node-csv</code> module is a suite of smaller modules used to read/parse, transform and write CSV data from and to files.
We've used the <code>csv-parse</code> module to read CSV files and the <code>csv-stringify</code> module to stringify data before writing it to a file using Node.js.

In this tutorial, we'll be using the node-csv suite, or rather, the csv-parse and csv-stringify modules to read and write CSV file in Node.js with examples.