Scott Robinson

YAML has recently become quite popular in the development industry. In this article, we'll see how you can use YAML files with a Node.js/JavaScript application by reading and writing to them.

Reading and Writing YAML to a File in Node.js/JavaScript

<h2 id="introduction">Introduction</h2>
Streams are a somewhat advanced concept to understand. So in this article, we will go along with some examples for a better understanding and introduce you to a few concepts along the way.
<h2 id="whatisastream">What is a Stream</h2>
In simple terms, streams are used to read from input or write to output sequentially. Most of the time, streams are used to read or write from a continuous source or a comparably big one.
For example, let's assume that you have to read a large file. If the file size is larger than your free memory space, you cannot read the whole file into the memory in order to process it. You have to read it piece by piece and process each chunk, which may be separated by a line for example.
Another example of a continuous source is network communication - like a chat application where data should be continuously flowing from the sender to the receiver.
<h2 id="streamsinnodejs">Streams in Node.js</h2>
The <code>Stream</code> module is a native module that is shipped by default in Node.js. The <code>Stream</code> is an instance of the <a rel="nofollow noopener" target="_blank" href="https://nodejs.org/docs/latest/api/events.html">EventEmitter</a> class, which handles events asynchronously in Node.js. Due to their superclass, streams are inherently event-based.
There are 4 types of streams in Node.js:
<ul>
<li>Writable: Used to write data sequentially</li>
<li>Readable: Used to read data sequentially</li>
<li>Duplex: Used to both read and write data sequentially</li>
<li>Transform: Where data can be modified when writing or reading. Take compression for an example, with a stream like this you can write compressed data and read decompressed data.</li>
</ul>
Let's take a look at a few examples of streams.
<h2 id="writablestreams">Writable Streams</h2>
First of all, let's create a writable stream and write some data into a file:
<pre><code class="hljs">const fs = require(&#x27;fs&#x27;);
const file = fs.createWriteStream(&#x27;file.txt&#x27;);

file.write(&#x27;hello world&#x27;);
file.end(&#x27;, from streams!&#x27;);
</code></pre>
In this code, we have used the file system module to create a writable stream to a file (<code>file.txt</code>) and write 2 separate chunks to it: <code>hello world</code> and <code>, from streams</code>.
Unlike the <code>fs.writeFile()</code> where we need to write the file content at once, using a stream we can write the content chunk by chunk.
To simulate a continuous input, we could do something along the lines of:
<pre><code class="hljs">const fs = require(&#x27;fs&#x27;);
const file = fs.createWriteStream(&#x27;file.txt&#x27;);

for (let i = 0; i &lt; 10000; i++) {
 file.write(&#x27;Hello world &#x27; + i);
}
file.end();
</code></pre>
This will write <code>Hello world + {i}</code> ten thousand times and then end the stream:
<pre><code class="hljs">Hello world 0
Hello world 1
Hello world 2
Hello world 3
Hello world 4
...
</code></pre>
Please remember to <code>.end()</code> your streams after you're done using them, since the <code>finish</code> event is sent after the <code>.end()</code> method has been called.
This signifies that the stream's body has been flushed into our file.
<h2 id="readablestreams">Readable Streams</h2>
Now let's take a look at another simple example by reading a file using a stream. We can read a file chunk by chunk, instead of reading the full content in to memory, using a readable stream:
<pre><code class="hljs">const fs = require(&#x27;fs&#x27;);

const readableStream = fs.createReadStream(&#x27;./article.md&#x27;, {
 highWaterMark: 10
});

readableStream.on(&#x27;readable&#x27;, () =&gt; {
 process.stdout.write(`[${readableStream.read()}]`);
});

readableStream.on(&#x27;end&#x27;, () =&gt; {
 console.log(&#x27;DONE&#x27;);
});
</code></pre>
Similar to creating a writable stream, we've created a readable stream by calling the <code>.createReadStream()</code> method.
While buffering (segmenting the data into chunks), the size of the buffer depends on the <code>highWaterMark</code> parameter, which is passed to the stream constructor.
The default value of this parameter is 16384 bytes (16kb) so if you don't override the parameter, the stream will read 16kb chunks and pass them to you to process.
Since we are using a small text file it makes more sense to use a small value for our example, so the text will be chucked by 10 characters.
In our example above, we simply printed the chunk of data we received, except with brackets around it so that you can easily see the different chunks. The output of our code looks like this:
<pre><code class="hljs">[### Introd][uction

St][reams are ][a somewhat][ advanced ][concept to][ understan][d. So in t][his articl][e, we will][ go along ][with some ][examples f][or a bette][r understa][nding and ][introduce ][you to a f][ew concept][s along th][e way.

##][# What is ][a Stream

][In simple ]...
</code></pre>
<h2 id="duplexstreams">Duplex Streams</h2>
With both writable and readable streams out of the way, we can jump into an example using duplex streams - which essentially combine both.
We'll be demonstrating them using a simple HTTP server built using Node.js' native <code>http</code> module. The example used here is from the official <a rel="nofollow noopener" target="_blank" href="https://nodejs.org/api/stream.html">Node.js documentation</a>.
Since servers receive requests and then send responses, they're a good example for duplex streams, which handle both - a readable stream will act as a continuous request and a writable stream will act as a response.
First, let's import the HTTP module:
<pre><code class="hljs">const http = require(&#x27;http&#x27;);
</code></pre>
Now let's create a simple HTTP server:
<pre><code class="hljs">const server = http.createServer((req, res) =&gt; {
 // `req` is an http.IncomingMessage, which is a Readable Stream.
 // `res` is an http.ServerResponse, which is a Writable Stream.

 let body = &#x27;&#x27;;

 // Get the data as utf8 strings.
 // If an encoding is not set, Buffer objects will be received.
 req.setEncoding(&#x27;utf8&#x27;);

 // Readable streams emit &#x27;data&#x27; events once a listener is added.
 req.on(&#x27;data&#x27;, (chunk) =&gt; {
 body += chunk;
 });

 // The &#x27;end&#x27; event indicates that the entire body has been received.
 req.on(&#x27;end&#x27;, () =&gt; {
 consol.log(body);

 try {
 // Send &#x27;Hello World&#x27; to the user
 res.write(&#x27;Hello World&#x27;);
 res.end();
 } catch (er) {
 res.statusCode = 400;
 return res.end(`error: ${er.message}`);
 }
 });
});
</code></pre>
The <code>req</code> parameter is a readable stream, which we will process upon receiving as an HTTP request. We'll then send <code>res</code> as a response, which is, again, a simple writable stream.
Then, using the <code>.on()</code> method, we read the body of the request in chunks of 64KB and store it into the <code>body</code>, triggered by the <code>data</code> event.
Please note the use of the <code>setEncoding()</code> method before reading from the stream.
This way, the stream will emit strings and it would emit <code>Buffer</code> objects otherwise. Though, you can also perform that conversation inside the <code>data</code> event callback if you prefer.
The <code>end</code> event is triggered when there is nothing left to read in a readable stream. We will talk about other useful events later in this article.
Now, let's listen to the server:
<pre><code class="hljs">server.listen(1337);
</code></pre>
Hitting <code>http://localhost:1337</code>, you should see a simple <code>Hello World</code> response from the HTTP server.
<h2 id="streampipelines">Stream Pipelines</h2>
Using stream pipes we can directly pipe readable streams to a writable stream without storing the buffer temporarily - so we can save memory space.
Consider a scenario where a user requests a large file from the server and there is no memory space to load it to the memory, or the same file is requested by a thousand different clients. In this case, we cannot read the content of the file to memory and then write it back to the client.
This is where the <code>pipe</code> method is useful, as we'll pipe a readable stream (a request) into a writable stream (a response) and serve it to the user without holding it up in the buffer.
First, let's do this without using streams:
<pre><code class="hljs">const fs = require(&#x27;fs&#x27;);
const server = require(&#x27;http&#x27;).createServer();

server.on(&#x27;request&#x27;, (req, res) =&gt; {
 fs.readFile(&#x27;./video.mkv&#x27;, (err, data) =&gt; {
 if (err) throw err;

 res.end(data);
 });
});

server.listen(1337);
</code></pre>
This method is directly reading the file into the memory using the <code>.readFile()</code> method and sends it to the user.
Open your web browser and go to <code>http://localhost:1337</code>, here's what happening behind the scenes:
<img src="https://stackabuse.s3.amazonaws.com/media/introduction-to-nodejs-streams-1.png" alt="web browser memory usage">
Now, let's serve the video using a stream:
<pre><code class="hljs">const http = require(&#x27;http&#x27;);
const fs = require(&#x27;fs&#x27;);

const server = http.createServer((req, res) =&gt; {
 const src = fs.createReadStream(&#x27;./video.mkv&#x27;);
 src.pipe(res);
});

server.listen(1337);
</code></pre>
In this code, we have created a readable stream to the file and directly pipe it to the HTTP response, so rather than loading it to the memory the input from the HDD disk is directly written into the network without consuming the memory.
Here is the screen-shot of memory usage while sending the file using a stream:
<img src="https://stackabuse.s3.amazonaws.com/media/introduction-to-nodejs-streams-2.png" alt="web browser memory usage with a stream">
As you can see the memory usage is way too low compared to the first method.
<h2 id="usefuleventsinastream">Useful Events in a Stream</h2>
Since the <code>Stream</code> class inherits the <code>EventEmitter</code> class, each stream will have its own type of events that you can subscribe to using the <code>EventEmitter</code>'s <code>on()</code> method. This event will depend on the stream type.
<h3 id="eventsinreadablestreams">Events in Readable Streams</h3>
<ul>
<li><code>data</code>: Emitted when a chunk of data is read from the stream. By default, the chunk will be a <code>Buffer</code> object. If you want to change it you can use the <code>.setEncoding()</code> method.</li>
<li><code>error</code>: Emitted when an error occurs during reading. This may happen if the writable stream is unable to generate data due to some internal failure or when an invalid chunk is pushed to the stream.</li>
<li><code>end</code>: Emitted when there is no more data in the stream.</li>
<li><code>close</code>: Emitted when the stream resource is closed and indicates that no more events will be emitted in the future.</li>
<li><code>readable</code>: Emitted when the data is available in the readable stream to read.</li>
</ul>
<h3 id="eventsinwritablestreams">Events in Writable Streams</h3>
<ul>
<li><code>close</code>: Emitted when the stream resource is closed and indicates that no more events will be emitted in the future.</li>
<li><code>error</code>: Emitted when an error occurs during reading. This may happen if the writable stream is unable to generate data due to some internal failure or when invalid chunk data is pushed to the stream.</li>
<li><code>finish</code>: Emitted when all the data has been flushed from the writable stream.</li>
<li><code>pipe</code>: Emitted when the writable stream is piped to a readable stream.</li>
<li><code>unpipe</code>: Emitted when the writable stream is un-piped from a readable stream.</li>
</ul>
<h2 id="conclusion">Conclusion</h2>
In simple terms, streams are used to read from input or write into output sequentially. Most of the time, streams are used to read or write from a continuous source or a comparably big one.
The Stream module is a native module that is shipped by default in Node.js. The <code>Stream</code> is an instance of the <code>EventEmitter</code> class, which handles events asynchronously in Node.js. Due to their superclass, streams are inherently event-based.
Transformation Streams weren't covered in this article, as they warrant their own article.
The source code of this project is available on <a rel="nofollow noopener" target="_blank" href="https://github.com/jkasun/stack-abuse-nodejs-streams">GitHub</a> as usual. Use this to compare your code if you got stuck along the tutorial.
If you want more information about streams and or advanced knowledge, it's recommended to follow the <a rel="nofollow noopener" target="_blank" href="https://nodejs.org/api/stream.html">official documentation</a> for Streams.

Janith Kasun

Streams represent continuous, sequential data flow, received by a potentially unbounded source. In this article, we'll use writable, readable and duplex streams in Node.js, as well as build a Stream Pipeline.

Introduction to Node.js Streams

Dário Alves

Coroutines are a special type of functions that deliberately yield control over to the caller - but they don't end their context in the process, maintaining it in an idle state.

In this article, we'll dive into Python coroutines.

Coroutines in Python

Robley Gori

In this article, we'll use Redis and Celery with Flask to create an asynchronous messaging system in order to create a simple reminder application.

Asynchronous Tasks Using Flask, Redis and Celery

Asynchronous Tasks Using Flask, Redis, and Celery

Usman Malik

Time-series data changes with time. In this article, we'll be using PyTorch to analyze time-series data and predict future values using deep learning.

Time Series Prediction using LSTM with PyTorch in Python

<h2 id="introduction">Introduction</h2>
Sorting is a crucial aspect of digesting data. For us humans, it's much more natural to sort things that have something in common like the date of publishing, alphabetical order, articles belonging to an author, from smallest to largest, etc...
This makes it a lot easier to comprehend the data as it's logically connected rather than dispersed all around.
Human sorting is intuitive and simple, and thus often inefficient. We're usually not working with more than two elements we wish to sort. Computers are able to store huge amounts of data and element locations in their memory, which allow them to sort collections in a way humans couldn't, not to mention the speed of accessing/moving elements.
<h2 id="bubblesort">Bubble Sort</h2>
Bubble Sort is, in most cases, the first sorting algorithm any computer science enthusiast will encounter. It's the simplest and most intuitive sorting algorithms, which is one of the main reasons why it's taught to novice programmers/students.
It works by swapping adjacent elements, according to an order criteria. For example, if we want to sort a collection's elements from the smallest to largest - if the first element is larger than the second, they're swapped. This simple exchange is repeated with adjacent indexes until the collection is eventually sorted.
The exit condition for the algorithm is when we iterate through the whole collection without swapping a single element - which means it's fully sorted.
Here's a visual representation of how bubble sort works:
<img src="https://stackabuse.s3.amazonaws.com/media/bubble-sort-in-java-1.gif" alt="bubble sort visualization animation gif">
As you can see, the name itself comes from the visual illusion of elements &quot;bubbling up&quot; to their desired place. If you follow a certain element, say, <code>8</code> - you can notice it &quot;bubbling up&quot; to it's correct place in this example.
<h2 id="implementation">Implementation</h2>
With a brief overview of the theory behind Bubble Sort out of the way, let's implement it by sorting two different types of collections. First, we'll sort a simple array, and afterwards we'll sort an <code>ArrayList</code> with a custom object and a <code>compareTo()</code> method.
<h3 id="sortingarrays">Sorting Arrays</h3>
Let's start out by sorting a simple array of integers:
<pre><code class="hljs">public void bubbleSort(int[] array) {
 boolean sorted = false;
 int temp;
 while (!sorted) {
 sorted = true;
 for (int i = 0; i &lt; array.length - 1; i++) {
 if (a[i] &gt; a[i+1]) {
 temp = a[i];
 a[i] = a[i+1];
 a[i+1] = temp;
 sorted = false;
 }
 }
 }
}
</code></pre>
The <code>sorted</code> flag is used to signal if the array is sorted or not. If there's no reason to swap any element, or rather <code>a[i]</code> is always less than <code>a[i+1]</code> in a given iteration, the <code>sorted</code> flag is never reset to <code>false</code>.
Since it stays <code>true</code>, the array is sorted and we break out of the loop.
Running this piece of code:
<pre><code class="hljs">int[] array = new int[]{5, 6, 7, 2, 4, 1, 7};
bubbleSort(array);
System.out.println(Arrays.toString(array));
</code></pre>
Will yield:
<pre><code class="hljs">[1, 2, 4, 5, 6, 7, 7]
</code></pre>
Note: Since arrays are treated as objects in Java, having a <code>void</code> return type is absolutely valid when sorting arrays, and the contents are not copied at face-value when using it as an argument. In this case the array is sorted &quot;in place&quot;.
<h3 id="sortingarraylists">Sorting ArrayLists</h3>
A more common scenario would be sorting an <code>ArrayList</code> populated by non-numerical objects. These can be employees, results from a database, users, etc. Since we don't know ahead of time how to sort these objects, it will need to be provided by the caller via the <code>comapreTo()</code> method.
Let's define a class for our objects that'll be stored in a collection:
<pre><code class="hljs">public class Element {
 private int id;

 public Element(int id) {
 this.id = id;
 }

 // Getters and setters

 public int compareTo(Element element) {
 int res = 0;
 if (this.id &lt; element.getId()) {
 res =- 1;
 }
 if (this.id &gt; element.getId()) {
 res = 1;
 }
 return res;
 }
}
</code></pre>
It's a very simple class with a single field - <code>id</code>. We can also <code>@Override</code> the <code>toString()</code> method if we'd like to print the results, but for the sake of brevity, let's not do that here and simply use the <code>getId()</code> method instead.
When it comes to sorting <code>ArrayList</code>s, since we're working with objects it's a little different than sorting simple arrays of primitive values since we can't just use relational operators.
Now, our <code>compareTo()</code> method simply compares the <code>id</code>s and returns <code>-1</code> if the <code>id</code> of the current element is lesser than the element we're comparing it to, <code>1</code> if the <code>id</code> of the current element is higher, or <code>0</code> if they are equal.
Really, you can implement the comparing functionality in any way you'd like.
Now, let's implement Bubble Sort again:
<pre><code class="hljs">public void bubbleSortArrayList(List&lt;Element&gt; list) {
 Element temp;
 boolean sorted = false;

 while (!sorted) {
 sorted = true;
 for (int i = 0; i &lt; list.size()-1; i++) {
 if (list.get(i).compareTo(list.get(i + 1)) &gt; 0) {
 temp = list.get(i);
 list.set(i, list.get(i + 1));
 list.set(i + 1, temp);
 sorted = false;
 }
 }
 }
}
</code></pre>
This is pretty straightforward code, practically the same as the code in the example with arrays, utilizing the methods provided by the <code>List</code> interface.
Now, let's populate an <code>ArrayList</code> with a few elements:
<pre><code class="hljs">List&lt;Element&gt; list = new ArrayList&lt;&gt;();

// Create elements w/ IDs 0-24
for (int i = 0; i &lt; 25; i++) {
 list.add(new Element(i));
}

// Move the elements to a random order
Collections.shuffle(list);
</code></pre>
And we can sort it:
<pre><code class="hljs">// Print list before sorting
list.forEach(e -&gt; System.out.print(e.getId() + &quot;, &quot;));

// Sort the list
bubbleSort.bubbleSortArrayList(list);

System.out.println();

// Print sorted list
list.forEach(e -&gt; System.out.print(e.getId() + &quot;, &quot;));
</code></pre>
As expected, the output is:
<pre><code class="hljs">17, 13, 14, 5, 15, 22, 24, 7, 3, 9, 21, 10, 1, 11, 18, 20, 12, 8, 4, 19, 0, 23, 16, 2, 6,
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24,
</code></pre>
<h2 id="thecollectionsapi">The Collections API</h2>
A great thing about the Collections API that ships with Java 8 is helper methods that allow you to quickly and easily sort collections. We just need to implement the <code>Comparable</code> interface for the elements we wish to sort later on.
Let's change our <code>Element</code> class so that it implements the <code>Comparable</code> interface:
<pre><code class="hljs">public class Element implements Comparable&lt;Element&gt; {
 private int id;

 // Constructor, getters and setters

 @Override
 public int compareTo(Element element) {
 int res = 0;
 if (this.id &lt; element.getId()) {
 res =- 1;
 }
 if (this.id &gt; element.getId()) {
 res = 1;
 }
 return res;
 }
}
</code></pre>
As you can see, the <code>Element</code> class is pretty much the same as before. This time, we implemented the <code>Comparable</code> interface and overridden it's <code>compareTo()</code> method with the same logic as before.
Now, we can simply call <code>Collections.sort()</code> on our list:
<pre><code class="hljs">List&lt;Element&gt; list = new ArrayList&lt;&gt;();
for (int i = 0; i &lt; 10000; i++) {
 list.add(new Element(i));
}

Collections.shuffle(list);

// Print shuffled list
list.forEach(e -&gt; System.out.print(e.getId() + &quot;, &quot;));

// Sort the list
Collections.sort(list);

System.out.println();

// Print sorted list
list.forEach(e -&gt; System.out.print(e.getId() + &quot;, &quot;));
</code></pre>
The <code>sort()</code> method from the Collections API uses Quick Sort to sort the given collection. This results in huge performance benefits compared to Bubble Sort, but we'll save that for another article.
<h2 id="timecomplexity">Time Complexity</h2>
The time complexity (both average and worst) of Bubble Sort is O(n^2). This is, realistically observing, horrible for a sorting algorithm.
Albeit horrible, here's how the algorithm performed when sorting 10,000 objects in a collection:
<pre><code class="hljs">List&lt;Element&gt; list = new ArrayList&lt;&gt;();
for (int i = 0; i &lt; 10000; i++) {
 list.add(new Element(i));
}

Collections.shuffle(list);

// Print shuffled collection
list.forEach(e -&gt; System.out.print(e.getId() + &quot;, &quot;));

long startTime = System.nanoTime();
bubbleSort.bubbleSortArrayList(list);
long endTime = System.nanoTime();

// Print sorted collection
list.forEach(e -&gt; System.out.print(e.getId() + &quot;, &quot;));

System.out.println();

// Print runtime in nanoseconds
System.out.println(&quot;Bubble Sort runtime: &quot; + (endTime - startTime));
</code></pre>
And here are the results in seconds after running it 10 times:
<table class="table table-striped" style="white-space:nowrap; font-size:15px;">
 <tr>
 <th>Bubble Sort</th>
 <th>time(s)</th>
 <th></th>
 </tr>
 <tr>
 <td>First Run</td>
 <td>0.5885202</td>
 </tr>
 <tr>
 <td>Second Run</td>
 <td>0.6647364</td>
 </tr>
 <tr>
 <td>Third Run</td>
 <td>0.5748066</td>
 </tr>
 <tr>
 <td>Fourth Run</td>
 <td>0.5266222</td>
 </tr>
 <tr>
 <td>Fifth Run</td>
 <td>0.522961</td>
 </tr>
 <tr>
 <td>Sixth Run</td>
 <td>0.5033268</td>
 </tr>
 <tr>
 <td>Seventh Run</td>
 <td>0.5044489</td>
 </tr>
 <tr>
 <td>Eight Run</td>
 <td>0.6409187</td>
 </tr>
 <tr>
 <td>Ninth Run</td>
 <td>0.6021427</td>
 </tr>
 <tr>
 <td>Tenth Run</td>
 <td>0.694294</td>
 </tr>
</table>
With an average runtime of ~0.5s for 10,000 objects, will you really need algorithms that perform better? Most of the time, not really, unless you have an application with a high load that requires a fast response time.
If you're doing more complex comparisons than just comparing <code>id</code>s and have a huge collection, much larger than this one - yes, using an advanced algorithm with a much better time complexity will affect your performance significantly.
For reference, the <code>sort()</code> method from the Collections API sorted this same array of 10,000 elements in just 0.01s consistently. So even if there's no real need to sort your collections faster than 0.5s, using a built-in sorter provided by the Collections API will both save you time when coding and improve your application.
<h2 id="conclusion">Conclusion</h2>
Bubble Sort is, in most cases, the first sorting algorithm any computer science enthusiast will encounter. It's the simplest and most intuitive sorting algorithm which is one of the main reasons why it's being taught at an early point.
We saw that this simple sorting algorithm works by swapping adjacent elements, according to a given order criteria. For example, if we want to sort a collection's elements from the smallest to largest - if the first element is larger than the second, they're swapped. This simple exchange is repeated for adjacent indexes until the collection is eventually sorted.
It's horribly inefficient and given that there are much more efficient algorithms built-in to Java in the Collections API, we'd advise that you don't use this algorithm for any production applications.

David Landup

Bubble Sort is in most cases the first sorting algorithm you'll come across. In this article, we'll be taking a deep dive into the algorithm, how it works and then implement it in Java.

Bubble Sort in Java

<h2 id="introduction">Introduction</h2>
Users don't only consume data, they also produce data and upload it. They can send data through applications like messengers or email to specific recipients or upload files to social networks and data streaming platforms such as Facebook or YouTube.
That being said, almost every interactive website today supports file uploads.
<h2 id="fileuploadinglibraries">File Uploading Libraries</h2>
There are several Node libraries available on NPM that can simplify the process of validating and uploading files to the server. Among them, the most popular choice these days are <a rel="nofollow noopener" target="_blank" href="https://www.npmjs.com/package/multer">Multer</a>, <a rel="nofollow noopener" target="_blank" href="https://www.npmjs.com/package/formidable">Formidable</a>, and <a rel="nofollow noopener" target="_blank" href="https://www.npmjs.com/package/multiparty">Multiparty</a>.
They all have stable versions and are backed by an online community of open source developers.
<h2 id="whatismulter">What is Multer?</h2>
Multer is a popular Node.js middleware used for handling <code>multipart/form-data</code> requests. It makes use of <a rel="nofollow noopener" target="_blank" href="https://github.com/mscdex/busboy">busboy</a> to parse any data that it received through an HTML form. This greatly enhances its performance because the busboy module is unmatched when it comes to analyzing form data.
Multer provides us control and flexibility when handling <code>multipart/form-data</code> requests - we get detailed information about each uploaded file, the ability to add a custom storage engine, validation of files according to our needs, the ability to set limits on uploaded files, etc.
<h2 id="projectsetup">Project Setup</h2>
Since we won't be storing our images in a database, but rather a simple folder for brevity and simplicity, let's make another folder within our project folder and name it, say, <code>uploads</code>.
Now, let's install Express:
<pre><code class="hljs">$ npm i express
</code></pre>
And finally, let's install Multer:
<pre><code class="hljs">$ npm i multer
</code></pre>
<h2 id="projectimplementation">Project Implementation</h2>
At this point, we are ready to write some code, starting off with the HTML forms that we'll use to harvest information.
Let's start off with the form for uploading a single file:
<pre><code class="hljs">&lt;form method=&quot;POST&quot; action=&quot;/upload-profile-pic&quot; enctype=&quot;multipart/form-data&quot;&gt;
 &lt;div&gt;
 &lt;label&gt;Select your profile picture:&lt;/label&gt;
 &lt;input type=&quot;file&quot; name=&quot;profile_pic&quot; /&gt;
 &lt;/div&gt;
 &lt;div&gt;
 &lt;input type=&quot;submit&quot; name=&quot;btn_upload_profile_pic&quot; value=&quot;Upload&quot; /&gt;
 &lt;/div&gt;
&lt;/form&gt;
</code></pre>
And then with a form that allows us to upload multiple files:
<pre><code class="hljs">&lt;form method=&quot;POST&quot; action=&quot;/upload-multiple-images&quot; enctype=&quot;multipart/form-data&quot;&gt;
 &lt;div&gt;
 &lt;label&gt;Select multiple images:&lt;/label&gt;
 &lt;input type=&quot;file&quot; name=&quot;multiple_images&quot; multiple /&gt;
 &lt;/div&gt;
 &lt;div&gt;
 &lt;input type=&quot;submit&quot; name=&quot;btn_upload_multiple_images&quot; value=&quot;Upload&quot; /&gt;
 &lt;/div&gt;
&lt;/form&gt;
</code></pre>
You can either put these forms on separate pages or on the same one. For the purpose of this tutorial, they're one after another:
<img src="https://stackabuse.s3.amazonaws.com/media/handling-file-uploads-in-node-with-express-and-multer-1.JPG" alt="HTML5 forms for submitting files to server">
The HTML forms are pretty straightforward, accepting <code>multipart/form-data</code> and routing to the adequate functions that handle their requests.
<h3 id="expressapplication">Express Application</h3>
With our forms ready, we can work on the actual logic for uploading and validating files through Express.
Let's create a file called <code>app.js</code> in the project root and start by importing the required modules:
<pre><code class="hljs">const express = require(&#x27;express&#x27;);
const multer = require(&#x27;multer&#x27;);
const path = require(&#x27;path&#x27;);
</code></pre>
Now, let's create our Express app:
<pre><code class="hljs">const app = express();
</code></pre>
And finally, let's set up the port on which it'll run:
<pre><code class="hljs">const port = process.env.PORT || 3000;
</code></pre>
The <code>public</code> directory off of our root folder contains the static files we want to serve, so let's set it as a static directory using <code>express.static</code>:
<pre><code class="hljs">app.use(express.static(__dirname + &#x27;/public&#x27;));
</code></pre>
At this point, let's define the storage location for our images:
<pre><code class="hljs">const storage = multer.diskStorage({
 destination: function(req, file, cb) {
 cb(null, &#x27;uploads/&#x27;);
 },

 // By default, multer removes file extensions so let&#x27;s add them back
 filename: function(req, file, cb) {
 cb(null, file.fieldname + &#x27;-&#x27; + Date.now() + path.extname(file.originalname));
 }
});
</code></pre>
And finally, let's run the app using the port we set earlier:
<pre><code class="hljs">app.listen(port, () =&gt; console.log(`Listening on port ${port}...`));
</code></pre>
<h3 id="filevalidationandupload">File Validation and Upload</h3>
For fundamental security reasons, we'll want to validate files before uploading them to our servers. Let's edit the <code>app.js</code> file and add both functionalities:
<pre><code class="hljs">app.post(&#x27;/upload-profile-pic&#x27;, (req, res) =&gt; {
 // &#x27;profile_pic&#x27; is the name of our file input field in the HTML form
 let upload = multer({ storage: storage, fileFilter: helpers.imageFilter }).single(&#x27;profile_pic&#x27;);

 upload(req, res, function(err) {
 // req.file contains information of uploaded file
 // req.body contains information of text fields, if there were any

 if (req.fileValidationError) {
 return res.send(req.fileValidationError);
 }
 else if (!req.file) {
 return res.send(&#x27;Please select an image to upload&#x27;);
 }
 else if (err instanceof multer.MulterError) {
 return res.send(err);
 }
 else if (err) {
 return res.send(err);
 }

 // Display uploaded image for user validation
 res.send(`You have uploaded this image: &lt;hr/&gt;&lt;img src=&quot;${req.file.path}&quot; width=&quot;500&quot;&gt;&lt;hr /&gt;&lt;a href=&quot;./&quot;&gt;Upload another image&lt;/a&gt;`);
 });
});
</code></pre>
Here, we've accepted an HTTP POST request, in which the image information is embodied. The function that actually takes care of the upload functionality is the <code>multer().single()</code> method.
You might have noticed the <code>fileFilter: helpers.imageFilter</code> but we haven't yet created/imported the <code>helpers</code> file. So, let's create a new file in our project directory and name it <code>helpers.js</code>. Here we will write some code that is used to check whether the submitted file is an image or not.
<pre><code class="hljs">const imageFilter = function(req, file, cb) {
 // Accept images only
 if (!file.originalname.match(/\.(jpg|JPG|jpeg|JPEG|png|PNG|gif|GIF)$/)) {
 req.fileValidationError = &#x27;Only image files are allowed!&#x27;;
 return cb(new Error(&#x27;Only image files are allowed!&#x27;), false);
 }
 cb(null, true);
};
exports.imageFilter = imageFilter;
</code></pre>
Of course, to use this module, we'll have to import it at the top of our <code>app.js</code> file:
<pre><code class="hljs">const helpers = require(&#x27;./helpers&#x27;);
</code></pre>
Now, we can run our application and validate that it's working properly:
<img src="https://stackabuse.s3.amazonaws.com/media/handling-file-uploads-in-node-with-express-and-multer-2.JPG" alt="Uploaded single image">
<h3 id="uploadingmultiplefiles">Uploading Multiple Files</h3>
Uploading multiple files is essentially the same as uploading a single file. Though, instead of the <code>multer().single()</code> function, we use the <code>multer().array()</code> function:
<pre><code class="hljs">app.post(&#x27;/upload-multiple-images&#x27;, (req, res) =&gt; {
 // 10 is the limit I&#x27;ve defined for number of uploaded files at once
 // &#x27;multiple_images&#x27; is the name of our file input field
 let upload = multer({ storage: storage, fileFilter: helpers.imageFilter }).array(&#x27;multiple_images&#x27;, 10);

 upload(req, res, function(err) {
 if (req.fileValidationError) {
 return res.send(req.fileValidationError);
 }
 else if (...) // The same as when uploading single images

 let result = &quot;You have uploaded these images: &lt;hr /&gt;&quot;;
 const files = req.files;
 let index, len;

 // Loop through all the uploaded images and display them on frontend
 for (index = 0, len = files.length; index &lt; len; ++index) {
 result += `&lt;img src=&quot;${files[index].path}&quot; width=&quot;300&quot; style=&quot;margin-right: 20px;&quot;&gt;`;
 }
 result += &#x27;&lt;hr/&gt;&lt;a href=&quot;./&quot;&gt;Upload more images&lt;/a&gt;&#x27;;
 res.send(result);
 });
});
</code></pre>
And now, to validate if everything is working correctly:
<img src="https://stackabuse.s3.amazonaws.com/media/handling-file-uploads-in-node-with-express-and-multer-3.JPG" alt="Uploaded multiple image">
<h2 id="conclusion">Conclusion</h2>
Users don't only consume data, they produce data and, in many cases, need to upload it to a web server. They can send data through applications like messengers or email to specific recipients, or they can upload files to social networks and data streaming platforms such as Facebook or YouTube.
In this article we've used Express.js and the Multer library to handle basic file uploading functionality in a simple web application.

Guest Contributor

Most interactive websites today support file uploads. In this article, we'll be using Multer, Formidable and Multiparty to facilitate file upload capabilities in Node.js.

Handling File Uploads in Node.js with Express and Multer

<h2 id="introduction">Introduction</h2>
Monitoring an application's health and metrics helps us manage it better, notice unoptimized behavior and get closer to its performance. This especially holds true when we're developing a system with many microservices, where monitoring each service can prove to be crucial when it comes to maintaining our system.
Based on this information, we can draw conclusions and decide which microservice needs to scale if further performance improvements can't be achieved with the current setup.
In this article, we'll cover how to monitor Spring Boot web applications. We will be using three projects to achieve this:
<ul>
<li><a rel="nofollow noopener" target="_blank" href="https://micrometer.io/">Micrometer</a>: Exposes the metrics from our application</li>
<li><a rel="nofollow noopener" target="_blank" href="https://prometheus.io/">Prometheus</a>: Stores our metric data</li>
<li><a rel="nofollow noopener" target="_blank" href="https://grafana.com">Grafana</a>: Visualizes our data in graphs</li>
</ul>
This might look like a lot, especially compared to just using the <a rel="nofollow noopener" target="_blank" href=" https://spring.io/guides/gs/actuator-service/">Spring Boot Actuator</a> project, but it's very easy to implement all of them with just a few configurations.
To make things even easier, we'll be using <a rel="nofollow noopener" target="_blank" href="https://www.docker.com/">Docker</a> to run Prometheus and Grafana since they both have official Docker images. If you're not familiar with Docker, you can check out our article <a target="_blank" href="/docker-a-high-level-introduction/">Docker: A High-Level Introduction</a>.
Please note that these metrics will give you aggregated information over an interval of time. If you want to check information about an individual request at a particular time and what happened to it, then this might not be the solution for you.
In that case, you probably need a distributed tracing system which we have covered in detail in <a target="_blank" href="/spring-cloud-distributed-tracing-with-sleuth">Distributed Tracing with Sleuth</a>.
<h2 id="springbootactuator">Spring Boot Actuator</h2>
We'll start off with a simple REST service using <a rel="nofollow noopener" target="_blank" href="https://start.spring.io/">Spring Initializr</a> that contains a single endpoint of <code>/hello</code> and running on the default port of <code>8080</code>.
Besides, this application also has the <code>spring-boot-starter-actuator</code> dependency, which provides <a rel="nofollow noopener" target="_blank" href="https://docs.spring.io/spring-boot/docs/current/reference/html/production-ready-endpoints.html">production-ready endpoints</a> that you can use for your application. These endpoints fall under a common prefix of <code>/actuator</code> and are, by default, protected.
Expose them individually, or all at once, by adding the following properties in <code>application.properties</code>:
<pre><code class="hljs">management.endpoints.web.exposure.include=*
</code></pre>
To check, let's navigate our browser to <code>http://localhost:8080/actuator</code>:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-1.png" alt="spring boot monitor actuator">
You can see all the endpoints that Actuator exposes such as <code>/health</code>, <code>/metrics</code>, <code>/mappings</code>, etc. Let's open up the <code>/metrics</code> endpoint of the Actuator by navigating our browser to <code>http://localhost:8080/actuator/metrics</code>:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-2.png" alt="spring boot monitor mappings and endpoints">
As you can see, there's a bunch of information about our application here, such as information about threads, Tomcat sessions, classes, the buffer, etc. Let's go deeper and retrieve information about the JVM memory used:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-3.png" alt="spring boot monitor actuator metrics memeory">
Now, using the Spring Boot Actuator like this does yield a lot of information about your application, but it's not very user-friendly. It can be integrated with <a rel="nofollow noopener" target="_blank" href="https://github.com/codecentric/spring-boot-admin">Spring Boot Admin</a> for visualization, but it has its limitations and is less popular.
Tools like Prometheus, <a rel="nofollow noopener" target="_blank" href="https://github.com/Netflix/atlas">Netflix Atlas</a>, and Grafana are more commonly used for the monitoring and visualization and are language/framework-independent.
Each of these tools has its own set of data formats and converting the <code>/metrics</code> data for each one would be a pain. To avoid converting them ourselves, we need a vendor-neutral data provider, such as Micrometer.
<h2 id="micrometer">Micrometer</h2>
To solve this problem of being a vendor-neutral data provider, Micrometer came to be. It exposes Actuator metrics to external monitoring systems such as Prometheus, Netflix Atlas, <a rel="nofollow noopener" target="_blank" href="https://aws.amazon.com/cloudwatch/">AWS CloudWatch</a>, and many more.
They quite correctly describe themselves as:
<blockquote>
Think SLF4J, but for metrics.
</blockquote>
Just as a refresher, SLF4J is a logging facade for other Java logging frameworks. SLF4J itself does not have any logging implementation. The idea is that you write code using SLF4J API and the real implementation of it comes from the framework you choose. It could be any of the popular frameworks like <a rel="nofollow noopener" target="_blank" href="https://logging.apache.org/log4j/2.x/">Log4j</a>, <a rel="nofollow noopener" target="_blank" href="http://logback.qos.ch/">Logback</a>, etc.
Similarly, Micrometer automatically exposes <code>/actuator/metrics</code> data into something your monitoring system can understand. All you need to do is include that vendor-specific micrometer dependency in your application.
Micrometer is a separate open-sourced project and is not in the Spring ecosystem, so we have to explicitly add it as a dependency. Since we will be using Prometheus, let's add it's specific dependency in our <code>pom.xml</code>:
<pre><code class="hljs">&lt;dependency&gt;
 &lt;groupId&gt;io.micrometer&lt;/groupId&gt;
 &lt;artifactId&gt;micrometer-registry-prometheus&lt;/artifactId&gt;
&lt;/dependency&gt;
</code></pre>
Rebuild and start the application and navigate our browser to <code>http://localhost:8080/actuator</code>:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-4.png" alt="spring boot monitor actuator prometheus">
This will generate a new endpoint - <code>/actuator/prometheus</code>. Opening it, you will see data formatted specific for Prometheus:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-5.png" alt="spring boot monitor actuator prometheus data">
<h2 id="prometheus">Prometheus</h2>
Prometheus is a time-series database that stores our metric data by pulling it (using a built-in data scraper) periodically over HTTP. The intervals between pulls can be configured, of course, and we have to provide the URL to pull from. It also has a simple user interface where we can visualize/query on all of the collected metrics.
Let's configure Prometheus, and more precisely the scrape interval, the targets, etc. To do that, we'll be using the <code>prometheus.yml</code> file:
<pre><code class="hljs">global:
 scrape_interval: 10s

scrape_configs:
 - job_name: &#x27;spring_micrometer&#x27;
 metrics_path: &#x27;/actuator/prometheus&#x27;
 scrape_interval: 5s
 static_configs:
 - targets: [&#x27;192.168.2.8:8080&#x27;]
</code></pre>
As you can see, we have a <code>scrape_configs</code> root key where we can define a list of jobs and specify the URL, metrics path, and the interval. If you'd like to read more about Prometheus configurations, feel free to visit the <a rel="nofollow noopener" target="_blank" href="https://prometheus.io/docs/prometheus/latest/configuration/configuration/">official documentation</a>.
Note: Since we are using Docker to run Prometheus, it will be running in a Docker network that won't understand <code>localhost/120.0.01</code>, as you might expect. Since our app is running on <code>localhost</code>, and for the Docker container, <code>localhost</code> means its own network, we have to specify our system IP in place of it.
So instead of using <code>locahost:8080</code>, <code>192.168.2.8:8080</code> is used where <code>192.168.2.8</code> is my PC IP at the moment.
To check your system IP you can run <code>ipconfig</code> or <code>ifconfig</code> in your terminal, depending upon your OS.
Now, we can run Prometheus using the Docker command:
<pre><code class="hljs">$ docker run -d -p 9090:9090 -v &lt;path-to-prometheus.yml&gt;:/etc/prometheus/prometheus.yml prom/prometheus
</code></pre>
<code>&lt;path-to-prometheus.yml&gt;</code> is where your own <code>prometheus.yml</code> is starting from the root. For an example, this works on my local Windows PC:
<pre><code class="hljs">$ docker run -d -p 9090:9090 -v $PWD/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus
</code></pre>
To see Prometheus dashboard, navigate your browser to <a rel="nofollow noopener" target="_blank" href="http://localhost:9090">http://localhost:9090</a>:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-6.png" alt="prometheus dashboard">
To check if Prometheus is actually listening to the Spring app, you can go to the <code>/targets</code> endpoint:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-7.png" alt="monitor prometheus target">
Let's go back to the home page and select a metric from the list and click Execute:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-8.png" alt="monitor metric from the list">
<h3 id="prometheusquerylanguagepromql">Prometheus Query Language - PromQL</h3>
Another thing to note is - Prometheus has its own query language called PromQL. It allows the user to select and aggregate time series data in real-time, storing it either in graph or tabular format. Alternatively, you can feed it to an external API through HTTP.
If you'd like to read more about PromQL, the <a rel="nofollow noopener" target="_blank" href="https://prometheus.io/docs/prometheus/latest/querying/basics/">official documentation</a> covers it quite nicely.
<h2 id="grafana">Grafana</h2>
While Prometheus does provide some crude visualization, Grafana offers a rich UI where you can build up custom graphs quickly and create a dashboard out of many graphs in no time. You can also import many community built dashboards for free and get going.
Grafana can pull data from various data sources like Prometheus, <a rel="nofollow noopener" target="_blank" href="https://www.elastic.co/">Elasticsearch</a>, <a rel="nofollow noopener" target="_blank" href="https://www.influxdata.com/">InfluxDB</a>, etc. It also allows you to set rule-based alerts, which then can notify you over Slack, Email, HipChat, and similar.
Let's start off by running Grafana using Docker:
<pre><code class="hljs">$ docker run -d -p 3000:3000 grafana/grafana
</code></pre>
If you visit <code>http://localhost:3000</code>, you will be redirected to a login page:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-9.png" alt="grafana login page">
The default username is <code>admin</code> and the default password is <code>admin</code>. You can change these in the next step, which is highly recommended:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-10.png" alt="grafana dashboard">
Since Grafana works with many data sources, we need to define which one we're relying on. Select Prometheus as your data source:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-11.png" alt="grafana datasource">
Now, add the URL that Prometheus is running on, in our case <code>http://localhost:9090</code> and select Access to be through a browser.
At this point, we can save and test to see if the data source is working correctly:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-12.png" alt="grafana datasource update">
As previously said, Grafana has a ton of pre-built <a rel="nofollow noopener" target="_blank" href="https://grafana.com/grafana/dashboards">dashboards</a>. For Spring Boot projects, <a rel="nofollow noopener" target="_blank" href="https://grafana.com/grafana/dashboards/4701">the JVM dashboard</a> is popular:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-13.png" alt="grafana datasource import">
Input the URL for the dashboard, select &quot;Already created Prometheus datasource&quot; and then click Import:
<img src="https://stackabuse.s3.amazonaws.com/media/monitoring-spring-boot-apps-with-micrometer-prometheus-and-grafana-14.png" alt="grafana datasource jvm">
<h2 id="conclusion">Conclusion</h2>
Monitoring an application's health and metrics helps us manage it better, notice unoptimized behavior, and better understand its performance. This especially holds true when we're developing a system with many microservices, where monitoring each service can prove to be crucial when it comes to maintaining our system.
Based on this information, we can draw conclusions and decide which microservice needs to scale if further performance improvements can't be achieved with the current setup.
In this article, we used Micrometer to reformat the metrics data provided by Spring Boot Actuator and expose it in a new endpoint. This data was then regularly pulled and stored by Prometheus, which is a time-series database. Ultimately, we've used Grafana to visualize this information with a user-friendly dashboard.
As always, the code for the examples used in this article can be found on <a rel="nofollow noopener" target="_blank" href="https://github.com/StackAbuse/microservices-monitoring">GitHub</a>.

Dhananjay Singh

Monitoring an app's health helps us manage it and improve optimization and performance. In this article, we'll be using Micrometer, Prometheus, and Grafana to closely monitor a Spring Boot application.

Monitoring Spring Boot Apps with Micrometer, Prometheus, and Grafana

PyTorch, alongside Tensorflow, is an extremely popular deep learning library for Python. In this article, we'll cover one of the basic tasks in machine learning - classification.