<h2 id="introduction">Introduction</h2>
There are numerous free hosting services available for getting your Node.js applications up and running publicly. One of these services is <a rel="nofollow" target="_blank" href="https://www.heroku.com/">Heroku</a>, that allows you to deploy, manage and scale your applications on the web.
In this article we'll be building a simple Node and Express.js application that shortens given links, and deploy it to Heroku.
<h2 id="creatinganodejsurlshortener">Creating a Node.js URL Shortener</h2>
To make the development process easier we'll be using <a rel="nofollow" target="_blank" href="https://expressjs.com">Express</a>, which is a lightweight web framework that is flexible and easily customizable. Express comes with an <a rel="nofollow" target="_blank" href="https://expressjs.com/en/starter/generator.html">application generator</a>, but we'll start from scratch.
<h3 id="creatingtheprojectdirectory">Creating the Project Directory</h3>
Let's <code>cd</code> into the workspace and create a new directory for the project:
<pre><code class="hljs">$ mkdir url-shortener
</code></pre>
<h3 id="initializingnpm">Initializing npm</h3>
Next let's initialize <a rel="nofollow" target="_blank" href="https://www.npmjs.com">npm</a>, and with it, start our project:
<pre><code class="hljs">$ npm init
</code></pre>
<code>npm init</code> will ask a series of questions (package name, version, description, etc). For simplicity's sake, let's skip all of these by hitting <kbd>RETURN</kbd> for each prompt. Node will then generate a <code>package.json</code> file and print it to the terminal:
<pre><code class="hljs">{
 &quot;name&quot;: &quot;url-shortener&quot;,
 &quot;version&quot;: &quot;1.0.0&quot;,
 &quot;description&quot;: &quot;&quot;,
 &quot;main&quot;: &quot;index.js&quot;,
 &quot;scripts&quot;: {
 &quot;start&quot;: &quot;node app.js&quot;,
 &quot;test&quot;: &quot;echo \&quot;Error: no test specified\&quot; &amp;&amp; exit 1&quot;
 },
 &quot;author&quot;: &quot;&quot;,
 &quot;license&quot;: &quot;ISC&quot;,
}
</code></pre>
The fields in <code>package.json</code> (package name, description, author, etc) can be updated at any time so you don't need to worry about it too much.
What we've added here manually is the <code>&quot;start&quot;: &quot;node app.js&quot;</code>. This will be extremely important later, when we deploy the app to Heroku.
Heroku needs to know how to run our application, and this is typically done via a Procfile. Since our application is so simple, a Procfile would be overkill, and it's enough to simply define that our app should be run by simply executing the <code>node app.js</code> command.
<h3 id="installingexpress">Installing Express</h3>
With that out of the way, we're ready to install Express:
<pre><code class="hljs">$ npm install express --save
</code></pre>
<h3 id="buildinganodeserver">Building a Node Server</h3>
With Express installed I'll create a very simple application in a file called <code>app.js</code>:
<pre><code class="hljs">const express = require(&#x27;express&#x27;);
const app = express();
const path = require(&#x27;path&#x27;);
const port = 3000;

app.get(&#x27;/&#x27;, function(req, res) {
 res.sendFile(path.join(__dirname + &#x27;/index.html&#x27;));
});

app.listen(port, () =&gt; console.log(`url-shortener listening on port ${port}!`));
</code></pre>
The application at this point is really only the Express boilerplate required to set it up, with a single route handler that returns an HTML page on port <code>3000</code>.
Let's go ahead and define the page, which will accept a URL we want to shorten. It'll only consist of a header and a simple form that accepts the URL:
<pre><code class="hljs">&lt;h1&gt;URL Shortener&lt;/h1&gt;

&lt;form method=&quot;POST&quot; action=&quot;/url&quot;&gt;
 &lt;input type=&quot;text&quot; name=&quot;url&quot; /&gt;
 &lt;input type=&quot;submit&quot; /&gt;
&lt;/form&gt;
</code></pre>
We can test if the application is running fine by executing:
<pre><code class="hljs">$ node app.js
</code></pre>
Since we've added the <code>&quot;start&quot; : &quot;node app.js&quot;</code> command to our <code>package.json</code> file, we can also run it using an NPM command:
<pre><code class="hljs">$ npm start
</code></pre>
And navigating to <code>http://localhost:3000</code>:
<img src="https://stackabuse.s3.amazonaws.com/media/deploying-a-node-app-to-heroku-1.png" alt="application homepage">
<h3 id="handlingformsubmission">Handling Form Submission</h3>
While the page is rendering correctly, there's no logic to actually handle the POST request sent via the form at the <code>/url</code> endpoint.
To snatch the URL from the request sent via the form, we'll be using the <a rel="nofollow" target="_blank" href="https://www.npmjs.com/package/body-parser">body-parser</a> module and extract it from the request body, so let's go ahead and <code>require</code> it as well and set it up for use:
<pre><code class="hljs">const bodyParser = require(&#x27;body-parser&#x27;);
app.use(bodyParser.urlencoded({extended: true}));
app.use(express.urlencoded());
</code></pre>
With that done, we can extract the body of the request via <code>req.body</code> so let's go ahead and make another request handler that packs the URL from the body and sends it back via the response:
<pre><code class="hljs">app.post(&#x27;/url&#x27;, function(req, res) {
 const url = req.body.url;

 res.send(url);
});
</code></pre>
This still isn't what we want as we're getting the entire URL back and not the shortened version.
<h3 id="shorteningtheurl">Shortening the URL</h3>
The only thing left to do for our Node app to be complete is to actually shorten the URL before sending it back. For this, we can use the <a rel="nofollow" target="_blank" href="https://www.npmjs.com/package/node-url-shortener">node-url-shortener</a> module. Let's install it via npm:
<pre><code class="hljs">$ npm install node-url-shortener --save
</code></pre>
Then require the package in the application:
<pre><code class="hljs">const urlShortener = require(&#x27;node-url-shortener&#x27;);
</code></pre>
And lastly, let's add another block of code that shortens the URL before sending it back:
<pre><code class="hljs">app.post(&#x27;/url&#x27;, function(req, res) {
 const url = req.body.url;

 urlShortener.short(url, function(err, shortUrl){
 res.send(shortUrl);
 });
});
</code></pre>
The <code>urlShortner</code> has a <code>short()</code> method that accepts two parameters: the original URL and a callback function. The shortened URL gets passed into the callback function and then into <code>res.send()</code>.
Let's start up the application and test it out:
<img src="https://stackabuse.s3.amazonaws.com/media/deploying-a-node-app-to-heroku-3.png" alt="input url">
After inputting a URL, we're greeted with:
<img src="https://stackabuse.s3.amazonaws.com/media/deploying-a-node-app-to-heroku-2.png" alt="url shortener result">
<h2 id="deployingtoheroku">Deploying to Heroku</h2>
<h3 id="signingupforherokuandinstallingthecli">Signing up for Heroku and Installing the CLI</h3>
Now that the application has been built, it's time to think about Heroku. If you don't have an account, head over to <a rel="nofollow" target="_blank" href="https://signup.heroku.com">signup.heroku.com</a> and create one.
Once you have an account, the next thing to do is install Heroku's CLI tools, which are used for deployment. Heroku provides installation instructions at <a rel="nofollow" target="_blank" href="https://devcenter.heroku.com/articles/heroku-cli#download-and-install">The Dev Center</a>.
<h3 id="pushingtoheroku">Pushing to Heroku</h3>
Deploying to Heroku works via <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Git">Git</a>. It's very simple once you understand how it works. In essence you can deploy to Heroku by pushing to a remote repo, just like pushing to GitHub.
Because this is a new project, Git needs to be initialized:
<pre><code class="hljs">$ git init
</code></pre>
Now, we're ready to add and commit all the project files. Though, before doing that, we'll add a <code>.gitignore</code> file because we don't want the <code>node_modules</code> directory to be committed. It should be generated by npm upon deployment rather than committed to source control.
In your <code>.gitignore</code> file, simply add:
<pre><code class="hljs">node_modules
</code></pre>
With that out of the way, we can add and commit:
<pre><code class="hljs">$ git add .
$ git commit -m &#x27;initial commit&#x27;
</code></pre>
Once the project is committed, we need to create a Heroku app that'll correspond to our Node app:
<pre><code class="hljs">$ heroku create
Creating app... !
 ▸ Invalid credentials provided.
heroku: Press any key to open up the browser to login or q to exit:
Opening browser to https://cli-auth.heroku.com/auth/browser/abcd1234-b6a7-4df4-bb42-0eaf987d0637
Logging in... done
Logged in as example@gmail.com
Creating app... done, ⬢ nameful-wolf-12818
https://nameful-wolf-12818.herokuapp.com/ | https://git.heroku.com/nameful-wolf-12818.git
</code></pre>
If this is your first time using the CLI, Heroku will ask you to login via the browser. The process is very simple, just follow the instructions.
Once complete, Heroku returns the newly created app name and URL (your app name will be different to mine). If you browse to the URL you'll see a Heroku welcome page.
This still isn't our URL shortener app, simply a public Heroku app to which we'll deploy our own.
<h3 id="deployingtheapp">Deploying the App</h3>
In the previous <code>heroku create</code> command, a new remote will have been added to your Git repository, allowing you to deploy to Heroku with a simple command like this:
<pre><code class="hljs">$ git push heroku master
Enumerating objects: 5, done.
Counting objects: 100% (5/5), done.
Delta compression using up to 4 threads
Compressing objects: 100% (3/3), done.
Writing objects: 100% (3/3), 326 bytes | 326.00 KiB/s, done.
Total 3 (delta 2), reused 0 (delta 0)
...
...
...
remote: Verifying deploy... done.
To https://git.heroku.com/nameful-wolf-12818.git
 5cb9118..dd0bacd master -&gt; master
</code></pre>
Since the application is successfully deployed, we can go ahead and start it up:
<img src="https://stackabuse.s3.amazonaws.com/media/deploying-a-node-app-to-heroku-4.png" alt="deployed application on heroku">
Though, we'll encounter an issue due to the mismatch in ports. In our app, we've hardcoded that it uses port <code>3000</code>, but Heroku works on a different port and this clash crashes our application.
In order to make the application work both locally and on Heroku, we'll change the port to be either <code>3000</code> or the <code>process.env.PORT</code>, which is set by Heroku:
<pre><code class="hljs">const port = process.env.PORT || 3000;
</code></pre>
Deploying the application again and refreshing the page, everything works just fine!
<img src="https://stackabuse.s3.amazonaws.com/media/deploying-a-node-app-to-heroku-5.png" alt="heroku deployed application homepage">
<img src="https://stackabuse.s3.amazonaws.com/media/deploying-a-node-app-to-heroku-6.png" alt="heroku deployed application result page">
<h2 id="conclusion">Conclusion</h2>
There are numerous free hosting services available for getting your applications up and running publicly. One of these services is <a rel="nofollow" target="_blank" href="https://www.heroku.com/">Heroku</a>, that allows you to deploy, manage and scale your applications on the web.
We've built a simple Node and Express application that shortens provided URLs via the <code>node-url-shortener</code> module. We've then prepared that app for deployment and finally got it up and running on Heroku.

Tom Kadwill

Heroku allows us to easily deploy, scale and get applications public on the web.
In this article, we'll build a Node/Express application and deploy it to Heroku.

Deploying a Node.js App to Heroku

<h2 id="introduction">Introduction</h2>
Once you've written a web application, there are dozens of offerings to get your app online and usable by other people. One well known offering is part of the Amazon Web Services (AWS) platform - <a rel="nofollow" target="_blank" href="https://aws.amazon.com/ec2/">Elastic Compute Cloud</a> (EC2).
EC2 is a core part of AWS, and a lot of AWS' other services are built on top of it - therefore it's good to get an understanding of what EC2 is and how to deploy to it.
In this tutorial, we're going to create a basic Node.js app with Docker, start and configure an EC2 instance, and deploy our app to it. At the end of this tutorial you'll have your Node app running on AWS, and a better understanding of how to interact with a core AWS service.
<h2 id="prerequisites">Prerequisites</h2>
<h3 id="awsaccount">AWS Account</h3>
<a rel="nofollow" target="_blank" href="https://aws.amazon.com/">Amazon Web Services</a> (AWS) is a collection of tools for building applications in the cloud. As EC2 is an AWS service, we'll need to set up an AWS account.
AWS has a free tier for a lot of awesome stuff, and EC2 is no exception - you're free to use 750 hours (31 days) of EC2 a month in the free tier for a whole year.
<h3 id="docker">Docker</h3>
<a rel="nofollow" target="_blank" href="https://www.docker.com/">Docker</a> allows us to bundle up our applications into small, easily deployable units that can be run anywhere where Docker is installed. This means no more of that 'but it works on my machine!'
This article will assume basic familiarity with Docker, and won't be going into any depth on it - however, if you'd like to do a deeper dive then check out <a target="_blank" href="/deploying-a-node-js-app-to-a-digitalocean-droplet-with-docker/">Deploying a Node.js App to a DigitalOcean Droplet with Docker</a>.
<h2 id="nodeapplication">Node Application</h2>
Let's make a really simple Node application that responds to a request. To do this, we'll open up a terminal and run:
<pre><code class="hljs">$ mkdir node-ec2
$ cd node-ec2
$ npm init
</code></pre>
This will create a new folder, move into that folder, and then initialize a new Node application.
Let's stick with the NPM defaults for now - this article will assume you left the entry point as <code>index.js</code>. This will generate our <code>package.json</code> file, which is essentially a configuration file for our app.
Once the <code>package.json</code> file is created, open it up and add the following line to the beginning of the <code>scripts</code> section:
<pre><code class="hljs">&quot;start&quot;: &quot;node index.js&quot;,
</code></pre>
By doing this, instead of running <code>node index.js</code>, we'll be using <code>npm start</code>, which will run everything in our script. In this specific case, it just runs <code>node index.js</code>, though in reality, it could be much more than that. For example, if we can add flags to the command without having to type it out each time, or we could set some environment variables like <code>NODE_ENV=production node index.js</code>.
To serve our requests, we're going to be using the <a rel="nofollow" target="_blank" href="https://expressjs.com">Express</a> framework - it's minimalistic, and easy to get started with:
<pre><code class="hljs">$ npm install express --save
</code></pre>
Our <code>package.json</code> should now look something like this:
<pre><code class="hljs">{
 &quot;name&quot;: &quot;app&quot;,
 &quot;version&quot;: &quot;1.0.0&quot;,
 &quot;description&quot;: &quot;&quot;,
 &quot;main&quot;: &quot;index.js&quot;,
 &quot;scripts&quot;: {
 &quot;start&quot;: &quot;node index.js&quot;
 },
 &quot;author&quot;: &quot;&quot;,
 &quot;license&quot;: &quot;ISC&quot;,
 &quot;dependencies&quot;: {
 &quot;express&quot;: &quot;^4.17.1&quot;
 }
}
</code></pre>
Wait for the install, and then we're going to open our preferred code editor to create another new file in the same location called <code>index.js</code>.
The file will set up Express and define a request handler:
<pre><code class="hljs">const express = require(&#x27;express&#x27;);
const app = express();
const port = 3000;

app.get(&#x27;/status&#x27;, (req, res) =&gt; res.send({status: &quot;I&#x27;m alive!&quot;}));

app.listen(port, () =&gt; console.log(`Example app listening on port ${port}!`));
</code></pre>
This app will start on port <code>3000</code>, and will serve an endpoint at <code>/status</code>. We can verify this works by running:
<pre><code class="hljs">$ npm start
Example app listening on port 3000!
</code></pre>
Heading to <code>http://localhost:3000/status</code> - we should get a response back with <code>{status: &quot;I'm alive!&quot;}</code>. Once that's successful, make sure to stop the server with <kbd>CTRL</kbd>+<kbd>C</kbd>.
With our simple Node application ready, let's turn it into a Docker image which we'll deploy to EC2.
<h3 id="dockerizingthenodeapplication">Dockerizing the Node Application</h3>
Create a new file in the same directory as your Node application, called <code>Dockerfile</code>:
<pre><code class="hljs">FROM node:13-alpine

WORKDIR /usr/src/app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 3000
CMD [ &quot;node&quot;, &quot;index.js&quot; ]
</code></pre>
This is a basic Dockerfile that can be used for most simple Node applications. Next, let's build the Docker image and then run it to verify it's working correctly:
<pre><code class="hljs">$ docker build . -t ec2-app
$ docker run -p 3000:3000 ec2-app
</code></pre>
If you navigate to <code>http://localhost:3000/status</code> again, you should see the same status response from earlier. Exit the process with <kbd>CTRL</kbd>+<kbd>C</kbd> again.
Finally, let's push our Docker image to <a rel="nofollow" target="_blank" href="https://hub.docker.com/">Docker Hub</a>:
<pre><code class="hljs">$ docker login # Use your Docker Hub credentials here
$ docker tag ec2-app &lt;YOUR_DOCKER_USERNAME&gt;/ec2-app
$ docker push &lt;YOUR_DOCKER_USERNAME&gt;/ec2-app
</code></pre>
<h2 id="settingupec2">Setting up EC2</h2>
With our application &quot;dockerized&quot;, we need to set up an EC2 instance for it to run on.
Head to AWS and log in.
Click the 'Services' dropdown menu at the top of the page, and search for 'EC2'. AWS is currently experimenting with their interface, so you should see a page that looks something like the one below, but the center of the page may look slightly different.
Click on the 'Instances' link on the left.
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-1.png" alt="EC2 Welcome Page">
On the next view, click the 'Launch Instance' button. You'll see a page that looks like this:
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-2.png" alt="EC2 AMI Selection">
<h3 id="amis">AMIs</h3>
This is where we select the Amazon Machine Image - or AMI for short. An AMI is an 'out of the box' server, and can come with multiple configurations.
For instance, we could select one of the Quick Start AMIs that have <a rel="nofollow" target="_blank" href="https://aws.amazon.com/amazon-linux-2/">Amazon Linux 2</a> on them, or if you scroll down, there are instances with Ubuntu running on them, etc.
Each AMI is a frozen image of a machine with an operating system and potentially some extra software installed.
To make things easy, we can use this to build an EC2 instance with Docker already configured for us!
To do this, we need to select 'AWS Marketplace' on the left, and then in the search box we want to enter 'ECS'. We should get a few results, but we want the 'ECS Optimized Amazon Linux 2' image.
This image comes with Docker, and is optimized for running containers. Hit 'Select' on the chosen image and we'll continue to the next page:
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-3.png" alt="Choosing an AMI">
<h3 id="instancetypes">Instance Types</h3>
On the next view, we select what type of instance we want. Generally, this dictates the resources available to the server that we're starting up, with scaling costs for more performant machines.
The <code>t2.micro</code> instance type is eligible for free tier, so it's recommended to use that:
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-4.png" alt="Selecting EC2 instance type">
Select the appropriate checkbox, and then click 'Review and Launch' in the bottom right corner. Click 'Launch' in the same place on the next page, and you'll get a pop-up to select or create a key-pair.
Select the first drop-down, and select 'Create a new key pair'. Under 'Key pair name', enter what you want to call your key pair.
Make sure to 'Download the Key Pair' on the right hand side - this is what we'll use to access our EC2 instance.
By selecting 'Launch Instance' again, your EC2 instance should get started up:
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-5.png" alt="Launching instance">
Click the highlighted link to be taken to the instance detail page.
<h3 id="securitygroups">Security Groups</h3>
Before we try running our application, we need to make sure that we'll be able to access the application.
Most AWS resources operate under 'Security Groups' - these groups dictate how resources can be accessed, on what port, and from which IP addresses.
Click the security group highlighted here:
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-6.png" alt="Security Group Link">
From here, you'll be able to see details about the security group, including it's inbound and outbound rules in various tabs. Under the inbound tab, you'll hopefully see this:
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-7.png" alt="Inbound rules">
What this means is that traffic that comes in through port <code>22</code>, using the TCP protocol, is allowed from anywhere (<code>0.0.0.0/0</code> meaning anywhere). We need to add another rule to allow anybody to access our app at port <code>3000</code>.
At the top of the page, click 'Actions' and then click 'Edit inbound rules'. In the dialog that opens, click 'Add rule'.
Set the port range of the rule to <code>3000</code>, and under Source, click the drop-down and select 'Anywhere'. The rest should be automatically populated.
Ultimately, you should end up with something like:
<img src="https://stackabuse.s3.amazonaws.com/media/getting-started-with-aws-ec2-with-nodejs-9.png" alt="Editing the inbound rules">
<h3 id="connectingtoyourec2instance">Connecting to Your EC2 Instance</h3>
Head back to the 'Instances' page (click the link on the left) and select the instance you created earlier. The address for your EC2 instance is located above the link to the security groups under the 'Public DNS' field.
Head back to the terminal, and navigate to the folder where the key-pair you downloaded earlier is located. It will be named as whatever you entered for the key-pair name, with a <code>.pem</code> as its extension.
Let's change the key's permissions and then SSH into the EC2 instance:
<pre><code class="hljs">$ chmod 400 &lt;NAME_OF_KEYPAIR_FILE&gt;
$ ssh -i &lt;NAME_OF_KEYPAIR_FILE&gt;ec2-user@&lt;PUBLIC_DNS&gt;
</code></pre>
From here, we just need to launch our app via Docker:
<pre><code class="hljs">$ docker run -p 3000:3000 &lt;YOUR_DOCKER_USERNAME&gt;/ec2-app
</code></pre>
You'll be able to reach the instance using the same address you used to SSH into the instance. Simply navigate in your browser to:
<pre><code class="hljs">&lt;PUBLIC_DNS&gt;:3000/status
</code></pre>
Your app should return the status endpoint to you that we saw earlier. Congratulations, you've just run your first app on EC2!
<h2 id="whatnext">What Next?</h2>
<h3 id="runyourappheadlessly">Run Your App Headlessly</h3>
A quick win, however, is to run the app &quot;headless&quot;. As of now, your app is currently running in your shell session - and as soon as you close that session, the app will terminate!
To start the app in a way that it'll keep running in the background, run the app with the additional <code>-d</code> flag:
<pre><code class="hljs">$ docker run -d -p 3000:3000 &lt;YOUR_DOCKER_USERNAME&gt;/ec2-app
</code></pre>
<h3 id="security">Security</h3>
You might want to go back and tighten up the security on the instance/experiment with different configurations - such as configuring it so that only we can access the SSH port, for example.
Change the 'Source' field on the first rule to 'My IP' - AWS will automatically figure out where you're accessing it from.
Note: If you're running through this tutorial on the move, or come back to it later, your computer might have a different IP than when you initially set 'My IP'. If you encounter any difficulties later on, make sure to come back here and select 'My IP' again!
<h3 id="otheramis">Other AMIs</h3>
There are hundreds of different AMIs, a lot from various communities, with applications already pre-installed - it's worth taking a look through to see if there's an easy way to set up something you've wanted to work with!
<h3 id="addingadomain">Adding a Domain</h3>
Now that you've got an app running on a server, you might want to set up a domain name and point that at your application.
<h2 id="conclusion">Conclusion</h2>
EC2 really is the backbone of a lot of AWS services - for instance, RDS (AWS' database service) is really just heavily optimized EC2 instances with a nice dashboard.
Understanding this core product in AWS' arsenal is bound to open doors to new ways of implementing ideas.
In this tutorial, we've created a simple Node.js application with the help of Express, dockerized it, set up EC2 for deployment and finally - deployed it to the EC2 instance.

Josh Simpson

AWS EC2 is a core service and a foundation for many other services. In this article, we'll be building a Node app with Docker, setting up an EC2 instance and deploying our app to it.

Deploying Node.js Apps to AWS EC2 with Docker

<h2 id="introduction">Introduction</h2>
System administrators and developers frequently turn to automation to reduce their workload and improve their processes. When working with servers, automated tasks are frequently scripted with shell scripts. However, a developer might prefer to use a more general higher-level language for complex tasks. Many applications also need to interact with the file system and other OS-level components, which is often more easily done with command line level utilities.
With Node.js, we can run shell commands and process their inputs and outputs using JavaScript. Therefore, we can write most of these complex operations in JavaScript instead of the shell scripting language, potentially making the program easier to maintain.
In this article, we will learn the various ways to execute shell commands in Node.js using the <code>child_process</code> module.
<h2 id="thechild_proccessmodule">The child_proccess Module</h2>
Node.js executes its main event loop in a single thread. However, that does not mean that all of its processing is done in that one thread. Asynchronous tasks in Node.js are executed in other internal threads. When they are complete, the code in the callback, or error, is returned to the main, single thread.
These various threads are run in the same Node.js process. However, it is sometimes desirable to create another process to execute code. When a new process is created, the Operating System determines which processor it uses and how to schedule its tasks.
The <code>child_process</code> module creates new child processes of our main Node.js process. We can execute shell commands with these child processes.
Using external processes can improve performance of your application if used correctly. For example, if a feature of a Node.js application is CPU intensive, as Node.js is single threaded it would block the other tasks from executing while it is running.
However, we can delegate that resource intensive code to a child process, let's say a very efficient C++ program. Our Node.js code will then execute that C++ program in a new process, not blocking its other activities, and when complete process its output.
Two functions that we will use to execute shell commands are <code>exec()</code> and <code>spawn()</code>.
<h2 id="theexecfunction">The <code>exec()</code> Function</h2>
The <code>exec()</code> function creates a new shell and executes a given command. The output from the execution is buffered, which means kept in memory, and is available for use in a callback.
Let's use the <code>exec()</code> function to list all folders and files in our current directory. In a new Node.js file called <code>lsExec.js</code>, write the following code:
<pre><code class="hljs">const { exec } = require(&quot;child_process&quot;);

exec(&quot;ls -la&quot;, (error, stdout, stderr) =&gt; {
 if (error) {
 console.log(`error: ${error.message}`);
 return;
 }
 if (stderr) {
 console.log(`stderr: ${stderr}`);
 return;
 }
 console.log(`stdout: ${stdout}`);
});
</code></pre>
First, we require the <code>child_process</code> module in our program, specifically using the <code>exec()</code> function (via ES6 destructuring). Next, we call the <code>exec()</code> function with two parameters:
<ul>
<li>A string with the shell command we want executed.</li>
<li>A callback function with three parameters: <code>error</code>, <code>stdout</code>, <code>stderr</code>.</li>
</ul>
The shell command we are running is <code>ls -la</code>, which should list all the files and folders in our current directory line by line, including hidden files/folders. The callback function logs whether we got an <code>error</code> while trying to execute the command or output on the shell's <code>stdout</code> or <code>stderr</code> streams.
<blockquote>
Note: The <code>error</code> object is different from <code>stderr</code>. The <code>error</code> object is not null when the <code>child_process</code> module fails to execute a command. This could happen if you try to execute another Node.js script in <code>exec()</code> but the file could not be found, for example. On the other hand, if the command successfully runs and writes a message to the standard error stream, then the <code>stderr</code> object would not be null.
</blockquote>
If you run that Node.js file, you should see output similar to:
<pre><code class="hljs">$ node lsExec.js
stdout: total 0
drwxr-xr-x@ 9 arpan arpan 0 Dec 7 00:14 .
drwxr-xr-x@ 4 arpan arpan 0 Dec 7 22:09 ..
-rw-r--r--@ 1 arpan arpan 0 Dec 7 15:10 lsExec.js

child process exited with code 0
</code></pre>
Now that we've understood how to run commands with <code>exec()</code>, let's learn another way to execute commands with <code>spawn()</code>.
<h2 id="thespawnfunction">The <code>spawn()</code> Function</h2>
The <code>spawn()</code> function executes a command in a new process. This function uses a <a target="_blank" href="/introduction-to-node-js-streams">Stream API</a>, so its output of the command is made available via listeners.
Similar to before, we will use the <code>spawn()</code> function to list all folders and files in our current directory. Let's create a new Node.js file, <code>lsSpawn.js</code>, and enter the following:
<pre><code class="hljs">const { spawn } = require(&quot;child_process&quot;);

const ls = spawn(&quot;ls&quot;, [&quot;-la&quot;]);

ls.stdout.on(&quot;data&quot;, data =&gt; {
 console.log(`stdout: ${data}`);
});

ls.stderr.on(&quot;data&quot;, data =&gt; {
 console.log(`stderr: ${data}`);
});

ls.on(&#x27;error&#x27;, (error) =&gt; {
 console.log(`error: ${error.message}`);
});

ls.on(&quot;close&quot;, code =&gt; {
 console.log(`child process exited with code ${code}`);
});
</code></pre>
We begin by requiring the <code>spawn()</code> function from the <code>child_process</code> module. Then, we create a new process that executes the <code>ls</code> command, passing <code>-la</code> as an argument. Note how the arguments are held in an array and not included in the command string.
We then set up our listeners. The <code>stdout</code> object of <code>ls</code>, fires a <code>data</code> event when the command writes to that stream. Similarly, the <code>stderr</code> also fires a <code>data</code> event when the command writes to that stream.
Errors are caught by listening for them directly on the object that stores the reference for the command. You will only get an error if <code>child_process</code> fails to run the command.
The <code>close</code> event occurs when the command has finished.
If we run this Node.js file, we should get output like before with <code>exec()</code>:
<pre><code class="hljs">$ node lsSpawn.js
stdout: total 0
drwxr-xr-x@ 9 arpan arpan 0 Dec 7 00:14 .
drwxr-xr-x@ 4 arpan arpan 0 Dec 7 22:09 ..
-rw-r--r--@ 1 arpan arpan 0 Dec 7 15:10 lsExec.js
-rw-r--r--@ 1 arpan arpan 0 Dec 7 15:40 lsSpawn.js

child process exited with code 0
</code></pre>
<h2 id="whentouseexecandspawn">When to use <code>exec()</code> and <code>spawn()</code>?</h2>
The key difference between <code>exec()</code> and <code>spawn()</code> is how they return the data. As <code>exec()</code> stores all the output in a buffer, it is more memory intensive than <code>spawn()</code>, which streams the output as it comes.
Generally, if you are not expecting large amounts of data to be returned, you can use <code>exec()</code> for simplicity. Good examples of use-cases are creating a folder or getting the status of a file. However, if you are expecting a large amount of output from your command, then you should use <code>spawn()</code>. A good example would be using commands to manipulate binary data and then loading it into your Node.js program.
<h2 id="conclusion">Conclusion</h2>
Node.js can run shell commands by using the standard <code>child_process</code> module. If we use the <code>exec()</code> function, our command will run and its output will be available to us in a callback. If we use the <code>spawn()</code> module, its output will be available via event listeners.
If our application expects a lot of output from our commands, we should prefer <code>spawn()</code> over <code>exec()</code>. If not, we might opt to use <code>exec()</code> for its simplicity.
Now that you can run tasks external to Node.js, what applications would you build?

Marcus Sanatan

Arpan Abhishek

With Node, we can run shell commands and process their I/O using JavaScript, instead of the shell scripting language. This makes the application easier to maintain and develop as we stay in the same environment.

Executing Shell Commands with Node.js

Darinka Zobenica

Dimitrije Stamenic

Design Patterns are battle-hardened, well-adopted approaches to solving common problems. In this article, we'll be exploring the Creational Design Patterns in Python.

Creational Design Patterns in Python

<h3 id="introduction">Introduction</h3>
Design Patterns are reusable models for solving known and common problems in software architecture.
They're best described as templates for dealing with a certain usual situation. An architect might have a template for designing certain kinds of door-frames which he fits into many of his projects, and a software engineer, or software architect, should know templates for solving frequent programming challenges.
A good presentation of a design pattern should include:
<ul>
<li>Name</li>
<li>Motivating problem</li>
<li>Solution</li>
<li>Consequences</li>
</ul>
<h3 id="equivalentproblems">Equivalent Problems</h3>
If you were thinking that that's a pretty <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Fuzzy_logic">fuzzy</a> concept, you'd be right. For instance, we could say that the following &quot;pattern&quot; solves all of your problems:
<ol>
<li>Fetch and prepare the necessary data and other resources</li>
<li>Do the calculations needed and perform the necessary work</li>
<li>Make logs of what you're doing</li>
<li>Release all resources</li>
<li>???</li>
<li>Profit</li>
</ol>
This is an example of thinking too abstract. You can't really call this a pattern because it's not really a good model for solving any problem, despite being technically applicable to any of them (including making dinner).
On the other extreme, you can have solutions that are just too concrete to be called a pattern. For instance, you could wonder whether <a target="_blank" href="/quicksort-in-python/">QuickSort</a> is a pattern for solving the sorting problem.
It certainly is a common programming problem, and QuickSort is a good solution for it. However, it can be applied to any sorting problem with little to no modification.
Once you have it in a library and you can call it, your only real job is to make your object comparable somehow, you don't really have to deal with the essence of it yourself to modify it to fit your particular problem.
Equivalent problems are somewhere between these concepts. These are different problems that are sufficiently similar that you can apply the same model to them, but sufficiently different that this model has to be customized considerably to be applicable in each case.
Patterns that could be applied to these sorts of problems are what we can meaningfully dub design patterns.
<h3 id="whyusedesignpatterns">Why use Design Patterns?</h3>
You're probably familiar with some design patterns already through practice of writing code. A lot of good programmers eventually gravitate towards them even without being explicitly taught or they just pick them up from seniors along the way.
Motivations for making, learning, and utilizing design patterns are manifold. They're a way to give names to complex abstract concepts to enable discussion and teaching.
They make communication within teams faster, because someone can just use the pattern's name instead of whipping out a white board. They enable you to learn from the experiences of people who came before you, rather than having to reinvent the wheel by going through the whole crucible of gradually improving practices yourself (and having to constantly cringe at your old code).
Bad solutions that tend to be commonly invented because they seem logical on the first glance are often called <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Anti-pattern">anti-patterns</a>. In order for something to justly be called an anti-pattern it needs to be commonly reinvented and there needs to be a pattern for the same problem which solves it better.
Despite the obvious utility in practice, design patterns are also useful for learning. They introduce you to many problems that you may not have considered and allow you to think about scenarios that you may not have had hands-on experience with in-depth.
They're a must-learn for all, and they're an exceptionally good learning resource for all aspiring architects and developers who may be at the beginning of their careers and lacking the first-hand experience of grappling with various problems the industry provides.
<h3 id="designpatternsinpython">Design Patterns in Python</h3>
Traditionally, design patterns have been classified into three main categories: <a href="#creationaldesignpatterns">Creational</a>, <a href="#structuraldesignpatterns">Structural</a>, and <a href="#behavioraldesignpatterns">Behavioral</a>. There are other categories, like <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Architectural_pattern">architectural</a> or <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Concurrency_pattern">concurrency</a> patterns, but they're beyond the scope of this article.
There are also Python-specific design patterns that are created specifically around the problems that the structure of the language itself provides or that deal with problems in special ways that are only allowed because of the structure of the language.
Creational Design Patterns deal with creation of classes or objects. They serve to abstract away the specifics of classes so that we'd be less dependent on their exact implementation, or so that we wouldn't have to deal with complex construction whenever we need them, or so we'd ensure some special instantiation properties. They're very useful for lowering levels of dependency and controlling how the user interacts with our classes.
Structural Design Patterns deal with assembling objects and classes into larger structures, while keeping those structures flexible and efficient. They tend to be really useful for improving readability and maintainability of the code, ensure functionalities are properly separated, encapsulated, and that there are effective minimal interfaces between interdependent things.
Behavioral Design Patterns deal with algorithms in general, and assignment of responsibility between interacting objects. For example, they're good practices in cases where you may be tempted to implement a naive solution, like <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Busy_waiting">busy waiting</a>, or load your classes with unnecessary code for one specific purpose that isn't the core of their functionality.
<h3 id="creationaldesignpatterns">Creational Design Patterns</h3>
<ul>
<li><a target="_blank" href="/creational-design-patterns-in-python#factorydesignpattern">Factory</a></li>
<li><a target="_blank" href="/creational-design-patterns-in-python#abstractfactorydesignpattern">Abstract Factory</a></li>
<li><a target="_blank" href="/creational-design-patterns-in-python#builderdesignpattern">Builder</a></li>
<li><a target="_blank" href="/creational-design-patterns-in-python#prototypedesignpattern">Prototype</a></li>
<li><a target="_blank" href="/creational-design-patterns-in-python#singletondesignpattern">Singleton</a></li>
<li><a target="_blank" href="/creational-design-patterns-in-python#objectpooldesignpattern">Object Pool</a></li>
</ul>
<h3 id="structuraldesignpatterns">Structural Design Patterns</h3>
<ul>
<li><a target="_blank" href="/structural-design-patterns-in-python/#adapterdesignpattern">Adapter</a></li>
<li><a target="_blank" href="/structural-design-patterns-in-python/#bridgedesignpattern">Bridge</a></li>
<li><a target="_blank" href="/structural-design-patterns-in-python/#compositedesignpattern">Composite</a></li>
<li><a target="_blank" href="/structural-design-patterns-in-python/#decoratordesignpattern">Decorator</a></li>
<li><a target="_blank" href="/structural-design-patterns-in-python/#facadedesignpattern">Facade</a></li>
<li><a target="_blank" href="/structural-design-patterns-in-python/#flyweightdesignpattern">Flyweight</a></li>
<li><a target="_blank" href="/structural-design-patterns-in-python/#proxy">Proxy</a></li>
</ul>
<h3 id="behavioraldesignpatterns">Behavioral Design Patterns</h3>
Coming soon!
<ul>
<li><a target="_blank" href="#">Chain of Responsibility</a></li>
<li><a target="_blank" href="#">Command</a></li>
<li><a target="_blank" href="#">Iterator</a></li>
<li><a target="_blank" href="#">Mediator</a></li>
<li><a target="_blank" href="#">Memento</a></li>
<li><a target="_blank" href="#">Observer</a></li>
<li><a target="_blank" href="#">State</a></li>
<li><a target="_blank" href="#">Strategy</a></li>
<li><a target="_blank" href="#">Visitor</a></li>
</ul>
<h3 id="pythonspecificdesignpatterns">Python-Specific Design Patterns</h3>
Coming soon!
<ul>
<li><a target="_blank" href="#">Global Object Pattern</a></li>
<li><a target="_blank" href="#">Prebound Method Pattern</a></li>
<li><a target="_blank" href="#">Sentinel Object Pattern</a></li>
</ul>

Design Patterns are a really important topic in Software Engineering. In this article, we'll be discussing what they are, what they aren't as well as dive into some of the most important categories of Design Patterns in Python.

Design Patterns in Python

Janith Kasun

Authentication in many cases allows users to personably perform actions on a website. In this article, we'll be handling authentication in Node with Express.js.

Handling Authentication in Express.js

Olivera Popović

Merge Sort is one of the most famous sorting algorithms due to its efficient, general-purpose usage. It's a classic example of a divide-and-conquer algorithm. We'll be implementing it in Python on multiple data types.

Merge Sort in Python

<h2 id="introduction">Introduction</h2>
Have you ever received a PDF or an image file from someone via email, only to see strange characters when you open it? This can happen if your email server was only designed to handle text data. Files with binary data, bytes that represent non-text information like images, can be easily corrupted when being transferred and processed to text-only systems.
Base64 encoding allows us to convert bytes containing binary or text data to ASCII characters. By encoding our data, we improve the chances of it being processed correctly by various systems.
In this tutorial, we would learn how Base64 encoding and decoding works, and how it can be used. We will then use Python to Base64 encode and decode both text and binary data.
<h2 id="whatisbase64encoding">What is Base64 Encoding?</h2>
Base64 encoding is a type of conversion of bytes into ASCII characters. In mathematics, the base of a number system refers to how many different characters represent numbers. The name of this encoding comes directly from the mathematical definition of bases - we have 64 characters that represent numbers.
The Base64 character set contains:
<ul>
<li>26 uppercase letters</li>
<li>26 lowercase letters</li>
<li>10 numbers</li>
<li><code>+</code> and <code>/</code> for new lines (some implementations may use different characters)</li>
</ul>
When the computer converts Base64 characters to binary, each Base64 character represents 6 bits of information.
Note: This is not an encryption algorithm, and should not be used for security purposes.
Now that we know what Base64 encoding is and how it is represented on a computer, let's look deeper into how it works.
<h2 id="howdoesbase64encodingwork">How Does Base64 Encoding Work?</h2>
We will illustrate how Base64 encoding works by converting text data, as it's more standard than the various binary formats to choose from. If we were to Base64 encode a string we would follow these steps:
<ol>
<li>Take the ASCII value of each character in the string</li>
<li>Calculate the 8-bit binary equivalent of the ASCII values</li>
<li>Convert the 8-bit chunks into chunks of 6 bits by simply re-grouping the digits</li>
<li>Convert the 6-bit binary groups to their respective decimal values.</li>
<li>Using a base64 encoding table, assign the respective base64 character for each decimal value.</li>
</ol>
Let's see how it works by converting the string &quot;Python&quot; to a Base64 string.
The ASCII values of the characters <code>P, y, t, h, o, n</code> are <code>15, 50, 45, 33, 40, 39</code> respectively. We can represent these ASCII values in 8-bit binary as follows:
<pre><code class="hljs">01010000 01111001 01110100 01101000 01101111 01101110
</code></pre>
Recall that Base64 characters only represent 6 bits of data. We now re-group the 8-bit binary sequences into chunks of 6 bits. The resultant binary will look like this:
<pre><code class="hljs">010100 000111 100101 110100 011010 000110 111101 101110
</code></pre>
Note: Sometimes we are not able to group the data into sequences of 6 bits. If that occurs, we have to pad the sequence.
With our data in groups of 6 bits, we can obtain the decimal value for each group. Using our last result, we get the following decimal values:
<pre><code class="hljs">20 7 37 52 26 6 61 46
</code></pre>
Finally, we will convert these decimals into the appropriate Base64 character using the Base64 conversion table:
<img src="https://stackabuse.s3.amazonaws.com/media/encoding-and-decoding-base64-strings-python-1.png" alt="Base64 Encoding Table">
As you can see, the value <code>20</code> corresponds to the letter <code>U</code>. Then we look at <code>7</code> and observe it's mapped to <code>H</code>. Continuing this lookup for all decimal values, we can determine that &quot;Python&quot; is represented as <code>UHl0aG9u</code> when Base64 encoded. You can verify this result with an online converter.
To Base64 encode a string, we convert it to binary sequences, then to decimal sequences, and finally, use a lookup table to get a string of ASCII characters. With that deeper understanding of how it works, let's look at why would we Base64 encode our data.
<h2 id="whyusebase64encoding">Why use Base64 Encoding?</h2>
In computers, all data of different types are transmitted as 1s and 0s. However, some communication channels and applications are not able to understand all the bits it receives. This is because the meaning of a sequence of 1s and 0s is dependent on the type of data it represents. For example, <code>10110001</code> must be processed differently if it represents a letter or an image.
To work around this limitation, you can encode your data to text, improving the chances of it being transmitted and processed correctly. Base64 is a popular method to get binary data into ASCII characters, which is widely understood by the majority of networks and applications.
A common real-world scenario where Base64 encoding is heavily used are in mail servers. They were originally built to handle text data, but we also expect them to send images and other media with a message. In those cases, your media data would be Base64 encoded when it is being sent. It will then be Base64 decoded when it is received so an application can use it. So, for example, the image in the HTML might look like this:
<pre><code class="hljs">&lt;img src=&quot;data:image/png;base64,aVRBOw0AKg1mL9...&quot;&gt;
</code></pre>
Understanding that data sometimes needs to be sent as text so it won't be corrupted, let's look at how we can use Python to Base64 encoded and decode data.
<h2 id="encodingstringswithpython">Encoding Strings with Python</h2>
Python 3 provides a <code>base64</code> module that allows us to easily encode and decode information. We first convert the string into a bytes-like object. Once converted, we can use the <code>base64</code> module to encode it.
In a new file <code>encoding_text.py</code>, enter the following:
<pre><code class="hljs">import base64

message = &quot;Python is fun&quot;
message_bytes = message.encode(&#x27;ascii&#x27;)
base64_bytes = base64.b64encode(message_bytes)
base64_message = base64_bytes.decode(&#x27;ascii&#x27;)

print(base64_message)
</code></pre>
In the code above, we first imported the <code>base64</code> module. The <code>message</code> variable stores our input string to be encoded. We convert that to a bytes-like object using the string's <code>encode</code> method and store it in <code>message_bytes</code>. We then Base64 encode <code>message_bytes</code> and store the result in <code>base64_bytes</code> using the <code>base64.b64encode</code> method. We finally get the string representation of the Base64 conversion by decoding the <code>base64_bytes</code> as ASCII.
Note: Be sure to use the same encoding format when converting from string to bytes, and from bytes to string. This prevents data corruption.
Running this file would provide the following output:
<pre><code class="hljs">$ python3 encoding_text.py
UHl0aG9uIGlzIGZ1bg==
</code></pre>
Now let's see how we can decode a Base64 string to its raw representation.
<h2 id="decodingstringswithpython">Decoding Strings with Python</h2>
Decoding a Base64 string is essentially a reverse of the encoding process. We decode the Base64 string into bytes of un-encoded data. We then convert the bytes-like object into a string.
In a new file called <code>decoding_text.py</code>, write the following code:
<pre><code class="hljs">import base64

base64_message = &#x27;UHl0aG9uIGlzIGZ1bg==&#x27;
base64_bytes = base64_message.encode(&#x27;ascii&#x27;)
message_bytes = base64.b64decode(base64_bytes)
message = message_bytes.decode(&#x27;ascii&#x27;)

print(message)
</code></pre>
Once again, we need the <code>base64</code> module imported. We then encode our message into a bytes-like object with <code>encode('ASCII')</code>. We continue by calling the <code>base64.b64decode</code> method to decode the <code>base64_bytes</code> into our <code>message_bytes</code> variable. Finally, we decode <code>message_bytes</code> into a string object <code>message</code>, so it becomes human readable.
Run this file to see the following output:
<pre><code class="hljs">$ python3 decoding_text.py
Python is fun
</code></pre>
Now that we can encode and decode string data, let's try to encode binary data.
<h2 id="encodingbinarydatawithpython">Encoding Binary Data with Python</h2>
As we mentioned previously, Base64 encoding is primarily used to represent binary data as text. In Python, we need to read the binary file, and Base64 encodes its bytes so we can generate its encoded string.
Let's see how we can encode this image:
<img src="https://stackabuse.s3.amazonaws.com/media/encoding-and-decoding-base64-strings-python-2.png" alt="Python logo">
Create a new file <code>encoding_binary.py</code> and add the following:
<pre><code class="hljs">import base64

with open(&#x27;logo.png&#x27;, &#x27;rb&#x27;) as binary_file:
 binary_file_data = binary_file.read()
 base64_encoded_data = base64.b64encode(binary_file_data)
 base64_message = base64_encoded_data.decode(&#x27;utf-8&#x27;)

 print(base64_message)
</code></pre>
Let's go over the code snippet above. We open the file using <code>open('my_image.png', 'rb')</code>. Note how we passed the <code>'rb'</code> argument along with the file path - this tells Python that we are reading a binary file. Without using <code>'rb'</code>, Python would assume we are reading a text file.
We then use the <code>read()</code> method to get all the data in the file into the <code>binary_file_data</code> variable. Similar to how we treated strings, we Base64 encoded the bytes with <code>base64.b64encode</code> and then used the <code>decode('utf-8')</code> on <code>base64_encoded_data</code> to get the Base64 encoded data using human-readable characters.
Executing the code will produce similar output to:
<pre><code class="hljs">$ python3 encoding_binary.py
iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAAsTAAALEwEAmpwYAAAB1klEQVQ4jY2TTUhUURTHf+fy/HrjhNEX2KRGiyIXg8xgSURuokXLxFW0qDTaSQupkHirthK0qF0WQQQR0UCbwCQyw8KCiDbShEYLJQdmpsk3895p4aSv92ass7pcfv/zP+fcc4U6kXKe2pTY3tjSUHjtnFgB0VqchC/SY8/293S23f+6VEj9KKwCoPDNIJdmr598GOZNJKNWTic7tqb27WwNuuwGvVWrAit84fsmMzE1P1+1TiKMVKvYUjdBvzPZXCwXzyhyWNBgVYkgrIow09VJMznpyebWE+Tdn9cEroBSc1JVPS+6moh5Xyjj65vEgBxafGzWetTh+rr1eE/c/TMYg8hlAOvI6JP4KmwLgJ4qD0TIbliTB+sunjkbeLekKsZ6Zc8V027aBRoBRHVoduDiSypmGFG7CrcBEyDHA0ZNfNphC0D6amYa6ANw3YbWD4Pn3oIc+EdL36V3od0A+MaMAXmA8x2Zyn+IQeQeBDfRcUw3B+2PxwZ/EdtTDpCPQLMh9TKx0k3pXipEVlknsf5KoNzGyOe1sz8nvYtTQT6yyvTjIaxsmHGB9pFx4n3jIEfDePQvCIrnn0J4B/gA5J4XcRfu4JZuRAw3C51OtOjM3l2bMb8Br5eXCsT/w/EAAAAASUVORK5CYII=
</code></pre>
Your output may vary depending on the image you've chosen to encode.
Now that we know how to Bas64 encode binary data in Python, let's move on to Base64 decoding binary data.
<h2 id="decodingbinarydatawithpython">Decoding Binary Data with Python</h2>
Base64 decoding binary is similar to Base64 decoding text data. The key difference is that after we Base64 decode the string, we save the data as a binary file instead of a string.
Let's see how to Base64 decode binary data in practice by creating a new file called <code>decoding_binary.py</code>. Type the following code into the Python file:
<pre><code class="hljs">import base64

base64_img = &#x27;iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAACXBIWXMAAAsTAAA&#x27; \
 &#x27;LEwEAmpwYAAAB1klEQVQ4jY2TTUhUURTHf+fy/HrjhNEX2KRGiyIXg8xgSURuokX&#x27; \
 &#x27;LxFW0qDTaSQupkHirthK0qF0WQQQR0UCbwCQyw8KCiDbShEYLJQdmpsk3895p4aS&#x27; \
 &#x27;v92ass7pcfv/zP+fcc4U6kXKe2pTY3tjSUHjtnFgB0VqchC/SY8/293S23f+6VEj&#x27; \
 &#x27;9KKwCoPDNIJdmr598GOZNJKNWTic7tqb27WwNuuwGvVWrAit84fsmMzE1P1+1TiK&#x27; \
 &#x27;MVKvYUjdBvzPZXCwXzyhyWNBgVYkgrIow09VJMznpyebWE+Tdn9cEroBSc1JVPS+&#x27; \
 &#x27;6moh5Xyjj65vEgBxafGzWetTh+rr1eE/c/TMYg8hlAOvI6JP4KmwLgJ4qD0TIbli&#x27; \
 &#x27;TB+sunjkbeLekKsZ6Zc8V027aBRoBRHVoduDiSypmGFG7CrcBEyDHA0ZNfNphC0D&#x27; \
 &#x27;6amYa6ANw3YbWD4Pn3oIc+EdL36V3od0A+MaMAXmA8x2Zyn+IQeQeBDfRcUw3B+2&#x27; \
 &#x27;PxwZ/EdtTDpCPQLMh9TKx0k3pXipEVlknsf5KoNzGyOe1sz8nvYtTQT6yyvTjIax&#x27; \
 &#x27;smHGB9pFx4n3jIEfDePQvCIrnn0J4B/gA5J4XcRfu4JZuRAw3C51OtOjM3l2bMb8&#x27; \
 &#x27;Br5eXCsT/w/EAAAAASUVORK5CYII=&#x27;

base64_img_bytes = base64_img.encode(&#x27;utf-8&#x27;)
with open(&#x27;decoded_image.png&#x27;, &#x27;wb&#x27;) as file_to_save:
 decoded_image_data = base64.decodebytes(base64_img_bytes)
 file_to_save.write(decoded_image_data)
</code></pre>
In the above code, we first convert our Base64 string data into a bytes-like object that can be decoded. When you are base64 decoding a binary file, you must know the type of data that is being decoded. For example, this data is only valid as a PNG file and not a MP3 file as it encodes an image.
Once the destination file is open, we Base64 decode the data with <code>base64.decodebytes</code>, a different method from <code>base64.b64decode</code> that was used with strings. This method should be used to decode binary data. Finally, we write the decoded data to a file.
In the same directory that you executed <code>decoding_binary.py</code>, you would now see a new <code>decoded_image.png</code> file that contains the original image that was encoded earlier.
<h2 id="conclusion">Conclusion</h2>
Base64 encoding is a popular technique to convert data in different binary formats to a string of ASCII characters. This is useful when transmitting data to networks or applications that cannot process raw binary data but would readily handle text.
With Python, we can use the <code>base64</code> module to Base64 encode and decode text and binary data.
What applications would you use to encode and decode Base64 data?

Abhishek Amin

Base64 encoding allows us to convert bytes containing binary or text data to ASCII characters. In this tutorial, we'll be encoding and decoding Base64 Strings in Python.

Encoding and Decoding Base64 Strings in Python

<h2 id="introduction">Introduction</h2>
Most of the user-facing software comes with a visually pleasing interface or via a decorated webpage. At other times, a program can be so small that it does not warrant an entire graphical user interface or web application to expose its functionality to the end-user.
In these cases, we can build programs that are accessible via a Command Line Interface, or CLI.
In this post, we will explore Python's <code>argparse</code> module and use it to build a simple command-line tool to help us shorten URLs swiftly.
<h2 id="commandlineinterface">Command Line Interface</h2>
A Command Line Interface is a text-based user interface that provides a means of interacting with a computer through textual commands. The program that facilitates this interaction by exposing the interface is known as a command-line interpreter or a shell.
It takes in commands in the form of text input, executes programs based on the input provided, then displays the output on the interface. There are many shells available, with the first popular ones being the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Bourne_shell">Bourne shell</a> and <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/C_shell">C shell</a> for Unix-based systems. The <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Bash_(Unix_shell)">Bourne Again Shell</a> (aka bash) is a hugely popular extension of the Bourne Shell, alongside the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/KornShell">KornShell</a> (<code>ksh</code>).
It is also worth noting that CLIs, like other software, require user accounts to work with them. The system enforces permissions on these users to help control the level of access and limit of what a user can achieve with the system. This distinction is required since the shell is an interface between the user and the operating system kernel that controls all computer operations. Access to this functionality needs to be restricted to prevent malicious use of the command-line interface.
CLIs offer a prompt, which is usually a dollar sign (<code>$</code>) that indicates that you can enter your command. However, this prompt also indicates that the command entered will be performed without root access.
When root access is granted to the current user interacting with the CLI, the prompt changes to a hash sign (<code>#</code>).
While Graphical User Interfaces (GUIs) are easier to learn and more visually intuitive, CLIs allow users to interact with software using only a keyboard, which can result in faster performance. CLIs also use up fewer computer resources as compared to GUIs, making them lighter and faster.
<h2 id="scripting">Scripting</h2>
Most command-line interpreters ship with some basic commands that can all be invoked through the command-line interface to perform specific tasks. Some common ones include:
<ul>
<li><code>uptime</code>: Indicates how long the computer has been on</li>
<li><code>date</code>: Returns the current date and time</li>
<li><code>ls</code>: Returns a list of all the files and folders in a directory</li>
<li><code>cd</code>: Used to move from one directory to another</li>
<li><code>pwd</code>: Used to show the current working directory</li>
<li><code>man</code>: Used to show the manual or instructions of any command</li>
<li><code>touch</code>: Used to create new, empty files</li>
<li><code>mkdir</code>: Used to create new directories</li>
</ul>
A shell script is a program designed to be executed by a command-line interpreter. It contains a series of commands, like the ones listed above, coupled with variables and conditions instructing the shell on what task or tasks to undertake.
Through shell scripts, a user can perform multiple commands in quick succession without the need to remember them all. They are mostly used to achieve repetitive operations without the repetitive entry of commands, therefore, reducing the effort required by the end-user.
We can write shell scripts that consist of the shell commands to be executed, but also, we can execute other high-level languages, such as Python and JavaScript.
<h2 id="whatisargparse">What is argparse?</h2>
A Python shell script is just a normal Python program that is executed by the command-line interpreter. When executing a shell script, arguments are passed into our script through <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/sys.html#sys.argv">sys.argv</a>. This variable is a list of the arguments passed to our program, including the script name, which is also the first argument.
Normally, we can write a simple script that does not require any extra arguments, like a simple script to display the current date. This will, however, limit the functionality we can offer. To make our scripts more versatile and widen the scope of their usage, we have to facilitate customization through arguments that give the user more control and options in terms of functionality.
The <code>argparse</code> module helps us parse the arguments passed with our script and process them in a more convenient way. It also adds customization features such as naming our program and adding descriptions in a simpler way.
<code>argparse</code> also provides a means for us to customize the usage instructions for our script and indicate which arguments are required and which are optional. To explore all these features and more, we'll build our own Python CLI utility in the next section.
<h2 id="demoapplication">Demo Application</h2>
Currently if we want to shorten a URL, we will need to fire up a browser and navigate to a URL shortening site in order to do the task. Our goal is to accelerate and enhance this URL shortening process through a script that we can fire up anytime on our terminal. We will only have to pass the URLs we need to shorten as arguments and receive the shortened URLs in response.
For this demo, we'll use <a rel="nofollow" target="_blank" href="https://shorte.st/">Shorte.st</a> as our provider since its API is simple and straightforward.
After creating an account, we can head over to the Link Tools section and select Developers API. Here we will find our access token and the URL which our script will use to shorten our URLs.
Once the user provides a URL to our command line utility to be shortened, we will send the URL to the Shorte.st API endpoint along with our access token. The response will be our shortened URL and a status message.
Let us start by creating a virtual environment and installing the <a target="_blank" href="/the-python-requests-module/">requests module</a>, which we'll use to send HTTP requests to the API:
<pre><code class="hljs">$ mkdir pyshortener &amp;&amp; cd pyshortener
$ virtualenv --python=python3 env --no-site-packages
$ source env/bin/activate
$ pip install requests
</code></pre>
In the first line above we have actually combined two commands in one using the double ampersand (<code>&amp;&amp;</code>). This allows us to execute the commands in sequence, unless the first command fails, which then prevents the second command from being executed.
After creating our virtual environment and activating it, we then install our Python dependency.
For this demo we will first build our shortening function and then wrap its functionality using <code>argparse</code> into the final script:
<pre><code class="hljs">import requests
from requests.exceptions import ConnectionError
import json

def shorten_url(url):
 try:
 response = requests.put(&quot;https://api.shorte.st/v1/data/url&quot;,
 {&quot;urlToShorten&quot;: url},
 headers={&quot;public-api-token&quot;: &quot;[MY-API-TOKEN]&quot;})

 api_response = json.loads(response.content)

 return {&quot;status&quot;: api_response[&#x27;status&#x27;],
 &quot;shortenedUrl&quot;: api_response[&#x27;shortenedUrl&#x27;],
 &quot;message&quot;: &quot;URL shortened successfully&quot;}

 except ConnectionError:
 return {&quot;status&quot;: &quot;error&quot;,
 &quot;shortenedUrl&quot;: None,
 &quot;message&quot;: &quot;Please ensure you are connected to the internet and try again.&quot;}

shorten_url(www.stackabuse.com)
</code></pre>
Our function takes in a URL and sends it to the Shorte.st API and returns the shortened URL. Let's shorten <code>www.stackabuse.com</code> by executing our script:
<pre><code class="hljs">$ python pyshortener.py
{&#x27;status&#x27;: &#x27;ok&#x27;, &#x27;shortenedUrl&#x27;: &#x27;http://gestyy.com/w6ph2J&#x27;, &#x27;message&#x27;: &#x27;URL shortened successfully&#x27;}
</code></pre>
As we can see, our function works but the output is less than ideal. We also have to hardcode the URL in the script itself, which gives us a fixed input and output.
We are going to take this a step further and allow users to pass in the URL as an argument when they execute the script. In order to do this we will now introduce <code>argparse</code> to help us parse the arguments provided by the user:
<pre><code class="hljs">import requests
from requests.exceptions import ConnectionError
import json
import argparse # Add the argparse import

def shorten_url(url):
 # Code stays the same...

# Create a parser
parser = argparse.ArgumentParser(description=&#x27;Shorten URLs on the terminal&#x27;)

# Add argument
parser.add_argument(&#x27;--url&#x27;, default=&quot;google.com&quot;, help=&quot;The URL to be shortened&quot;)
args = vars(parser.parse_args())
print(args)
</code></pre>
In this version of the script, we don't call <code>shorten_url</code>, but instead just print out the arguments captured and parsed by <code>argparse</code>.
We start by creating an <code>ArgumentParser</code> object using <code>argparse</code>, which will hold all the info required to transform the arguments passed into Python data types we can work with.
After creating the parser, we can now add arguments using <code>parser.add_argument()</code>. This function allows to specify the following information about our arguments:
<ul>
<li>The first argument is a name or a flag used to identify our arguments. Optional arguments are identified by the <code>-</code> prefix, in our case <code>--url</code> is an optional argument.</li>
<li>The <code>default</code> option allows specifying a default value when the user has not provided the argument.</li>
<li>The <code>help</code> option briefly describes what the argument is.</li>
<li>We can also use the <code>choice</code> option to specify allowable values for an argument, such as yes and no.</li>
<li>Through a <code>type</code> option we can also specify the type to which our command will be converted to, for instance converting arguments into integers.</li>
</ul>
When we run our script without providing any arguments, the URL defaults to &quot;google.com&quot;, just like we've set in the <code>add_argument</code> method:
<pre><code class="hljs">$ python pyshortener.py
{&#x27;url&#x27;: &#x27;google.com&#x27;}
</code></pre>
Now when we pass <code>www.stackabuse.com</code> using the <code>--url</code> flag, it is set as the value to the <code>url</code> key:
<pre><code class="hljs">$ python pyshortener.py --url www.stackabuse.com
{&#x27;url&#x27;: &#x27;www.stackabuse.com&#x27;}
</code></pre>
We can now receive a user's URL via the command line and shorten it by modifying our script:
<pre><code class="hljs">if args.get(&#x27;url&#x27;):
 print(shorten_url(args[&#x27;url&#x27;]))
</code></pre>
When we run it and pass a URL we should then receive the output from the Shorte.st API:
<pre><code class="hljs">$ python pyshortener.py --url stackabuse.com
{&#x27;status&#x27;: &#x27;ok&#x27;, &#x27;shortenedUrl&#x27;: &#x27;http://gestyy.com/w6pk2R&#x27;, &#x27;message&#x27;: &#x27;URL shortened successfully&#x27;}
</code></pre>
Although our output is not as friendly as we'd want it too, so let us create a function to format our output in a more desirable manner:
<pre><code class="hljs">def handle_output(result):
 &quot;&quot;&quot; Function to format and print the output
 &quot;&quot;&quot;
 if result[&quot;status&quot;] == &quot;ok&quot;:
 print(f&quot;{result[&#x27;message&#x27;]}. Your shortened URL is:\n&quot;
 f&quot;\t{result[&#x27;shortenedUrl&#x27;]}&quot;)
 elif result[&quot;status&quot;] == &quot;error&quot;:
 print(f&quot;{result[&#x27;message&#x27;]}&quot;)

# Receive and process the argument
args = vars(parser.parse_args())

if args.get(&#x27;url&#x27;):
 result = shorten_url(args[&#x27;url&#x27;])
 handle_output(result)
</code></pre>
When we run our script one more time:
<pre><code class="hljs">$ python pyshortener.py --url www.stackabuse.com
URL shortened successfully. Your shortened URL is:
 http://gestyy.com/w6pk2R
</code></pre>
Our output is now more user-friendly due to the addition of the <code>handle_output()</code> function. To see how <code>argparse</code> has generated the help text for our script, we can execute our script with the <code>-h</code> flag to display the help text as follows:
<pre><code class="hljs">$ python pyshortener.py -h
usage: pyshortener.py [-h] [--url URL]

Shorten URLs on the terminal

optional arguments:
 -h, --help show this help message and exit
 --url URL The URL to be shortened
</code></pre>
<h2 id="conclusion">Conclusion</h2>
We have built a shell script using Python to help us shorten URLs quickly on the terminal. We have used the <code>argparse</code> module to parse the arguments that are passed into our script and even defined a default value in case the argument is provided.
Our script now also has a beautiful help message that can be displayed using the <code>-h</code> flag which was generated by the <code>argparse</code> module meaning we did not have to write it down manually.
The next step would be to enhance our script to accept a list of URLs or read URLs from a text file to facilitate batch shortening of URLs.
The source code for the script in this project can be found <a rel="nofollow" target="_blank" href="https://github.com/ro6ley/PyShortener">here on GitHub</a>.

Robley Gori

At times, apps are small enough that they really don't need a GUI. In such cases, a Command Line Interface is sufficient. In this article, we'll be using the argparse module to create some CLI apps in Python.