<h2 id="introduction">Introduction</h2>
For web developers, validating user inputs in various types of forms is of crucial importance. Since that is the starting point of data being sent between the client and the server, you need to make sure that everything starts off on the right foot - lest you end up with robust validation on the server end, which is oftentimes a bigger hassle than doing it on the front-end.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Additionally, input can be malicious - in which case, you have to take security into consideration as well. It's better to avoid it altogether by validating input on the front-end.

 </div>
 </div>
 </div>
 <blockquote>
In this article, we'll take a look at how to validate email addresses in JavaScript using Regular Expressions.
</blockquote>
<h2 id="regularexpressionsinjavascript">Regular Expressions in JavaScript</h2>
For anyone unfamiliar with regular expressions, or anyone feeling like they need a quick reminder, here it is!
<blockquote>
Regular expressions are sequences of metacharacters, denoting a pattern. These patterns can be of various kinds: a mix of letters with digits, special characters and even different language characters. An abbreviation for regular expression is RegEx or RegExp.
</blockquote>
Through metacharacters, quantifiers, groups and escape characters - you can express just about any pattern. For instance, this expression denotes a sequence of characters that contains any valid letter between A-Z (both lowercase and uppercase) or digits, in any combination:
<pre><code class="hljs">^([A-Za-z]|[0-9])+$
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 This is also known as checking whether a sequence is alphanumeric.

 </div>
 </div>
 </div>
 For the rest of the guide, we will assume that you are somewhat familiar with Regular Expressions.
<blockquote>
If you aren't familiar with Regular Expressions and would like to learn more - read our <a target="_blank" href="https://stackabuse.com/guide-to-regular-expressions-and-matching-strings-in-javascript/">Guide to Regular Expressions and Matching Strings in JavaScript</a>!
</blockquote>
<h2 id="matchingemailformatsinjavascriptwithregularexpressions">Matching Email Formats in JavaScript with Regular Expressions</h2>
First and foremost, a regular expression that matches all the possible valid email addresses doesn't exist. However, the one that matches 99.9%, does. When validating emails, or really any input, a good practice, which can more or less guarantee that the user will match the RegEx, is to limit the user input upfront.
For example, mandatory usage of gmail.com or yahoo.com and straight-up rejecting the non-supported providers (though, you do run into the issue of scalability and staying up-to-date with this approach).
Another question is raised:
<blockquote>
What is the format of an email?
</blockquote>
It's surprisingly a loose definition, as we'll shortly see - and you can go simple or robust on this. We'll cover the most general regular expressions for validating email, as well as those which are more specific in the guide.
Before we get into the code, let's preview the email formats that we will be looking into:
<ul>
<li>General format - (something)@(some_domain).(some_toplevel_domain)</li>
<li>Specific hosts or domains - referring to a specific type of domain, or top-level domain</li>
<li>RFC 5322 - Internet Message Format, covering 99.9% of email addresses</li>
</ul>
<h2 id="generalemailformatregularexpression">General Email Format Regular Expression</h2>
After many attempts at validating with robust Regular Expressions, many engineers fall back to the good old &quot;general&quot; format that works most of the time. Whether this is a good thing or not is debatable.
What does an email address entail? It has to have a <code>@</code> symbol, as well as some string preceding it, and some string proceeding it. Additionally, the second string needs to contain a dot, which has an additional 2-3 characters after that.
In conclusion, this is a rough sketch:
<pre><code class="hljs">(randomString)@(randomString2).(2-3 characters)
</code></pre>
This follows the general intuition of these emails being valid:
<pre><code class="hljs">someone@gmail.com
john.doe.1@yahoo.com
david@davitech.io
</code></pre>
With that in mind, to generally validate an email address in JavaScript via Regular Expressions, we translate the rough sketch into a <code>RegExp</code>:
<pre><code class="hljs">let regex = new RegExp(&#x27;[a-z0-9]+@[a-z]+\.[a-z]{2,3}&#x27;);

let testEmails = [&quot;notanemail.com&quot;, &quot;workingexample@email.com&quot;, &quot;another_working@somethingelse.org&quot;, &quot;notworking@1.com&quot;];

testEmails.forEach((address) =&gt; {
 console.log(regex.test(address))
});
</code></pre>
The first string can contain any lowercase alphanumeric characters - <code>john.doe.1</code>, <code>workingemail</code>, etc.
This results in:
<pre><code class="hljs">false
true
true
false
</code></pre>
Will this always work? No. There will be some malformed emails that pass-through. You also can't perform spam-detection using this Regular Expression so an email address that intuitively looks like a spam passes this expression just fine:
<pre><code class="hljs">console.log(regex.test(&quot;aaa@aa.aa&quot;)); // true
</code></pre>
Though, even the most robust, complex email address validation expressions fail on this - they're there to validate the form, not whether the email exists.
<blockquote>
Technically speaking, <code>aaa@aa.aa</code> could exist, so who are we to say it doesn't based on just a Regular Expression?
</blockquote>
<h2 id="specificemailaddresses">Specific Email Addresses</h2>
Lowering the amount of uncertainty helps. The less uncertainty there is, the less restrictions you need to impose using an expression. This makes specific email address validation more accurate using the same general formats as we've just seen - you don't have to cover as many edge cases.
Let's take a look at some general cases referring to the domain and top-level domain.
<h3 id="validatinganemailaddressdomainwithjavascript">Validating an Email Address Domain with JavaScript</h3>
Say, you work in a company called Stack Abuse. All the staff have an email ending with <code>@stackabuse.com</code> and the user string is the one changing. The rough sketch would look like this:
<pre><code class="hljs">(randomString)@stackabuse.com
</code></pre>
This makes our task a lot easier, as some of the variables such as the domain name and organization type are now fixed. These two are the typical problem-causing variables, as domain names can vary wildly.
To validate an email address pertaining to a specific domain thus becomes an easy task, utilizing the <code>RegExp</code> class:
<pre><code class="hljs">let regex = new RegExp(&#x27;[a-z0-9]+@stackabuse.com&#x27;);

let testEmails = [&quot;notanemail.com&quot;, &quot;workingexample@stackabuse.com&quot;, &quot;not_working@stackabuse.org&quot;];

testEmails.forEach((address) =&gt; {
 console.log(regex.test(address))
});
</code></pre>
This results in:
<pre><code class="hljs">false
true
false
</code></pre>
With this approach, you can change any literal string to match it according to your needs. As always, the first part of the regular expression can be changed to match cases with uppercase letters, including special characters such as <code>+</code> or <code>_</code>, etc.
<h3 id="validatingemailaddresstopleveldomainsinjavascript">Validating Email Address Top-Level Domains in JavaScript</h3>
This case is pretty similar to the previous one, except that we will be limiting the last two or three characters of the email. 
These can be literally any of: .com, .org, .edu, .eu, .us, etc. Let's match only emails containing <code>.edu</code> since it is never solely this top-level domain, but instead something like <code>example@yale.edu.com</code>.
<pre><code class="hljs">let regex = new RegExp(&#x27;[a-z0-9]+@[a-z]+\.edu\.[a-z]{2,3}&#x27;);

let testEmails = [&quot;notanemail.com&quot;, &quot;someone@stackabuse.com&quot;, &quot;example@yale.edu.com&quot;];

testEmails.forEach((address) =&gt; {
 console.log(regex.test(address))
});
</code></pre>
An invalid email, as well as a valid one fail - because they don't contain an <code>edu</code> in their top-level domain, though, the made-up Yale address works:
<pre><code class="hljs">false
false
true
</code></pre>
<h2 id="rfc5322format">RFC 5322 Format</h2>
The RFC 5322 Format is an Internet Message Format (classic format of an email message). The RFC 5322 only dictates what should be allowed - it isn't an expression itself.
There are multiple expressions that implement the rules laid out, and these can get pretty complex.
<blockquote>
If implemented correctly, the RFC 5322-compliant Regular Expression should validate 99.99% of the valid email addresses.
</blockquote>
A short-hand version is:
<pre><code class="hljs">let regex = new RegExp(&quot;([!#-&#x27;*+/-9=?A-Z^-~-]+(\.[!#-&#x27;*+/-9=?A-Z^-~-]+)*|\&quot;\(\[\]!#-[^-~ \t]|(\\[\t -~]))+\&quot;)@([!#-&#x27;*+/-9=?A-Z^-~-]+(\.[!#-&#x27;*+/-9=?A-Z^-~-]+)*|\[[\t -Z^-~]*])&quot;);
</code></pre>
While an extended version that covers additional edge-cases is:
<pre><code class="hljs">(?:[a-z0-9!#$%&amp;&#x27;*+/=?^_`{|}~-]+(?:\.[a-z0-9!#$%&amp;&#x27;*+/=?^_`{|}~-]+)*|&quot;(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21\x23-\x5b\x5d-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])*&quot;)@(?:(?:[a-z0-9](?:[a-z0-9-]*[a-z0-9])?\.)+[a-z0-9](?:[a-z0-9-]*[a-z0-9])?|\[(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?|[a-z0-9-]*[a-z0-9]:(?:[\x01-\x08\x0b\x0c\x0e-\x1f\x21-\x5a\x53-\x7f]|\\[\x01-\x09\x0b\x0c\x0e-\x7f])+)\])
</code></pre>
These expressions aren't particularly easy to comprehend unless you break them down into groups and spend some time reading through them. Though, an easier way is to visualize it:
<img src="https://s3.stackabuse.com/media/articles/java-regular-expressions-how-to-validate-emails.png" alt="">
*Image and claim of accuracy are courtesy of <a target="_blank" rel="nofollow noopener noreferrer" href="http://emailregex.com/">EmailRegex.com</a>.
That being said, let's use this expression instead to validate a couple of addresses:
<pre><code class="hljs">let regex = new RegExp(&quot;([!#-&#x27;*+/-9=?A-Z^-~-]+(\.[!#-&#x27;*+/-9=?A-Z^-~-]+)*|\&quot;\(\[\]!#-[^-~ \t]|(\\[\t -~]))+\&quot;)@([!#-&#x27;*+/-9=?A-Z^-~-]+(\.[!#-&#x27;*+/-9=?A-Z^-~-]+)*|\[[\t -Z^-~]*])&quot;);

let testEmails = [&quot;notanemail.com&quot;, &quot;workingexample@stackabuse.com&quot;, &quot;example@yale.edu.com&quot;];

testEmails.forEach((address) =&gt; {
 console.log(regex.test(address))
});
</code></pre>
This results in:
<pre><code class="hljs">false
true
true
</code></pre>
You can go ahead and test this expression interactively through a beautiful interface at <a target="_blank" rel="nofollow noopener noreferrer" target="_regex101.com">regex101</a>.
<h2 id="conclusion">Conclusion</h2>
In conclusion, there is really not a single &quot;proper&quot; way to validate email addresses using regular expressions. However, there is a wrong way - if you don't cover the cases that shouldn't be correct.
For those who want to make sure that literally, almost everything is covered - use the RFC 5322 Format.

David Landup

Lazar Ristic

In this guide, we'll take a look at how to validate an email address in JavaScript, using Regular Expressions (RegEx), through practical examples, covering all edge cases.

Validate Email Addresses with Regular Expressions in JavaScript

<h2 id="introduction">Introduction</h2>
<blockquote>
In this article, we'll be taking a look at how to round a number to an integer (whole number) in JavaScript.
</blockquote>
JavaScript provides three different methods to achieve this goal: the <code>round()</code> method, the <code>ceil()</code> method and the <code>floor()</code> method.
All of these are static methods of the <code>Math</code> object. This means that they will be called directly via <code>Math.method_name()</code>, not as a method of an instance of <code>Math</code> object - <code>Math</code> has no constructor.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: These methods can only be called on numbers, everything else will result in a <code>NaN</code> value. The only exception is when it's called on <code>null</code> - calling methods on <code>null</code> will always return 0.

 </div>
 </div>
 </div>
 <h2 id="theceilmethod">The ceil() Method</h2>
The name of this method is actually an abbreviation of the word ceiling - so its name automatically rings a bell that it rounds a number to the closest integer value larger than the current number.
If the number is an integer already - there's nothing to round, so it'll simply return the integer instead:
<pre><code class="hljs">let x = 4.7
console.log(Math.ceil(x))
// Output: 5

let y = -3.2
console.log(Math.ceil(y))
// Output: -3

let z = &quot;something not a number&quot;
console.log(Math.ceil(z))
// Output: NaN

console.log(Math.ceil(null))
// Output: 0
</code></pre>
You'll use <code>ceil()</code> when you specifically want to round to the next upper bound. <code>5.1</code> is rounded to <code>6</code>, so if you want to round to the closest integer, you'll use the <code>round()</code> function, covered in a later section of this guide.
<h2 id="thefloormethod">The floor() Method</h2>
Similar to <code>ceil()</code>, <code>floor()</code>'s method is also carefully chosen to fit its purpose. It rounds the integer to the closest integer smaller than the current one:
<pre><code class="hljs">let x = 4.7
console.log(Math.floor(x))
// Output: 4

let y = -3.2
console.log(Math.floor(y))
// Output: -4

let z = &quot;something not a number&quot;
console.log(Math.floor(z))
// Output: NaN

console.log(Math.floor(null))
// Output: 0
</code></pre>
<h2 id="theroundmethod">The round() Method</h2>
The <code>round()</code> method can be viewed as a two-in-one method, containing both <code>ceil()</code> and <code>floor()</code>. It rounds the number to the closest integer - it either &quot;ceils&quot; the number, or &quot;floors&quot; it, based on its value:
<pre><code class="hljs">let x = 4.7
console.log(Math.round(x))
// Output: 5

let y = 4.2 
console.log(Math.round(y))
// Output: 4

let z = 4.5
console.log(Math.round(z))
// Output: 5

console.log(Math.round(null))
// Output: 0
</code></pre>
Everything up to x.49 will be rounded down to the lower value, while everything higher than that will be rounded to the higher value.
<h2 id="conclusion">Conclusion</h2>
In this quick article, we had a look at some of the methods that can be used to round a non-integer in JavaScript. Some key notes to take away from this article are:
<ul>
<li>Math.ceil() - rounds the number to a higher value</li>
<li>Math.floor() - rounds the number to a lower value</li>
<li>Math.round() - rounds the number to either lower or higher value, depending on the number</li>
<li>When called with <code>null</code> as an argument, the methods always return 0</li>
<li>When called with anything but a number or null as an argument, the methods always return <code>NaN</code></li>
</ul>

In this guide, we'll take a look at how to round a number to an integer (whole number) in JavaScript, using ceil(), floor() and round(), with practical examples.

Rounding Numbers in JavaScript using ceil(), floor() and round()

<h2 id="introduction">Introduction</h2>
Scikit-Learn is one of the most widely-used Machine Learning library in Python. It's optimized and efficient - and its high-level API is simple and easy to use.
Scikit-Learn has a plethora of convenience tools and methods that make preprocessing, evaluating and other painstaking processes as easy as calling a single method - and splitting data between a training and testing set is no exception.
Generally speaking, the rule-of-thumb for splitting data is 80/20 - where 80% of the data is used for training a model, while 20% is used for testing it. This depends on the dataset you're working with, but an 80/20 split is very common and would get you through most datasets just fine.
In this guide - we'll take a look at how to use the <code>split_train_test()</code> method in Scikit-Learn, and how to configure the parameters so that you have control over the splitting process.
<h2 id="installingscikitlearn">Installing Scikit-Learn</h2>
Assuming it isn't already installed - Scikit-Learn can easily be installed via <code>pip</code>:
<pre><code class="hljs">$ pip install scikit-learn
</code></pre>
Once installed, you can import the library itself via:
<pre><code class="hljs">import sklearn
</code></pre>
In most cases, people avoid importing the entire library, as it's pretty vast, and import specific classes or modules that they'll be using specifically.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: This tends to mean that people have a hefty import list when using Scikit-Learn.

 </div>
 </div>
 </div>
 <h2 id="importanceoftrainingandtestingsets">Importance of Training and Testing Sets</h2>
The most common procedure when training a (basic) model in Machine Learning follows the same rough outline:
<ul>
<li>Acquiring and processing data which we'll feed into a model.</li>
</ul>
Scikit-Learn has various datasets to be loaded and used for training the model (iris, diabetes, digits...), mainly for benchmarking/learning.
<ul>
<li>Splitting sets into training and test sets</li>
<li>Building a model and defining the architecture</li>
<li>Compiling the model</li>
<li>Training the model</li>
<li>Verifying the results</li>
</ul>
The training set is a subset of the whole dataset and we generally don't train a model on the entirety of the data. In non-generative models, a training set usually contains around 80% of the main dataset's data. As the name implies, it is used for training the model. This procedure is also referred to as fitting the model.
There are exceptions to this rule, though.
For instance, when training Generative Adversarial Networks (GANs) that generate images - how do you test the results? They're highly subjective in some cases, as they represent new instances that were never seen before. In most generative models, at least as of now, a human is typically required to judge the outputs, in which cases, a test set is totally redundant.
Additionally, sometimes you need more or less than 20% for testing, and if you're using techniques such as cross-validation, you might want to have a tiny bit less testing data so as to not &quot;take away&quot; too much from the training data. For instance, if you have 1.000.000 instances in a dataset, holding out just 5% for a testing set amounts to 50.000 instances, which is most likely more than enough for any model to be tested on.
The test set is a subset of the whole dataset, and is used to evaluate the model and check how well it learned from the training set.
<blockquote>
The model mustn't interact or see the test set before evaluating. The data must be unknown when first evaluating, otherwise it's not really testing the model.
</blockquote>
<h2 id="whataboutvalidationsets">What About Validation Sets?</h2>
Validation sets are a common sight in professional and academic models. Validation sets are taken out of the training set, and used during training to validate the model's accuracy approximately.
The testing set is fully disconnected until the model is finished training - but the validation set is used to validate it during training.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: The validation set isn't used for training, and the model doesn't train on the data. It just validates the current epoch. This way - it indirectly trains on the data, as it does affect its prior beliefs, so the validation set can't be used for testing.

 </div>
 </div>
 </div>
 Similar to how you'll learn more about your own knowledge if you hear it's incorrect - even if you don't know why. This is why validation sets approximate a model’s accuracy, and testing sets are still required even when using a validation set.
They help with approximating a model's actual performance during training, so you don't end up with an illusory overfit model without realizing it after testing it via a test set. You can also use validation sets to tune models, and approximately evaluate their ability without exposing them to a testing set.
Deep Learning frameworks such as Keras can display a <code>val_accuracy</code> besides your regular training <code>accuracy</code> as a good sign of overfitting. If they start diverging, your model is overfitting during training, and you don't need to waste time training it further when they diverge enough. Additionally, callbacks such as <code>EarlyStopping</code> can be used to automatically stop a model's training if the <code>val_accuracy</code> doesn't improve after <code>n</code> epochs, even if the <code>accuracy</code> is increasing.
<blockquote>
Creating a validation set is easy.
</blockquote>
You can, quite literally, just run the <code>train_test_split()</code> method on the training set, which was already split by the <code>train_test_split()</code> method and extract a validation set from it. Alternatively, you can manually create a validation set.
The validation set size is typically split similar to a testing set - anywhere between 10-20% of the training set is typical. For huge datasets, you can do much lower than this, but for small datasets, you can take out too much, making it hard for the model to fit the data in the training set.
<blockquote>
In the proceeding sections, we'll also take out a validation set using the same <code>train_test_split()</code> method.
</blockquote>
<h2 id="scikitlearnsdatasetsmodule">Scikit-Learn's datasets Module</h2>
Several clean and popular datasets are available built-into Scikit-Learn, typically used during learning and for benchmarking models on simple tasks.
If you've ever read resources regarding Machine Learning in Python - you've probably seen some of these most popular datasets:
<ul>
<li>Iris - set of 3 classes (flowers), with 50 samples per class, used for classification.</li>
<li>Diabetes - set with a total of 442 samples, used for regression.</li>
<li>Digits - set of 10 classes (hand-written digits), with ~180 samples per class, used for classification.</li>
<li>Wine - set of 3 classes (of wine), with total of 178 samples, used for classification.</li>
</ul>
Each of these datasets can be loaded in through the <code>datasets</code> module with their respective function:
<pre><code class="hljs">from sklearn import datasets

X_iris, y_iris = datasets.load_iris(return_X_y=True)
X_diabetes, y_diabetes = datasets.load_diabetes(return_X_y=True)
X_digits, y_digits = datasets.load_digits(return_X_y=True)
X_wine, y_wine = datasets.load_wine(return_X_y=True)
</code></pre>
Alternatively, you can load in the specific functions instead:
<pre><code class="hljs">from sklearn.datasets import load_iris
from sklearn.datasets import load_diabetes
from sklearn.datasets import load_digits
from sklearn.datasets import load_wine

X_iris, y_iris = load_iris(return_X_y=True)
X_diabetes, y_diabetes = load_diabetes(return_X_y=True)
X_digits, y_digits = load_digits(return_X_y=True)
X_wine, y_wine = load_wine(return_X_y=True)
</code></pre>
By default, these methods return a <code>Bunch</code> object, containing the data and the targets (data and their classes), however, if you set the <code>return_X_y</code> argument to <code>True</code>, a tuple of <code>(data, targets)</code> is returned, denoting the data you'll be training on and the target classes you want your classifier/regression model to hit.
<h2 id="splittingadatasetwithtrain_test_split">Splitting a Dataset with train_test_split()</h2>
The <code>train_test_split()</code> method resides in the <code>sklearn.model_selection</code> module:
<pre><code class="hljs">from sklearn.model_selection import train_test_split
</code></pre>
There are a couple of arguments we can set while working with this method - and the default is very sensible and performs a 75/25 split. In practice, all of Scikit-Learn's default values are fairly reasonable and set to serve well for most tasks. However, it's worth noting what these defaults are, in the cases they don't work that well.
The main two arguments are <code>train_size</code> and <code>test_size</code>, where their values range between <code>0</code> and <code>1</code> and their sum has to be <code>1</code>. Their values denote the percentage proportion of the dataset, so even if you define just one, such as <code>train_size</code>, the <code>test_size</code> is equal to <code>1 - train_size</code>, and vice versa.
<h3 id="settingthetrain_sizeandtest_sizearguments">Setting the train_size and test_size Arguments</h3>
This is the most common approach, which, leaves us with 4 subsets - <code>X_train</code>, <code>X_test</code>, <code>y_train</code> and <code>y_test</code>:
<pre><code class="hljs">from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(X, y)

print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)
</code></pre>
Without setting either <code>train_size</code> or <code>test_size</code> the default values kick in, setting the <code>test_size</code> to <code>0.25</code>, and a complementary (<code>0.75</code>) <code>train_size</code>:
<pre><code class="hljs">(112, 4)
(38, 4)
(112,)
(38,)
</code></pre>
As you can see, train and test sets are split 75%/25%, as there are 112 instances in the <code>X_train</code> set, and 38 instances in the <code>X_test</code> set.
Some other split proportions are: 80%/20% (very common), 67%/33% and more rarely 50%/50%.
Setting any of these boils down to defining either one or both of the arguments in the <code>train_test_split()</code> method:
<pre><code class="hljs">X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8)
X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8, test_size=0.2)
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2)
</code></pre>
All three of these splits would result in the same split of:
<pre><code class="hljs">(120, 4)
(30, 4)
(120,)
(30,)
</code></pre>
<h3 id="creatingavalidationsetwithtrain_test_split">Creating a Validation Set with train_test_split()</h3>
Validation sets are really useful during training and make your life as a Data Scientist significantly easier.
<blockquote>
Whenever possible, try to use a validation set.
</blockquote>
There is no built-in function to extract a validation set from a training set, however, since this boils down to just splitting it like before - why not use the same <code>train_test_split()</code> method?
Let's re-use it to get our hands on a validation set, taking 10% of the data from the training set:
<pre><code class="hljs">from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, train_size=0.8)

X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, train_size=0.9)

print(X_train.shape)
print(X_test.shape)
print(X_valid.shape)
</code></pre>
This won't create a 70%-20%-10% split, as we're splitting 10% from the already split <code>X_train</code> so we're actually ending up with a 72%-20%-8% split here:
<pre><code class="hljs">(108, 4)
(30, 4)
(12, 4)
</code></pre>
To account for this, you can either manually set a different number, expecting this, or you could define your proportions upfront, and calculate an updated split to reference the original size, instead of the already truncated size.
To split the data proportionally into a training, testing and validation set - we need to set the <code>test_size</code> argument on the second function call to:
$$ 
test_s = validation_r/(train_r+test_r) 
$$
Let's load in the Diabetes dataset, as it has more instances (due to rounding, small datasets oftentimes produce slightly different splits even with same ratios):
<pre><code class="hljs">from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

X, y = load_diabetes(return_X_y=True)

print(X.shape)
</code></pre>
<pre><code class="hljs">(442, 10)
</code></pre>
Say we're aiming for an 80%/10%/10% split - we'd want to have <code>352</code>, <code>45</code> and <code>45</code> instances respectively. Let's define these rations and split the dataset into a training, testing and validation set with the <code>train_test_split()</code> function:
<pre><code class="hljs">from sklearn.datasets import load_diabetes
from sklearn.model_selection import train_test_split

train_ratio = 0.80
test_ratio = 0.10
validation_ratio = 0.10

X, y = load_diabetes(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=test_ratio)

X_train, X_valid, y_train, y_valid = train_test_split(X_train, y_train, test_size=validation_ratio/(train_ratio+test_ratio))

print(X_train.shape)
print(X_test.shape)
print(X_valid.shape)
</code></pre>
This results in:
<pre><code class="hljs">(352, 10)
(45, 10)
(45, 10)
</code></pre>
Awesome! Our dataset has successfully been split into three sets, which we can now feed into a model and perform validation during training to tune the hyperparameters.
<h3 id="stratifiedsplit">Stratified Split</h3>
Sometimes, there are different numbers of samples for each class in a dataset. Say, one class has 100 samples, the second one has 50, the third one 30, etc. Splitting without this in mind creates an issue when you're training a classification model (though, regression models don't suffer from this).
<blockquote>
It is best to somehow split the set, so that it preserves the proportions of the classes. This is a stratified split.
</blockquote>
Luckily, the method <code>train_test_split</code> has an argument called <code>stratify</code> which takes an array which defines the number of samples by class, when splitting, to stay proportional:
<pre><code class="hljs">from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split

X, y = load_iris(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y)
</code></pre>
In a lot of cases, you can simply use the <code>y</code> NumPy array from your dataset for a good <code>stratify</code> split array. This ensures that your model can fight the lack of balance between instances of classes and becomes less biased towards some.
<h2 id="conclusion">Conclusion</h2>
In this guide, we got familiar with some of the Scikit-Learn library and its <code>datasets</code> module. You've learned what training, testing and validation sets are, where they're applied and the benefits of validating your models.
We've taken a look at how to employ the <code>train_test_split()</code> method to split your data into a training and testing set, as well as how to separate a validation set, dynamically preserving the ratios of these sets.

In this guide, we'll take a look at how to split a dataset into a training, testing and validation set using Scikit-Learn's train_test_split() method, with practical examples and tips for best practices.

Scikit-Learn's train_test_split() - Training, Testing and Validation Sets

<h2 id="introduction">Introduction</h2>
In this guide, we will explore Heap Sort - the theory behind it and how to implement Heap Sort in JavaScript.
We will start off with what data structure it's based on (massive foreshadow here: it's a heap!), how to perform operations on that data structure, and how that data structure can be used as means of an efficient sorting algorithm.
Data structures and sorting algorithms are core concepts in programming. A computer program consistently deals with large datasets, retrieving and injecting data ad nauseam. The way we organize these datasets and operate on them is of great importance as it directly impacts the ease and speed with which the user interacts with our applications.
A sorting algorithm is evaluated based on two characteristics: the time and the space the algorithm uses as a function of the dataset's size. These are known as the Time Complexity and Space Complexity respectively, and allow us to &quot;pit&quot; algorithms against each other in average and best-case scenarios.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Heap Sort is regarded as an efficient algorithm, with average time complexity of θ(n log(n)).

 </div>
 </div>
 </div>
 Though there exist other algorithms outperforming Heap Sort in the average scenario, its significance relies on its power to perform with the same efficacy in the worst-case scenario as it does in the best, giving it a stable runtime over varying datasets, while some algorithms may suffer from large or small ones - depending on their underlying mechanism.
<h2 id="heapsortinjavascript">Heap Sort in JavaScript</h2>
<blockquote>
Heap Sort is an in-place, non-stable, comparison-based sorting algorithm.
</blockquote>
It does not require auxiliary data structures - it sorts the data in place and affects the original data (in-place). It doesn't preserve the relative order or equal elements. If you have two elements with the same value in an unsorted collection, their relative order might be changed (or stay the same) in the sorted collection (non-stable). Finally, the elements are compared to each other to find their order (comparison-based).
Although Heap Sort is in-place (doesn't require an auxiliary data structure), to make the implementation a bit clear, we will recruit an additional array during sorting.
The mechanism underlying Heap Sort is fairly simple and some even call it &quot;Improved Selection Sort&quot;.
<blockquote>
If you'd like to read more about Selection Sort, read our <a target="_blank" href="https://stackabuse.com/selection-sort-in-javascript/">Selection Sort in JavaScript</a>!
</blockquote>
It starts by converting the unsorted array into a heap - either a max-heap or min-heap. In the case of a max-heap, each parent holds a greater value than its descendants, making the root element the largest among the heap and vice versa.
Heap Sort relies on this heap condition.
At each iteration, the algorithm removes the root of the heap and pushes it into an empty array. After each removal, the heap restores itself, bubbling its second-largest (or second-smallest) element up to the root to preserve its heap condition. This process is also known as heapifying and you'll oftentimes see people refer to methods doing this as heapify.
Heap Sort continues shifting the newly located root elements into the sorted array until there is none left.
Using a max-heap in this manner will result in an array with elements in descending order. For the array to be in ascending order, one has to opt for a min-heap.
This sort of self-sorting and selective removal is reminiscent of Selection Sort (sans the self-sorting part) hence the parallel people draw.
<h3 id="whatisaheap">What Is a Heap?</h3>
A heap is a tree-like data structure. The type of heap we will use for our purposes will be a binary tree (a data structure that resembles a tree branch and is bound to start with one node and if were to branch out, is allowed a maximum of two successors extending from each node). While there exist few types of heaps, there are two distinctive features of a heap:
<ol>
<li>A heap must be complete, meaning that each level of the tree should be filled from left to right, and one is not allowed to create another level of the tree without filling all the possible nodes remaining on the last level.</li>
</ol>
<img src="https://s3.stackabuse.com/media/articles/heap-sort-in-javascript-1.png" alt="heap data structure">
<ol start="2">
<li>Each node has to hold a value that is greater than or equal to (in the case of a min-heap, smaller than or equal to) the value of every one of its descendants. This is called the &quot;heap condition&quot;.</li>
</ol>
<h3 id="mappingaheaptoanarray">Mapping a Heap to an Array</h3>
What we have defined and depicted as a heap up until this point is merely a diagram, a collection of circles and lines. To use this structure in a JavaScript-based computer program, we need to rework it into an array or a list.
Luckily, this is a fairly straightforward operation that mimics the way we build the heap in the first place. We read and shift the elements off of the heap into an array in the same order we have placed them into the heap: from left to right and level by level.
An example of a heap and its array counterpart, after this shift:
<img src="https://s3.stackabuse.com/media/articles/heap-sort-in-javascript-2.png" alt="heap to array data structure">
This way, not only can we manage to express a heap in code, but we also gain a compass with which to navigate inside that heap. We can deduct three equations that, given each node's index, will point us to the location of its parent and its right and left children inside the array:
<img src="https://s3.stackabuse.com/media/articles/heap-sort-in-javascript-3.png" alt="heap to array data structure">
<h3 id="creatingaheapinjavascript">Creating a Heap In JavaScript</h3>
Now that a detailed definition of a heap is in place, we can go ahead and implement it as a JavaScript class.
In this guide, we will create and employ a max-heap. Since the difference between a max-heap and a min-heap is trivial and does not affect the general logic behind the Heap Sort algorithm, the implementation of the min-heap and, therefore, creation of an ascending order via heap sort is a matter of changing the comparison operators.
Let's go ahead and define a <code>MaxHeap</code> class:
<pre><code class="hljs">class MaxHeap{
 constructor(){
 this.heap = [];
 }

 parentIndex(index){
 return Math.floor((index-1)/2);
 }

 leftChildIndex(index){
 return (2*index + 1);
 }

 rightChildIndex(index){
 return (2*index + 2);
 }
}
</code></pre>
In the <code>MaxHeap</code> class, we have defined a constructor that initializes an empty array. Later on, we will create additional functions to populate a heap inside this array.
For the time being, however, we have only created helper functions that will return the index of the parent and children of a given node.
<h3 id="insertingelementstoaheap">Inserting Elements to a Heap</h3>
Whenever a new element is inserted into a heap, it is placed next to the rightmost node on the bottom level (the last empty space in the array representation) or, if the bottom level is already full, at the leftmost node of a new level. In this scenario, the heap's first requirement: completeness of the tree, is ensured.
Moving forward, the heap property, which has likely been disturbed, needs to be reestablished. To move the new element to its proper place on the heap it is compared to its parent, and if the new element is larger than its parent, the elements are swapped.
The new element is bubbled up in the heap, whilst being compared to its parent at each level until finally the heap property is restored:
<img src="https://s3.stackabuse.com/media/articles/heap-sort-in-javascript-4.png" alt="array representation of a heap in javascript">
Let's add this functionality to the MaxHeap class we have previously created:
<pre><code class="hljs"> swap(a, b) {
 let temp = this.heap[a];
 this.heap[a] = this.heap[b];
 this.heap[b] = temp;
 }

 insert(item) {
 this.heap.push(item);
 var index = this.heap.length - 1;
 var parent = this.parentIndex(index);
 while(this.heap[parent] &amp;&amp; this.heap[parent] &lt; this.heap[index]) {
 this.swap(parent, index);
 index = this.parentIndex(index);
 parent = this.parentIndex(index);
 }
}
</code></pre>
<code>swap()</code> is added as a helper method to save us some redundancy in the code since while inserting the new element, we may have to perform this action several times - a number between zero and log(n) (in the case where the new element is larger than the root of the heap, and we have to make it climb the entire tree which has a height of log(the-total-number-of-its-elements) - which in other words, is a lot.
<code>insert()</code> operates as follows:
<ol>
<li>Appends the given element to the <code>heap</code> using the built-in JavaScript method: <code>push()</code>.</li>
<li>Marks the last element of the <code>heap</code> as <code>index</code> and its parent as <code>parent</code>.</li>
<li>While there exists an element of the heap at the index <code>parent</code> (<code>this.heap[parent]</code>), and that element happens to be smaller than the one at <code>index</code> (<code>this.heap[parent] &lt; this.heap[index</code>), the <code>insert()</code> method goes on to swap the two (<code>this.swap(parent, index)</code>) and moves its cursor one level up.</li>
</ol>
<h3 id="removingelementsfromtheheap">Removing Elements From the Heap</h3>
A heap only allows the deletion of the root element, which afterward leaves us with a completely distorted heap. Thereon, we first have to reinstate the complete binary tree property by moving the last node of the heap to the root. Then we need to bubble this misplaced value down until the heap property is back in place:
<img src="https://s3.stackabuse.com/media/articles/heap-sort-in-javascript-5.png" alt="removing elements from heap for heap sort in javascript">
<pre><code class="hljs">delete() {
 var item = this.heap.shift();
 this.heap.unshift(this.heap.pop());
 var index = 0;
 var leftChild = this.leftChildIndex(index);
 var rightChild = this.rightChildIndex(index);
 while(this.heap[leftChild] &amp;&amp; this.heap[leftChild] &gt; this.heap[index] || this.heap[rightChild] &gt; this.heap[index]){
 var max = leftChild;
 if(this.heap[rightChild] &amp;&amp; this.heap[rightChild] &gt; this.heap[max]){
 max = rightChild
 }
 this.swap(max, index);
 index = max;
 leftChild = this.leftChildIndex(max);
 rightChild = this.rightChildIndex(max);
 }
 return item;
}
</code></pre>
The <code>delete()</code> method, which we create inside the <code>MaxHeap</code> class, operates in the following manner:
<ol>
<li>The method starts by harvesting the largest element -therefore, the first element in the array representation of the heap. The built-in <code>shift()</code> method removes the first element of the array and returns the removed element, which we then store in the <code>item</code> variable.</li>
<li>The last element of the <code>heap</code> gets removed via <code>pop()</code> and gets placed to the recently emptied first space of <code>heap</code> via <code>unshift()</code>. <code>unshift()</code> is a built-in JavaScript method that works as the counterpart to <code>shift()</code>. While <code>shift()</code> removes the first element of the array and shifts the rest of the elements one space back, <code>unshift()</code> pushes an element to the beginning of the array and shifts the rest of the elements one space forward.</li>
<li>To be able to bubble the new root downward, pointers to the location of it, which is initially 0, and its two children (<code>index</code>, <code>rightChild</code>, <code>leftChild</code>) gets created.</li>
<li>The <code>while()</code> loop checks whether there exists a left child to the <code>index</code> node to ensure the existence of another level below (does not check for a right child yet) and if any of the children in this level is bigger than the node at [<code>index</code>].</li>
<li>If the condition inside the while loop is met, a <code>max</code> variable is created to declare that the left node is the maximum value the method has encountered yet. Then inside the loop, in an <code>if</code> clause, we check whether a right child exists, and if it does, whether it is bigger than the left child we first checked. If the value of the right child is indeed bigger, its index replaces the value in <code>max</code>.</li>
<li>Whichever child holding the bigger value gets swapped with its parent via <code>this.swap(max, index)</code>.</li>
<li>The method moves its imaginary cursor one level down at the end of the while loop and goes on to execute the code inside the while loop over and over until its condition no longer holds.</li>
</ol>
<h3 id="implementingheapsortinjavascript">Implementing Heap Sort in JavaScript</h3>
Finally, to achieve what this guide has promised, we create a <code>heapSort()</code> function (this time outside the <code>MaxHeap</code> class), and supply it with an array we'd like to sort:
<pre><code class="hljs">function heapSort(arr){
 var sorted = [];
 var heap1 = new MaxHeap();
 
 for(let i=0; i&lt;arr.length; i++){
 heap1.insert(arr[i]);
 }
 
 for(let i=0; i&lt;arr.length; i++){
 sorted.push(heap1.delete());
 }
 return sorted;
}
</code></pre>
The <code>heapSort()</code> takes the array to be sorted as its argument. Then, it creates an empty array to place the sorted version, as well as an empty heap via which to perform the sort.
Then, <code>heap1</code> is populated with the elements of <code>arr</code> and are deleted one by one, pushing the removed elements into the sorted array. The <code>heap1</code> self-organizes with each removal, so just pushing the elements off of it into the sorted array nets us with a sorted array.
Let's create an array and test this out:
<pre><code class="hljs">let arr = [1, 6, 2, 3, 7, 3, 4, 6, 9];
arr = heapSort(arr);

console.log(arr);
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this guide, we've learned about heap data structure and how Heap Sort operates.
While not being the fastest possible algorithm, Heap Sort can be advantageous when data is partially sorted or when there is a need for a stable algorithm.
Even though we have implemented it using an additional data structure, Heap Sort is essentially an in-place sorting algorithm and, for that reason, can also be used at times when memory usage is a concern.

Cansın Güler

In this detailed guide - we'll take a look at the theory behind as well as how to implement Heap Sort in JavaScript, with hands-on code.

Heap Sort in JavaScript

The Portable Document Format (PDF) is not a WYSIWYG (What You See is What You Get) format. It was developed to be platform-agnostic, independent of the underlying operating system and rendering engines.
To achieve this, PDF was constructed to be interacted with via something more like a programming language, and relies on a series of instructions and operations to achieve a result. In fact, PDF is based on a scripting language - <a rel="nofollow noreferrer noopener" target="_blank" href="https://www.adobe.com/products/postscript.html">PostScript</a>, which was the first device-independent Page Description Language.
In this guide, we'll be using <a rel="nofollow noreferrer noopener" target="_blank" href="https://borbpdf.com/">borb</a> - a Python library dedicated to reading, manipulating and generating PDF documents. It offers both a low-level model (allowing you access to the exact coordinates and layout if you choose to use those) and a high-level model (where you can delegate the precise calculations of margins, positions, etc to a layout manager).
<blockquote>
In this guide, we'll take a look at how to apply Optical Character Recognition (OCR) on a scanned PDF document.
</blockquote>
<h2 id="installingborb">Installing borb</h2>
borb can be downloaded from source on <a rel="nofollow noopener" target="_blank" href="https://github.com/jorisschellekens/borb">GitHub</a>, or installed via <code>pip</code>:
<pre><code class="hljs">$ pip install borb
</code></pre>
<h2 id="mypdfdocumenthasnotext">“My PDF Document Has No Text!”</h2>
This is by far one of the most classic questions on any programming-forum, or help desk:
<blockquote>
&quot;My document does not seem to have text in it. Help?&quot;
</blockquote>
Or:
<blockquote>
&quot;Your text-extraction code sample does not work for my document. How come?&quot;
</blockquote>
The answer is often as straightforward as &quot;your scanner hates you&quot;.
Most of the documents for which this doesn't work are PDF documents that are essentially glorified images. They contain all the meta-data needed to constitute a PDF, but their pages are just large (often low-quality) images, created by scanning physical papers.
As a consequence, there are no text-rendering instructions in these documents. And most PDF libraries will not be able to handle them. <code>borb</code>, however, loves to help and can be applied in these cases, with built-in support for OCR.
In this section we'll be using a special <code>EventListener</code> implementation called <code>OCRAsOptionalContentGroup</code>. This class uses <code>tesseract</code> (or rather <code>pytesseract</code>) to perform OCR (optical character recognition) on the <code>Document</code>.
<blockquote>
If you'd like to read more about OCR in Python, read our <a target="_blank" href="https://stackabuse.com/pytesseract-simple-python-optical-character-recognition/">Guide to Simple Optical Character Recognition with PyTesseract</a>!
</blockquote>
Once finished, the recognized text is re-inserted in each Page as a special &quot;layer&quot; (in PDF this is called an &quot;optional content group&quot;).
With the content now restored, the usual tricks (<code>SimpleTextExtraction</code>) yield the expected results.
You'll start by creating a method that builds a PIL Image with some text in it. This Image will then be inserted in a PDF.
<h2 id="creatinganimage">Creating an Image</h2>
<pre><code class="hljs">import typing
from pathlib import Path

from PIL import Image as PILImage # Type: ignore [import]
from PIL import ImageDraw, ImageFont

def create_image() -&gt; PILImage:
 # Create new Image
 img = PILImage.new(&quot;RGB&quot;, (256, 256), color=(255, 255, 255))

 # Create ImageFont
 # CAUTION: you may need to adjust the path to your particular font directory
 font = ImageFont.truetype(&quot;/usr/share/fonts/truetype/ubuntu/UbuntuMono-B.ttf&quot;, 24)

 # Draw text
 draw = ImageDraw.Draw(img)
 draw.text((10, 10),
 &quot;Hello World!&quot;,
 fill=(0, 0, 0),
 font=font)

 # Return
 return img
</code></pre>
Now let's build a PDF with this image, to represent our scanned document, that isn't parsable, as it doesn't contain metadata:
<pre><code class="hljs">import typing
# New imports
from borb.pdf.canvas.layout.image.image import Image
from borb.pdf.canvas.layout.page_layout.multi_column_layout import SingleColumnLayout
from borb.pdf.canvas.layout.page_layout.page_layout import PageLayout
from borb.pdf.canvas.layout.text.paragraph import Paragraph
from borb.pdf.document import Document
from borb.pdf.page.page import Page
from borb.pdf.pdf import PDF

# Main method to create the document
def create_document():

 # Create Document
 d: Document = Document()

 # Create/add Page
 p: Page = Page()
 d.append_page(p)

 # Set PageLayout
 l: PageLayout = SingleColumnLayout(p)

 # Add Paragraph
 l.add(Paragraph(&quot;Lorem Ipsum&quot;))

 # Add Image
 l.add(Image(create_image()))

 # Write
 with open(&quot;output_001.pdf&quot;, &quot;wb&quot;) as pdf_file_handle:
 PDF.dumps(pdf_file_handle, d)
</code></pre>
The resulting document should look like this:
<img src="https://s3.stackabuse.com/media/articles/applying-ocr-to-a-scanned-pdf-in-python-using-borb-1.png" alt="pdf document with image">
When you select the text in this document, you'll see immediately that only the top line is actually text. The rest is an Image with text (the Image you created):
<img src="https://s3.stackabuse.com/media/articles/applying-ocr-to-a-scanned-pdf-in-python-using-borb-2.png" alt="image is not selectable pdf">
Now, let's apply OCR to this document, and overlay actual text so that it becomes parsable:
<pre><code class="hljs"># New imports
from pathlib import Path
from borb.toolkit.ocr.ocr_as_optional_content_group import OCRAsOptionalContentGroup
from borb.toolkit.text.simple_text_extraction import SimpleTextExtraction

def apply_ocr_to_document():

 # Set up everything for OCR
 tesseract_data_dir: Path = Path(&quot;/home/joris/Downloads/tessdata-master/&quot;)
 assert tesseract_data_dir.exists()
 l: OCRAsOptionalContentGroup = OCRAsOptionalContentGroup(tesseract_data_dir)

 # Read Document
 doc: typing.Optional[Document] = None
 with open(&quot;output_001.pdf&quot;, &quot;rb&quot;) as pdf_file_handle:
 doc = PDF.loads(pdf_file_handle, [l])

 assert doc is not None

 # Store Document
 with open(&quot;output_002.pdf&quot;, &quot;wb&quot;) as pdf_file_handle:
 PDF.dumps(pdf_file_handle, doc)
</code></pre>
You can see this created an extra layer in the PDF. This layer is named &quot;OCR by borb&quot;, and contains the rendering instructions <code>borb</code> re-inserted in the <code>Document</code>.
You can toggle the visibility of this layer (this can be handy when debugging):
<img src="https://s3.stackabuse.com/media/articles/applying-ocr-to-a-scanned-pdf-in-python-using-borb-3.png" alt="hidden layer turned on borb ocr pdf">
<img src="https://s3.stackabuse.com/media/articles/applying-ocr-to-a-scanned-pdf-in-python-using-borb-4.png" alt="hidden layer turned off borb ocr pdf">
You can see that borb re-inserted the postscript rendering command to ensure &quot;Hello World!&quot; is in the `Document. Let's hide this layer again.
<blockquote>
Keep in mind OCR is a heuristic. The location and matched text may not always be 100% correct. That's just the way it goes. Typically, you'll keep the layer hidden (but selectable) so the original image is in place, and you can select/copy an approximation of it.
</blockquote>
Now (even with the layer hidden), you can select the text:
<img src="https://s3.stackabuse.com/media/articles/applying-ocr-to-a-scanned-pdf-in-python-using-borb-5.png" alt="ocr text is selectable, even when invisible pdf borb">
And if you apply <code>SimpleTextExtraction</code> now, you should be able to retrieve all the text in the <code>Document</code>.
<pre><code class="hljs"># New imports
from borb.toolkit.text.simple_text_extraction import SimpleTextExtraction

def read_modified_document():

 doc: typing.Optional[Document] = None
 l: SimpleTextExtraction = SimpleTextExtraction()
 with open(&quot;output_002.pdf&quot;, &quot;rb&quot;) as pdf_file_handle:
 doc = PDF.loads(pdf_file_handle, [l])

 print(l.get_text_for_page(0))


def main():
 create_document()
 apply_ocr_to_document()
 read_modified_document()

 
if __name__ == &quot;__main__&quot;:
 main()
</code></pre>
This prints:
<pre><code class="hljs">Lorem Ipsum
Hello World!
</code></pre>
Awesome!
<h2 id="conclusion">Conclusion</h2>
In this guide you've learned how to apply OCR to PDF documents, ensuring your scanned documents are searchable and ready for future processing.

Joris Schellekens

In this guide, we'll take a look at how to apply OCR to scanned PDF documents (images) and overlay layers to contain parsable text in Python using borb.

Perform OCR on a Scanned PDF in Python Using borb

<h2 id="introduction">Introduction</h2>
Bash scripts take in command-line arguments as inputs both sequentially and also, parsed as options. The command-line utilities use these arguments to conditionally trigger functions in a Bash script or selectively choose between environments to execute the script. In Bash, these are configured in different ways.
<blockquote>
In this article, we will be exploring several ways of creating these arguments for consumption and demonstrate a practical example for the implementation of command-line arguments.
</blockquote>
<h2 id="positionalcommandlinearguments">Positional Command-Line Arguments</h2>
Command-line arguments are read in a positional manner, from position <code>$1, $2, ..$n</code>. The pattern <code>$</code> followed by an <code>integer</code> is a reserved combination to represent the command-line arguments.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 <code>$0</code> denotes the name of the script.

 </div>
 </div>
 </div>
 Here's a pictorial representation of positional command-line arguments used to invoke a script:
<img src="https://stackabuse.s3.amazonaws.com/media/how-to-parse-command-line-arguments-in-bash-01.png" alt="Representation of positional command-line arguments">
Consider the following script, <code>arg_intro.sh</code>, where the arguments are printed according to their positions:
<pre><code class="hljs">#!/bin/bash

echo &quot;Script name is: $0&quot;
echo &quot;Arg1 is $1&quot;
echo &quot;Arg1 is $2&quot;
echo &quot;Arg1 is $3&quot;
echo &quot;-----------&quot;
echo &quot;All args: $*&quot;
echo &quot;All args count: $#&quot;
</code></pre>
This shell script is executed with the arguments as shown below:
<pre><code class="hljs">$ bash arg_intro.sh runtime inputs
Script name is: ./arg_intro.sh
Arg1 is runtime
Arg1 is inputs
Arg1 is 
-----------
All args: runtime inputs
All args count: 2
</code></pre>
<h2 id="parsingcomplexargumentsusinggetopt">Parsing Complex Arguments using getopt</h2>
In the previous section, you understood how the positional arguments are passed into the script. This method doesn't hold well if there's an increase in the number of arguments or there's a conditional requirement for the assignment of the variables. In such cases, a solid framework needs to be in place.
The command-line utility <code>getopt</code> solves this problem by providing syntax and options to define and parse the arguments.
Here's a gist on how to define arguments using the <code>getopt</code>.
<h3 id="definingtheoptions">Defining the Options</h3>
There are two kinds of arguments when passing them to a command-line utility. These include:
<ul>
<li>Short Arguments - These are defined by a single character, prepended by a hyphen. For example, <code>-h</code> may denote help, <code>-l</code> may denote a list command.</li>
<li>Long Arguments - These are whole strings, prepended by two hyphens. For example, <code>--help</code> denotes help, and <code>--list</code> denotes list.</li>
</ul>
Consider this script <code>test_options.sh</code>, where set up arguments with the <code>getopt</code> utility:
<pre><code class="hljs">#!/bin/bash

SHORT=c:,d:,h
LONG=city1:,city2:,help
OPTS=$(getopt --alternative --name weather --options $SHORT --longoptions $LONG -- &quot;$@&quot;) 
</code></pre>
The short arguments are passed to the <code>--options</code> flag of the <code>getopt</code> utility while the long arguments are passed to the <code>--longoptions</code> flag. In the code above, we have three short options:
<ul>
<li><code>c</code> corresponds to city1</li>
<li><code>d</code> corresponds to city2</li>
<li><code>h</code> corresponds to help.</li>
</ul>
The <code>help</code> option in the command-line utilities doesn’t take any values, and hence it doesn't have a colon attached to it.
<ul>
<li>Single colon (:) - Value is required for this option</li>
<li>Double colon (::) - Value is optional</li>
<li>No colons - No values are required</li>
</ul>
In the code above, the arguments <code>c</code> and <code>d</code> (followed by a colon) require the value to be sent, while the option <code>h</code> doesn't require any arguments (no colon).
<h3 id="shiftingtootherarguments">Shifting to Other Arguments</h3>
The <code>getopt</code> utility puts the input arguments to an organized positional output.
For example, let's tweak <code>test_options.sh</code> to print our options:
<pre><code class="hljs">#!/bin/bash

SHORT=c:,d:,h
LONG=city1:,city2:,help
OPTS=$(getopt -a -n weather --options $SHORT --longoptions $LONG -- &quot;$@&quot;)
echo $OPTS
</code></pre>
Upon passing arguments to this script, it results in the following output:
<pre><code class="hljs">$ bash test_options.sh --city1 Paris --city2 NewYork
--city1 &#x27;Paris&#x27; --city2 &#x27;NewYork&#x27; --
</code></pre>
Here's a visual representation of the output generated by the script and a demonstration of the QA environments:
<img src="https://stackabuse.s3.amazonaws.com/media/how-to-parse-command-line-arguments-in-bash-02.png" alt="Representation of how shifting the arguments work in command-line utilities">
Notice that the utility has added a trailing double hyphen (<code>--</code>) denoting the end of the output. Now, this output can be treated as positional arguments, and loops are written to iterate through each of them. This is where the <code>shift</code> keyword is helpful.
The <code>shift</code> keyword takes an additional argument of how many positions of the arguments to move the cursor to. To parse the next argument, the shift command has a number <code>2</code> assigned to it, which would switch to the next argument.
Consider the <code>test_options_looping.sh</code> script which uses <code>shift</code> to capture and print the values of the options in our arguments:
<pre><code class="hljs">#!/bin/bash

SHORT=c:,d:,h
LONG=city1:,city2:,help
OPTS=$(getopt -a -n weather --options $SHORT --longoptions $LONG -- &quot;$@&quot;)

eval set -- &quot;$OPTS&quot;

while :
do
 case &quot;$1&quot; in
 -c | --city1 )
 city1=&quot;$2&quot;
 shift 2
 ;;
 -d | --city2 )
 city2=&quot;$2&quot;
 shift 2
 ;;
 -h | --help)
 &quot;This is a weather script&quot;
 exit 2
 ;;
 --)
 shift;
 break
 ;;
 *)
 echo &quot;Unexpected option: $1&quot;
 ;;
 esac
done

echo $city1, $city2
</code></pre>
When the script is called by passing the valid arguments, it results in the printing of the two assigned variables as:
<pre><code class="hljs">$ bash test_options_looping.sh --city1 Paris --city2 NewYork
Paris, NewYork
</code></pre>
Notice how the script exited when it encountered a double hyphen (<code>--</code>).
With the learnings from above, let's make an example CLI weather app that accepts and parses the city names as arguments.
<h2 id="examplebuildingacommandlineweatherapp">Example - Building a Command-Line Weather App</h2>
The command-line weather app is retrieves data from <a href="https://wttr.in" target="_blank" rel="noopener nofollow noreferrer">https://wttr.in</a>, a public and free weather web app created by Igor Chubin. Our shell script makes a <code>GET</code> request with the <code>curl</code> utility to retrieve a weather report. If two arguments are passed, it gives a comparison between the weather in two cities.
The script contains all the above-mentioned components, including the <code>getopt</code> utility call. Based on the input arguments, there is a check on the number of valid arguments.
If no arguments are passed, the <code>help()</code> function is executed and exited. If not, the script proceeds and checks for the options in the switch case.
Based on the inputs, the <code>if-else</code> condition is evaluated and the <code>curl</code> to the URL is made by replacing the string argument with inputs.
Let's create a script called <code>weather.sh</code> with the following code:
<pre><code class="hljs">#!/bin/bash

help()
{
 echo &quot;Usage: weather [ -c | --city1 ]
 [ -d | --city2 ]
 [ -h | --help ]&quot;
 exit 2
}

SHORT=c:,d:,h
LONG=city1:,city2:,help
OPTS=$(getopt -a -n weather --options $SHORT --longoptions $LONG -- &quot;$@&quot;)

VALID_ARGUMENTS=$# # Returns the count of arguments that are in short or long options

if [ &quot;$VALID_ARGUMENTS&quot; -eq 0 ]; then
 help
fi

eval set -- &quot;$OPTS&quot;

while :
do
 case &quot;$1&quot; in
 -c | --city1 )
 city1=&quot;$2&quot;
 shift 2
 ;;
 -d | --city2 )
 city2=&quot;$2&quot;
 shift 2
 ;;
 -h | --help)
 help
 ;;
 --)
 shift;
 break
 ;;
 *)
 echo &quot;Unexpected option: $1&quot;
 help
 ;;
 esac
done

if [ &quot;$city1&quot; ] &amp;&amp; [ -z &quot;$city2&quot; ]
then
 curl -s &quot;https://wttr.in/${city1}&quot;
elif [ -z &quot;$city1&quot; ] &amp;&amp; [ &quot;$city2&quot; ]
then
 curl -s &quot;https://wttr.in/${city2}&quot;
elif [ &quot;$city1&quot; ] &amp;&amp; [ &quot;$city2&quot; ]
then
 diff -Naur &lt;(curl -s &quot;https://wttr.in/${city1}&quot; ) &lt;(curl -s &quot;https://wttr.in/${city2}&quot; )
else
 curl -s https://wttr.in
fi
</code></pre>
The script is tested with the following cases:
i) Test case 1 - When a single city is provided
<pre><code class="hljs">$ bash weather.sh -city1 NewYork
</code></pre>
<img src="https://stackabuse.s3.amazonaws.com/media/how-to-parse-command-line-arguments-in-bash-03.png" alt="Figure 1: Passing the city1 argument to the shell script">
ii) Test case 2 - When two cities are provided:
<pre><code class="hljs">$ bash weather.sh -city1 NewYork -city2 NewDelhi
</code></pre>
<img src="https://stackabuse.s3.amazonaws.com/media/how-to-parse-command-line-arguments-in-bash-04.png" alt="Figure 2: Passing the city1 and city2 arguments to the shell script">
iii) Test case 3 - When none of the city arguments are mentioned:
<pre><code class="hljs">$ bash weather.sh
</code></pre>
<img src="https://stackabuse.s3.amazonaws.com/media/how-to-parse-command-line-arguments-in-bash-05.png" alt="Figure 3: Calling the shell script without any arguments">
As expected, our script works fine and it returns the weather report for the input cities. When no arguments are passed, the script calls the <code>help()</code> function to return the list of supported arguments.
<h2 id="conclusion">Conclusion</h2>
In this article, we have covered the basics of passing the positional and parsing complex optional arguments from a shell script.
The example utility that we created calls the weather API based on the arguments that we provided to the script.

Marcus Sanatan

Sathiya Sarathi Gunasekaran

In this article, we will learn to handle CLI arguments in Bash. We'll begin by reading positional arguments, and then more intricate setups with getopts

How to Parse Command Line Arguments in Bash

Dragan Bjekic

Get started with using GitHub actions! In this guide, you'll learn what they are and how they work, topped with a practical example workflow of a benchmarker built using Python.

Getting Started with GitHub Actions - Software Automation

In this guide, we'll take a look at how to upload files to a Django web application, using Python, building a small application and exploring storage and file type options along the way.

Handling File Uploads with Django

<h2 id="introduction">Introduction</h2>
As we've covered in many of our <a rel="noopener" target="_blank" href="/tag/git/">previous articles</a>, Git is a very useful tool for developers, albeit, it tends to be a bit confusing to newcomers.
And all the while many know the basics of Git and know how to perform common operations, such as <a rel="noopener" target="_blank" href="/git-merge-branch-into-master/">merging commits</a>, <a rel="noopener" target="_blank" href="/git-create-a-new-branch/">branching</a>, or <a rel="noopener" target="_blank" href="/git-clone-a-repository/">cloning repos</a> - there is a long-tail subset of things you'll likely need to do over time that you just won't know off the top of your head.
<blockquote>
For instance, all commits contain some amount of metadata. How do we find that exactly?
</blockquote>
In this short article, we'll see how to extract the date of a specific Git commit.
<h2 id="getdateofacommitusinggit">Get Date of a Commit using Git</h2>
For many use-cases, you can easily find information like this in the repo's hosting provider repository (such as GitHub), but that's not always viable.
In that case, you can use the command line. There are a number of different ways to achieve this, but for now we'll focus on the following command:
<pre><code class="hljs">$ git show -s --format=%ci &lt;commit&gt;
</code></pre>
Using this in actual repository, you'll get something along the lines of:
<pre><code class="hljs">$ git show -s --format=%ci d4d20c2
2021-09-28 11:13:46 -0500
</code></pre>
You can also omit the commit hash to get the date for the latest commit instead:
<pre><code class="hljs">$ git show -s --format=%ci
2021-09-29 10:04:27 -0500
</code></pre>
For what seems like a simple task, there is a lot going on here. Let's break it down to better understand what's going on.
First, we use the <code>show</code> command. According to the man page, this command is used to &quot;Show various types of objects&quot;. That sounds a bit vague, but that's exactly what it does.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 You can use the <code>show</code> command to extract various metadata about blobs, trees, tags, and commits.

 </div>
 </div>
 </div>
 So when you want to find out just about any specific information about a commit, for example, this is the command you'll likely want to use.
The next option in our command is <code>-s</code>. This option simply suppresses the diff output. Without this option, the <code>show</code> command displays the commit patch by default, thus cluttering your output:
<pre><code class="hljs">$ git show --format=%ci d4d20c2
2021-09-28 11:13:46 -0500


diff --git a/routes/api.js b/routes/api.js
index d09cae0..bf6004f 100644
--- a/routes/api.js
+++ b/routes/api.js
@@ -111,7 +111,7 @@ exports.edit = async (req, res) =&gt; {
 }
 
 if (!user.hasPermission(&#x27;content_edit&#x27;)) {
- response.failed(&#x27;You don\&#x27;t have permission to edit this user.&#x27;);
+ response.failed(&#x27;You don\&#x27;t have permission to edit this content.&#x27;);
 return res.json(response.done());
 }

</code></pre>
We get a lot more information from the output than what we care for, so <code>-s</code> is a useful option to remove this.
The next option is the <code>--format=%ci</code>. This tells the <code>show</code> command exactly what information we're requesting about the object. In this case, <code>%ci</code> specifies the committer date, specifically in ISO 8601-like format. Other similar formats are the date in RFC2822 style (<code>%cD</code>) or Unix timestamp (<code>%ct</code>).
The <code>--format</code> option is important as it's where you can specify a wide range of metadata to show. Other than the commit date, here are a few other examples of metadata that you can show with this command:
<ul>
<li>Commit hash (<code>%H</code>)</li>
<li>Tree hash (<code>%T</code>)</li>
<li>Author name (<code>%an</code>)</li>
<li>Author email (<code>%ae</code>)</li>
<li>Subject (<code>%s</code>)</li>
<li>Commit notes (<code>%N</code>)</li>
</ul>
And of course, the last argument in our command is <code>&lt;commit&gt;</code>, which is fairly self-explanatory. This is an optional argument where you can specify the commit you're interested in. Since the <code>show</code> command works on many different types of objects, this argument can be used to specify any of the supported objects, like tags or trees.
For example, if you wanted to specify a tag, you'd simply use the tag name instead of commit hash:
<pre><code class="hljs">$ git show -s --format=%ci 
2021-09-28 11:13:46 -0500

$ git show -s --format=%ci v0.1.2
tag v0.1.2
Tagger: Scott Robinson &lt;scott@example.com&gt;

Fix failure message when user doesn&#x27;t have content_edit permission
</code></pre>
<h2 id="extractingcommitdatewithlogandformattingdates">Extracting Commit Date with log and Formatting Dates</h2>
You might not actually care about all of the different ways you can extract information like this from Git, but it is helpful to see other approaches as it will give you a good idea of what all is possible with Git.
<h3 id="formattingdates">Formatting Dates</h3>
While there are multiple date formats directly supported by the <code>--format</code> option, it doesn't support all possible style of dates you might want to get as output. A more versatile option is to use <code>--format=%cd</code>, which returns the committer date while also respecting the date format provided in the <code>--date=</code> option. This option can be given values like <code>relative</code>, <code>local</code>, <code>default</code>, <code>iso</code>, and others.
The <code>--date</code> option can also be used to specify more fine-grained date formats using formatting tokens:
<pre><code class="hljs">$ git show -s --format=%cd --date=format:&#x27;%Y-%m-%d %H:%M:%S&#x27; d4d20c2
2021-09-29 10:04:27
</code></pre>
Using the formatter tokens, you can format the date in any form and shape you'd like.
<h3 id="usingthelogcommand">Using the log Command</h3>
Along with the <code>show</code> command, we can also use the <code>log</code> command to get the date for a commit. This is a redundant method, in this case, but it has its own uses in other respects.
According to the man pages, <code>log</code> &quot;Shows the commit logs&quot;, which may seem obvious. If you read through the man pages, you'll notice that there are a lot of overlapping features with <code>show</code>.
The main difference is that <code>show</code> is mainly used for inspecting a single commit, whereas <code>log</code> is used for showing a range of commits. <code>log</code> also has many more features in terms of returning and formatting commits, like searching/skipping/limiting returned commit messages.
To get the commit date, we can use the <code>log</code> command, with the same arguments as before, but with the addition of <code>-1</code>:
<pre><code class="hljs">$ git log -1 --format=%ci d4d20c2
2021-09-29 10:04:27 -0500
</code></pre>
The only difference is the <code>-1</code> parameter, which tells this command to return date information for only 1 commit. Using <code>-2</code> here would return 2 commits, and so-on. The rest of the parameters are identical to <code>show</code>.
<h2 id="conclusion">Conclusion</h2>
In this article we looked, in detail, into how we can use the <code>show</code> command to retrieve metadata about a commit, specifically getting its commit date. We also saw how it can be used to retrieve a number of other pieces of information, as well as other commands available to use for similar uses.

Scott Robinson

In this short guide, we'll take a look at how to use Git to get the date of a commit using the show and log commands, as well as how to format the output.