<h2 id="introduction">Introduction</h2>
Object detection is a large field in computer vision, and one of the more important applications of computer vision &quot;in the wild&quot;. On one end, it can be used to build autonomous systems that navigate agents through environments - be it robots performing tasks or self-driving cars, but this requires intersection with other fields. However, anomaly detection (such as defective products on a line), locating objects within images, facial detection and various other applications of object detection can be done without intersecting other fields.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice This short guide is based on a small part of a much larger lesson on object detection belonging to our <a target="_blank" href="https://stackabuse.com/courses/practical-deep-learning-for-computer-vision-with-python/">&quot;Practical Deep Learning for Computer Vision with Python&quot;</a> course.

 </div>
 </div>
 </div>
 Object detection isn't as standardized as image classification, mainly because most of the new developments are typically done by individual researchers, maintainers and developers, rather than large libraries and frameworks. It's difficult to package the necessary utility scripts in a framework like TensorFlow or PyTorch and maintain the API guidelines that guided the development so far.
This makes object detection somewhat more complex, typically more verbose (but not always), and less approachable than image classification. One of the major benefits of being in an ecosystem is that it provides you with a way to not search for useful information on good practices, tools and approaches to use. With object detection - most have to do way more research on the landscape of the field to get a good grip.
<h2 id="objectdetectionwithpytorchtorchvisionsretinanet">Object Detection with PyTorch/TorchVision's RetinaNet</h2>
<code>torchvision</code> is PyTorch's Computer Vision project, and aims to make the development of PyTorch-based CV models easier, by providing transformation and augmentation scripts, a model zoo with pre-trained weights, datasets and utilities that can be useful for a practitioner.
While still in beta and very much experimental - <code>torchvision</code> offers a relatively simple Object Detection API with a few models to choose from:
<ul>
<li>Faster R-CNN</li>
<li>RetinaNet</li>
<li>FCOS (Fully convolutional RetinaNet)</li>
<li>SSD (VGG16 backbone... yikes)</li>
<li>SSDLite (MobileNetV3 backbone)</li>
</ul>
While the API isn't as polished or simple as some other third-party APIs, it's a very decent starting point for those who'd still prefer the safety of being in an ecosystem they're familiar with. Before going forward, make sure you install PyTorch and Torchvision:
<pre><code class="hljs">$ pip install torch torchvision
</code></pre>
Let's load in some of the utility functions, such as <code>read_image()</code>, <code>draw_bounding_boxes()</code> and <code>to_pil_image()</code> to make it easier to read, draw on and output images, followed by importing RetinaNet and its pre-trained weights (MS COCO):
<pre><code class="hljs">from torchvision.io.image import read_image
from torchvision.utils import draw_bounding_boxes
from torchvision.transforms.functional import to_pil_image
from torchvision.models.detection import retinanet_resnet50_fpn_v2, RetinaNet_ResNet50_FPN_V2_Weights

import matplotlib.pyplot as plt
</code></pre>
RetinaNet uses a ResNet50 backbone and a Feature Pyramid Network (FPN) on top of it. While the name of the class is verbose, it's indicative of the architecture. Let's fetch an image using the <code>requests</code> library and save it as a file on our local drive:
<pre><code class="hljs">import requests
response = requests.get(&#x27;https://i.ytimg.com/vi/q71MCWAEfL8/maxresdefault.jpg&#x27;)
open(&quot;obj_det.jpeg&quot;, &quot;wb&quot;).write(response.content)

img = read_image(&quot;obj_det.jpeg&quot;)
</code></pre>
With an image in place - we can instantiate our model and weights:
<pre><code class="hljs">weights = RetinaNet_ResNet50_FPN_V2_Weights.DEFAULT
model = retinanet_resnet50_fpn_v2(weights=weights, score_thresh=0.35)
# Put the model in inference mode
model.eval()
# Get the transforms for the model&#x27;s weights
preprocess = weights.transforms()
</code></pre>
The <code>score_thresh</code> argument defines the threshold at which an object is detected as an object of a class. Intuitively, it's the confidence threshold, and we won't classify an object to belong to a class if the model is less than 35% confident that it belongs to a class.
Let's preprocess the image using the transforms from our weights, create a batch and run inference:
<pre><code class="hljs">batch = [preprocess(img)]
prediction = model(batch)[0]
</code></pre>
That's it, our <code>prediction</code> dictionary holds the inferred object classes and locations! Now, the results aren't very useful for us in this form - we'll want to extract the labels with respect to the metadata from the weights and draw bounding boxes, which can be done via <code>draw_bounding_boxes()</code>:
<pre><code class="hljs">labels = [weights.meta[&quot;categories&quot;][i] for i in prediction[&quot;labels&quot;]]

box = draw_bounding_boxes(img, boxes=prediction[&quot;boxes&quot;],
 labels=labels,
 colors=&quot;cyan&quot;,
 width=2, 
 font_size=30,
 font=&#x27;Arial&#x27;)

im = to_pil_image(box.detach())

fig, ax = plt.subplots(figsize=(16, 12))
ax.imshow(im)
plt.show()
</code></pre>
This results in:
<img src="https://s3.stackabuse.com/media/ebooks/deep+learning+computer+vision/lesson_9/object-detection-4.png" alt="">
RetinaNet actually classified the person peeking behind the car! That's a pretty difficult classification.
You can switch out RetinaNet to an FCOS (fully convolutional RetinaNet) by replacing <code>retinanet_resnet50_fpn_v2</code> with <code>fcos_resnet50_fpn</code>, and use the <code>FCOS_ResNet50_FPN_Weights</code> weights:
<pre><code class="hljs">from torchvision.io.image import read_image
from torchvision.utils import draw_bounding_boxes
from torchvision.transforms.functional import to_pil_image
from torchvision.models.detection import fcos_resnet50_fpn, FCOS_ResNet50_FPN_Weights

import matplotlib.pyplot as plt
import requests
response = requests.get(&#x27;https://i.ytimg.com/vi/q71MCWAEfL8/maxresdefault.jpg&#x27;)
open(&quot;obj_det.jpeg&quot;, &quot;wb&quot;).write(response.content)

img = read_image(&quot;obj_det.jpeg&quot;)
weights = FCOS_ResNet50_FPN_Weights.DEFAULT
model = fcos_resnet50_fpn(weights=weights, score_thresh=0.35)
model.eval()

preprocess = weights.transforms()
batch = [preprocess(img)]
prediction = model(batch)[0]

labels = [weights.meta[&quot;categories&quot;][i] for i in prediction[&quot;labels&quot;]]

box = draw_bounding_boxes(img, boxes=prediction[&quot;boxes&quot;],
 labels=labels,
 colors=&quot;cyan&quot;,
 width=2, 
 font_size=30,
 font=&#x27;Arial&#x27;)

im = to_pil_image(box.detach())

fig, ax = plt.subplots(figsize=(16, 12))
ax.imshow(im)
plt.show()
</code></pre>

David Landup

In this short guide, learn how to perform object detection inference, using a pre-trained MS COCO RetinaNet detector, using Python, PyTorch and torchvision, with practical code examples.

RetinaNet Object Detection in Python with PyTorch and torchvision

<h2 id="introduction">Introduction</h2>
There are plenty of guides explaining how transformers work, and for building an intuition on a key element of them - token and position embedding.
Positionally embedding tokens allowed transformers to represent non-rigid relationships between tokens (usually, words), which is much better at modeling our context-driven speech in language modeling. While the process is relatively simple, it's fairly generic, and the implementations quickly become boilerplate.
<blockquote>
In this short guide, we'll take a look at how we can use KerasNLP, the official Keras add-on, to perform <code>PositionEmbedding</code> and <code>TokenAndPositionEmbedding</code>.
</blockquote>
<h2 id="kerasnlp">KerasNLP</h2>
KerasNLP is a horizontal addition for NLP. As of writing, it's still very young, at version 0.3, and the documentation is still fairly brief, but the package is more than just usable already.
It provides access to Keras layers, such as <code>TokenAndPositionEmbedding</code>, <code>TransformerEncoder</code> and <code>TransformerDecoder</code>, which makes building custom transformers easier than ever.
To use KerasNLP in our project, you can install it via <code>pip</code>:
<pre><code class="hljs">$ pip install keras_nlp
</code></pre>
Once imported into the project, you can use any <code>keras_nlp</code> layer as a standard Keras layer.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 If you're interested in applying KerasNLP layers further and building a Transformer-decoder text generator, read our <a rel="nofollow noopener noreferrer" target="_blank" href="https://stackabuse.com/gpt-style-text-generation-in-python-with-tensorflowkeras/">&quot;5-Line GPT-Style Text Generation in Python with TensorFlow/Keras&quot;</a>!

 </div>
 </div>
 </div>
 <h2 id="tokenization">Tokenization</h2>
Computers work with numbers. We voice our thoughts in words. To allow computers to crunch through them, we'll have to map words to numbers in some form.
A common way to do this is to simply map words to numbers where each integer represents a word. A corpus of words creates a vocabulary, and each word in the vocabulary gets an index. Thus, you can turn a sequence of words into a sequence of indices known as tokens:
<pre><code class="hljs">def tokenize(sequence):
 # ...
 return tokenized_sequence

sequence = [&#x27;I&#x27;, &#x27;am&#x27;, &#x27;Wall-E&#x27;]
sequence = tokenize(sequence)
print(sequence) # [4, 26, 472]
</code></pre>
With Keras, tokenization is typically done via the <code>TextVectorization</code> layer, which works wonderfully for a wide variety of inputs and supports several output modes (the default one being <code>int</code> which works as previously described):
<pre><code class="hljs"># Instantiate
vectorize = keras.layers.TextVectorization(
 max_tokens=max_features,
 output_mode=&#x27;int&#x27;,
 output_sequence_length=max_len)

# Adapt to text
vectorize.adapt(text_dataset)
# Vectorize new input
vectorized_text = vectorize([&#x27;some input&#x27;])
</code></pre>
You can use this layer as a standalone layer for preprocessing or as part of a Keras model, to make the preprocessing truly end-to-end, and supply raw input to the model. This guide is aimed at token embedding, not tokenization, so I won't dive further into the layer, which will be the main topic of another guide.
This sequence of tokens can then be embedded into a dense vector that defines the tokens in latent space:
<pre><code class="hljs">[[4], [26], [472]] -&gt; [[0.5, 0.25], [0.73, 0.2], [0.1, -0.75]]
</code></pre>
This is typically done with the <code>Embedding</code> layer in Keras. Transformers don't encode only using a standard <code>Embedding</code> layer. They perform <code>Embedding</code> and <code>PositionEmbedding</code>, and add them together, displacing the regular embeddings by their position in latent space.
With KerasNLP - performing <code>TokenAndPositionEmbedding</code> combines regular token embedding (<code>Embedding</code>) with positional embedding (<code>PositionEmbedding</code>).
<h2 id="positionembedding">PositionEmbedding</h2>
Let's take a look at <code>PositionEmbedding</code> first. It accepts tensors and ragged tensors, and assumes that the final dimension represents the features, while the second-to-last dimension represents the sequence.
<pre><code class="hljs"># Seq
(5, 10)
 # Features
</code></pre>
The layer accepts a <code>sequence_length</code> argument, denoting, well, the length of the input and output sequence. Let's go ahead and positionally embed a random uniform tensor:
<pre><code class="hljs">seq_length = 5
input_data = tf.random.uniform(shape=[5, 10])

input_tensor = keras.Input(shape=[None, 5, 10])
output = keras_nlp.layers.PositionEmbedding(sequence_length=seq_length)(input_tensor)
model = keras.Model(inputs=input_tensor, outputs=output)
 
model(input_data)
</code></pre>
This results in:
<pre><code class="hljs">&lt;tf.Tensor: shape=(5, 10), dtype=float32, numpy=
array([[ 0.23758471, -0.16798696, -0.15070847, 0.208067 , -0.5123104 ,
 -0.36670157, 0.27487397, 0.14939266, 0.23843127, -0.23328197],
 [-0.51353353, -0.4293166 , -0.30189738, -0.140344 , -0.15444171,
 -0.27691704, 0.14078277, -0.22552207, -0.5952263 , -0.5982155 ],
 [-0.265581 , -0.12168896, 0.46075982, 0.61768025, -0.36352775,
 -0.14212841, -0.26831496, -0.34448475, 0.4418767 , 0.05758983],
 [-0.46500492, -0.19256318, -0.23447984, 0.17891657, -0.01812166,
 -0.58293337, -0.36404118, 0.54269964, 0.3727749 , 0.33238482],
 [-0.2965023 , -0.3390794 , 0.4949159 , 0.32005525, 0.02882379,
 -0.15913549, 0.27996767, 0.4387421 , -0.09119213, 0.1294356 ]],
 dtype=float32)&gt;
</code></pre>
<h2 id="tokenandpositionembedding">TokenAndPositionEmbedding</h2>
Token and position embedding boils down to using <code>Embedding</code> on the input sequence, <code>PositionEmbedding</code> on the embedded tokens, and then adding these two results together, effectively displacing the token embeddings in space to encode their relative meaningful relationships.
This can technically be done as:
<pre><code class="hljs">seq_length = 10
vocab_size = 25
embed_dim = 10

input_data = tf.random.uniform(shape=[5, 10])

input_tensor = keras.Input(shape=[None, 5, 10])
embedding = keras.layers.Embedding(vocab_size, embed_dim)(input_tensor)
position = keras_nlp.layers.PositionEmbedding(seq_length)(embedding)
output = keras.layers.add([embedding, position])
model = keras.Model(inputs=input_tensor, outputs=output)
 
model(input_data).shape # ([5, 10, 10])
</code></pre>
The inputs are embedded, and then positionally embedded, after which they're added together, producing a new positionally embedded shape. Alternatively, you can leverage the <code>TokenAndPositionEmbedding</code> layer, which does this under the hood:
<pre><code class="hljs">... # https://github.com/keras-team/keras-nlp/blob/master/keras_nlp/layers/token_and_position_embedding.py
def call(self, inputs):
 embedded_tokens = self.token_embedding(inputs)
 embedded_positions = self.position_embedding(embedded_tokens)
 outputs = embedded_tokens + embedded_positions
 return outputs
</code></pre>
This makes it much cleaner to perform <code>TokenAndPositionEmbedding</code>:
<pre><code class="hljs">seq_length = 10
vocab_size = 25
embed_dim = 10

input_data = tf.random.uniform(shape=[5, 10])

input_tensor = keras.Input(shape=[None, 5, 10])
output = keras_nlp.layers.TokenAndPositionEmbedding(vocabulary_size=vocab_size, 
 sequence_length=seq_length, 
 embedding_dim=embed_dim)(input_tensor)
model = keras.Model(inputs=input_tensor, outputs=output)
 
model(input_data).shape # ([5, 10, 10])
</code></pre>
The data we've passed into the layer is now positionally embedded in a latent space of 10 dimensions:
<pre><code class="hljs">model(input_data)
</code></pre>
<pre><code class="hljs">&lt;tf.Tensor: shape=(5, 10, 10), dtype=float32, numpy=
array([[[-0.01695484, 0.7656435 , -0.84340465, 0.50211895,
 -0.3162892 , 0.16375223, -0.3774369 , -0.10028353,
 -0.00136751, -0.14690581],
 [-0.05646318, 0.00225556, -0.7745967 , 0.5233861 ,
 -0.22601983, 0.07024342, 0.0905793 , -0.46133494,
 -0.30130145, 0.451248 ],
 ...
</code></pre>
<h2 id="conclusions">Conclusions</h2>
Transformers have made a large wave since 2017, and many great guides offer insight into how they work, yet, they are still elusive to many due to the overhead of custom implementations. KerasNLP addresses this problem, providing building blocks that let you build flexible, powerful NLP systems, rather than providing pre-packaged solutions.
In this guide, we've taken a look at token and position embedding with Keras and KerasNLP.

Transformer Token and Position Embedding with Keras

In a URL, query string values often provide information about the request, like parameters for a search or the ID of an object you're using. If any of the business or request logic is handled in the frontend, it's important to know how to retrieve the query string values from the URL. There are a number of ways to achieve this, a few of which we'll see here.
<h2 id="urlsearchparams">URLSearchParams</h2>
The <code>URLSearchParams</code> interface is supported by all major browser versions except IE 11. It works by parsing the query string of a URL and providing a way to access the values. For example:
<pre><code class="hljs">let params = new URLSearchParams(&#x27;q=node&amp;page=2&#x27;);
params.get(&#x27;q&#x27;); // &#x27;node&#x27;
params.get(&#x27;page&#x27;); // &#x27;2&#x27;
</code></pre>
One of the downsides of this interface is that you must pass it only the query string of a URL. If you're working with the current browser URL, that's easy to do since you can just pass <code>window.location.search</code>. If you're working with any other URL, you'll need to parse out and pass the query string separately.
To parse the query parameters into an object, use <code>URL.searchParams</code>'s <code>.entries()</code> method, which returns an <code>Iterator</code> of key/value pairs, and <code>Object.fromEntries</code> to convert it into an object.
<pre><code class="hljs">let params = new URLSearchParams(&#x27;q=node&amp;page=2&#x27;);
let entries = params.entries();
Object.fromEntries(entries); // {q: &#x27;node&#x27;, page: &#x27;2&#x27;}
</code></pre>
<h2 id="urlobject">URL Object</h2>
The <code>URL</code> API is also supported by all major browser versions except IE 11. It offers a more flexible way to parse URLs, and it also provides a way to access the query string values. For example:
<pre><code class="hljs">const url = new URL(&#x27;https://stackabuse.com/search?q=node&amp;page=2&#x27;);
const searchParams = url.searchParams;

searchParams.get(&#x27;q&#x27;); // &#x27;node&#x27;
searchParams.get(&#x27;page&#x27;); // &#x27;2&#x27;
</code></pre>
<code>url.searchParams</code> is the same type of instance object returned by <code>URLSearchParams</code>.
The <code>url</code> object above also has all parts of the URL broken out into its parts. For example:
<pre><code class="hljs">url.href; // &#x27;https://stackabuse.com/search?q=node&amp;page=2&#x27;
url.origin; // &#x27;https://stackabuse.com&#x27;
url.protocol; // &#x27;https:&#x27;
url.host; // &#x27;stackabuse.com&#x27;
url.hostname; // &#x27;stackabuse.com&#x27;
url.port; // &#x27;&#x27;
url.pathname; // &#x27;/search&#x27;
url.search; // &#x27;?q=node&amp;page=2&#x27;
url.hash; // &#x27;&#x27;
</code></pre>
<h2 id="purejavascript">Pure JavaScript</h2>
If for any reason you're not able to access the APIs above or want to have more control over the parsing, you can use the following code to parse the query string into an object.
<pre><code class="hljs">function getQueryParams(url) {
 const paramArr = url.slice(url.indexOf(&#x27;?&#x27;) + 1).split(&#x27;&amp;&#x27;);
 const params = {};
 paramArr.map(param =&gt; {
 const [key, val] = param.split(&#x27;=&#x27;);
 params[key] = decodeURIComponent(val);
 })
 return params;
}
</code></pre>
Note: There are many ways to parse query params in plain JS, some more complicated (and robust) than others. This is just one way, and was adapted from <a rel="nofollow noopener" target="_blank" href="https://gist.github.com/pirate/9298155edda679510723">this gist</a>.
We can then use this plain JS function to parse a single query param into a string:
<pre><code class="hljs">getQueryParams(&#x27;https://stackabuse.com/search?q=node&amp;page=2&#x27;)
// { q: &#x27;node&#x27;, page: &#x27;2&#x27; }
</code></pre>

Scott Robinson

Get Query String Values in JavaScript

Mistakes are common, we all make them. That's why they put erasers on pencils. And this is no different with tools like Git. While some changes can be difficult to undo, there is usually a way.
So what if you want to modify an existing commit message? Luckily this is simple to do in Git, but the method may change a bit depending on some factors.
<h2 id="fixlastcommitnotyetpushedtoremote">Fix Last Commit, Not Yet Pushed to Remote</h2>
This is the simplest case since there is no need to modify a remote repository, you just need to modify your own. In this case, you can use the following command:
<pre><code class="hljs">$ git commit --amend
</code></pre>
Running this command will open up an editor where you can modify the last commit message. Once you're done and save/close the editor, the commit message will be changed.
Personally, I prefer to do most things on the command line, which you can do by adding an argument and message to the command above:
<pre><code class="hljs">$ git commit --amend -m &quot;Added a new file&quot;
</code></pre>
Doing it this way won't open the editor, but instead it'll just modify the commit message immediately.
<h2 id="fixlastcommitpushedtoremote">Fix Last Commit, Pushed to Remote</h2>
This is another common case, and is a bit more difficult to fix, not only because we need to modify our own repository, but we also need to modify the remote repository, which could be further ahead than your local branch.
The first step is to amend the last commit, just like we did in the previous section:
<pre><code class="hljs">$ git commit --amend -m &quot;Added a new file&quot;
</code></pre>
Then, you need to push these changes to the remote repository. However, this must be done using the <code>--force</code> flag.
<pre><code class="hljs">$ git push &lt;remote&gt; &lt;branch&gt; --force
</code></pre>
We need to do it this way in order to overwrite the remote repository with our own local state.
Be aware that by forcing the push, you'll also lose any commits on the local branch that you don't already have in your local branch. Proceed with caution!
If you can catch the mistake fast enough, then making the change won't be a problem. Otherwise, you may need to find another solution to avoid losing any changes made after the commit message you're wanting to change.
<h2 id="useinteractiverebase">Use Interactive Rebase</h2>
Another option, which is more flexible than the previous solutions described, is to use an interactive rebase. Doing this allows you to inspect the last N commits and make changes to them, including picking, squashing, etc.
To start the rebase, you can use the following command:
<pre><code class="hljs">$ git rebase -i HEAD~n
</code></pre>
Here, <code>n</code> is the number of commits you want to inspect and edit. You'll then see a list of commits, similar to this:
<pre><code class="hljs">$ $ git rebase -i HEAD~4
pick 788ebf0 Oops, I forgot to add a new file
pick ced1329 Fixed a super critical bug
pick 5e9cdc5 Added a new file

# Rebase d05209d..5e9cdc5 onto d05209d (4 commands)
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like &quot;squash&quot;, but discard this commit log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out
</code></pre>
As you can see all of the last N commits are listed, and you can modify their messages as needed.

Git: Modify an Existing Commit Message

Missing values are common and occur either due to human error, instrument error, processing from another team, or otherwise just a lack of data for a certain observation.
<blockquote>
In this Byte, we'll take a look at how to fill NaNs in a <code>DataFrame</code>, if you choose to handle NaNs by filling them.
</blockquote>
First off, let's create a mock <code>DataFrame</code> with some random values dropped out:
<pre><code class="hljs">import numpy as np
array = np.random.randn(25, 3)
mask = np.random.choice([1, 0], array.shape, p=[.3, .7]).astype(bool)
array[mask] = np.nan

df = pd.DataFrame(array, columns=[&#x27;Col1&#x27;, &#x27;Col2&#x27;, &#x27;Col3&#x27;])
</code></pre>
<pre><code class="hljs"> Col1 Col2 Col3 
0 -0.671603 -0.792415 0.783922
1 0.207720 NaN 0.996131
2 -0.892115 -1.282333 NaN
3 -0.315598 -2.371529 -1.959646
4 NaN NaN -0.584636
5 0.314736 -0.692732 -0.303951
6 0.355121 NaN NaN
7 NaN -1.900148 1.230828
8 -1.795468 0.490953 NaN
9 -0.678491 -0.087815 NaN
10 0.755714 0.550589 -0.702019
11 0.951908 -0.529933 0.344544
12 NaN 0.075340 -0.187669
13 NaN 0.314342 -0.936066
14 NaN 1.293355 0.098964
</code></pre>
Let's plot, say, the third column:
<pre><code class="hljs">plt.plot(df[&#x27;Col3&#x27;])
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-3.png" alt="">
When filled with various techniques - this NaN-filled graph can be replaced with:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-0.png" alt="">
<h2 id="fillnameanmedianmode">fillna() - Mean, Median, Mode</h2>
You can fill these values into a new column and assign it to the column you wish to fill, or in-place using the <code>inplace</code> argument. Here, we'll be extracting the filled values in a new column for ease of inspection:
<pre><code class="hljs">mean = df[&#x27;Col3&#x27;].fillna(df[&#x27;Col3&#x27;].mean(), inplace=False)
median = df[&#x27;Col3&#x27;].fillna(df[&#x27;Col3&#x27;].median(), inplace=False)
mode = df[&#x27;Col3&#x27;].fillna(df[&#x27;Col3&#x27;].mode(), inplace=False)
</code></pre>
The median, mean and mode of the column are <code>-0.187669</code>, <code>-0.110873</code> and <code>0.000000</code> and these values will be used for each NaN respectively. This is effectively filled with constant values, where the value being input depends on the entirety of the column.
First, filling with median values results in:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-4.png" alt="">
With mean values:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-5.png" alt="">
With mode values:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-6.png" alt="">
<h2 id="fillnaconstantvalue">fillna() - Constant Value</h2>
You can also fill with a constant value instead:
<pre><code class="hljs">constant = df[&#x27;Col3&#x27;].fillna(0, inplace=False
</code></pre>
This results in a constant value (0) being put instead of each NaN. <code>0</code> is close to our median and mean and equal to the mode, so the filled values will resemble that method closely for our mock dataset:
<pre><code class="hljs">0 0.783922
1 0.996131
2 0.000000
3 -1.959646
4 -0.584636
5 -0.303951
6 0.000000
7 1.230828
8 0.000000
9 0.000000
10 -0.702019
11 0.344544
12 -0.187669
13 -0.936066
14 0.098964
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-7.png" alt="">
<h2 id="fillnaforwardandbackwardfill">fillna() - Forward and Backward Fill</h2>
On each row - you can do a forward or backward fill, taking the value either from the row before or after:
<pre><code class="hljs">ffill = df[&#x27;Col3&#x27;].fillna(method=&#x27;ffill&#x27;)
bfill = df[&#x27;Col3&#x27;].fillna(method=&#x27;bfill&#x27;)
</code></pre>
With forward-filling, since we're missing from row 2 - the value from row 1 is taken to fill the second one. The values propagate forward:
<pre><code class="hljs">0 0.783922
1 0.996131
2 0.996131
3 -1.959646
4 -0.584636
5 -0.303951
6 -0.303951
7 1.230828
8 1.230828
9 1.230828
10 -0.702019
11 0.344544
12 -0.187669
13 -0.936066
14 0.098964
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-8.png" alt="">
With backward-filling, the opposite happens. Row 2 is filled with the value from row 3:
<pre><code class="hljs">0 0.783922
1 0.996131
2 -1.959646
3 -1.959646
4 -0.584636
5 -0.303951
6 1.230828
7 1.230828
8 -0.702019
9 -0.702019
10 -0.702019
11 0.344544
12 -0.187669
13 -0.936066
14 0.098964
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-9.png" alt="">
Though, if there's more than one <code>NaN</code> in a sequence - these won't do well and can cascade NaNs further down, skewing the data and removing actually recorded values.
<h2 id="interpolate">interpolate()</h2>
The <code>interpolate()</code> method delegates the interpolation of values to SciPy's <a rel="nofollow noopener noreferrer" target="_blank" href="https://docs.scipy.org/doc/scipy/reference/interpolate.html">suite of methods for interpolating values</a>. It accepts a wide variety of arguments, including, <code>nearest</code>, <code>zero</code>, <code>slinear</code>, <code>quadratic</code>, <code>cubic</code>, <code>spline</code>, <code>barycentric</code>, <code>polynomial</code>, <code>krogh</code>, <code>piecewise_polynomial</code>, <code>spline</code>, <code>pchip</code>, <code>akima</code>, <code>cubicspline</code>, etc.
<blockquote>
Interpolation is much more flexible and &quot;smart&quot; than just filling values with constants or half-variables such as previous methods.
</blockquote>
Interpolation can properly fill a sequence in a way that no other methods can, such as:
<pre><code class="hljs">s = pd.Series([0, 1, np.nan, np.nan, np.nan, 5])
s.fillna(s.mean()).values
# array([0., 1., 2., 2., 2., 5.])
s.fillna(method=&#x27;ffill&#x27;).values
# array([0., 1., 1., 1., 1., 5.])
s.interpolate().values
# array([0., 1., 2., 3., 4., 5.])
</code></pre>
The default interpolation is linear, and assuming that <code>1...5</code> is likely a <code>1, 2, 3, 4, 5</code> sequence isn't far-fetched (but isn't guaranteed). Both constant filling and forward or backward-filling fail miserably here. Generally speaking - interpolation is usually going to be a good friend when it comes to filling NaNs in noisy signals, or corrupt datasets.
Experimenting with types of interpolation may yield better results.
Here are two interpolation methods (<code>splice</code> and <code>polynomial</code> require an <code>order</code> argument):
<pre><code class="hljs">nearest = df[&#x27;Col3&#x27;].interpolate(method=&#x27;nearest&#x27;)
polynomial = df[&#x27;Col3&#x27;].interpolate(method=&#x27;polynomial&#x27;, order=3)
</code></pre>
These result in:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-10.png" alt="">
And:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-11.png" alt="">

In this tutorial - learn how to fill NaNs in a Pandas DataFrame using the mean, median, mode, constant value, backward and forward-fill methods as well as how to interpolate missing values with Python and Scipy.

How to Fill NaNs in a Pandas DataFrame

Dimitrije Stamenic

Joel Olawanle

In this tutorial, you will learn how to perform POST HTTP requests in React using the two primary methods - Fetch API and Axios. You will also get to know how to do this in functional and class components in React.

POST HTTP Request in React

K-means clustering is an unsupervised learning algorithm that groups data based on each point euclidean distance to a central point called centroid. The centroids are defined by the means of all points that are in the same cluster. The algorithm first chooses random points as centroids and then iterates adjusting them until full convergence.
<blockquote>
An important thing to remember when using K-means, is that the number of clusters is a hyperparameter, it will be defined before running the model.
</blockquote>
K-means can be implemented using Scikit-Learn with just 3 lines of code. Scikit-Learn also already has a centroid optimization method available, kmeans++, that helps the model converge faster.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice If you'd like to read an in-depth guide to K-Means Clustering, read our <a href="https://stackabuse.com/k-means-clustering-with-scikit-learn/" target="_blank">Definitive Guide to K-Means Clustering with Scikit-Learn&quot;!</a>

 </div>
 </div>
 </div>
 To apply the K-means clustering algorithm, let's load the Palmer Penguins dataset, choose the columns that will be clustered, and use Seaborn to plot a scatter plot with color coded clusters.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: You can download the dataset from this <a href="https://gist.github.com/cassiasamp/197b4e070f5f4da890ca4d226d088d1f" target="_blank">link</a>.

 </div>
 </div>
 </div>
 Let's import the libraries and load the Penguins dataset, trimming it to the chosen columns and dropping rows with missing data (there were only 2):
<pre><code class="hljs">import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
from sklearn.cluster import KMeans

df = pd.read_csv(&#x27;penguins.csv&#x27;)
print(df.shape) # (344, 9)
df = df[[&#x27;bill_length_mm&#x27;, &#x27;flipper_length_mm&#x27;]]
df = df.dropna(axis=0)
</code></pre>
We can use the Elbow method to have an indication of clusters for our data. It consists in the interpretation of a line plot with an elbow shape. The number of clusters is where the elbow bends. The x axis of the plot is the number of clusters and the y axis is the Within Clusters Sum of Squares (WCSS) for each number of clusters:
<pre><code class="hljs">wcss = []

for i in range(1, 11):
 clustering = KMeans(n_clusters=i, init=&#x27;k-means++&#x27;, random_state=42)
 clustering.fit(df)
 wcss.append(clustering.inertia_)
 
ks = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sns.lineplot(x = ks, y = wcss);
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/k-means-clustering-with-the-elbow-method-1.png" alt="">
The elbow method indicates our data has 2 clusters. Let's plot the data before and after clustering:
<pre><code class="hljs">fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(15,5))
sns.scatterplot(ax=axes[0], data=df, x=&#x27;bill_length_mm&#x27;, y=&#x27;flipper_length_mm&#x27;).set_title(&#x27;Without clustering&#x27;)
sns.scatterplot(ax=axes[1], data=df, x=&#x27;bill_length_mm&#x27;, y=&#x27;flipper_length_mm&#x27;, hue=clustering.labels_).set_title(&#x27;Using the elbow method&#x27;);
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/k-means-clustering-with-the-elbow-method-2.png" alt="">
<blockquote>
This example shows how the Elbow method is only a reference when used to choose the number of clusters. We already know that we have 3 types of penguins in the dataset, but if we were to determine their number by using the Elbow method, 2 clusters would be our result.
</blockquote>
Since K-means is sensitive to data variance, let's look at the descriptive statistics of the columns we are clustering:
<pre><code class="hljs">df.describe().T # T is to transpose the table and make it easier to read
</code></pre>
This results in:
<pre><code class="hljs"> 					count 	mean 		std 		min 	25% 	50% 	75% 	max
bill_length_mm 		342.0 	43.921930 	5.459584 	32.1 	39.225 	44.45 	48.5 	59.6
flipper_length_mm 	342.0 	200.915205 	14.061714 	172.0 	190.000 197.00 	213.0 	231.0
</code></pre>
Notice that the mean is far from the standard deviation (std), this indicates high variance. Let's try to reduce it by scaling the data with Standard Scaler:
<pre><code class="hljs">from sklearn.preprocessing import StandardScaler

ss = StandardScaler()
scaled = ss.fit_transform(df)
</code></pre>
Now, let's repeat the Elbow method process for the scaled data:
<pre><code class="hljs">wcss_sc = []

for i in range(1, 11):
 clustering_sc = KMeans(n_clusters=i, init=&#x27;k-means++&#x27;, random_state=42)
 clustering_sc.fit(scaled)
 wcss_sc.append(clustering_sc.inertia_)
 
ks = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
sns.lineplot(x = ks, y = wcss_sc);
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/k-means-clustering-with-the-elbow-method-3.png" alt="">
This time, the suggested number of clusters is 3. We can plot the data with the cluster labels again along with the two former plots for comparison:
<pre><code class="hljs">fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(15,5))
sns.scatterplot(ax=axes[0], data=df, x=&#x27;bill_length_mm&#x27;, y=&#x27;flipper_length_mm&#x27;).set_title(&#x27;Without clustering&#x27;)
sns.scatterplot(ax=axes[1], data=df, x=&#x27;bill_length_mm&#x27;, y=&#x27;flipper_length_mm&#x27;, hue=clustering.labels_).set_title(&#x27;With the Elbow method&#x27;)
sns.scatterplot(ax=axes[2], data=df, x=&#x27;bill_length_mm&#x27;, y=&#x27;flipper_length_mm&#x27;, hue=clustering_sc.labels_).set_title(&#x27;With the Elbow method and scaled data&#x27;);
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/k-means-clustering-with-the-elbow-method-4.png" alt="">
When using K-means Clustering, you need to predetermine the number of clusters. As we have seen when using a method to choose our k number of clusters, the result is only a suggestion and can be impacted by the amount of variance in data. It is important to conduct an in-depth analysis and generate more than one model with different _k_s when clustering.
If there is no prior indication of how many clusters are in the data, visualize it, test it and interpret it to see if the clustering results make sense. If not, cluster again. Also, look at more than one metric and instantiate different clustering models - for K-means, look at silhouette score and maybe Hierarchical Clustering to see if the results stay the same.

Cássia Sampaio

In this short guide - learn how to apply the Elbow method to determine the optimal K (number of clusters) for K-means clustering using Python and Scikit-Learn.

K-Means Clustering with the Elbow method

<h2 id="introduction">Introduction</h2>
As pointed out in a previous article that deals with <a href="/reading-files-with-python/">reading data from files</a>, file handling is essential knowledge of every professional and hobbyist Python programmer. This feature is a core part of the Python language, and no extra module needs to be loaded to do it properly.
<blockquote>
In this article, we will take a look at how to write data to a file line by line, as a list of lines, and append data at the end of a file.
</blockquote>
<h2 id="basicsofwritingfilesinpython">Basics of Writing Files in Python</h2>
There are three common functions to operate with files in Python:
<ul>
<li><code>open()</code> to open a file,</li>
<li><code>seek()</code> to set the file's current position at the given offset,</li>
<li><code>close()</code> to close the file afterwards.</li>
</ul>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: <code>open()</code> is a built-in Python function that returns a file handle that represents a <a rel="nofollow" target="_blank" href="https://docs.python.org/3/glossary.html#term-file-object">file object</a> to be used to access the file for reading, writing, or appending.

 </div>
 </div>
 </div>
 Writing to a file requires a few decisions - the name of the file in which to store data and the access mode of the file. Available are two modes, writing to a new file (and overwriting any existing data) and appending data at the end of a file that already exists. The according abbreviations are &quot;w&quot;, and &quot;a&quot;, and have to be specified before opening a file.

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Besides mentioned &quot;w&quot; and &quot;a&quot; flags, there is another, more powerful one - the &quot;w+&quot; flag, which opens file for both reading and writing.

 </div>
 </div>
 </div>
 <h2 id="writingasinglelinetoafile">Writing a Single Line to a File</h2>
This first example is pretty similar to writing to files with the popular programming languages C and C++. The process is pretty straightforward. First, we open the file using the built-in <code>open()</code> function for writing, write a single line of text to the file using the <code>write()</code> method, and then close the file using the <code>close()</code> method. Keep in mind that due to the way we opened the <code>helloworld.txt</code> file it will either be created if it does not exist yet, or it will be completely overwritten:
<pre><code class="hljs">filehandle = open(&#x27;helloworld.txt&#x27;, &#x27;w&#x27;)
filehandle.write(&#x27;Hello, world!\n&#x27;)
filehandle.close()
</code></pre>
This entire process can be shortened using the <code>with</code> statement:
<pre><code class="hljs">with open(&#x27;helloworld.txt&#x27;, &#x27;w&#x27;) as filehandle:
 filehandle.write(&#x27;Hello, world!\n&#x27;)
</code></pre>
As already said before, keep in mind that opening the <code>helloworld.txt</code> file this way will either create if it does not exist yet or completely overwrite it.
<h2 id="writingalistoflinestoafile">Writing a List of Lines to a File</h2>
In reality, a file does not consist only of a single line, but much more data. Therefore, the contents of the file are stored in a list that represents a file buffer. To write the entire file buffer we'll use the <code>writelines()</code> method:
<pre><code class="hljs">filehandle = open(&quot;helloworld.txt&quot;, &quot;w&quot;)
filebuffer = [&quot;a first line of text&quot;, &quot;a second line of text&quot;, &quot;a third line&quot;]
filehandle.writelines(filebuffer)
filehandle.close()
</code></pre>
Running the previous Python program and then using the <code>cat</code> command we can see that the file <code>helloworld.txt</code> has the following content:
<pre><code class="hljs">$ cat helloworld.txt
a first line of texta second line of texta third line
</code></pre>
This happens because the <code>writelines()</code> method does not automatically add any line separators when writing the data, even though intuitively, it should write individual lines.
This can easily be remedied by adding a <code>\n</code> (newline) character at the end of each line. Additionally, we can simplify the loop using a generator expression:
<pre><code class="hljs">with open(&#x27;helloworld.txt&#x27;, &#x27;w&#x27;) as filehandle:
 filebuffer = [&quot;a line of text&quot;, &quot;another line of text&quot;, &quot;a third line&quot;]
 filehandle.writelines(f&quot;{line for line in filebuffer}\n&quot;)
</code></pre>
Now, the output file <code>helloworld.txt</code> has the desired content:
<pre><code class="hljs">$ cat helloworld.txt
a first line of text
a second line of text
a third line
</code></pre>
<h2 id="appendingdatatoafile">Appending Data to a File</h2>
So far, we have stored data in new files or in overwritten data in existing files. But what if we want to append data to the end of an existing file? In this case, we would need to open the existing file using a different access mode. We change the access code to <code>'a'</code> instead of <code>'w'</code>:
<pre><code class="hljs">filehandle = open(&#x27;helloworld.txt&#x27;,&#x27;a&#x27;)
filehandle.write(&#x27;\n&#x27; + &#x27;Hello, world!\n&#x27;)
filehandle.close()
</code></pre>
As before, we can also rewrite the previous code using the <code>with</code> statement:
<pre><code class="hljs">with open(&#x27;helloworld.txt&#x27;, &#x27;a&#x27;) as filehandle:
 filehandle.write(f&#x27;\n Hello, world!\n&#x27;)
</code></pre>
Using the same <code>helloworld.txt</code> file from before, running this code will produce the following file contents:
<pre><code class="hljs">$ cat helloworld.txt
Hello, world
a second line
and a third line

Hello, world!
</code></pre>
<h2 id="conclusion">Conclusion</h2>
Writing plain text data to files, or appending data to existing files, is as easy as reading from files in Python. As soon as a file is closed after writing or appending data, Python triggers a synchronization call. As a result, the updated file is immediately written to disk.

Frank Hofmann

In this short tutorial, we'll take a look at how to write data to a file in Python. We'll be taking a look at a couple of different illustrative examples - writing to a file line by line, writing multiple lines at once, and appending to the existing file.

Writing Files using Python

In this guide, we'll take a comprehensive look at how to cluster a dataset in Python using the K-Means algorithm with the Scikit-Learn library, how to use the elbow method, find optimal cluster number and implement K-Means from scratch.