Deep Learning frameworks like Keras lower the barrier to entry for the masses and democratize the development of DL models to unexperienced folk, who can rely on reasonable defaults and simplified APIs to bear the brunt of heavy lifting, and produce decent results.
A common confusion arises between newer deep learning practitioners when using Keras loss functions for classification, such as <code>CategoricalCrossentropy</code> and <code>SparseCategoricalCrossentropy</code>:
<pre><code class="hljs">loss = keras.losses.SparseCategoricalCrossentropy(from_logits=True)
# Or
loss = keras.losses.SparseCategoricalCrossentropy(from_logits=False)
</code></pre>
<blockquote>
What does the <code>from_logits</code> flag refer to?
</blockquote>
The answer is fairly simple, but requires a look at the output of the network we're trying to grade using the loss function.
<h2 id="logitsandsoftmaxprobabilities">Logits and SoftMax Probabilities</h2>
Long story short:
<blockquote>
Probabilities are normalized - i.e. have a range between <code>[0..1]</code>. Logits aren't normalized, and can have a range between <code>[-inf...+inf]</code>.
</blockquote>
Depending on the output layer of your network:
<pre><code class="hljs">output = keras.layers.Dense(n, activation=&#x27;softmax&#x27;)(x)
# Or
output = keras.layers.Dense(n)(x)
</code></pre>
The output of the <code>Dense</code> layer will either return:
<ul>
<li>probabilities: The output is passed through a SoftMax function which normalizes the output into a set of probabilities over <code>n</code>, that all add up to <code>1</code>.</li>
<li>logits: <code>n</code> activations.</li>
</ul>
This misconception possibly arises from the short-hand syntax that allows you to add an activation to a layer, seemingly as a single layer, even though it's just shorthand for:
<pre><code class="hljs">output = keras.layers.Dense(n, activation=&#x27;softmax&#x27;)(x)
# Equivalent to
dense = keras.layers.Dense(n)(x)
output = keras.layers.Activation(&#x27;softmax&#x27;)(dense)
</code></pre>
Your loss function has to be informed as to whether it should expect a normalized distribution (output passed through a SoftMax function) or logits. Hence, the <code>from_logits</code> flag!
<h2 id="whenshouldfrom_logitstrue">When Should from_logits=True?</h2>
<blockquote>
If your output layer has a <code>'softmax'</code> activation, <code>from_logits</code> should be <code>False</code>. If your output layer doesn't have a <code>'softmax'</code> activation, <code>from_logits</code> should be <code>True</code>.
</blockquote>
If your network normalizes the output probabilities, your loss function should set <code>from_logits</code> to <code>False</code>, as it's not accepting logits. This is also the default value of all loss classes that accept the flag, as most people add an <code>activation='softmax'</code> to their output layers:
<pre><code class="hljs">model = keras.Sequential([
 keras.layers.Input(shape=(10, 1)),
 # Outputs normalized probability - from_logits=False
 keras.layers.Dense(10, activation=&#x27;softmax&#x27;) 
])

input_data = tf.random.uniform(shape=[1, 1])
output = model(input_data)
print(output)
</code></pre>
This results in:
<pre><code class="hljs">tf.Tensor(
[[[0.12467965 0.10423233 0.10054766 0.09162105 0.09144577 0.07093797
 0.12523937 0.11292477 0.06583504 0.11253635]]], shape=(1, 1, 10), dtype=float32)
</code></pre>
Since this network results in a normalized distribution - when comparing the outputs with target outputs, and grading them via a classification loss function (for the appropriate task) - you should set <code>from_logits</code> to <code>False</code>, or let the default value stay.
On the other hand, if your network doesn't apply SoftMax on the output:
<pre><code class="hljs">model = keras.Sequential([
 keras.layers.Input(shape=(10, 1)),
 # Outputs logits - from_logits=True
 keras.layers.Dense(10)
])

input_data = tf.random.uniform(shape=[1, 1])
output = model(input_data)
print(output)
</code></pre>
This results in:
<pre><code class="hljs">tf.Tensor(
[[[-0.06081138 0.04154852 0.00153442 0.0705068 -0.01139916
 0.08506121 0.1211026 -0.10112958 -0.03410497 0.08653068]]], shape=(1, 1, 10), dtype=float32)
</code></pre>
You'd need to set <code>from_logits</code> to <code>True</code> for the loss function to properly treat the outputs.
<h2 id="whentousesoftmaxontheoutput">When to Use SoftMax on the Output?</h2>
Most practitioners apply SoftMax on the output to give a normalized probability distribution, as this is in many cases what you'll use a network for - especially in simplified educational material. However, in some cases, you don't want to apply the function to the output, to process it in a different way before applying either SoftMax or another function.
A notable example comes from NLP models, in which a really the probability over a large vocabulary can be present in the output tensor. Applying SoftMax over all of them and greedily getting the <code>argmax</code> typically doesn't produce very good results.
However, if you observe the logits, extract the Top-K (where K can be any number but is typically somewhere between <code>[0...10]</code>), and only then applying SoftMax to the top-k possible tokens in the vocabulary shifts the distribution significantly, and usually produces more realistic results.
This is known as Top-K sampling, and while it isn't the ideal strategy, usually significantly outperforms greedy sampling.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice: To see this principle in action in an end-to-end project that teaches you the crux of building autoregressive language models with TensorFlow and Keras - read our <a target="_blank" href="https://stackabuse.com/gpt-style-text-generation-in-python-with-tensorflowkeras/">&quot;5-Line GPT-Style Text Generation in Python with TensorFlow/Keras&quot;</a>!

 </div>
 </div>
 </div>
 <h2 id="conclusion">Conclusion</h2>
In this short guide, we've taken a look at the <code>from_logits</code> argument for Keras loss classes, which oftentimes raise questions with newer practitioners.
The confusion possibly arises from the short-hand syntax that allows the addition of activation layers on top of other layers, within the definition of a layer itself. We've finally taken a look at when the argument should be set to <code>True</code> or <code>False</code>, and when an output should be left as logits or passed through an activation function such as SoftMax.

David Landup

In this short Python guide, learn what the from_logits argument means and does in Keras/TensorFlow loss functions, such as CategoricalCrossentropy and SparseCategoricalCrossentropy, as well as when you should set it to True or False.

What is 'from_logits=True' in Keras/TensorFlow Loss Functions?

Usman Malik

Dimitrije Stamenic

In this guide - learn the intuition behind and how to perform algorithmic complexity analysis - including what Big-O, Big-Omega and Big-Theta are, how to calculate Big-O and understand the notation, with practical Python examples.

Big O Notation and Algorithm Analysis with Python Examples

Cássia Sampaio

In this detailed definitive guide - learn how K-Nearest Neighbors works, and how to implement it for regression, classification and anomaly detection with Python and Scikit-Learn, through practical code examples and best practicecs.

Guide to the K-Nearest Neighbors Algorithm in Python and Scikit-Learn

<h2 id="introduction">Introduction</h2>
Object detection is a large field in computer vision, and one of the more important applications of computer vision &quot;in the wild&quot;. On one end, it can be used to build autonomous systems that navigate agents through environments - be it robots performing tasks or self-driving cars, but this requires intersection with other fields. However, anomaly detection (such as defective products on a line), locating objects within images, facial detection and various other applications of object detection can be done without intersecting other fields.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice This short guide is based on a small part of a much larger lesson on object detection belonging to our <a target="_blank" href="https://stackabuse.com/courses/practical-deep-learning-for-computer-vision-with-python/">&quot;Practical Deep Learning for Computer Vision with Python&quot;</a> course.

 </div>
 </div>
 </div>
 Object detection isn't as standardized as image classification, mainly because most of the new developments are typically done by individual researchers, maintainers and developers, rather than large libraries and frameworks. It's difficult to package the necessary utility scripts in a framework like TensorFlow or PyTorch and maintain the API guidelines that guided the development so far.
This makes object detection somewhat more complex, typically more verbose (but not always), and less approachable than image classification. One of the major benefits of being in an ecosystem is that it provides you with a way to not search for useful information on good practices, tools and approaches to use. With object detection - most have to do way more research on the landscape of the field to get a good grip.
<blockquote>
In this short guide, we'll be performing Object Detection and Instance Segmentation, using a Mask R-CNN, in Python, with the Detectron2 Platform, written in PyTorch.
</blockquote>
<h2 id="metaaisdetectron2instancesegmentationandobjectdetection">Meta AI's Detectron2 - Instance Segmentation and Object Detection</h2>
<a target="_blank" rel="nofollow noopener noreferrer" href="https://github.com/facebookresearch/detectron2">Detectron2</a> is Meta AI (formerly FAIR - Facebook AI Research)'s open source object detection, segmentation and pose estimation package - all in one. Given an input image, it can return the labels, bounding boxes, confidence scores, masks and skeletons of objects. This is well-represented on the repository's page:
<img src="https://s3.stackabuse.com/media/ebooks/deep+learning+computer+vision/lesson_9/object-detection-7.png" alt="">
It's meant to be used as a library on the top of which you can build research projects. It offers a <a rel="nofollow noopener noreferrer" target="_blank" href="https://github.com/facebookresearch/detectron2/blob/main/MODEL_ZOO.md">model zoo</a> with most implementations relying on Mask R-CNN and R-CNNs in general, alongside RetinaNet. They also have a pretty decent <a rel="nofollow noopener noreferrer" target="_blank" href="https://detectron2.readthedocs.io/en/latest/tutorials/index.html">documentation</a>. Let's run an exemplary inference script!
First, let's install the dependencies:
<pre><code class="hljs">$ pip install pyyaml==5.1
$ pip install &#x27;git+https://github.com/facebookresearch/detectron2.git&#x27;
</code></pre>
Next, we'll import the Detectron2 utilities - this is where framework-domain knowledge comes into play. You can construct a detector using the <code>DefaultPredictor</code> class, by passing in a configuration object that sets it up. The <code>Visualizer</code> offers support for visualizing results. <code>MetadataCatalog</code> and <code>DatasetCatalog</code> belong to Detectron2's data API and offer information on built-in datasets as well as their metadata.
Let's import the classes and functions we'll be using:
<pre><code class="hljs">import torch, detectron2
from detectron2.utils.logger import setup_logger
setup_logger()

from detectron2 import model_zoo
from detectron2.engine import DefaultPredictor
from detectron2.config import get_cfg
from detectron2.utils.visualizer import Visualizer
from detectron2.data import MetadataCatalog, DatasetCatalog
</code></pre>
Using <code>requests</code>, we'll download an image and save it to our local drive:
<pre><code class="hljs">import matplotlib.pyplot as plt
import requests
response = requests.get(&#x27;http://images.cocodataset.org/val2017/000000439715.jpg&#x27;)
open(&quot;input.jpg&quot;, &quot;wb&quot;).write(response.content)
 
im = cv2.imread(&quot;./input.jpg&quot;)
fig, ax = plt.subplots(figsize=(18, 8))
ax.imshow(cv2.cvtColor(im, cv2.COLOR_BGR2RGB))
</code></pre>
This results in:
<img src="https://s3.stackabuse.com/media/ebooks/deep+learning+computer+vision/lesson_9/object-detection-8.png" alt="">
Now, we load the configuration, enact changes if need be (the models run on GPU by default, so if you don't have a GPU, you'll want to set the device to <code>&quot;cpu&quot;</code> in the config):
<pre><code class="hljs">cfg = get_cfg()

cfg.merge_from_file(model_zoo.get_config_file(&quot;COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml&quot;))
cfg.MODEL.ROI_HEADS.SCORE_THRESH_TEST = 0.5
cfg.MODEL.WEIGHTS = model_zoo.get_checkpoint_url(&quot;COCO-InstanceSegmentation/mask_rcnn_R_50_FPN_3x.yaml&quot;)
# If you don&#x27;t have a GPU and CUDA enabled, the next line is required
# cfg.MODEL.DEVICE = &quot;cpu&quot;
</code></pre>
Here, we specify which model we'd like to run from the <code>model_zoo</code>. We've imported an instance segmentation model, based on the Mask R-CNN architecture, and with a ResNet50 backbone. Depending on what you'd like to achieve (keypoint detection, instance segmentation, panoptic segmentation or object detection), you'll load in the appropriate model.
Finally, we can construct a predictor with this <code>cfg</code> and run it on the inputs! The <code>Visualizer</code> class is used to draw predictions on the image (in this case, segmented instances, classes and bounding boxes:
<pre><code class="hljs">predictor = DefaultPredictor(cfg)
outputs = predictor(im)

v = Visualizer(im[:, :, ::-1], MetadataCatalog.get(cfg.DATASETS.TRAIN[0]), scale=1.2)
out = v.draw_instance_predictions(outputs[&quot;instances&quot;].to(&quot;cpu&quot;))
fig, ax = plt.subplots(figsize=(18, 8))
ax.imshow(out.get_image()[:, :, ::-1])
</code></pre>
Finally, this results in:
<img src="https://s3.stackabuse.com/media/ebooks/deep+learning+computer+vision/lesson_9/object-detection-9.png" alt="">

In this short guide - learn how to perform object detection and instance segmentation on input images with Detectron2 and Python/PyTorch examples.

Object Detection and Instance Segmentation in Python with Detectron2

<h2 id="introduction">Introduction</h2>
Object detection is a large field in computer vision, and one of the more important applications of computer vision &quot;in the wild&quot;. On one end, it can be used to build autonomous systems that navigate agents through environments - be it robots performing tasks or self-driving cars, but this requires intersection with other fields. However, anomaly detection (such as defective products on a line), locating objects within images, facial detection and various other applications of object detection can be done without intersecting other fields.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Advice This short guide is based on a small part of a much larger lesson on object detection belonging to our <a target="_blank" href="https://stackabuse.com/courses/practical-deep-learning-for-computer-vision-with-python/">&quot;Practical Deep Learning for Computer Vision with Python&quot;</a> course.

 </div>
 </div>
 </div>
 Object detection isn't as standardized as image classification, mainly because most of the new developments are typically done by individual researchers, maintainers and developers, rather than large libraries and frameworks. It's difficult to package the necessary utility scripts in a framework like TensorFlow or PyTorch and maintain the API guidelines that guided the development so far.
This makes object detection somewhat more complex, typically more verbose (but not always), and less approachable than image classification. One of the major benefits of being in an ecosystem is that it provides you with a way to not search for useful information on good practices, tools and approaches to use. With object detection - most have to do way more research on the landscape of the field to get a good grip.
<h2 id="objectdetectionwithpytorchtorchvisionsretinanet">Object Detection with PyTorch/TorchVision's RetinaNet</h2>
<code>torchvision</code> is PyTorch's Computer Vision project, and aims to make the development of PyTorch-based CV models easier, by providing transformation and augmentation scripts, a model zoo with pre-trained weights, datasets and utilities that can be useful for a practitioner.
While still in beta and very much experimental - <code>torchvision</code> offers a relatively simple Object Detection API with a few models to choose from:
<ul>
<li>Faster R-CNN</li>
<li>RetinaNet</li>
<li>FCOS (Fully convolutional RetinaNet)</li>
<li>SSD (VGG16 backbone... yikes)</li>
<li>SSDLite (MobileNetV3 backbone)</li>
</ul>
While the API isn't as polished or simple as some other third-party APIs, it's a very decent starting point for those who'd still prefer the safety of being in an ecosystem they're familiar with. Before going forward, make sure you install PyTorch and Torchvision:
<pre><code class="hljs">$ pip install torch torchvision
</code></pre>
Let's load in some of the utility functions, such as <code>read_image()</code>, <code>draw_bounding_boxes()</code> and <code>to_pil_image()</code> to make it easier to read, draw on and output images, followed by importing RetinaNet and its pre-trained weights (MS COCO):
<pre><code class="hljs">from torchvision.io.image import read_image
from torchvision.utils import draw_bounding_boxes
from torchvision.transforms.functional import to_pil_image
from torchvision.models.detection import retinanet_resnet50_fpn_v2, RetinaNet_ResNet50_FPN_V2_Weights

import matplotlib.pyplot as plt
</code></pre>
RetinaNet uses a ResNet50 backbone and a Feature Pyramid Network (FPN) on top of it. While the name of the class is verbose, it's indicative of the architecture. Let's fetch an image using the <code>requests</code> library and save it as a file on our local drive:
<pre><code class="hljs">import requests
response = requests.get(&#x27;https://i.ytimg.com/vi/q71MCWAEfL8/maxresdefault.jpg&#x27;)
open(&quot;obj_det.jpeg&quot;, &quot;wb&quot;).write(response.content)

img = read_image(&quot;obj_det.jpeg&quot;)
</code></pre>
With an image in place - we can instantiate our model and weights:
<pre><code class="hljs">weights = RetinaNet_ResNet50_FPN_V2_Weights.DEFAULT
model = retinanet_resnet50_fpn_v2(weights=weights, score_thresh=0.35)
# Put the model in inference mode
model.eval()
# Get the transforms for the model&#x27;s weights
preprocess = weights.transforms()
</code></pre>
The <code>score_thresh</code> argument defines the threshold at which an object is detected as an object of a class. Intuitively, it's the confidence threshold, and we won't classify an object to belong to a class if the model is less than 35% confident that it belongs to a class.
Let's preprocess the image using the transforms from our weights, create a batch and run inference:
<pre><code class="hljs">batch = [preprocess(img)]
prediction = model(batch)[0]
</code></pre>
That's it, our <code>prediction</code> dictionary holds the inferred object classes and locations! Now, the results aren't very useful for us in this form - we'll want to extract the labels with respect to the metadata from the weights and draw bounding boxes, which can be done via <code>draw_bounding_boxes()</code>:
<pre><code class="hljs">labels = [weights.meta[&quot;categories&quot;][i] for i in prediction[&quot;labels&quot;]]

box = draw_bounding_boxes(img, boxes=prediction[&quot;boxes&quot;],
 labels=labels,
 colors=&quot;cyan&quot;,
 width=2, 
 font_size=30,
 font=&#x27;Arial&#x27;)

im = to_pil_image(box.detach())

fig, ax = plt.subplots(figsize=(16, 12))
ax.imshow(im)
plt.show()
</code></pre>
This results in:
<img src="https://s3.stackabuse.com/media/ebooks/deep+learning+computer+vision/lesson_9/object-detection-4.png" alt="">
RetinaNet actually classified the person peeking behind the car! That's a pretty difficult classification.
You can switch out RetinaNet to an FCOS (fully convolutional RetinaNet) by replacing <code>retinanet_resnet50_fpn_v2</code> with <code>fcos_resnet50_fpn</code>, and use the <code>FCOS_ResNet50_FPN_Weights</code> weights:
<pre><code class="hljs">from torchvision.io.image import read_image
from torchvision.utils import draw_bounding_boxes
from torchvision.transforms.functional import to_pil_image
from torchvision.models.detection import fcos_resnet50_fpn, FCOS_ResNet50_FPN_Weights

import matplotlib.pyplot as plt
import requests
response = requests.get(&#x27;https://i.ytimg.com/vi/q71MCWAEfL8/maxresdefault.jpg&#x27;)
open(&quot;obj_det.jpeg&quot;, &quot;wb&quot;).write(response.content)

img = read_image(&quot;obj_det.jpeg&quot;)
weights = FCOS_ResNet50_FPN_Weights.DEFAULT
model = fcos_resnet50_fpn(weights=weights, score_thresh=0.35)
model.eval()

preprocess = weights.transforms()
batch = [preprocess(img)]
prediction = model(batch)[0]

labels = [weights.meta[&quot;categories&quot;][i] for i in prediction[&quot;labels&quot;]]

box = draw_bounding_boxes(img, boxes=prediction[&quot;boxes&quot;],
 labels=labels,
 colors=&quot;cyan&quot;,
 width=2, 
 font_size=30,
 font=&#x27;Arial&#x27;)

im = to_pil_image(box.detach())

fig, ax = plt.subplots(figsize=(16, 12))
ax.imshow(im)
plt.show()
</code></pre>

In this short guide, learn how to perform object detection inference, using a pre-trained MS COCO RetinaNet detector, using Python, PyTorch and torchvision, with practical code examples.

RetinaNet Object Detection in Python with PyTorch and torchvision

<h2 id="introduction">Introduction</h2>
There are plenty of guides explaining how transformers work, and for building an intuition on a key element of them - token and position embedding.
Positionally embedding tokens allowed transformers to represent non-rigid relationships between tokens (usually, words), which is much better at modeling our context-driven speech in language modeling. While the process is relatively simple, it's fairly generic, and the implementations quickly become boilerplate.
<blockquote>
In this short guide, we'll take a look at how we can use KerasNLP, the official Keras add-on, to perform <code>PositionEmbedding</code> and <code>TokenAndPositionEmbedding</code>.
</blockquote>
<h2 id="kerasnlp">KerasNLP</h2>
KerasNLP is a horizontal addition for NLP. As of writing, it's still very young, at version 0.3, and the documentation is still fairly brief, but the package is more than just usable already.
It provides access to Keras layers, such as <code>TokenAndPositionEmbedding</code>, <code>TransformerEncoder</code> and <code>TransformerDecoder</code>, which makes building custom transformers easier than ever.
To use KerasNLP in our project, you can install it via <code>pip</code>:
<pre><code class="hljs">$ pip install keras_nlp
</code></pre>
Once imported into the project, you can use any <code>keras_nlp</code> layer as a standard Keras layer.

 <div class="alert alert-reference">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-link-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 If you're interested in applying KerasNLP layers further and building a Transformer-decoder text generator, read our <a rel="nofollow noopener noreferrer" target="_blank" href="https://stackabuse.com/gpt-style-text-generation-in-python-with-tensorflowkeras/">&quot;5-Line GPT-Style Text Generation in Python with TensorFlow/Keras&quot;</a>!

 </div>
 </div>
 </div>
 <h2 id="tokenization">Tokenization</h2>
Computers work with numbers. We voice our thoughts in words. To allow computers to crunch through them, we'll have to map words to numbers in some form.
A common way to do this is to simply map words to numbers where each integer represents a word. A corpus of words creates a vocabulary, and each word in the vocabulary gets an index. Thus, you can turn a sequence of words into a sequence of indices known as tokens:
<pre><code class="hljs">def tokenize(sequence):
 # ...
 return tokenized_sequence

sequence = [&#x27;I&#x27;, &#x27;am&#x27;, &#x27;Wall-E&#x27;]
sequence = tokenize(sequence)
print(sequence) # [4, 26, 472]
</code></pre>
With Keras, tokenization is typically done via the <code>TextVectorization</code> layer, which works wonderfully for a wide variety of inputs and supports several output modes (the default one being <code>int</code> which works as previously described):
<pre><code class="hljs"># Instantiate
vectorize = keras.layers.TextVectorization(
 max_tokens=max_features,
 output_mode=&#x27;int&#x27;,
 output_sequence_length=max_len)

# Adapt to text
vectorize.adapt(text_dataset)
# Vectorize new input
vectorized_text = vectorize([&#x27;some input&#x27;])
</code></pre>
You can use this layer as a standalone layer for preprocessing or as part of a Keras model, to make the preprocessing truly end-to-end, and supply raw input to the model. This guide is aimed at token embedding, not tokenization, so I won't dive further into the layer, which will be the main topic of another guide.
This sequence of tokens can then be embedded into a dense vector that defines the tokens in latent space:
<pre><code class="hljs">[[4], [26], [472]] -&gt; [[0.5, 0.25], [0.73, 0.2], [0.1, -0.75]]
</code></pre>
This is typically done with the <code>Embedding</code> layer in Keras. Transformers don't encode only using a standard <code>Embedding</code> layer. They perform <code>Embedding</code> and <code>PositionEmbedding</code>, and add them together, displacing the regular embeddings by their position in latent space.
With KerasNLP - performing <code>TokenAndPositionEmbedding</code> combines regular token embedding (<code>Embedding</code>) with positional embedding (<code>PositionEmbedding</code>).
<h2 id="positionembedding">PositionEmbedding</h2>
Let's take a look at <code>PositionEmbedding</code> first. It accepts tensors and ragged tensors, and assumes that the final dimension represents the features, while the second-to-last dimension represents the sequence.
<pre><code class="hljs"># Seq
(5, 10)
 # Features
</code></pre>
The layer accepts a <code>sequence_length</code> argument, denoting, well, the length of the input and output sequence. Let's go ahead and positionally embed a random uniform tensor:
<pre><code class="hljs">seq_length = 5
input_data = tf.random.uniform(shape=[5, 10])

input_tensor = keras.Input(shape=[None, 5, 10])
output = keras_nlp.layers.PositionEmbedding(sequence_length=seq_length)(input_tensor)
model = keras.Model(inputs=input_tensor, outputs=output)
 
model(input_data)
</code></pre>
This results in:
<pre><code class="hljs">&lt;tf.Tensor: shape=(5, 10), dtype=float32, numpy=
array([[ 0.23758471, -0.16798696, -0.15070847, 0.208067 , -0.5123104 ,
 -0.36670157, 0.27487397, 0.14939266, 0.23843127, -0.23328197],
 [-0.51353353, -0.4293166 , -0.30189738, -0.140344 , -0.15444171,
 -0.27691704, 0.14078277, -0.22552207, -0.5952263 , -0.5982155 ],
 [-0.265581 , -0.12168896, 0.46075982, 0.61768025, -0.36352775,
 -0.14212841, -0.26831496, -0.34448475, 0.4418767 , 0.05758983],
 [-0.46500492, -0.19256318, -0.23447984, 0.17891657, -0.01812166,
 -0.58293337, -0.36404118, 0.54269964, 0.3727749 , 0.33238482],
 [-0.2965023 , -0.3390794 , 0.4949159 , 0.32005525, 0.02882379,
 -0.15913549, 0.27996767, 0.4387421 , -0.09119213, 0.1294356 ]],
 dtype=float32)&gt;
</code></pre>
<h2 id="tokenandpositionembedding">TokenAndPositionEmbedding</h2>
Token and position embedding boils down to using <code>Embedding</code> on the input sequence, <code>PositionEmbedding</code> on the embedded tokens, and then adding these two results together, effectively displacing the token embeddings in space to encode their relative meaningful relationships.
This can technically be done as:
<pre><code class="hljs">seq_length = 10
vocab_size = 25
embed_dim = 10

input_data = tf.random.uniform(shape=[5, 10])

input_tensor = keras.Input(shape=[None, 5, 10])
embedding = keras.layers.Embedding(vocab_size, embed_dim)(input_tensor)
position = keras_nlp.layers.PositionEmbedding(seq_length)(embedding)
output = keras.layers.add([embedding, position])
model = keras.Model(inputs=input_tensor, outputs=output)
 
model(input_data).shape # ([5, 10, 10])
</code></pre>
The inputs are embedded, and then positionally embedded, after which they're added together, producing a new positionally embedded shape. Alternatively, you can leverage the <code>TokenAndPositionEmbedding</code> layer, which does this under the hood:
<pre><code class="hljs">... # https://github.com/keras-team/keras-nlp/blob/master/keras_nlp/layers/token_and_position_embedding.py
def call(self, inputs):
 embedded_tokens = self.token_embedding(inputs)
 embedded_positions = self.position_embedding(embedded_tokens)
 outputs = embedded_tokens + embedded_positions
 return outputs
</code></pre>
This makes it much cleaner to perform <code>TokenAndPositionEmbedding</code>:
<pre><code class="hljs">seq_length = 10
vocab_size = 25
embed_dim = 10

input_data = tf.random.uniform(shape=[5, 10])

input_tensor = keras.Input(shape=[None, 5, 10])
output = keras_nlp.layers.TokenAndPositionEmbedding(vocabulary_size=vocab_size, 
 sequence_length=seq_length, 
 embedding_dim=embed_dim)(input_tensor)
model = keras.Model(inputs=input_tensor, outputs=output)
 
model(input_data).shape # ([5, 10, 10])
</code></pre>
The data we've passed into the layer is now positionally embedded in a latent space of 10 dimensions:
<pre><code class="hljs">model(input_data)
</code></pre>
<pre><code class="hljs">&lt;tf.Tensor: shape=(5, 10, 10), dtype=float32, numpy=
array([[[-0.01695484, 0.7656435 , -0.84340465, 0.50211895,
 -0.3162892 , 0.16375223, -0.3774369 , -0.10028353,
 -0.00136751, -0.14690581],
 [-0.05646318, 0.00225556, -0.7745967 , 0.5233861 ,
 -0.22601983, 0.07024342, 0.0905793 , -0.46133494,
 -0.30130145, 0.451248 ],
 ...
</code></pre>
<h2 id="conclusions">Conclusions</h2>
Transformers have made a large wave since 2017, and many great guides offer insight into how they work, yet, they are still elusive to many due to the overhead of custom implementations. KerasNLP addresses this problem, providing building blocks that let you build flexible, powerful NLP systems, rather than providing pre-packaged solutions.
In this guide, we've taken a look at token and position embedding with Keras and KerasNLP.

Transformer Token and Position Embedding with Keras

In a URL, query string values often provide information about the request, like parameters for a search or the ID of an object you're using. If any of the business or request logic is handled in the frontend, it's important to know how to retrieve the query string values from the URL. There are a number of ways to achieve this, a few of which we'll see here.
<h2 id="urlsearchparams">URLSearchParams</h2>
The <code>URLSearchParams</code> interface is supported by all major browser versions except IE 11. It works by parsing the query string of a URL and providing a way to access the values. For example:
<pre><code class="hljs">let params = new URLSearchParams(&#x27;q=node&amp;page=2&#x27;);
params.get(&#x27;q&#x27;); // &#x27;node&#x27;
params.get(&#x27;page&#x27;); // &#x27;2&#x27;
</code></pre>
One of the downsides of this interface is that you must pass it only the query string of a URL. If you're working with the current browser URL, that's easy to do since you can just pass <code>window.location.search</code>. If you're working with any other URL, you'll need to parse out and pass the query string separately.
To parse the query parameters into an object, use <code>URL.searchParams</code>'s <code>.entries()</code> method, which returns an <code>Iterator</code> of key/value pairs, and <code>Object.fromEntries</code> to convert it into an object.
<pre><code class="hljs">let params = new URLSearchParams(&#x27;q=node&amp;page=2&#x27;);
let entries = params.entries();
Object.fromEntries(entries); // {q: &#x27;node&#x27;, page: &#x27;2&#x27;}
</code></pre>
<h2 id="urlobject">URL Object</h2>
The <code>URL</code> API is also supported by all major browser versions except IE 11. It offers a more flexible way to parse URLs, and it also provides a way to access the query string values. For example:
<pre><code class="hljs">const url = new URL(&#x27;https://stackabuse.com/search?q=node&amp;page=2&#x27;);
const searchParams = url.searchParams;

searchParams.get(&#x27;q&#x27;); // &#x27;node&#x27;
searchParams.get(&#x27;page&#x27;); // &#x27;2&#x27;
</code></pre>
<code>url.searchParams</code> is the same type of instance object returned by <code>URLSearchParams</code>.
The <code>url</code> object above also has all parts of the URL broken out into its parts. For example:
<pre><code class="hljs">url.href; // &#x27;https://stackabuse.com/search?q=node&amp;page=2&#x27;
url.origin; // &#x27;https://stackabuse.com&#x27;
url.protocol; // &#x27;https:&#x27;
url.host; // &#x27;stackabuse.com&#x27;
url.hostname; // &#x27;stackabuse.com&#x27;
url.port; // &#x27;&#x27;
url.pathname; // &#x27;/search&#x27;
url.search; // &#x27;?q=node&amp;page=2&#x27;
url.hash; // &#x27;&#x27;
</code></pre>
<h2 id="purejavascript">Pure JavaScript</h2>
If for any reason you're not able to access the APIs above or want to have more control over the parsing, you can use the following code to parse the query string into an object.
<pre><code class="hljs">function getQueryParams(url) {
 const paramArr = url.slice(url.indexOf(&#x27;?&#x27;) + 1).split(&#x27;&amp;&#x27;);
 const params = {};
 paramArr.map(param =&gt; {
 const [key, val] = param.split(&#x27;=&#x27;);
 params[key] = decodeURIComponent(val);
 })
 return params;
}
</code></pre>
Note: There are many ways to parse query params in plain JS, some more complicated (and robust) than others. This is just one way, and was adapted from <a rel="nofollow noopener" target="_blank" href="https://gist.github.com/pirate/9298155edda679510723">this gist</a>.
We can then use this plain JS function to parse a single query param into a string:
<pre><code class="hljs">getQueryParams(&#x27;https://stackabuse.com/search?q=node&amp;page=2&#x27;)
// { q: &#x27;node&#x27;, page: &#x27;2&#x27; }
</code></pre>

Scott Robinson

Get Query String Values in JavaScript

Mistakes are common, we all make them. That's why they put erasers on pencils. And this is no different with tools like Git. While some changes can be difficult to undo, there is usually a way.
So what if you want to modify an existing commit message? Luckily this is simple to do in Git, but the method may change a bit depending on some factors.
<h2 id="fixlastcommitnotyetpushedtoremote">Fix Last Commit, Not Yet Pushed to Remote</h2>
This is the simplest case since there is no need to modify a remote repository, you just need to modify your own. In this case, you can use the following command:
<pre><code class="hljs">$ git commit --amend
</code></pre>
Running this command will open up an editor where you can modify the last commit message. Once you're done and save/close the editor, the commit message will be changed.
Personally, I prefer to do most things on the command line, which you can do by adding an argument and message to the command above:
<pre><code class="hljs">$ git commit --amend -m &quot;Added a new file&quot;
</code></pre>
Doing it this way won't open the editor, but instead it'll just modify the commit message immediately.
<h2 id="fixlastcommitpushedtoremote">Fix Last Commit, Pushed to Remote</h2>
This is another common case, and is a bit more difficult to fix, not only because we need to modify our own repository, but we also need to modify the remote repository, which could be further ahead than your local branch.
The first step is to amend the last commit, just like we did in the previous section:
<pre><code class="hljs">$ git commit --amend -m &quot;Added a new file&quot;
</code></pre>
Then, you need to push these changes to the remote repository. However, this must be done using the <code>--force</code> flag.
<pre><code class="hljs">$ git push &lt;remote&gt; &lt;branch&gt; --force
</code></pre>
We need to do it this way in order to overwrite the remote repository with our own local state.
Be aware that by forcing the push, you'll also lose any commits on the local branch that you don't already have in your local branch. Proceed with caution!
If you can catch the mistake fast enough, then making the change won't be a problem. Otherwise, you may need to find another solution to avoid losing any changes made after the commit message you're wanting to change.
<h2 id="useinteractiverebase">Use Interactive Rebase</h2>
Another option, which is more flexible than the previous solutions described, is to use an interactive rebase. Doing this allows you to inspect the last N commits and make changes to them, including picking, squashing, etc.
To start the rebase, you can use the following command:
<pre><code class="hljs">$ git rebase -i HEAD~n
</code></pre>
Here, <code>n</code> is the number of commits you want to inspect and edit. You'll then see a list of commits, similar to this:
<pre><code class="hljs">$ $ git rebase -i HEAD~4
pick 788ebf0 Oops, I forgot to add a new file
pick ced1329 Fixed a super critical bug
pick 5e9cdc5 Added a new file

# Rebase d05209d..5e9cdc5 onto d05209d (4 commands)
#
# Commands:
# p, pick = use commit
# r, reword = use commit, but edit the commit message
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
# f, fixup = like &quot;squash&quot;, but discard this commit log message
# x, exec = run command (the rest of the line) using shell
# d, drop = remove commit
#
# These lines can be re-ordered; they are executed from top to bottom.
#
# If you remove a line here THAT COMMIT WILL BE LOST.
#
# However, if you remove everything, the rebase will be aborted.
#
# Note that empty commits are commented out
</code></pre>
As you can see all of the last N commits are listed, and you can modify their messages as needed.

Git: Modify an Existing Commit Message

Missing values are common and occur either due to human error, instrument error, processing from another team, or otherwise just a lack of data for a certain observation.
<blockquote>
In this Byte, we'll take a look at how to fill NaNs in a <code>DataFrame</code>, if you choose to handle NaNs by filling them.
</blockquote>
First off, let's create a mock <code>DataFrame</code> with some random values dropped out:
<pre><code class="hljs">import numpy as np
array = np.random.randn(25, 3)
mask = np.random.choice([1, 0], array.shape, p=[.3, .7]).astype(bool)
array[mask] = np.nan

df = pd.DataFrame(array, columns=[&#x27;Col1&#x27;, &#x27;Col2&#x27;, &#x27;Col3&#x27;])
</code></pre>
<pre><code class="hljs"> Col1 Col2 Col3 
0 -0.671603 -0.792415 0.783922
1 0.207720 NaN 0.996131
2 -0.892115 -1.282333 NaN
3 -0.315598 -2.371529 -1.959646
4 NaN NaN -0.584636
5 0.314736 -0.692732 -0.303951
6 0.355121 NaN NaN
7 NaN -1.900148 1.230828
8 -1.795468 0.490953 NaN
9 -0.678491 -0.087815 NaN
10 0.755714 0.550589 -0.702019
11 0.951908 -0.529933 0.344544
12 NaN 0.075340 -0.187669
13 NaN 0.314342 -0.936066
14 NaN 1.293355 0.098964
</code></pre>
Let's plot, say, the third column:
<pre><code class="hljs">plt.plot(df[&#x27;Col3&#x27;])
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-3.png" alt="">
When filled with various techniques - this NaN-filled graph can be replaced with:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-0.png" alt="">
<h2 id="fillnameanmedianmode">fillna() - Mean, Median, Mode</h2>
You can fill these values into a new column and assign it to the column you wish to fill, or in-place using the <code>inplace</code> argument. Here, we'll be extracting the filled values in a new column for ease of inspection:
<pre><code class="hljs">mean = df[&#x27;Col3&#x27;].fillna(df[&#x27;Col3&#x27;].mean(), inplace=False)
median = df[&#x27;Col3&#x27;].fillna(df[&#x27;Col3&#x27;].median(), inplace=False)
mode = df[&#x27;Col3&#x27;].fillna(df[&#x27;Col3&#x27;].mode(), inplace=False)
</code></pre>
The median, mean and mode of the column are <code>-0.187669</code>, <code>-0.110873</code> and <code>0.000000</code> and these values will be used for each NaN respectively. This is effectively filled with constant values, where the value being input depends on the entirety of the column.
First, filling with median values results in:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-4.png" alt="">
With mean values:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-5.png" alt="">
With mode values:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-6.png" alt="">
<h2 id="fillnaconstantvalue">fillna() - Constant Value</h2>
You can also fill with a constant value instead:
<pre><code class="hljs">constant = df[&#x27;Col3&#x27;].fillna(0, inplace=False
</code></pre>
This results in a constant value (0) being put instead of each NaN. <code>0</code> is close to our median and mean and equal to the mode, so the filled values will resemble that method closely for our mock dataset:
<pre><code class="hljs">0 0.783922
1 0.996131
2 0.000000
3 -1.959646
4 -0.584636
5 -0.303951
6 0.000000
7 1.230828
8 0.000000
9 0.000000
10 -0.702019
11 0.344544
12 -0.187669
13 -0.936066
14 0.098964
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-7.png" alt="">
<h2 id="fillnaforwardandbackwardfill">fillna() - Forward and Backward Fill</h2>
On each row - you can do a forward or backward fill, taking the value either from the row before or after:
<pre><code class="hljs">ffill = df[&#x27;Col3&#x27;].fillna(method=&#x27;ffill&#x27;)
bfill = df[&#x27;Col3&#x27;].fillna(method=&#x27;bfill&#x27;)
</code></pre>
With forward-filling, since we're missing from row 2 - the value from row 1 is taken to fill the second one. The values propagate forward:
<pre><code class="hljs">0 0.783922
1 0.996131
2 0.996131
3 -1.959646
4 -0.584636
5 -0.303951
6 -0.303951
7 1.230828
8 1.230828
9 1.230828
10 -0.702019
11 0.344544
12 -0.187669
13 -0.936066
14 0.098964
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-8.png" alt="">
With backward-filling, the opposite happens. Row 2 is filled with the value from row 3:
<pre><code class="hljs">0 0.783922
1 0.996131
2 -1.959646
3 -1.959646
4 -0.584636
5 -0.303951
6 1.230828
7 1.230828
8 -0.702019
9 -0.702019
10 -0.702019
11 0.344544
12 -0.187669
13 -0.936066
14 0.098964
</code></pre>
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-9.png" alt="">
Though, if there's more than one <code>NaN</code> in a sequence - these won't do well and can cascade NaNs further down, skewing the data and removing actually recorded values.
<h2 id="interpolate">interpolate()</h2>
The <code>interpolate()</code> method delegates the interpolation of values to SciPy's <a rel="nofollow noopener noreferrer" target="_blank" href="https://docs.scipy.org/doc/scipy/reference/interpolate.html">suite of methods for interpolating values</a>. It accepts a wide variety of arguments, including, <code>nearest</code>, <code>zero</code>, <code>slinear</code>, <code>quadratic</code>, <code>cubic</code>, <code>spline</code>, <code>barycentric</code>, <code>polynomial</code>, <code>krogh</code>, <code>piecewise_polynomial</code>, <code>spline</code>, <code>pchip</code>, <code>akima</code>, <code>cubicspline</code>, etc.
<blockquote>
Interpolation is much more flexible and &quot;smart&quot; than just filling values with constants or half-variables such as previous methods.
</blockquote>
Interpolation can properly fill a sequence in a way that no other methods can, such as:
<pre><code class="hljs">s = pd.Series([0, 1, np.nan, np.nan, np.nan, 5])
s.fillna(s.mean()).values
# array([0., 1., 2., 2., 2., 5.])
s.fillna(method=&#x27;ffill&#x27;).values
# array([0., 1., 1., 1., 1., 5.])
s.interpolate().values
# array([0., 1., 2., 3., 4., 5.])
</code></pre>
The default interpolation is linear, and assuming that <code>1...5</code> is likely a <code>1, 2, 3, 4, 5</code> sequence isn't far-fetched (but isn't guaranteed). Both constant filling and forward or backward-filling fail miserably here. Generally speaking - interpolation is usually going to be a good friend when it comes to filling NaNs in noisy signals, or corrupt datasets.
Experimenting with types of interpolation may yield better results.
Here are two interpolation methods (<code>splice</code> and <code>polynomial</code> require an <code>order</code> argument):
<pre><code class="hljs">nearest = df[&#x27;Col3&#x27;].interpolate(method=&#x27;nearest&#x27;)
polynomial = df[&#x27;Col3&#x27;].interpolate(method=&#x27;polynomial&#x27;, order=3)
</code></pre>
These result in:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-10.png" alt="">
And:
<img src="https://s3.stackabuse.com/media/articles/how-to-fill-nans-in-a-pandas-dataframe-11.png" alt="">

In this tutorial - learn how to fill NaNs in a Pandas DataFrame using the mean, median, mode, constant value, backward and forward-fill methods as well as how to interpolate missing values with Python and Scipy.