Lucas Otero

Building GraphQL APIs with Vue.js and Apollo Client

<h2 id="introduction">Introduction</h2>
Deploying an application requires developers to put thought and consideration into how it is configured. Many apps are deployed in a development environment before being deployed to the production environment. We need to ensure each environment is configured correctly, it could be disastrous if our production application was using our development database, for example.
Environment variables allow us to manage the configuration of our applications separate from our codebase. Separating configurations make it easier for our application to be deployed in different environments.
Large applications tend to have many environment variables. To better manage them we can use the <a rel="nofollow" target="_blank" href="https://www.npmjs.com/package/dotenv">dotenv</a> library, which allows us to load environment variables from a file.
<h2 id="whatareenvironmentvariables">What are Environment Variables?</h2>
From programming we know that variables are stored values that can be changed. They're mutable and can vary - hence the name variables.
Environment variables are variables external to our application which reside in the OS or the container of the app is running in. An environment variable is simply a name mapped to a value.
By convention, the name is capitalized e.g. <code>SENDER_EMAIL_ADDRESS=me@thisemail.com</code>. The values are strings.
If you open the terminal or command line application in Linux, Mac OS, or Windows and enter <code>set</code>, you will see a list of all environment variables for your user.
<h2 id="whyuseenvironmentvariables">Why use Environment Variables?</h2>
Environment variables are excellent for decoupling application configurations. Typically, our applications require many variables to be set in order for them to work. By relying on external configurations, your app can easily be deployed on different environments. These changes are independent of code changes, so they do not require your application to be rebuilt to change.
Data which changes depending on the environment your app is running on should be set as environment variables. Some common examples are:
<ul>
<li>HTTP Port and Address</li>
<li>Database, cache, and other storage connection information</li>
<li>Location of static files/folders</li>
<li>Endpoints of external services
<ul>
<li>For example, in a development environment your app will point to a test API URL, whereas in a production environment your app will point to the live API URL.</li>
</ul>
</li>
</ul>
Sensitive data like API keys should not be in the source code, or known to persons who do not need access to those external services.
<h2 id="environmentvariablesinnodejs">Environment Variables in Node.js</h2>
Consider a hello world <a rel="nofollow" target="_blank" href="https://nodejs.org/">Node.js</a> application with environment variables for the host and port the app runs on.
Create a new file called <code>hello.js</code> in a workspace of your choosing and add the following:
<pre><code class="hljs">const http = require(&#x27;http&#x27;);

// Read the host address and the port from the environment
const hostname = process.env.HOST;
const port = process.env.PORT;

// Return JSON regardless of HTTP method or route our web app is reached by
const server = http.createServer((req, res) =&gt; {
 res.statusCode = 200;
 res.setHeader(&#x27;Content-Type&#x27;, &#x27;application/json&#x27;);
 res.end(`{&quot;message&quot;: &quot;Hello World&quot;}`);
});

// Start a TCP server listening for connections on the given port and host
server.listen(port, hostname, () =&gt; {
 console.log(`Server running at http://${hostname}:${port}/`);
});
</code></pre>
Node.js provides a global variable <code>process.env</code>, an object that contains all environment variables available to the user running the application. It is expecting the hostname and the port that the app will run on to be defined by the environment.
You can run this application by entering this command in the terminal, <code>HOST=localhost PORT=3000 node hello.js</code>, granted you have Node.js installed. You will notice the following message on your console:
<pre><code class="hljs">Server running at http://localhost:3000/
</code></pre>
Creating and reading environment variables is that easy. Let's say we are writing a micro-service which communicates with a <a rel="nofollow" target="_blank" href="https://www.mysql.com/">MySQL</a> and <a rel="nofollow" target="_blank" href="https://redis.io/">Redis</a> instance, we would like to have the connection details available in our environment variables as well.
We might end up with a command like this:
<pre><code class="hljs">$ DB_HOST=localhost DB_PORT=3306 DB_USER=test DB_PASSWORD=verySecret123!$ DB_MAX_CONNECTIONS=15 CACHE_ADDR=localhost CACHE_PORT= 6379 HOST=localhost PORT=3000 node hello.js
</code></pre>
Developing with multiple environment variables quickly becomes unwieldy. It would be better if the environment variables we had to configure were stored in one central place to the application, like a file.
<h2 id="thedotenvlibrary">The <code>dotenv</code> Library</h2>
This library does one simple task: loads environment variables from a <code>.env</code> file into the <code>process.env</code> variable in Node.js. Let's use <code>dotenv</code> for the previous example.
First, we need to install it via <code>npm</code>:
<pre><code class="hljs">$ npm init # Optional, creates a local package.json that prevents global installs
$ npm install dotenv --save
</code></pre>
And then we make one minimal code change, add this line to the top of <code>hello.js</code>:
<pre><code class="hljs">require(&#x27;dotenv&#x27;).config()
</code></pre>
Now in the same directory of your app, create a new file called <code>.env</code> and add the following:
<pre><code class="hljs">HOST=localhost
PORT=3000
</code></pre>
Your environment variables are declared just as if you were entering them in the shell before running the node command. Instead of separating each variable assignment by a space, we separate them by a newline. Now in your shell, if you enter <code>node hello.js</code>, the app will be running on port <code>3000</code> on your <code>localhost</code>!
Even if <code>dotenv</code> is used in the application, it is completely optional. If no <code>.env</code> file is found, the library fails silently. You can continue to use environment variables defined outside the file.
Note: There are also many variations of <code>dotenv</code> across other programming languages and frameworks. A very popular guideline for building modern web applications is <a rel="nofollow" target="_blank" href="https://12factor.net/">The Twelve-Factor App</a>. The document was created by many developers who created Software-as-a-Service applications. One of the 12 considerations of a modern, scalable app is configuration, specifically the use of environment variables to store configuration. The popularity of this methodology has helped fuel the creation and popularity of <code>dotenv</code> and similar modules.
<h3 id="productionusage">Production Usage</h3>
Storing your environment variables in a file comes with one golden rule: never commit it to the source code repository. You do not want outsiders gaining access to secrets, like API keys. If you are using <code>dotenv</code> to help manage your environment variables, be sure to include the <code>.env</code> file in your <code>.gitignore</code> or the appropriate blacklist for your version control tool.
If you cannot commit the <code>.env</code> file, then there needs to be some way for a developer to know what environment variables are required to run the software. It is common for developers to list the environment variables necessary to run the program in a <code>README</code> or similar internal documentation.
Some developers create and maintain a <code>.sample-env</code> file in the source code repository. This sample file would list all the environment variables used by the app, for example:
<pre><code class="hljs">HOST=
PORT=
</code></pre>
A developer would then clone the repository, copy the <code>.sample-env</code> file into a new <code>.env</code> file and fill in the values.
If your app is running on a physical machine or a virtual machine (for example, <a rel="nofollow" target="_blank" href="https://www.digitalocean.com/products/droplets/">Digital Ocean droplets</a>, <a rel="nofollow" target="_blank" href="https://aws.amazon.com/ec2/">Amazon EC2</a> and <a rel="nofollow" target="_blank" href="https://azure.microsoft.com/en-us/services/virtual-machines/">Azure Virtual Machines</a>), then you can create a <code>.env</code> while logged into the server and it would run just as it is done on your local machine.
If your app is running on a docker container or a Platform-as-a-Service provider like <a rel="nofollow" target="_blank" href="https://www.heroku.com/">Heroku</a> or <a rel="nofollow" target="_blank" href="https://www.openshift.com/">OpenShift</a>, then you will be able to configure environment variables without having to use the <code>.env</code> file.
Remember, it fails silently so it would not affect the running of the app if the file is missing.
<h2 id="conclusion">Conclusion</h2>
Environment variables exist outside our application's code, they are available where our application is running. They can be used to decouple our application's configuration from its code, which allows our apps to be easily deployed across different environments.
With Node.js apps, environment variables are available through the <code>process.env</code> global variable. We can set the environment variables before we run the <code>node</code> command, or we can use the <code>dotenv</code> library which allows us to define our environment variables in a <code>.env</code> file.
The <code>.env</code> file should never be in the source code repository.

Marcus Sanatan

Managing Environment Variables in Node.js with dotenv

This article is the second in a series on working with PDFs in Python:
<ul>
<li><a target="_blank" href="/working-with-pdfs-in-python-adding-images-and-watermarks/">Reading and Splitting Pages</a></li>
<li>Adding Images and Watermarks (you are here)</li>
<li><a target="_blank" href="/working-with-pdfs-in-python-inserting-deleting-and-reordering-pages/">Inserting, Deleting, and Reordering Pages</a></li>
</ul>
<h2 id="introduction">Introduction</h2>
Today, a world without the Portable Document Format (PDF) seems to be unthinkable. It has become one of the most commonly used data formats ever. Up to PDF version 1.4, displaying a PDF document in an according PDF viewer works fine. Unfortunately, the features from the newer PDF revisions, such as forms, are tricky to implement, and still require further work to be fully functional in the tools. Using various Python libraries you can create your own application in a comparable easy way.
This article is part two of a little series on PDFs with Python. In <a target="_blank" href="https://stackabuse.com/working-with-pdfs-in-python-reading-and-splitting-pages/">part one</a> we already gave you an introduction into reading PDF documents using Python, and started with a summary of the various Python libraries. An introduction followed that showed how to manipulate existing PDFs, and how to read and extract the content - both the text and images. Furthermore, we showed you how to split documents into its single pages.
In this article you will learn how to add images to your PDF in the form of watermarks, stamps, and barcodes. For example this is quite helpful in order to stamp or mark documents that are intended to be read by a specific audience, only, or have a draft quality, or to simply add a barcode for identification purposes.
<h2 id="addingawatermarkviacommandlinewithpdftk">Adding a Watermark via Command Line with <code>pdftk</code></h2>
In order to add a watermark to an existing PDF on a Unix/Linux command-line we can use <a rel="nofollow" target="_blank" href="https://www.pdflabs.com/tools/pdftk-the-pdf-toolkit/">pdftk</a>. The name abbreviates &quot;PDF Toolkit&quot;, and describes itself as &quot;a simple tool for doing everyday things with PDF documents&quot;. <code>pdftk</code> has been ported to Java, and made available as an according <a rel="nofollow" target="_blank" href="https://packages.debian.org/buster/pdftk-java">package</a> for Debian GNU/Linux.
In order for this to work you need to have a background image available that comes with the word &quot;DRAFT&quot; on a transparent layer, which you can apply to an existing single-page PDF as follows:
<pre><code class="hljs">$ pdftk input.pdf background background.pdf output output.pdf
</code></pre>
The <code>pdftk</code> tool takes in the PDF file <code>input.pdf</code>, merges it with <code>background.pdf</code>, and outputs the result to the file <code>output.pdf</code>. Figure 1 shows the output of this action.
<img class="img-responsive" alt="Adding a watermark to an existing PDF" src="https://s3.amazonaws.com/stackabuse/media/working-with-pdfs-python-adding-images-watermarks-1.png">
For more complex actions, like stamping a document with different stamps per page, have a look at the description at the PDF Labs <a rel="nofollow" target="_blank" href="https://www.pdflabs.com/docs/how-to-add-headers-footers-watermarks-and-stamps-to-pdf/">project page</a>. We also show the stamping use-case in this article below, although our example uses the library <code>pdfrw</code> instead of <code>pdftk</code>.
<h2 id="addingawatermarkwithpypdf2">Adding a Watermark with PyPDF2</h2>
The <a rel="nofollow" target="_blank" href="https://pypi.org/project/PyPDF2/">PyPDF library</a> provides a method called <code>mergepage()</code> that accepts another PDF to be used as a watermark or stamp.
In the example below we start with reading the first page of the original PDF document and the watermark. To read the file we use the <code>PdfFileReader()</code> class. As a second step we merge the two pages by using the <code>mergepage()</code> method. Finally, we will write the output to the output file. This is done in three steps - creating an object based on the <code>PdfFileWriter()</code> class, adding the merged page to this object using the <code>addPage()</code> method, and writing the new content to the output page using the <code>write()</code> method.
<img class="img-responsive" alt="Adding a watermark to an existing PDF using PyPDF" src="https://s3.amazonaws.com/stackabuse/media/working-with-pdfs-python-adding-images-watermarks-2.png">
<pre><code class="hljs"># !/usr/bin/python
# Adding a watermark to a single-page PDF

import PyPDF2

input_file = &quot;example.pdf&quot;
output_file = &quot;example-drafted.pdf&quot;
watermark_file = &quot;draft.pdf&quot;

with open(input_file, &quot;rb&quot;) as filehandle_input:
 # read content of the original file
 pdf = PyPDF2.PdfFileReader(filehandle_input)
 
 with open(watermark_file, &quot;rb&quot;) as filehandle_watermark:
 # read content of the watermark
 watermark = PyPDF2.PdfFileReader(filehandle_watermark)
 
 # get first page of the original PDF
 first_page = pdf.getPage(0)
 
 # get first page of the watermark PDF
 first_page_watermark = watermark.getPage(0)
 
 # merge the two pages
 first_page.mergePage(first_page_watermark)
 
 # create a pdf writer object for the output file
 pdf_writer = PyPDF2.PdfFileWriter()
 
 # add page
 pdf_writer.addPage(first_page)
 
 with open(output_file, &quot;wb&quot;) as filehandle_output:
 # write the watermarked file to the new file
 pdf_writer.write(filehandle_output)
</code></pre>
<h2 id="addinganimagewithpymupdf">Adding an Image with PyMuPDF</h2>
<a rel="nofollow" target="_blank" href="https://github.com/pymupdf/PyMuPDF">PyMuPDF</a> are the Python bindings for MuPDF, which is a lightweight PDF and XPS viewer. In your Python script the module that needs to be imported is named <code>fitz</code>, and this name goes back to the previous name of PyMuPDF.
For this section we are going to show how to add an image by using a barcode as an example since this is a pretty common task. Although the same steps can be applied to adding any kind of image to a PDF.
In order to decorate a PDF document with a barcode we simply add an image as another PDF layer at the desired position. As for image formats, PyMuPDF accepts PNG or JPEG, but not SVG.
The position of the image is defined as a rectangle using the method <code>fitz.Rect()</code> that requires two pairs of coordinates - (x1,y1) and (x2,y2). PyMuPDF interprets the upper-left corner of the page as (0,0).
Having opened the input file and extracted the first page from it, the image containing the barcode is added using the method <code>insertImage()</code>. This method requires two parameters - the position delivered via <code>imageRectangle</code>, and the name of the image file to be inserted. Using the <code>save()</code> method the modified PDF is stored to disk. Figure 2 shows the barcode after it was added to the example PDF.
<img class="img-responsive" alt="Adding a barcode to an existing PDF using PyMuPDF" src="https://s3.amazonaws.com/stackabuse/media/working-with-pdfs-python-adding-images-watermarks-3.png">
<pre><code class="hljs"># !/usr/bin/python

import fitz

input_file = &quot;example.pdf&quot;
output_file = &quot;example-with-barcode.pdf&quot;
barcode_file = &quot;barcode.png&quot;

# define the position (upper-right corner)
image_rectangle = fitz.Rect(450,20,550,120)

# retrieve the first page of the PDF
file_handle = fitz.open(input_file)
first_page = file_handle[0]

# add the image
first_page.insertImage(image_rectangle, fileName=barcode_file)

file_handle.save(output_file)
</code></pre>
<h2 id="addingstampswithpdfrw">Adding Stamps with <code>pdfrw</code></h2>
<a rel="nofollow" target="_blank" href="https://pypi.org/project/pdfrw/">pdfrw</a> is a pure Python-based PDF parser to read and write PDF documents. It faithfully reproduces vector formats without rasterization. For Debian GNU/Linux, the package repository contains releases for both Python 2 and 3.
The following example will demonstrate how to add a barcode or watermark to an existing PDF that contains multiple pages. From the <code>pdfrw</code> package it is sufficient that you import the three classes <code>PdfReader</code>, <code>PdfWriter</code>, and <code>PageMerge</code>. Next, you establish the according reader and writer objects to access the contents of both the PDF, and the watermark. For each page in the original document you continue creating a <code>PageMerge</code> object to which you add the watermark, and which is rendered using the <code>render()</code> method. Finally, you write the modified pages to the output file. Figure 3 shows the modified document next to the code that made the addition possible.
<img class="img-responsive" alt="Adding a watermark to an existing PDF using pdfrw" src="https://s3.amazonaws.com/stackabuse/media/working-with-pdfs-python-adding-images-watermarks-4.png">
<pre><code class="hljs"># !/usr/bin/python
# Adding a watermark to a multi-page PDF

from pdfrw import PdfReader, PdfWriter, PageMerge

input_file = &quot;example.pdf&quot;
output_file = &quot;example-drafted.pdf&quot;
watermark_file = &quot;barcode.pdf&quot;

# define the reader and writer objects
reader_input = PdfReader(input_file)
writer_output = PdfWriter()
watermark_input = PdfReader(watermark_file)
watermark = watermark_input.pages[0]

# go through the pages one after the next
for current_page in range(len(reader_input.pages)):
 merger = PageMerge(reader_input.pages[current_page])
 merger.add(watermark).render()

# write the modified content to disk
writer_output.write(output_file, reader_input)
</code></pre>
<h2 id="conclusion">Conclusion</h2>
Adding images, watermarks, or stamps to a PDF file is quite simple. With a few lines of code this complex-sounding task is solved within minutes. No matter which of the given libraries you choose it works very well.
Part three of this series will exclusively focus on writing/creating PDFs, and will also include both deleting and re-combining single pages into a new document.

Frank Hofmann

Working with PDFs in Python: Adding Images and Watermarks

Robley Gori

Spring Security: Forgot Password Functionality

<h2 id="introduction">Introduction</h2>
Reinforcement Learning is definitely one of the most active and stimulating areas of research in AI.
The interest in this field grew exponentially over the last couple of years, following great (and greatly publicized) advances, such as <a rel="nofollow" target="_blank" href="https://www.deepmind.com/blog/alphago-zero-starting-from-scratch">DeepMind's AlphaGo</a> beating the word champion of GO, and OpenAI AI models <a rel="nofollow" target="_blank" href="https://venturebeat.com/2019/04/22/openais-dota-2-bot-defeated-99-4-of-players-in-public-matches/">beating professional DOTA players</a>.
Thanks to all of these advances, Reinforcement Learning is now being applied in a variety of different fields, from health-care to finance, from chemistry to resource management.
In this article, we will introduce the fundamental concepts and terminology of Reinforcement Learning, and we will apply them in a practical example.
<h2 id="whatisreinforcementlearning">What is Reinforcement Learning?</h2>
<a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Reinforcement_learning">Reinforcement Learning</a> (RL) is a branch of machine learning concerned with actors, or <a target="_blank" rel="noopener noreferrer" href="https://writesonic.com/blog/what-is-an-ai-agent">AI agents</a>, taking actions in some kind of environment in order to maximize some type of reward that they collect along the way.
This is deliberately a very loose definition, which is why reinforcement learning techniques can be applied to a very wide range of real-world problems.
Imagine someone playing a video game. The player is the agent, and the game is the environment. The rewards the player gets (i.e. beat an enemy, complete a level), or doesn't get (i.e. step into a trap, lose a fight) will teach him how to be a better player.
As you've probably noticed, reinforcement learning doesn't really fit into the categories of supervised/unsupervised/semi-supervised learning.
In supervised learning, for example, each decision taken by the model is independent, and doesn't affect what we see in the future.
In reinforcement learning, instead, we are interested in a long term strategy for our agent, which might include sub-optimal decisions at intermediate steps, and a trade-off between exploration (of unknown paths), and exploitation of what we already know about the environment.
<h2 id="briefhistoryofreinforcementlearning">Brief History of Reinforcement Learning</h2>
For several decades (since the 1950s!), reinforcement learning followed two separate threads of research, one focusing on trial and error approaches, and one based on optimal control.
Optimal control methods are aimed at designing a controller to minimize a measure of a dynamical system's behavior over time. To achieve this, they mainly used dynamic programming algorithms, which we will see are the foundations of modern reinforcement learning techniques.
Trial-and-error approaches, instead, have deep roots in the psychology of animal learning and neuroscience, and this is where the term reinforcement comes from: actions followed (reinforced) by good or bad outcomes have the tendency to be reselected accordingly.
Arising from the interdisciplinary study of these two fields came a field called <a rel="nofollow" target="_blank" href="http://www.scholarpedia.org/article/TD-learning">Temporal Difference (TD) Learning</a>.
The modern machine learning approaches to RL are mainly based on TD-Learning, which deals with rewards signals and a value function (we'll see more in detail what these are in the following paragraphs).
<h2 id="terminology">Terminology</h2>
We will now take a look at the main concepts and terminology of Reinforcement Learning.
<h3 id="agent">Agent</h3>
A system that is embedded in an environment, and takes actions to change the state of the environment. Examples include mobile robots, software agents, or industrial controllers.
<h3 id="environment">Environment</h3>
The external system that the agent can &quot;perceive&quot; and act on.
Environments in RL are defined as Markov Decision Processes (MDPs). A MDP is a tuple:
$$ 
(S, A, P, R, \gamma) 
$$
where:
<ul>
<li>S is a finite set of states</li>
<li>A is a finite set of actions</li>
<li>P is a state transition probability matrix</li>
</ul>
<div>
$$
P_{ss'}^{a} = \mathbb{P}[S_{t+1} = s'| S_t = s, A_t = a]
$$
</div>
<ul>
<li>R is a reward function</li>
</ul>
<div>
$$
R_s^a = \mathbb{E}[R_{t+1}|S_t=s, A_t = a]
$$
</div>
<ul>
<li>γ is a discount factor, γ ∈ [0,1]</li>
</ul>
<img class="img-responsive" alt="Markov decision process" src="https://s3.amazonaws.com/stackabuse/media/introduction-to-reinforcement-learning-with-python-1.png">
A lot of real-world scenarios can be represented as <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Markov_decision_process">Markov Decision Processes</a>, from a simple chess board to a much more complex video game.
In a chess environment, the states are all the possible configurations of the board (there are a lot). The actions refer to moving the pieces, surrendering, etc.
The rewards are based on whether we win or lose the game, so that winning actions have higher return than losing ones.
State transition probabilities enforce the game rules. For example, an illegal action (move a rook diagonally) will have zero probability.
<h3 id="rewardfunction">Reward Function</h3>
The reward function maps states to their rewards. This is the information that the agents use to learn how to navigate the environment.
A lot of research goes into designing a good reward function and overcoming the problem of sparse rewards, when the often sparse nature of rewards in the environment doesn't allow the agent to learn properly from it.
Return Gt is defined as the discounted sum of rewards from time step t.
<div>
$$
G_t=\sum_{k=0}^{\infty} \gamma^k R_{t+k+1}
$$
</div>
γ is called the discount factor, and it works by reducing the amount of the rewards as we move into the future.
Discounting rewards allows us to represent uncertainty about the future, but it also helps us model human behavior better, since it has been shown that humans/animals have a preference for immediate rewards.
<h3 id="valuefunction">Value Function</h3>
The value function is probably the most important piece of information we can hold about a RL problem.
Formally, the value function is the expected return starting from state s. In practice, the value function tells us how good it is for the agent to be in a certain state. The higher the value of a state, the higher the amount of reward we can expect:
<div>
$$
v_\pi (s) = \mathbb{E}_\pi [G_t|S_t = s]
$$
</div>
The actual name for this function is state-value function, to distinguish it from another important element in RL: the action-value function.
The action-value function gives us the value, i.e. the expected return, for using action a in a certain state s:
<div>
$$
q_\pi (s, a) = \mathbb{E}_\pi [G_t|S_t = s, A_t = a]
$$
</div>
<h3 id="policy">Policy</h3>
The policy defines the behavior of our agent in the MDP.
Formally, policies are distributions over actions given states. A policy maps states to the probability of taking each action from that state:
<div>
$$
\pi (a|s) = \mathbb{P}[A_t = a|S_t=s]
$$
</div>
The ultimate goal of RL is to find an optimal (or a good enough) policy for our agent. In the video game example, you can think of the policy as the strategy that the player follows, i.e, the actions the player takes when presented with certain scenarios.
<h2 id="mainapproaches">Main Approaches</h2>
A lot of different models and algorithms are being applied to RL problems.
Really, <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Reinforcement_learning#Comparison_of_reinforcement_learning_algorithms">a lot</a>.
However, all of them more or less fall into the same two categories: policy-based, and value-based.
<h3 id="policybasedapproach">Policy-Based Approach</h3>
In policy-based approaches to RL, our goal is to learn the best possible policy. Policy models will directly output the best possible move from the current state, or a distribution over the possible actions.
<h3 id="valuebasedapproach">Value-Based Approach</h3>
In value-based approaches, we want to find the optimal value function, which is the maximum value function over all policies.
We can then choose which actions to take (i.e. which policy to use) based on the values we get from the model.
<h2 id="explorationvsexploitation">Exploration vs Exploitation</h2>
The trade-off between exploration and exploitation has been widely studied in the RL literature.
Exploration refers to the act of visiting and collecting information about states in the environment that we have not yet visited, or about which we still don't have much information. The idea is that exploring our MDP might lead us to better decisions in the future.
On the other side, exploitation consists of making the best decision given current knowledge, comfortable in the bubble of the already known.
We will see in the following example how these concepts apply to a real problem.
<h2 id="amultiarmedbandit">A Multi-Armed Bandit</h2>
We will now look at a practical example of a Reinforcement Learning problem - the multi-armed bandit problem.
The multi-armed bandit is one of the most popular problems in RL:
<blockquote>
You are faced repeatedly with a choice among k different options, or actions. After each choice you receive a numerical reward chosen from a stationary probability distribution that depends on the action you selected. Your objective is to maximize the expected total reward over some time period, for example, over 1000 action selections, or time steps.
</blockquote>
You can think of it in analogy to a slot machine (a one-armed bandit). Each action selection is like a play of one of the slot machine’s levers, and the rewards are the payoffs for hitting the jackpot.
<img class="img-responsive" alt="Bandit machine" src="https://s3.amazonaws.com/stackabuse/media/introduction-to-reinforcement-learning-with-python-2.jpg">
Solving this problem means that we can come come up with an optimal policy: a strategy that allows us to select the best possible action (the one with the highest expected return) at each time step.
<h3 id="actionvaluemethods">Action-Value Methods</h3>
A very simple solution is based on the action value function. Remember that an action value is the mean reward when that action is selected:
<div>
$$
q(a) = E[R_t \mid A=a]
$$
</div>
We can easily estimate q using the sample average:
<div>
$$
Q_t(a) = \frac{\text{sum of rewards when "a" taken prior to "t"}}{\text{number of times "a" taken prior to "t"}}
$$
</div>
If we collect enough observations, our estimate gets close enough to the real function. We can then act greedily at each timestep, i.e. select the action with the highest value, to collect the highest possible rewards.
<h3 id="dontbetoogreedy">Don't be too Greedy</h3>
Remember when we talked about the trade-off between exploration and exploitation? This is one example of why we should care about it.
As a matter of fact, if we always act greedily as proposed in the previous paragraph, we never try out sub-optimal actions which might actually eventually lead to better results.
To introduce some degree of exploration in our solution, we can use an ε-greedy strategy: we select actions greedily most of the time, but every once in a while, with probability ε, we select a random action, regardless of the action values.
It turns out that this simple exploration method works very well, and it can significantly increase the rewards we get.
One final caveat - to avoid from making our solution too computationally expensive, we compute the average incrementally according to this formula:
<div>
$$
Q_{n+1} = Q_n + \frac{1}{n}[R_n - Q_n]
$$
</div>
<h3 id="pythonsolutionwalkthrough">Python Solution Walkthrough</h3>
<pre><code class="hljs">import numpy as np

# Number of bandits
k = 3

# Our action values
Q = [0 for _ in range(k)]

# This is to keep track of the number of times we take each action
N = [0 for _ in range(k)]

# Epsilon value for exploration
eps = 0.1

# True probability of winning for each bandit
p_bandits = [0.45, 0.40, 0.80]

def pull(a):
 &quot;&quot;&quot;Pull arm of bandit with index `i` and return 1 if win, 
 else return 0.&quot;&quot;&quot;
 if np.random.rand() &lt; p_bandits[a]:
 return 1
 else:
 return 0

while True:
 if np.random.rand() &gt; eps:
 # Take greedy action most of the time
 a = np.argmax(Q)
 else:
 # Take random action with probability eps
 a = np.random.randint(0, k)
 
 # Collect reward
 reward = pull(a)
 
 # Incremental average
 N[a] += 1
 Q[a] += 1/N[a] * (reward - Q[a])
</code></pre>
Et voilà! If we run this script for a couple of seconds, we already see that our action values are proportional to the probability of hitting the jackpots for our bandits:
<pre><code class="hljs">0.4406301434281669, 
0.39131455399060977, 
0.8008844354479673
</code></pre>
This means that our greedy policy will correctly favor actions from which we can expect higher rewards.
<h2 id="conclusion">Conclusion</h2>
Reinforcement Learning is a growing field, and there is a lot more to cover. In fact, we still haven't looked at general-purpose algorithms and models (e.g. dynamic programming, Monte Carlo, Temporal Difference).
The most important thing right now is to get familiar with concepts such as value functions, policies, and MDPs. In the Resources section of this article, you'll find some awesome resources to gain a deeper understanding of this kind of material.
<h2 id="resources">Resources</h2>
<ul>
<li><a rel="nofollow" target="_blank" href="https://www.youtube.com/watch?v=2pWv7GOvuf0">David Silver class on RL</a></li>
<li><a rel="nofollow" target="_blank" href="http://incompleteideas.net/book/the-book-2nd.html">Sutton and Barto - Reinforcement Learning: An Introduction</a></li>
</ul>

Daniele Paliotta

Introduction to Reinforcement Learning with Python

Usman Malik

Python for NLP: Working with the Gensim Library (Part 1)

Test Driven Development with pytest

Dan Nelson

Overview of Classification Methods in Python with Scikit-Learn

<h2 id="introduction">Introduction</h2>
Lambda functions have been an addition that came with <a rel="nofollow" target="_blank" href="https://www.oracle.com/technetwork/java/javase/overview/java8-2100321.html">Java 8</a>, and was the language's first step towards <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Functional_programming">functional programming</a>, following a general trend toward implementing useful features of various compatible <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Programming_paradigm">paradigms</a>.
The motivation for introducing lambda functions was mainly to reduce the cumbersome repetitive code that went into passing along class instances to simulate anonymous functions of other languages.
Here's an example:
<pre><code class="hljs">String[] arr = { &quot;family&quot;, &quot;illegibly&quot;, &quot;acquired&quot;, &quot;know&quot;, &quot;perplexing&quot;, &quot;do&quot;, &quot;not&quot;, &quot;doctors&quot;, &quot;where&quot;, &quot;handwriting&quot;, &quot;I&quot; };

Arrays.sort(arr, new Comparator&lt;String&gt;() {
 @Override public int compare(String s1, String s2) {
 return s1.length() - s2.length();
 }
});

System.out.println(Arrays.toString(arr));
</code></pre>
As you can see, the whole bit of instancing a new <a rel="nofollow" target="_blank" href="/sorting-algorithms-in-java/#sortinginjava">Comparator</a> class and overriding its contents is a snippet of repetitive code that we may as well do without, since it's always the same.
The entire <code>Arrays.sort()</code> line can be replaced by something much shorter and sweeter, but functionally equivalent:
<pre><code class="hljs">Arrays.sort(arr, (s1,s2) -&gt; s1.length() - s2.length());
</code></pre>
These short and sweet bits of code that do the same thing as their verbose counterparts are called <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Syntactic_sugar">syntactic sugar</a>. This is because they don't add functionality to a language, but instead make it more compact and readable. Lambda functions are an example of syntactic sugar for Java.
Although I highly suggest reading this article in order, if you're unfamiliar with the topic then here's a quick list of what we'll cover for easier reference:
<ul>
<li><a href="#lambdasasobjects">Lambdas as Objects</a>
<ul>
<li><a href="#singlemethodinterfacematching">Single-method Interface Matching</a></li>
</ul>
</li>
<li><a href="#implementation">Implementation</a>
<ul>
<li><a href="#parameters">Parameters</a></li>
<li><a href="#body">Body</a></li>
<li><a href="#variablecapture">Variable Capture</a></li>
<li><a href="#methodreferencing">Method Referencing</a>
<ul>
<li><a href="#staticmethodreference">Static Method Reference</a></li>
<li><a href="#parametermethodreference">Parameter Method Reference</a></li>
<li><a href="#instancemethodreference">Instance Method Reference</a></li>
<li><a href="#constructormethodreference">Constructor Method Reference</a></li>
</ul>
</li>
</ul>
</li>
</ul>
<h2 id="lambdasasobjects">Lambdas as Objects</h2>
Before we get into the nitty-gritty of lambda syntax itself, we should take a look at what lambda functions are in the first place and how are they used.
As mentioned, they're simply syntactic sugar, but they're syntactic sugar specifically for objects implementing a single method interface.
In those objects, the lambda implementation is considered the said method's implementation. If the lambda and the interface match, the lambda function can be assigned to a variable of that interface's type.
<h3 id="singlemethodinterfacematching">Single-Method Interface Matching</h3>
In order to match a lambda to a single method interface, also called a &quot;functional interface&quot;, several conditions need to be met:
<ul>
<li>The functional interface has to have exactly one unimplemented method, and that method (naturally) has to be abstract. The interface can contain static and default methods implemented within it, but what's important is that there's exactly one abstract method.</li>
<li>The abstract method has to accept arguments, in the same order, that correspond to the parameters lambda accepts.</li>
<li>The return type of both the method and the lambda function have to match.</li>
</ul>
If all of that is satisfied, all the conditions for matching have been made and you can assign your lambda to the variable.
Let's define our interface:
<pre><code class="hljs">public interface HelloWorld {
 abstract void world();
}
</code></pre>
As you can see, we have a pretty useless functional interface.
It contains exactly one function, and that function can do anything at all, as long as it accepts no arguments and returns no values.
We're going to make a simple Hello World program using this, though imagination is the limit if you want to play with it:
<pre><code class="hljs">public class Main {
 public static void main(String[] args) {
 HelloWorld hello = () -&gt; System.out.println(&quot;Hello World!&quot;);
 hello.world();
 }
}
</code></pre>
As we can see if we run this, our lambda function has successfully matched to the <code>HelloWorld</code> interface, and the object <code>hello</code> can now be used to access its method.
The idea behind this is that you can use lambdas wherever you'd otherwise use functional interfaces to pass along functions. If you remember our <code>Comparator</code> example, <code>Comparator&lt;T&gt;</code> is actually a functional interface, implementing a single method - <code>compare()</code>.
That's why we could replace it with a lambda that behaves similar to that method.
<h2 id="implementation">Implementation</h2>
The basic idea behind lambda functions is the same as the basic idea behind methods - they take parameters in and use them within the body consisting of expressions.
The implementation is just a bit different. Let's take the example of our <code>String</code> sorting lambda:
<pre><code class="hljs">(s1,s2) -&gt; s1.length() - s2.length()
</code></pre>
Its syntax can be understood as:
<pre><code class="hljs">parameters -&gt; body
</code></pre>
<h3 id="parameters">Parameters</h3>
Parameters are the same as function parameters, those are values passed to a lambda function for it to do something with.
Parameters are usually enclosed in brackets and separated by commas, although in the case of a lambda, which receives only one parameter, the brackets can be omitted.
A lambda function can take any number of parameters, including zero, so you could have something like this:
<pre><code class="hljs">() -&gt; System.out.println(&quot;Hello World!&quot;)
</code></pre>
This lambda function, when matched to a corresponding interface, will work the same as the following function:
<pre><code class="hljs">static void printing(){
 System.out.println(&quot;Hello World!&quot;);
}
</code></pre>
Similarly, we can have lambda functions with one, two, or more parameters.
A classic example of a function with one parameter is working on each element of a collection in a <code>forEach</code> loop:
<pre><code class="hljs">public class Main {
 public static void main(String[] args) {
 LinkedList&lt;Integer&gt; childrenAges = new LinkedList&lt;Integer&gt;(Arrays.asList(2, 4, 5, 7));
 childrenAges.forEach( age -&gt; System.out.println(&quot;One of the children is &quot; + age + &quot; years old.&quot;));
 }
}
</code></pre>
Here, the sole parameter is <code>age</code>. Note that we removed parentheses around it here, because that's allowed when we have only one parameter.
Using more parameters works similarly, they're just separated by a comma and enclosed in parentheses. We've already seen two-parameter lambda when we matched it to <code>Comparator</code> to sort Strings.
<h3 id="body">Body</h3>
A body of a lambda expression consists of a single expression or a statement block.
If you specify only a single expression as the body of a lambda function (whether in a statement block or by itself), the lambda will automatically return the evaluation of that expression.
If you have multiple lines in your statement block, or if you just want to (it's a free country), you can explicitly use a return statement from within a statement block:
<pre><code class="hljs">// just the expression
(s1,s2) -&gt; s1.length() - s2.length()

// statement block
(s1,s2) -&gt; { s1.length() - s2.length(); }

// using return
(s1,s2) -&gt; {
 s1.length() - s2.length();
 return; // because forEach expects void return
}
</code></pre>
You could try substituting any of these into our sorting example at the beginning of the article, and you'll find that they all work exactly the same.
<h3 id="variablecapture">Variable Capture</h3>
Variable capture enables lambdas to use variables declared outside of the lambda itself.
There are three very similar types of variable capture:
<ul>
<li>local variable capture</li>
<li>instance variable capture</li>
<li>static variable capture</li>
</ul>
The syntax is almost identical to how you would access these variables from any other function, but conditions under which you can do so are different.
You can access a local variable only if it's effectively final, which means that it does not change its value after assignment. It doesn't have to be explicitly declared as final, but it's advisable to do so to avoid confusion. If you use it in a lambda function and then change its value, the compiler will start whining.
The reason you can't do this is because the lambda can't reliably reference a local variable, because it may be destroyed before you execute the lambda. Because of this, it makes a <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Object_copying#Deep_copy">deep copy</a>. Changing the local variable may lead to some confusing behavior, as the programmer might expect the value within the lambda to change, so to avoid confusion, it's explicitly forbidden.
When it comes to instance variables, if your lambda is within the same class as the variable you're accessing, you can simply use <code>this.field</code> to access a field in that class. Moreover, the field does not have to be final, and can be changed later during the course of the program.
This is because if a lambda is defined within a class, it's instanced along with that class and tied to that class instance, and can thus easily refer to the value of the field it needs.
Static variables are captured much like instance variables, except for the fact that you wouldn't use <code>this</code> to refer to them. They can be changed and don't need to be final for the same reasons.
<h3 id="methodreferencing">Method Referencing</h3>
Sometimes, lambdas are just stand-ins for a specific method. In the spirit of making the syntax short and sweet, you don't actually have to type out the entire syntax when that is the case. For example:
<pre><code class="hljs">s -&gt; System.out.println(s)
</code></pre>
is equivalent to:
<pre><code class="hljs">System.out::println
</code></pre>
The <code>::</code> syntax will let the compiler know that you just want a lambda that passes along the given argument to <code>println</code>. You always just preface the method name with <code>::</code> where you would write a lambda function, otherwise accessing the method as you would normally, meaning you still have to specify the owner class before the double colon.
There are various types of method references, depending on the type of method you're calling:
<ul>
<li>static method reference</li>
<li>parameter method reference</li>
<li>instance method reference</li>
<li>constructor method reference</li>
</ul>
<h4 id="staticmethodreference">Static Method Reference</h4>
We need an interface:
<pre><code class="hljs">public interface Average {
 abstract double average(double a, double b);
}
</code></pre>
A static function:
<pre><code class="hljs">public class LambdaFunctions {
 static double averageOfTwo(double a, double b){
 return (a+b)/2;
 }
}
</code></pre>
And our lambda function and call in <code>main</code>:
<pre><code class="hljs">Average avg = LambdaFunctions::averageOfTwo;
System.out.println(avg.average(20.3, 4.5));
</code></pre>
<h4 id="parametermethodreference">Parameter Method Reference</h4>
Again, we're typing in <code>main</code>.
<pre><code class="hljs">Comparator&lt;Double&gt; cmp = Double::compareTo;
Double a = 20.3;
System.out.println(cmp.compare(a, 4.5));
</code></pre>
The <code>Double::compareTo</code> lambda is equivalent to:
<pre><code class="hljs">Comparator&lt;Double&gt; cmp = (a, b) -&gt; a.compareTo(b)
</code></pre>
<h4 id="instancemethodreference">Instance Method Reference</h4>
If we take our <code>LambdaFunctions</code> class and our function <code>averageOfTwo</code> (from <a href="#staticmethodreference">Static Method Reference</a>) and make it non-static, we'll get the following:
<pre><code class="hljs">public class LambdaFunctions {
 double averageOfTwo(double a, double b){
 return (a+b)/2;
 }
}
</code></pre>
To access this we now need an instance of the class, so we'd have to do this in <code>main</code>:
<pre><code class="hljs">LambdaFunctions lambda = new LambdaFunctions();
Average avg = lambda::averageOfTwo;
System.out.println(avg.average(20.3, 4.5));
</code></pre>
<h4 id="constructormethodreference">Constructor Method Reference</h4>
If we have a class called <code>MyClass</code> and want to call it's constructor through a lambda function, our lambda will look like this:
<pre><code class="hljs">MyClass::new
</code></pre>
It will accept as many arguments as it can match to one of the constructors.
<h2 id="conclusion">Conclusion</h2>
In conclusion, lambdas are a useful feature for making our code simpler, shorter, and more readable.
Some people avoid using them when there's a lot of Juniors on the team, so I'd advise consulting with your team before you refactor all of your code, but when everyone's on the same page they're a great tool.
<h3 id="seealso">See Also</h3>
Here's some further reading into how and where to apply lambda functions:
<ul>
<li><a rel="nofollow" target="_blank" href="/introduction-to-java-8-streams/">Streams</a></li>
<li><a rel="nofollow" target="_blank" href="https://www.dummies.com/programming/java/how-to-use-lambda-expressions-to-handle-events-in-java/">Event Listeners</a></li>
<li><a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/tutorial/java/javaOO/lambdaexpressions.html">Lambda Expressions on Oracle</a></li>
</ul>

Darinka Zobenica

Lambda Expressions in Java