<h2 id="introduction">Introduction</h2>
Temporary files, or &quot;temp files&quot;, are mainly used to store intermediate information on disk for an application. These files are normally created for different purposes such as temporary backup or if the application is dealing with a large dataset bigger than the system's memory, etc. Ideally, these files are located in a separate directory, which varies on different operating systems, and the name of these files are unique. The data stored in temporary files is not always required after the application quits, so you may want these files to be deleted after use.
Python provides a module known as <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/tempfile.html">tempfile</a>, which makes creating and handling temporary files easier. This module provides a few methods to create temporary files and directories in different ways. <code>tempfile</code> comes in handy whenever you want to use temporary files to store data in a Python program. Let's take a look at a couple of different examples on how the <code>tempfile</code> module can be used.
<h2 id="creatingatemporaryfile">Creating a Temporary File</h2>
Suppose your application needs a temporary file for use within the program, i.e. it will create one file, use it to store some data, and then delete it after use. To achieve this, we can use the <code>TemporaryFile()</code> function.
This function will create one temporary file to the default <code>tempfile</code> location. This location may be different between operating systems. The best part is that the temporary file created by <code>TemporaryFile()</code> will be removed automatically whenever the file is closed. Also, it does not create any reference to this file in the system's file system table. This makes it private to the current application i.e. no other program will be able to open the file.
Let's take a look at the below Python program to see how it works:
<pre><code class="hljs">import tempfile #1

print(&quot;Creating one temporary file...&quot;)

temp = tempfile.TemporaryFile() #2

try:
 print(&quot;Created file is:&quot;, temp) #3
 print(&quot;Name of the file is:&quot;, temp.name) #4
finally:
 print(&quot;Closing the temp file&quot;)
 temp.close() #5
</code></pre>
It will print the below output:
<pre><code class="hljs">$ python3 temp-file.py
Creating one temporary file...
Created file is: &lt;_io.BufferedRandom name=4&gt;
Name of the file is: 4
Closing the temp file
</code></pre>
<ol>
<li>To create one temporary file in Python, you need to import the <code>tempfile</code> module.</li>
<li>As explained above, we have created the temporary file using the <code>TemporaryFile()</code> function.</li>
<li>From the output, you can see that the created object is actually not a file, it is a file-like object. And, the <code>mode</code> parameter (not shown in our example) of the created file is <code>w+b</code>, i.e. you can read and write without being closed.</li>
<li>The temporary file created has no name.</li>
<li>Finally, we are closing the file using the <code>close()</code> method. It will be destroyed after it is closed.</li>
</ol>
One thing we should point out is that the file created using the <code>TemporaryFile()</code> function may or may not have a visible name in the file system. On Unix, the directory entry for the file is removed automatically after it is created, although this is not supported on other platforms. Normally <code>TemporaryFile()</code> is the ideal way to create one temporary storage area for any program in Python.
<h2 id="createanamedtemporaryfile">Create a Named Temporary File</h2>
In our previous example, we have seen that the temporary file created using the <code>TemporaryFile()</code> function is actually a file-like object without an actual file name. Python also provides a different method, <code>NamedTemporaryFile()</code>, to create a file with a visible name in the file system. Other than providing a name to the tempfile, <code>NamedTemporaryFile()</code> works the same as <code>TemporaryFile()</code>. Now let's use the same above example to create a named temporary file:
<pre><code class="hljs">import tempfile

print(&quot;Creating one named temporary file...&quot;)

temp = tempfile.NamedTemporaryFile()

try:
 print(&quot;Created file is:&quot;, temp)
 print(&quot;Name of the file is:&quot;, temp.name)
finally:
 print(&quot;Closing the temp file&quot;)
 temp.close()
</code></pre>
Running this code will print output similar to the following:
<pre><code class="hljs">$ python3 named-temp-file.py
Creating one named temporary file...
Created file is: &lt;tempfile._TemporaryFileWrapper object at 0x103f22ba8&gt;
Name of the file is: /var/folders/l7/80bx27yx3hx_0_p1_qtjyyd40000gn/T/tmpa3rq8lon
Closing the temp file
</code></pre>
So, the created file actually has a name this time. The advantage of <code>NamedTemporaryFile()</code> is that we can save the name of the created temp files and use them later before closing or destroying it. If the <code>delete</code> parameter is set to <code>False</code>, then we can close the file without it being destroyed, allowing us to reopen it later on.
<h2 id="providingasuffixorprefixtothename">Providing a Suffix or Prefix to the Name</h2>
Sometimes we need to add a prefix or suffix to a temp-file's name. It will help us to identify all temp files created by our program.
To achieve this, we can use the same <code>NamedTemporaryFile</code> function defined above. The only thing we need to add is two extra parameters while calling this function: <code>suffix</code> and <code>prefix</code>
<pre><code class="hljs">import tempfile

temp = tempfile.NamedTemporaryFile(prefix=&quot;dummyPrefix_&quot;, suffix=&quot;_dummySuffix&quot;)

try:
 print(&quot;Created file is:&quot;, temp)
 print(&quot;Name of the file is:&quot;, temp.name)
finally:
 temp.close()
</code></pre>
Running this code will print the following output:
<pre><code class="hljs">$ python3 prefix-suffix-temp-file.py
Created file is: &lt;tempfile._TemporaryFileWrapper object at 0x102183470&gt;
Name of the file is: /var/folders/tp/pn3dvz_n7cj7nfs0y2szsk9h0000gn/T/dummyPrefix_uz63brcp_dummySuffix
</code></pre>
So, if we pass the two extra arguments <code>suffix</code> and <code>prefix</code> to the <code>NamedTemporaryFile()</code> function, it will automatically add those in the start and end of the file name.
<h2 id="findingthedefaultlocationoftempfiles">Finding the Default Location of Temp Files</h2>
The <code>tempfile.tempdir</code> variable holds the default location for all temporary files. If the value of <code>tempdir</code> is <code>None</code> or unset, Python will search a standard list of directories and set <code>tempdir</code> to the first directory value, but only if the calling program can create a file in it. The following are the list of directories it will scan, in this order:
<ol>
<li>The directory named by the TMPDIR environment variable</li>
<li>The directory named by the TEMP environment variable</li>
<li>The directory named by the TMP environment variable</li>
<li>Platform-specific directories:
<ol>
<li>On windows, C:\TEMP, C:\TMP, \TEMP, and \TMP, in the same order.</li>
<li>On other platforms, /tmp, /var/tmp, and /usr/tmp, in the same order.</li>
</ol>
</li>
<li>The current working directory.</li>
</ol>
To find out the default location of temporary files, we can call the <code>tempfile.gettempdir()</code> method. It will return the value of <code>tempdir</code> if it is not <code>None</code>. Otherwise it will first search for the directory location using the steps mentioned above and then return the location.
<pre><code class="hljs">import tempfile

print(&quot;Current temp directory:&quot;, tempfile.gettempdir())

tempfile.tempdir = &quot;/temp&quot;

print(&quot;Temp directory after change:&quot;, tempfile.gettempdir())
</code></pre>
If you will run the above program, it will print an output similar to the following:
<pre><code class="hljs">$ python3 dir-loc-temp-file.py
Current temp directory: /var/folders/tp/pn3dvz_n7cj7nfs0y2szsk9h0000gn/T
Temp directory after change: /temp
</code></pre>
You can see that the first temp directory location is the system-provided directory location and the second temp directory is the same value as the one that we have defined.
<h2 id="readingandwritingdatafromtempfiles">Reading and Writing Data from Temp Files</h2>
We have learned how to create a temporary file, create a temporary file with a name, and how to create a temporary file with a suffix and/or prefix. Now, let's try to understand how to actually read and write data from a temporary file in Python.
Reading and writing data from a temporary file in Python is pretty straightforward. For writing, you can use the <code>write()</code> method and for reading, you can use the <code>read()</code> method. For example:
<pre><code class="hljs">import tempfile

temp = tempfile.TemporaryFile()

try:
 temp.write(b&#x27;Hello world!&#x27;)
 temp.seek(0)

 print(temp.read())
finally:
 temp.close()
</code></pre>
This will print the output as <code>b'Hello world!'</code> since the <code>write()</code> method takes input data in bytes (hence the <code>b</code> prefix on the string).
If you want to write text data into a temp file, you can use the <code>writelines()</code> method instead. For using this method, we need to create the tempfile using <code>w+t</code> mode instead of the default <code>w+b</code> mode. To do this, a <code>mode</code> param can be passed to <code>TemporaryFile()</code> to change the mode of the created temp file.
<pre><code class="hljs">import tempfile

temp = tempfile.TemporaryFile(mode=&#x27;w+t&#x27;)

try:
 temp.writelines(&quot;Hello world!&quot;)
 temp.seek(0)

 print(temp.read())
finally:
 temp.close()
</code></pre>
Unlike the previous example, this will print &quot;Hello World&quot; as the output.
<h2 id="createatemporarydirectory">Create a Temporary Directory</h2>
If your program has several temporary files, it may be more convenient to create one temporary directory and put all of your temp files inside of it. To create a temporary directory, we can use the <code>TemporaryDirectory()</code> function. After all temp files are closed, we need to delete the directory manually.
<pre><code class="hljs">import tempfile

with tempfile.TemporaryDirectory() as tmpdirname:
 print(&#x27;Created temporary directory:&#x27;, tmpdirname)

# Both the directory and its contents have been deleted
</code></pre>
It will print the below output:
<pre><code class="hljs">$ python3 mk-dir-temp-file.py
Created temporary directory: /var/folders/l7/80bx27yx3hx_0_p1_qtjyyd40000gn/T/tmpn_ke7_rk
</code></pre>
<h2 id="createasecuretemporaryfileanddirectory">Create a Secure Temporary File and Directory</h2>
By using <code>mkstemp()</code>, we can create a temporary file in the most secure manner possible. The temporary file created using this method is readable and writable only by the creating user ID. We can pass <code>prefix</code> and <code>suffix</code> arguments to add prefix and suffix to the created file name. By default, it opens the file in binary mode. To open it in text mode, we can pass <code>text=True</code> as an argument to the function. Unlike <code>TemporaryFile()</code>, the file created by <code>mkstemp()</code> doesn't get deleted automatically after closing it.
As you can see in the example below, the user is responsible for deleting the file.
<pre><code class="hljs">import tempfile
import os

temp_directory = tempfile.TemporaryDirectory()

print(&quot;Directory name:&quot;, temp_directory)

os.removedirs(temp_directory)
</code></pre>
<pre><code class="hljs">$ python3 mk-secure-dir-temp-file.py
Directory name: /var/folders/tp/pn3dvz_n7cj7nfs0y2szsk9h0000gn/T/tmpf8f6xc53
</code></pre>
Similar to <code>mkstemp()</code>, we can create a temporary directory in the most secure manner possible using the <code>mkdtemp()</code> method. And again, like <code>mkstemp()</code>, it also supports <code>prefix</code> and <code>suffix</code> arguments for adding a prefix and suffix to the directory name.
<h2 id="conclusion">Conclusion</h2>
In this article we have learned different ways to create temporary files and directories in Python. You can use temp files in any Python program you want. But just make sure to delete it if the particular method used doesn't automatically delete it on its own. Also keep in mind that behavior may different between operating systems, like the output directory names and file names.
All of these functions we have explained above work with many different arguments, although we have not covered in detail what type of arguments each function takes. If you want to learn more about the <code>tempfile</code> module, you should check out the Python 3 <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/tempfile.html">official documentation</a>.

N Kaushik

The Python tempfile Module

Python as a scripting language is quite simple and compact. Compared to other languages, you only have a relatively low number of keywords to internalize in order to write proper Python code. Furthermore, both simplicity as well as readability of the code are preferred, which is what Python prides itself on. In order to achieve both goals, it is helpful that you follow the language's specific guidelines.
This article focuses on the guidelines mentioned above to write valid code that represents a more Pythonic way of programming. It is a selection of guidelines that focuses on practical usage, and further guidelines can be read in <a class="amazon-link" rel="nofollow" target="_blank" href="http://stackabu.se/hitchhikers-guide-to-python">The Hitchhiker's Guide to Python</a> and the <a rel="nofollow" target="_blank" href="https://www.python.org/dev/peps/pep-0008/">PEP8 Style Guide</a>.
Tim Peters - an American Python developer - combines the principles of the language humorously in the Zen of Python. These rules comprise the main targets and style of the language. Hopefully, these rules help you to orient as a developer.
Zen of Python
<pre><code>
 Beautiful is better than ugly.
 Explicit is better than implicit.
 Simple is better than complex.
 Complex is better than complicated.
 Flat is better than nested.
 Sparse is better than dense.
 Readability counts.
 Special cases aren't special enough to break the rules.
 Although practicality beats purity.
 Errors should never pass silently.
 Unless explicitly silenced.
 In the face of ambiguity, refuse the temptation to guess.
 There should be one-- and preferably only one --obvious way to do it.
 Although that way may not be obvious at first unless you're Dutch.
 Now is better than never.
 Although never is often better than right now.
 If the implementation is hard to explain, it's a bad idea.
 If the implementation is easy to explain, it may be a good idea.
 Namespaces are one honking great idea -- let's do more of those!
 --Tim Peters
</pre></code>
<h2 id="generalprogrammingguidelines">General Programming Guidelines</h2>
Following the Zen of Python, readability of code counts. To ensure properly-formatted code, the Python language has some programming guidelines outlined in PEP8 - for example consistent indentation, a specific line length, writing one statement per line only, and formulating pieces of code in a rather explicit than implicit way. We will explain these rules below step by step.
<h3 id="indentation">Indentation</h3>
Indentation is required for classes, functions (or methods), loops, conditions, and lists. You may use either tabulators, or spaces, but you must not combine both of them in the same script. For Python 3, spaces are the preferred indentation method, and more specifically four spaces are desired. As an example, a list is recommended to be defined in one of these two ways as follows:
Writing lists
<pre><code class="hljs"># version 1
numbers = [
 1, 2, 3,
 4, 5, 6
 ]

# version 2
numbers = [
 1, 2, 3,
 4, 5, 6
]
</code></pre>
As pointed out in PEP8, the closing bracket can either be lined-up under the first non-whitespace character of the last line of the list, as in &quot;version 1&quot;, or under the first character of the line that starts the list as in &quot;version 2&quot;.
Using spaces requires us to work with the same number of spaces per indentation level. The next example shows you how not to write your code, which mixes tabulators and a different number of spaces on each line.
Bad example
<pre><code class="hljs">def draw_point(x, y):
 &quot;&quot;&quot;draws a point at position x,y&quot;&quot;&quot;
 
 if (x &gt; 0):
 set_point(x, y)
 return
</code></pre>
In order to indent code blocks properly the next example uses four spaces per indentation level, consequently:
Good example
<pre><code class="hljs">def draw_point(x, y):
 &quot;&quot;&quot;draws a point at position x,y&quot;&quot;&quot;

 if (x &gt; 0):
 set_point(x, y)
 return
</code></pre>
<h3 id="onestatementperline">One Statement per Line</h3>
The example above follows another important rule regarding writing code: use one statement per line, only. Although, the Python language allows you to write several statements per line that are separated by a semi-colon as follows:
Bad
<pre><code class="hljs">print (&quot;Berlin&quot;); print (&quot;Cape Town&quot;)

if x == 1: print (&quot;Amsterdam&quot;)
</code></pre>
For a better clarity write the code like that, instead:
Good
<pre><code class="hljs">print (&quot;Berlin&quot;)
print (&quot;Cape Town&quot;)

if x == 1:
 print (&quot;Amsterdam&quot;)
</code></pre>
This refers to using Python modules as well. Many programming examples show two or more modules that are imported on a single line as follows:
Bad practice
<pre><code class="hljs">import sys, os
</code></pre>
It is much better to import one module per line, instead:
Good practice
<pre><code class="hljs">import sys
import os
</code></pre>
Place the <code>import</code> statements at the beginning of the file, after the copyright information, and the docstrings. Furthermore, it is common to group the <code>import</code> statements into standard modules from the Python library, related third-party modules, and finally library-specific imports. Inserting a blank line as well as comments help the readability and to understand the code in a better way.
Importing External Modules
<pre><code class="hljs"># use operating-system specific routines
import os

# use regular expressions routines
import re

# use SAX XML library/parser
from xml.sax import make_parser, handler
...
</code></pre>
<h3 id="linelength">Line Length</h3>
A single line should not exceed the number of 79 characters, whereas a docstring or comment is no longer than 72 characters. Lines of code can be wrapped by using a backslash (<code>\</code>) as follows:
Code with a Line-Break
<pre><code class="hljs">with open(&#x27;/path/to/some/file/you/want/to/read&#x27;) as file_1, \
 open(&#x27;/path/to/some/file/being/written&#x27;, &#x27;w&#x27;) as file_2:
 file_2.write(file_1.read())
</code></pre>
<h3 id="explicitvsimplicitcode">Explicit vs Implicit Code</h3>
Python as a scripting language is flexible enough that it allows you to use &quot;tricks&quot; throughout your code. Although you should take into account that many times your code is read by other developers. To improve the readability, it is better to write explicit code instead of making implicit assumptions, like using one-liners or &quot;tricks&quot;.
In the example below the function <code>calculation()</code> hides the two values <code>x</code> and <code>y</code> in a single parameter named <code>args</code>. This way of writing also allows callers to pass more or less than these values to the function if desired, but it is not obvious at first sight.
Bad
<pre><code class="hljs">def calculation(*args):
 &quot;&quot;&quot;calculation of the total&quot;&quot;&quot;

 x, y = args
 return (x+y)

print(calculation(3, 4))
</code></pre>
For higher clarity it is recommended to write it like this, instead:
Good
<pre><code class="hljs">def calculation(x,y):
 &quot;&quot;&quot;calculation of the total&quot;&quot;&quot;

 total = x + y
 return (total)

print(calculation(3, 4))
</code></pre>
<h2 id="namingconventions">Naming Conventions</h2>
There exist quite a few variations to name modules, classes, methods/functions, and variables. This includes the usage of lowercase and uppercase letters with or without underscores, capitalized words, and mixed styles. Due to the huge diversity of developers you will find all these styles, and there is little consistency among the modules.
Naming Style Variations
<pre><code class="hljs">shoppingcart = [] # lowercase
shopping_cart = [] # lowercase with underscores
SHOPPINGCART = [] # uppercase
SHOPPING_CART = [] # uppercase with underscores
ShoppingCart = [] # capitalized words
shoppingCart = [] # mixed style
</code></pre>
Which of the styles you use is up to you. Again, be consistent, and use the same style in your entire code. According to PEP8, the following main rules apply:
<ul>
<li>Names of identifiers have to be ASCII compatible</li>
<li>Modules are required to have short, all-lowercase names</li>
<li>Classes follow the capitalized words convention</li>
<li>Exceptions follow the capitalized words convention, and are expected to have the <code>Error</code> suffix if they refer to errors</li>
<li>Constants are written in uppercase letters</li>
</ul>
For more details have a look at the PEP8 standard.
We should also point out that it is considered more &quot;Pythonic&quot; to use the &quot;lowercase with underscores&quot; approach when naming variables in Python, although any approach is allowed.
<h2 id="codestylevalidation">Code Style Validation</h2>
Guidelines are great to achieve code that follows certain conditions. As a programmer you want to make sure that you follow them as much as possible. Automated tools are great to help you to validate your code.
As mentioned above, the guidelines are described in PEP8. Consequently, the Python language contains a correesponding command line tool to help you check your code against the guidelines. Originally known as <code>pep8</code>, this code checker was renamed to <a rel="nofollow" target="_blank" href="http://pycodestyle.pycqa.org/">pycodestyle</a> in 2016. It is maintained by the Python Code Quality Authority, and belongs to a number of tools like the source code analyzers <a rel="nofollow" target="_blank" href="https://www.pylint.org/">pylint</a> and <a rel="nofollow" target="_blank" href="https://github.com/PyCQA/pyflakes">pyflakes</a>, the complexity checker <a rel="nofollow" target="_blank" href="http://pypi.python.org/pypi/mccabe">mccabe</a> as well as the docstring checker <a rel="nofollow" target="_blank" href="http://pydocstyle.org">pydocstyle</a>.
<code>pycodestyle</code> analyzes your Python code, and reports violations that cover indentation errors, blank lines that are unnecessary, and the usage of tabulators instead of spaces. The following example contains a sample output with some typical errors and warnings:
<pre><code class="hljs">$ pycodestyle --first stack.py
stack.py:3:1: E265 block comment should start with &#x27;# &#x27;
stack.py:12:1: E302 expected 2 blank lines, found 1
stack.py:13:1: W191 indentation contains tabs
</code></pre>
In Debian GNU/Linux, the tool is available as the packages <code>python-pycodestyle</code> (for Python 2.x), and <code>python3-pycodestyle</code> (for Python 3.x). Both of them come with a number of useful parameters, for example:
<ul>
<li><code>--first</code>: Show first occurrence of each error (as seen above). The output shows the file the error was detected in as well as the line number and the column.</li>
<li><code>--show-source</code>: Show source code for each error</li>
</ul>
<pre><code class="hljs">$ pycodestyle --show-source stack.py
stack.py:3:1: E265 block comment should start with &#x27;# &#x27;
#o
^
stack.py:12:1: E302 expected 2 blank lines, found 1
class Stack:
^
stack.py:13:1: W191 indentation contains tabs
 def __init__(self):
^
...
</code></pre>
<ul>
<li><code>--statistics</code>: Count errors and warnings. In the following example, <code>pycodestyle</code> detected two errors - E265 and E302 - as well as 30 warnings (W191).</li>
</ul>
<pre><code class="hljs">$ pycodestyle --statistics stack.py
...
1 E265 block comment should start with &#x27;# &#x27;
1 E302 expected 2 blank lines, found 1
30 W191 indentation contains tabs
</code></pre>
The same tool is also <a rel="nofollow" target="_blank" href="http://pep8online.com/">available online</a>. Just copy and paste your code into the tool, and see the validation result.
<img class="img-responsive" alt="pep8online.com validation result" src="https://s3.amazonaws.com/stackabuse/media/introduction-python-coding-style-1.png">
<h2 id="conclusion">Conclusion</h2>
Writing proper Python code is not always easy. But luckily there are guidelines that help, as well as command line tools to ensure that your code meets these guidelines. With the various resources available it can be very easy :)
<h2 id="acknowledgements">Acknowledgements</h2>
The author would like to thank Zoleka Hatitongwe for her support while preparing the article.

Frank Hofmann

Introduction to the Python Coding Style

In our previous article <a target="_blank" href="/implementing-pca-in-python-with-scikit-learn/">Implementing PCA in Python with Scikit-Learn</a>, we studied how we can reduce dimensionality of the feature set using PCA. In this article we will study another very important dimensionality reduction technique: <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Linear_discriminant_analysis">linear discriminant analysis</a> (or LDA). But first let's briefly discuss how PCA and LDA differ from each other.
<h2 id="pcavsldawhatsthedifference">PCA vs LDA: What's the Difference?</h2>
Both PCA and LDA are linear transformation techniques. However, PCA is unsupervised while LDA is a supervised dimensionality reduction technique.
PCA has no concern with the class labels. In simple words, PCA summarizes the feature set without relying on the output. PCA tries to find the directions of the maximum variance in the dataset. In a large feature set, there are many features that are merely duplicate of the other features or have a high correlation with the other features. Such features are basically redundant and can be ignored. The role of PCA is to find such highly correlated or duplicate features and to come up with a new feature set where there is minimum correlation between the features or in other words feature set with maximum variance between the features. Since the variance between the features doesn't depend upon the output, therefore PCA doesn't take the output labels into account.
Unlike PCA, LDA tries to reduce dimensions of the feature set while retaining the information that discriminates output classes. LDA tries to find a decision boundary around each cluster of a class. It then projects the data points to new dimensions in a way that the clusters are as separate from each other as possible and the individual elements within a cluster are as close to the centroid of the cluster as possible. The new dimensions are ranked on the basis of their ability to maximize the distance between the clusters and minimize the distance between the data points within a cluster and their centroids. These new dimensions form the linear discriminants of the feature set.
Let us now see how we can implement LDA using Python's Scikit-Learn.
<h2 id="implementingldawithscikitlearn">Implementing LDA with Scikit-Learn</h2>
Like PCA, the <a rel="nofollow" target="_blank" href="http://scikit-learn.org/stable/index.html">Scikit-Learn</a> library contains built-in classes for performing LDA on the dataset. In this section we will apply LDA on the Iris dataset since we used the same dataset for the PCA article and we want to compare results of LDA with PCA. The information about the Iris dataset is available at the following link:
<a rel="nofollow" target="_blank" href="https://archive.ics.uci.edu/ml/datasets/iris">https://archive.ics.uci.edu/ml/datasets/iris</a>
The rest of the sections follows our traditional machine learning pipeline:
<h3 id="importinglibraries">Importing Libraries</h3>
<pre><code class="hljs">import numpy as np
import pandas as pd
</code></pre>
<h3 id="importingthedataset">Importing the Dataset</h3>
<pre><code class="hljs">url = &quot;https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data&quot;
names = [&#x27;sepal-length&#x27;, &#x27;sepal-width&#x27;, &#x27;petal-length&#x27;, &#x27;petal-width&#x27;, &#x27;Class&#x27;]
dataset = pd.read_csv(url, names=names)
</code></pre>
<h3 id="datapreprocessing">Data Preprocessing</h3>
Once a dataset is loaded into a pandas data frame object, the first step is to divide the dataset into features and corresponding labels and then divide the resultant dataset into training and test sets. The following code divides data into labels and feature set:
<pre><code class="hljs">X = dataset.iloc[:, 0:4].values
y = dataset.iloc[:, 4].values
</code></pre>
The above script assigns the first four columns of the dataset i.e. the feature set to <code>X</code> variable while the values in the fifth column (labels) are assigned to the <code>y</code> variable.
The following code divides data into training and test sets:
<pre><code class="hljs">from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
</code></pre>
<h3 id="featurescaling">Feature Scaling</h3>
As was the case with PCA, we need to perform feature scaling for LDA too. Execute the following script to do so:
<pre><code class="hljs">from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
</code></pre>
<h3 id="performinglda">Performing LDA</h3>
It requires only four lines of code to perform LDA with Scikit-Learn. The <code>LinearDiscriminantAnalysis</code> class of the <code>sklearn.discriminant_analysis</code> library can be used to Perform LDA in Python. Take a look at the following script:
<pre><code class="hljs">from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA

lda = LDA(n_components=1)
X_train = lda.fit_transform(X_train, y_train)
X_test = lda.transform(X_test)
</code></pre>
In the script above the <code>LinearDiscriminantAnalysis</code> class is imported as <code>LDA</code>. Like PCA, we have to pass the value for the <code>n_components</code> parameter of the LDA, which refers to the number of linear discriminants that we want to retrieve. In this case we set the <code>n_components</code> to 1, since we first want to check the performance of our classifier with a single linear discriminant. Finally we execute the <code>fit</code> and <code>transform</code> methods to actually retrieve the linear discriminants.
Notice, in the case of LDA, the <code>transform</code> method takes two parameters: the <code>X_train</code> and the <code>y_train</code>. However in the case of PCA, the <code>transform</code> method only requires one parameter i.e. <code>X_train</code>. This reflects the fact that LDA takes the output class labels into account while selecting the linear discriminants, while PCA doesn't depend upon the output labels.
<h3 id="trainingandmakingpredictions">Training and Making Predictions</h3>
Since we want to compare the performance of LDA with one linear discriminant to the performance of PCA with one principal component, we will use the same Random Forest classifier that we used to evaluate performance of PCA-reduced algorithms.
Execute the following code:
<pre><code class="hljs">from sklearn.ensemble import RandomForestClassifier

classifier = RandomForestClassifier(max_depth=2, random_state=0)

classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
</code></pre>
<h3 id="evaluatingtheperformance">Evaluating the Performance</h3>
As always, the last step is to evaluate the performance of the algorithm with the help of a confusion matrix and find the accuracy of the prediction. Execute the following script:
<pre><code class="hljs">from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

cm = confusion_matrix(y_test, y_pred)
print(cm)
print(&#x27;Accuracy&#x27; + str(accuracy_score(y_test, y_pred)))
</code></pre>
The output of the script above looks like this:
<pre><code class="hljs">[[11 0 0]
 [ 0 13 0]
 [ 0 0 6]]
Accuracy 1.0
</code></pre>
You can see that with one linear discriminant, the algorithm achieved an accuracy of 100%, which is greater than the accuracy achieved with one principal component, which was 93.33%.
<h2 id="pcavsldawhattochoosefordimensionalityreduction">PCA vs LDA: What to Choose for Dimensionality Reduction?</h2>
In the case of uniformly distributed data, LDA almost always performs better than PCA. However if the data is highly skewed (irregularly distributed) then it is advised to use PCA since LDA can be biased towards the majority class.
Finally, it is beneficial that PCA can be applied to labeled as well as unlabeled data since it doesn't rely on the output labels. On the other hand, LDA requires output classes for finding linear discriminants and hence requires labeled data.

Usman Malik

Linear Discriminant Analysis (LDA) in Python with Scikit-Learn

Adam McQuistan

A SQLite Tutorial with Node.js

With the availability of high performance CPUs and GPUs, it is pretty much possible to solve every regression, classification, clustering and other related problems using machine learning and deep learning models. However, there are still various factors that cause performance bottlenecks while developing such models. Large number of features in the dataset is one of the factors that affect both the training time as well as accuracy of machine learning models. You have different options to deal with a huge number of features in a dataset.
<ol>
<li>Try to train the models on the original number of features, which take days or weeks if the number of features is too high.</li>
<li>Reduce the number of variables by merging correlated variables.</li>
<li>Extract the most important features from the dataset that are responsible for maximum variance in the output. Different statistical techniques are used for this purpose e.g. linear discriminant analysis, factor analysis, and principal component analysis.</li>
</ol>
In this article, we will see how <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Principal_component_analysis">principal component analysis</a> can be implemented using Python's <a target="_blank" href="/tag/scikit-learn/">Scikit-Learn</a> library.
<h2 id="principalcomponentanalysis">Principal Component Analysis</h2>
Principal component analysis, or PCA, is a statistical technique to convert high dimensional data to low dimensional data by selecting the most important features that capture maximum information about the dataset. The features are selected on the basis of variance that they cause in the output. The feature that causes highest variance is the first principal component. The feature that is responsible for the second highest variance is considered the second principal component, and so on. It is important to mention that principal components do not have any correlation with each other.
<h3 id="advantagesofpca">Advantages of PCA</h3>
There are two main advantages of <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Dimensionality_reduction">dimensionality reduction</a> with PCA.
<ol>
<li>The training time of the algorithms reduces significantly with less number of features.</li>
<li>It is not always possible to analyze data in high dimensions. For instance if there are 100 features in a dataset. Total number of scatter plots required to visualize the data would be <code>100(100-1)2 = 4950</code>. Practically it is not possible to analyze data this way.</li>
</ol>
<h3 id="normalizationoffeatures">Normalization of Features</h3>
It is imperative to mention that a feature set must be normalized before applying PCA. For instance if a feature set has data expressed in units of Kilograms, Light years, or Millions, the variance scale is huge in the training set. If PCA is applied on such a feature set, the resultant loadings for features with high variance will also be large. Hence, principal components will be biased towards features with high variance, leading to false results.
Finally, the last point to remember before we start coding is that PCA is a statistical technique and can only be applied to numeric data. Therefore, categorical features are required to be converted into numerical features before PCA can be applied.
<h2 id="implementingpcawithscikitlearn">Implementing PCA with Scikit-Learn</h2>
In this section we will implement PCA with the help of Python's <a rel="nofollow" target="_blank" href="http://scikit-learn.org/stable/">Scikit-Learn</a> library. We will follow the classic machine learning pipeline where we will first import libraries and dataset, perform exploratory data analysis and preprocessing, and finally train our models, make predictions and evaluate accuracies. The only additional step will be to perform PCA to find out the optimal number of features before we train our models. These steps have been implemented as follows:
<h3 id="importinglibraries">Importing Libraries</h3>
<pre><code class="hljs">import numpy as np
import pandas as pd
</code></pre>
<h3 id="importingdataset">Importing Dataset</h3>
The dataset we are going to use in this article is the famous <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Iris_flower_data_set">Iris data set</a>. Some additional information about the Iris dataset is available at:
<a rel="nofollow" target="_blank" href="https://archive.ics.uci.edu/ml/datasets/iris">https://archive.ics.uci.edu/ml/datasets/iris</a>
The dataset consists of 150 records of Iris plant with four features: <code>sepal-length</code>, <code>sepal-width</code>, <code>petal-length</code>, and <code>petal-width</code>. All of the features are numeric. The records have been classified into one of the three classes i.e. <code>Iris-setosa</code>, <code>Iris-versicolor</code>, or <code>Iris-virginica</code>.
Execute the following script to download the dataset using <code>pandas</code>:
<pre><code class="hljs">url = &quot;https://archive.ics.uci.edu/ml/machine-learning-databases/iris/iris.data&quot;
names = [&#x27;sepal-length&#x27;, &#x27;sepal-width&#x27;, &#x27;petal-length&#x27;, &#x27;petal-width&#x27;, &#x27;Class&#x27;]
dataset = pd.read_csv(url, names=names)
</code></pre>
Let's take a look at what our dataset looks like:
<pre><code class="hljs">dataset.head()
</code></pre>
Executing the above command will display the first five rows of our dataset as shown below:
<table class="table table-striped">
 <thead>
 <tr>
 <th></th>
 <th>sepal-length</th>
 <th>sepal-width</th>
 <th>petal-length</th>
 <th>petal-width</th>
 <th>Class</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>0</td>
 <td>5.1</td>
 <td>3.5</td>
 <td>1.4</td>
 <td>0.2</td>
 <td>Iris-setosa</td>
 </tr>
 <tr>
 <td>1</td>
 <td>4.9</td>
 <td>3.0</td>
 <td>1.4</td>
 <td>0.2</td>
 <td>Iris-setosa</td>
 </tr>
 <tr>
 <td>2</td>
 <td>4.7</td>
 <td>3.2</td>
 <td>1.3</td>
 <td>0.2</td>
 <td>Iris-setosa</td>
 </tr>
 <tr>
 <td>3</td>
 <td>4.6</td>
 <td>3.1</td>
 <td>1.5</td>
 <td>0.2</td>
 <td>Iris-setosa</td>
 </tr>
 <tr>
 <td>4</td>
 <td>5.0</td>
 <td>3.6</td>
 <td>1.4</td>
 <td>0.2</td>
 <td>Iris-setosa</td>
 </tr>
 </tbody>
</table>
<h3 id="preprocessing">Preprocessing</h3>
The first preprocessing step is to divide the dataset into a feature set and corresponding labels. The following script performs this task:
<pre><code class="hljs">X = dataset.drop(&#x27;Class&#x27;, 1)
y = dataset[&#x27;Class&#x27;]
</code></pre>
The script above stores the feature sets into the <code>X</code> variable and the series of corresponding labels in to the <code>y</code> variable.
The next preprocessing step is to divide data into training and test sets. Execute the following script to do so:
<pre><code class="hljs"># Splitting the dataset into the Training set and Test set
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)
</code></pre>
As mentioned earlier, PCA performs best with a normalized feature set. We will perform standard scalar normalization to normalize our feature set. To do this, execute the following code:
<pre><code class="hljs">from sklearn.preprocessing import StandardScaler

sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
</code></pre>
<h3 id="applyingpca">Applying PCA</h3>
It is only a matter of three lines of code to perform PCA using Python's Scikit-Learn library. The <code>PCA</code> class is used for this purpose. PCA depends only upon the feature set and not the label data. Therefore, PCA can be considered as an unsupervised machine learning technique.
Performing PCA using Scikit-Learn is a two-step process:
<ol>
<li>Initialize the <code>PCA</code> class by passing the number of components to the constructor.</li>
<li>Call the <code>fit</code> and then <code>transform</code> methods by passing the feature set to these methods. The <code>transform</code> method returns the specified number of principal components.</li>
</ol>
Take a look at the following code:
<pre><code class="hljs">from sklearn.decomposition import PCA

pca = PCA()
X_train = pca.fit_transform(X_train)
X_test = pca.transform(X_test)
</code></pre>
In the code above, we create a <code>PCA</code> object named <code>pca</code>. We did not specify the number of components in the constructor. Hence, all four of the features in the feature set will be returned for both the training and test sets.
The PCA class contains <code>explained_variance_ratio_</code> which returns the variance caused by each of the principal components. Execute the following line of code to find the &quot;explained variance ratio&quot;.
<pre><code class="hljs">explained_variance = pca.explained_variance_ratio_
</code></pre>
The <code>explained_variance</code> variable is now a float type array which contains variance ratios for each principal component. The values for the <code>explained_variance</code> variable looks like this:
<table class="table table-striped">
 <tbody>
 <tr>
 <td>0.722265</td>
 </tr>
 <tr>
 <td>0.239748</td>
 </tr>
 <tr>
 <td>0.0333812</td>
 </tr>
 <tr>
 <td>0.0046056</td>
 </tr>
 <tbody>
</table>
It can be seen that the first principal component is responsible for 72.22% variance. Similarly, the second principal component causes 23.9% variance in the dataset. Collectively we can say that (72.22 + 23.9) 96.21% percent of the classification information contained in the feature set is captured by the first two principal components.
Let's first try to use 1 principal component to train our algorithm. To do so, execute the following code:
<pre><code class="hljs">from sklearn.decomposition import PCA

pca = PCA(n_components=1)
X_train = pca.fit_transform(X_train)
X_test = pca.transform(X_test)
</code></pre>
The rest of the process is straightforward.
<h3 id="trainingandmakingpredictions">Training and Making Predictions</h3>
In this case we'll use random forest classification for making the predictions.
<pre><code class="hljs">from sklearn.ensemble import RandomForestClassifier

classifier = RandomForestClassifier(max_depth=2, random_state=0)
classifier.fit(X_train, y_train)

# Predicting the Test set results
y_pred = classifier.predict(X_test)
</code></pre>
<h3 id="performanceevaluation">Performance Evaluation</h3>
<pre><code class="hljs">from sklearn.metrics import confusion_matrix
from sklearn.metrics import accuracy_score

cm = confusion_matrix(y_test, y_pred)
print(cm)
print(&#x27;Accuracy&#x27; + accuracy_score(y_test, y_pred))
</code></pre>
The output of the script above looks like this:
<pre><code class="hljs">[[11 0 0]
 [ 0 12 1]
 [ 0 1 5]]
0.933333333333
</code></pre>
It can be seen from the output that with only one feature, the random forest algorithm is able to correctly predict 28 out of 30 instances, resulting in 93.33% accuracy.
<h3 id="resultswith2and3principalcomponents">Results with 2 and 3 Principal Components</h3>
Now let's try to evaluate classification performance of the random forest algorithm with 2 principal components. Update this piece of code:
<pre><code class="hljs">from sklearn.decomposition import PCA

pca = PCA(n_components=2)
X_train = pca.fit_transform(X_train)
X_test = pca.transform(X_test)
</code></pre>
Here the number of components for PCA has been set to 2. The classification results with 2 components are as follows:
<pre><code class="hljs">[[11 0 0]
 [ 0 10 3]
 [ 0 2 4]]
0.833333333333
</code></pre>
With two principal components the classification accuracy decreases to 83.33% compared to 93.33% for 1 component.
With three principal components, the result looks like this:
<pre><code class="hljs">[[11 0 0]
 [ 0 12 1]
 [ 0 1 5]]
0.933333333333
</code></pre>
With three principal components the classification accuracy again increases to 93.33%
<h3 id="resultswithfullfeatureset">Results with Full Feature Set</h3>
Let's try to find the results with a full feature set. To do so, simply remove the PCA part from the script that we wrote above. The results with full feature set, without applying PCA looks like this:
<pre><code class="hljs">[[11 0 0]
 [ 0 13 0]
 [ 0 2 4]]
0.933333333333
</code></pre>
The accuracy received with the full feature set for random forest algorithm is also 93.33%.
<h2 id="discussion">Discussion</h2>
From the above experimentation we achieved optimal level of accuracy while significantly reducing the number of features in the dataset. We saw that accuracy achieved with only 1 principal component is equal to the accuracy achieved with the feature set i.e. 93.33%. It is also pertinent to mention that the accuracy of a classifier doesn't necessarily improve with an increased number of principal components. From the results we can see that the accuracy achieved with one principal component (93.33%) was greater than the one achieved with two principal components (83.33%).
The number of principal components to retain in a feature set depends on several conditions such as storage capacity, training time, performance, etc. In some dataset all the features are contributing equally to the overall variance, therefore all the principal components are crucial to the predictions and none can be ignored. A general rule of thumb is to take the number of principal components that contribute to significant variance and ignore those with diminishing variance returns. A good way is to plot the variance against principal components and ignore the principal components with diminishing values as shown in the following graph:
<img class="img-responsive" alt="Variance of PCs" src="https://s3.amazonaws.com/stackabuse/media/implementing-pca-python-scikit-learn-1.png">
For instance, in the chart above, we can see that after the third principal component the change in variance almost diminishes. Therefore, the first three components can be selected.

Principal Component Analysis (PCA) in Python with Scikit-Learn

Chidume Nnamdi

Beginner's Guide to ngrx and Angular

One of the basic elements of programming languages are variables. Simply speaking a variable is an abstraction layer for the memory cells that contain the actual value. For us, as a developer, it is easier to remember the name of the memory cell than it is to remember its physical memory address. A valid name can consist of characters from 'a' to 'z' (in both lower and upper cases) as well as digits. No spaces or special characters, like umlauts and hyphens, are allowed in the name.
Furthermore, variables have a specific data type like strings (characters), digits, lists or references to other variables. In Python, we may reuse the same variable to store values of any type. The type is automatically determined by the value that is assigned to the name. In order to define a variable with a specific value, simply assign this value to a name as follows:
<pre><code class="hljs">age = 42
name = &quot;Dominic&quot;
places = [&quot;Berlin&quot;, &quot;Cape Town&quot;, &quot;New York&quot;]
</code></pre>
The Python interpreter creates the three variables <code>age</code>, <code>name</code>, and <code>places</code>, and assigns the value 42 to the first and &quot;Dominic&quot; to the second variable, and <code>places</code> becomes a list of three elements that contains the strings &quot;Berlin&quot;, &quot;Cape Town&quot;, and &quot;New York&quot;.
<h2 id="namespaces">Namespaces</h2>
All the variables from above are part of the same namespace and therefore have the same scope. Unless redefined as a local variable later on, a variable defined in the main program belongs to the global namespace that can be accessed by any function in your Python program. The following example code demonstrates that and uses the two variables <code>name</code> and <code>age</code> in the function <code>info()</code>.
<pre><code class="hljs">age = 42
name = &quot;Dominic&quot;
places = [&quot;Berlin&quot;, &quot;Cape Town&quot;, &quot;New York&quot;]

def info():
 print(&quot;%s is %i years old.&quot; % (name, age))
 return

info()
</code></pre>
The output consists of the single line that comes from the <code>print</code> statement in function <code>info()</code>:
<pre><code class="hljs">$ python3 global.py
Dominic is 42 years old.
</code></pre>
To be more precise, every module, class and function has its own namespace and variables are locally bound to that. In the next example we make use of two namespaces - the outer, global one from the main program and the inner, local one from the function simply named <code>output()</code>. The variable <code>place</code> exists in the main program (line 6) and is redefined as a local variable with a new value in line 2 of the function <code>output()</code>.
<pre><code class="hljs">def output():
 place = &quot;Cape Town&quot;
 print(&quot;%s lives in %s.&quot; % (name, place))
 return

place = &quot;Berlin&quot;
name = &quot;Dominic&quot;
print(&quot;%s lives in %s.&quot; % (name, place))
output()
</code></pre>
The output consists of these two lines, whereas the first line originates from the main program (line 8) and the second line from the <code>print</code> statement in line 3 in the function <code>output()</code>. At first the two variables <code>name</code> and <code>place</code> are defined in the main program (lines 6 and 7) and printed using <code>stdout</code>. Calling the <code>output()</code> function, the variable <code>place</code> is locally redefined in line 2 and <code>name</code> comes from the global namespace, instead. This leads to the output as shown below.
<pre><code class="hljs">$ python3 localscope.py
Dominic lives in Berlin.
Dominic lives in Cape Town.
</code></pre>
<h2 id="modifyingglobalvariablesinadifferentnamespace">Modifying Global Variables in a Different Namespace</h2>
The value of a global variable can be accessed throughout the entire program. In order to achieve that from within functions, Python offers the usage of the keyword <code>global</code>. The function below demonstrates how to use it and imports the variable <code>name</code> into the namespace of the function:
<pre><code class="hljs">def location():
 global place
 place = &quot;Cape Town&quot;
 return

place = &quot;Berlin&quot;
print(place)
location()
print(place)
</code></pre>
The variable <code>place</code> is already defined in the main program (line 6). Using the keyword <code>global</code> in line 2, the variable becomes available in the function <code>location()</code> and can be set to a different value, immediately (line 3). The output of the the code is shown here:
<pre><code class="hljs">$ python3 globalscope.py
Berlin
Cape Town
</code></pre>
Without using the keyword <code>global</code> as seen in line 2, the variable <code>place</code> would be treated as a local variable in the function <code>location()</code> instead and the variable <code>place</code> from the main program is unchanged then.
<h2 id="detectthescopeofavariable">Detect the Scope of a Variable</h2>
Python has two built-in methods named <code>globals()</code> and <code>locals()</code>. They allow you to determine whether a variable is either part of the global namespace or the local namespace. The following example shows how to use these methods:
<pre><code class="hljs">def calculation():
 &quot;do a complex calculation&quot;
 
 global place
 place = &quot;Cape Town&quot;
 name = &quot;John&quot;
 
 print(&quot;place in global:&quot;, &#x27;place&#x27; in globals())
 print(&quot;place in local :&quot;, &#x27;place&#x27; in locals())
 print(&quot;name in global :&quot;, &#x27;name&#x27; in globals())
 print(&quot;name in local :&quot;, &#x27;name&#x27; in locals())
 return

place = &quot;Berlin&quot;
print(place)
calculation()
</code></pre>
The output is as follows and shows the scope of the two variables <code>place</code> and <code>name</code> inside the function <code>calculation()</code>:
<pre><code class="hljs">$ python3 variablelist.py
Berlin
place in global: True
place in local : False
name in global : False
name in local : True
</code></pre>
<h2 id="usingglobalvariablesinpractice">Using Global Variables in Practice</h2>
Using and modifying global variables from inside functions is seen as a very bad programming style, as it causes side effects, which are rather difficult to detect. It is strongly recommended to use proper function parameters, instead.
<h2 id="acknowledgements">Acknowledgements</h2>
The author would like to thank Mandy Neumeyer for her support while preparing the article.
<h2 id="linksandreferences">Links and References</h2>
<ul>
<li>Bernd Klein: Global, Local and nonlocal Variables, <a rel="nofollow" target="_blank" href="https://www.python-course.eu/python3_global_vs_local_variables.php">https://www.python-course.eu/python3_global_vs_local_variables.php</a></li>
<li>Variable scope and lifetime, University of Cape Town, <a rel="nofollow" target="_blank" href="http://python-textbok.readthedocs.io/en/1.0/Variables_and_Scope.html">http://python-textbok.readthedocs.io/en/1.0/Variables_and_Scope.html</a></li>
<li>Python 3: globals(), <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/functions.html#globals">https://docs.python.org/3/library/functions.html#globals</a></li>
<li>Python 3: locals(), <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/functions.html#locals">https://docs.python.org/3/library/functions.html#locals</a></li>
</ul>

Local and Global Variables in Python

<h2 id="deploymenttoavirtualprivateserver">Deployment to a Virtual Private Server</h2>
Welcome to the seventh and final installment to this multi-part tutorial series on full-stack web development using Vue.js and Flask. In this post I will be demonstrating how to deploy the application built throughout this series.
The code for this post can be found on my GitHub account under the branch <a rel="nofollow" target="_blank" href="https://github.com/amcquistan/flask-vuejs-survey/tree/SeventhPost">SeventhPost</a>.
<h2 id="seriescontent">Series Content</h2>
<ol>
<li><a href="/single-page-apps-with-vue-js-and-flask-setting-up-vue-js/">Seup and Getting to Know VueJS</a></li>
<li><a href="/single-page-apps-with-vue-js-and-flask-navigating-vue-router/">Navigating Vue Router</a></li>
<li><a href="/single-page-apps-with-vue-js-and-flask-state-management-with-vuex/">State Management with Vuex</a></li>
<li><a href="/single-page-apps-with-vue-js-and-flask-restful-api-with-flask/">RESTful API with Flask</a></li>
<li><a href="/single-page-apps-with-vue-js-and-flask-ajax-integration/">AJAX Integration with REST API</a></li>
<li><a href="/single-page-apps-with-vue-js-and-flask-jwt-authentication/">JWT Authentication</a></li>
<li>Deployment to a Virtual Private Server (you are here)</li>
</ol>
<h2 id="overviewofthetechnologies">Overview of the Technologies</h2>
This tutorial will be covering several technologies necessary to deploy a distributed multi-tier Flask REST API and Vue.js SPA application. Below I have listed the technologies and their uses:
<ul>
<li>Ubuntu LTS 16.04: host server for running various applications and servers</li>
<li>uWSGI: Web Server Gateway Interface (WSGI) container server for executing Python applications (Flask in this case)</li>
<li>Nginx: Highly performant non-blocking HTTP web server capable of reverse proxying to uWSGI</li>
<li>Node.js / NPM: Javascript environment for building the Vue.js SPA application</li>
</ul>
<h2 id="gettingthecodereadyfordeployment">Getting the Code Ready for Deployment</h2>
There are a couple of changes that need to be made to the code to make it more maintainable once the application has been deployed to my production environment.
For example, in <code>api/index.js</code> of the <code>survey-spa</code> Vue.js application I have hardcoded a variable called <code>API_URL</code> to point to the dev server <code>http://127.0.0.1:5000/api</code>. Doing this I will need to remember to change this to the production server's IP address every time I need to deploy.
Experience has taught me that there will always be changes to the application requiring future deployments where I am likely to forget to update this IP address. A better approach is to remove the risk of me forgetting to update this and instead utilizing configurations in the build process to handle this for me resulting in less that I have to remember (ie, fewer steps needed) during deployment. This significantly reduces the risk of an unsuccessful deployment on future updates.
I accomplish this by moving over to the survey-spa/config directory and modifying the <code>dev.env.js</code> and <code>prod.env.js</code> files by defining a variable called <code>API_URL</code> which are assigned a value of <code>http://localhost:5000/api</code> for dev and <code>http://${process.env.BASE_URL}/api</code> for prod as shown below:
<pre><code class="hljs">// dev.env.js

&#x27;use strict&#x27;
const merge = require(&#x27;webpack-merge&#x27;)
const prodEnv = require(&#x27;./prod.env&#x27;)

module.exports = merge(prodEnv, {
 NODE_ENV: &#x27;&quot;development&quot;&#x27;,
 API_URL: JSON.stringify(`http://localhost:5000/api`)
})
</code></pre>
<pre><code class="hljs">// prod.env.js
&#x27;use strict&#x27;
module.exports = {
 NODE_ENV: &#x27;&quot;production&quot;&#x27;,
 API_URL: JSON.stringify(`http://${process.env.BASE_URL}/api`)
}
</code></pre>
Note: the value of <code>process.env.BASE_URL</code> is an environment variable that I will add to the Ubuntu server's user <code>.bash\_profile</code> and set it equal to the IP address of the server.
Then over in <code>api/index.js</code> I modify the line <code>const API_URL = 'http://127.0.0.1:5000/api'</code> and set it equal to <code>process.env.API_URL</code>.
Next, in the Flask application I need to add a new module named <code>wsgi.py</code> to serve as the entry point to the Flask REST API. The <code>wsgi.py</code> module looks quite similar to the <code>appserver.py</code> module except for that it does not have any calls to the <code>run(...)</code> method of the app object. This is because the app object will serve as a callable for the uWSGI container server to execute against using its fast binary protocol rather than the regular development server that gets created when <code>app.run(...)</code> is called.
<pre><code class="hljs"># backend/wsgi.py

from surveyapi.application import create_app
app = create_app()
</code></pre>
With this finished I can push my changes to version control and hop onto my production server to pull down the project and set up the programs I will use to run the application on the production server.
<h2 id="readyingtheubuntuserver">Readying the Ubuntu Server</h2>
Next I'll get onto my production Ubuntu virtual private server which could be hosted by one of the many Cloud services such as AWS, DigitalOcean, Linode, etc... and begin installing all of the goodies I listed in the Overview of the Technologies section.
<pre><code class="hljs">$ apt-get update
$ apt-get install python3-pip python3-dev python3-venv nginx nodejs npm
</code></pre>
With those installs out of the way I can now create a user called &quot;survey&quot; to execute the application under and house the code.
<pre><code class="hljs">$ adduser survey
$ usermod -aG sudo survey
$ su survey
$ cd
</code></pre>
I should now be in the &quot;survey&quot; user's home directory at /home/survey.
With the survey user created I can update the <code>.bash\_profile</code> file to contain the IP address of my production server by adding this line to the end of the file. Note that 123.45.67.89 represents a fake IP address of my server. Replace it with your true IP address if you are following along.
<pre><code class="hljs">export BASE_URL=123.45.67.89
</code></pre>
Next I want to tell the firewall (<a rel="nofollow" target="_blank" href="https://wiki.ubuntu.com/UncomplicatedFirewall">ufw</a>) that OpenSSH is acceptable and enable it.
<pre><code class="hljs">$ sudo ufw allow OpenSSH
$ sudo ufw enable
</code></pre>
With this done I will now clone the repo onto the server so I can build and deploy it.
<pre><code class="hljs">$ git clone https://github.com/amcquistan/flask-vuejs-survey.git
</code></pre>
Now I will cd into <code>flask-vuejs-survey/frontend/survey-spa</code> and install the frontend dependencies as well as build the production application.
<pre><code class="hljs">$ cd flask-vuejs-survey/frontend/survey-spa
$ npm install
$ npm run build
</code></pre>
This creates a new directory called &quot;dist&quot;, which will contain an index.html page and a directory called &quot;static&quot; that contains all the compiled CSS and JavaScript files. These are what I will have Nginx server up to constitute the SPA's front-end application.
Next up I will create a virtual environment in the /home/survey directory for an isolated Python3 interpreter to run the Python application. Once installed I activate it and move into the backend project directory to install its dependency packages specified in the <code>requirements.txt</code> file.
<pre><code class="hljs">$ python3 -m venv venv
$ source venv/bin/activate
(venv) $ cd flask-vuejs-survey/backend
(venv) $ pip install -r requirements.txt
</code></pre>
Now I can initialize the <a target="_blank" href="/a-sqlite-tutorial-with-python/">SQLite database</a> and run the migrations to create the various database tables required by the REST API.
<pre><code class="hljs">(venv) $ python manage.py db upgrade
</code></pre>
At this point I would like to fire up the Flask dev server to make sure that all is working as expected. Before doing so I need to tell the <code>ufw</code> service to allow traffic in on port 5000.
<pre><code class="hljs">(venv) $ sudo ufw allow 5000
(venv) $ python appserver.py
</code></pre>
In a browser I can now go to <code>http://123.45.67.89:5000/api/surveys/</code> and I should see a simple JSON response of <code>[]</code> because there are no surveys in this database yet, but this does indicate that a successful request was made. Additionally, in the terminal connected to the server there should be a logged message for the GET request issued from my browser.
I key in Ctrl+C in the terminal to kill the Flask dev server and move on to configuring uWSGI to control the execution of my Flask REST API. If you are wondering where uWSGI came from it is specified as a requirement in the <code>requirements.txt</code> file that I pip installed with earlier.
<h2 id="settingupuwsgicontainerserver">Setting up uWSGI Container Server</h2>
Similar to what I just did with the Flask dev server I will now test that the uWSGI server can serve up the application as follows.
<pre><code class="hljs">(venv) $ uwsgi --socket 0.0.0.0:5000 --protocol=http -w wsgi:app
</code></pre>
Again, going to my browser and refreshing the same request I made previously should return an empty JSON array response. Once satisfied with my progress I can again key Ctrl+C into the terminal and move on.
There are two more steps I would like to do to complete the configuration of the uWSGI container server. One step is to create a configuration file that uWSGI will read in which will replace many of those command line flags and arguments I used above. The second step is to create a <code>systemd</code> service file to manage the uWSGI container server as a service like many of the others already running on the Ubuntu server.
In the backend directory I make a file called <code>surveyapi.ini</code> and fill it with the following:
<pre><code class="hljs">[uwsgi]
module = wsgi:app

master = true
processes = 4

socket = myproject.sock
chmod-socket = 660
vacuum = true

die-on-term = true
</code></pre>
This config file lets uWSGI know that the callable is the app object inside of the <code>wsgi.py</code> module. It also tells it to spawn and use four processes to handle application requests communicated over a socket file called <code>surveyapi.sock</code> which has a loose enough permission to allow the Nginx web server to read and write from it. The <code>vacuum</code> and <code>die-on-term</code> settings are to ensure proper cleanup.
For the <code>systemd</code> service file, I need to create a file called <code>surveyapi.service</code> in the <code>/etc/systemd/system</code> directory and add some descriptors plus access, write, and execution commands like so:
<pre><code class="hljs">(venv) $ sudo nano /etc/systemd/system/surveyapi.service
</code></pre>
Then populate it with the following:
<pre><code class="hljs">[Unit]
Description=uWSGI Python container server
After=network.target

[Service]
User=survey
Group=www-data
WorkingDirectory=/home/survey/flask-vuejs-survey/backend
Environment=&quot;PATH=/home/survey/venv/bin&quot;
ExecStart=/home/survey/venv/bin/uwsgi --ini surveyapi.ini

[Install]
WantedBy=multi-user.target
</code></pre>
Now I can start the service and check its status and make sure the backend directory now contains <code>surveyapi.sock</code>.
<pre><code class="hljs">(venv) $ sudo systemctl start surveyapi
(venv) $ sudo systemctl status surveyapi
 Loaded: loaded (/etc/systemd/system/surveyapi.service; disabled; vendor preset: enabled)
 Active: active (running) since Mon 2018-04-23 19:23:01 UTC; 2min 28s ago
 Main PID: 11221 (uwsgi)
 Tasks: 6
 Memory: 28.1M
 CPU: 384ms
 CGroup: /system.slice/surveyapi.service
 ├─11221 /home/survey/venv/bin/uwsgi --ini surveyapi.ini
 ├─11226 /home/survey/venv/bin/uwsgi --ini surveyapi.ini
 ├─11227 /home/survey/venv/bin/uwsgi --ini surveyapi.ini
 ├─11228 /home/survey/venv/bin/uwsgi --ini surveyapi.ini
 ├─11229 /home/survey/venv/bin/uwsgi --ini surveyapi.ini
 └─11230 /home/survey/venv/bin/uwsgi --ini surveyapi.ini

Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: mapped 437520 bytes (427 KB) for 5 cores
Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: *** Operational MODE: preforking ***
Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: WSGI app 0 (mountpoint=&#x27;&#x27;) ready in 0 seconds on interpreter 0x8b4c30 pid: 112
Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: *** uWSGI is running in multiple interpreter mode ***
Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: spawned uWSGI master process (pid: 11221)
Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: spawned uWSGI worker 1 (pid: 11226, cores: 1)
Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: spawned uWSGI worker 2 (pid: 11227, cores: 1)
Apr 23 19:23:01 ubuntu-s-1vcpu-2gb-sfo2-01 uwsgi[11221]: spawned uWSGI worker 3 (pid: 11228, cores: 1)
lines 1-23
(venv) $ ls -l /home/survey/flask-vuejs-survey/backend
-rw-rw-r-- 1 survey survey 201 Apr 23 18:18 appserver.py
-rw-rw-r-- 1 survey survey 745 Apr 23 17:55 manage.py
drwxrwxr-x 4 survey survey 4096 Apr 23 18:06 migrations
drwxrwxr-x 2 survey survey 4096 Apr 23 18:52 __pycache__
-rw-rw-r-- 1 survey survey 397 Apr 23 18:46 requirements.txt
drwxrwxr-x 3 survey survey 4096 Apr 23 18:06 surveyapi
-rw-rw-r-- 1 survey survey 133 Apr 23 19:04 surveyapi.ini
srw-rw---- 1 survey www-data 0 Apr 23 19:23 surveyapi.sock
-rw-r--r-- 1 survey survey 10240 Apr 23 18:19 survey.db
-rw-rw-r-- 1 survey survey 84 Apr 23 18:42 wsgi.py
</code></pre>
Excellent! The last thing I should do is enable automatic starting each time the system boots up ensuring that the application is always up.
<pre><code class="hljs">(venv) $ sudo systemctl enable surveyapi
</code></pre>
<h2 id="settingupnginx">Setting Up Nginx</h2>
I will utilize Nginx to serve static content such as HTML, CSS, and JavaScript as well as to reverse proxy REST API calls to the Flask / uWSGI application. To set up nginx to accomplish these things I will need to create a config file which defines how to manage these various requests.
Over in /etc/nginx/sites-available I will create a file called survey which will contain the following:
<pre><code class="hljs">server {
 listen 80;
 server_name 123.45.67.89;

 location /api {
 include uwsgi_params;
 uwsgi_pass unix:/home/survey/flask-vuejs-survey/backend/surveyapi.sock;
 }

 location / {
 root /home/survey/flask-vuejs-survey/frontend/survey-spa/dist;
 try_files $uri $uri/ /index.html;
 }
}
</code></pre>
This file creates a new server block configuration which says to listen to IP address 123.45.67.89 on the standard HTTP port of 80. Then it says look for any URI paths beginning with <code>/api</code> and reverse proxy that to the Flask / uWSGI REST API server using the previously defined socket file. Lastly, the config says to catch everything else under / and serve up the index.html file in the dist directory created when I built the Vue.js front-end SPA application prior.
With this config file created I need to let Nginx know that it is an available site by creating a symbolic link to the /etc/nginx/sites-enabled directory like so:
<pre><code class="hljs">$ sudo ln -s /etc/nginx/sites-available/survey /etc/nginx/sites-enabled 
</code></pre>
To allow traffic over the HTTP port and bind to Nginx I will issue the following update to <code>ufw</code> as well as close the previously opened 5000 port.
<pre><code class="hljs">$ sudo ufw delete allow 5000
$ sudo ufw allow &#x27;Nginx Full&#x27;
</code></pre>
Following this command I will need to restart the Nginx service like so for the updates to take effect.
<pre><code class="hljs">$ sudo systemctl restart nginx
</code></pre>
Now I can go to my browser again and visit <code>http://123.454.67.89</code> and I am presented with the survey application I've shown in prior articles.
<h2 id="conclusion">Conclusion</h2>
Well this is the concluding post to this multi-part tutorial series on how to utilize Flask and Vue.js to build a REST API enabled SPA application. I have attempted to cover most of the important topics that are common to many web application use cases assuming very little prior knowledge of the Flask and Vue.js technologies used.
I thank you for following along with this series and please do not be shy about commenting or critiquing below.

Single Page Apps with Vue.js and Flask: Deployment

In general, network services follow the traditional client/server model. One computer acts as a server to provide a certain service and another computer represents the client side which makes use of this service. In order to communicate over the network a <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Network_socket">network socket</a> comes into play, mostly only referred to as a socket. This kind of socket communication can even be used internally in a computer for <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Inter-process_communication">inter-process communication</a> (IPC).
This article explains how to write a simple client/server application that communicates via network socket using the Python programming language. For simplicity, our example server only outputs the received data using <code>stdout</code>. The idea behind the client/server application is a sensor in a weather station, which collects temperature data over time and sends the collected data to a server application, where the data gets processed further.
<h2 id="whatisasocket">What is a Socket?</h2>
A network socket is an endpoint of a two-way communication link between two programs or processes - client and server in our case - which are running on the network. This can be on the same computer as well as on different systems which are connected via the network.
Both parties communicate with each other by writing to or reading from the network socket. The technical equivalent in reality is a telephone communication between two participants. The network socket represents the corresponding number of the telephone line, or a contract in case of cell phones.
<h2 id="example">Example</h2>
In order to make use of the socket functionality, only the Python <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/socket.html">socket module</a> is necessary. In the example code shown below the Python <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/time.html">time module</a> is imported as well in order to simulate the weather station and to simplify time calculations.
In this case both the client and the server run on the same computer. A socket has a corresponding port number, which is 23456 in our case. If desired, you may choose a different port number from the unrestricted number range between 1024 and 65535.
<h3 id="theserver">The Server</h3>
Having loaded the additional Python <code>socket</code> module an Internet streaming socket is created using the <code>socket.socket</code> class with the two parameters <code>socket.AF_INET</code> and <code>socket.SOCK_STREAM</code>. The retrieval of the hostname, the fully qualified domain name, and the IP address is done by the methods <code>gethostname()</code>, <code>getfqdn()</code>, and <code>gethostbyname()</code>, respectively. Next, the socket is bound to the IP address and the port number 23456 with the help of the <code>bind()</code> method.
With the help of the <code>listen()</code> method the server listens for incoming connections on the specified port. In the while loop the server waits for incoming requests and accepts them using the <code>accept()</code> method. The data submitted by the client is read via <code>recv()</code> method as chunks of 64 bytes, and simply output using <code>stdout</code>. Finally, the current connection is closed if no further data is sent from the client.
<pre><code class="hljs"># load additional Python module
import socket

# create TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# retrieve local hostname
local_hostname = socket.gethostname()

# get fully qualified hostname
local_fqdn = socket.getfqdn()

# get the according IP address
ip_address = socket.gethostbyname(local_hostname)

# output hostname, domain name and IP address
print (&quot;working on %s (%s) with %s&quot; % (local_hostname, local_fqdn, ip_address))

# bind the socket to the port 23456
server_address = (ip_address, 23456)
print (&#x27;starting up on %s port %s&#x27; % server_address)
sock.bind(server_address)

# listen for incoming connections (server mode) with one connection at a time
sock.listen(1)

while True:
 # wait for a connection
 print (&#x27;waiting for a connection&#x27;)
 connection, client_address = sock.accept()

 try:
 # show who connected to us
 print (&#x27;connection from&#x27;, client_address)

 # receive the data in small chunks and print it
 while True:
 data = connection.recv(64)
 if data:
 # output received data
 print (&quot;Data: %s&quot; % data)
 else:
 # no more data -- quit the loop
 print (&quot;no more data.&quot;)
 break
 finally:
 # Clean up the connection
 connection.close()
</code></pre>
<h3 id="theclient">The Client</h3>
Now we will have a look at the client side. The Python code is mostly similar to the server side, except for the usage of the socket - the client uses the <code>connect()</code> method, instead. In a <code>for</code> loop the temperature data is sent to the server using the <code>sendall()</code> method. The call of the <code>time.sleep(2)</code> method pauses the client for two seconds before it sends another temperature reading. After all the temperature data is sent from the list the connection is finally closed using the <code>close()</code> method.
<pre><code class="hljs"># load additional Python modules
import socket
import time

# create TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

# retrieve local hostname
local_hostname = socket.gethostname()

# get fully qualified hostname
local_fqdn = socket.getfqdn()

# get the according IP address
ip_address = socket.gethostbyname(local_hostname)

# bind the socket to the port 23456, and connect
server_address = (ip_address, 23456)
sock.connect(server_address)
print (&quot;connecting to %s (%s) with %s&quot; % (local_hostname, local_fqdn, ip_address))

# define example data to be sent to the server
temperature_data = [&quot;15&quot;, &quot;22&quot;, &quot;21&quot;, &quot;26&quot;, &quot;25&quot;, &quot;19&quot;]
for entry in temperature_data:
 print (&quot;data: %s&quot; % entry)
 new_data = str(&quot;temperature: %s\n&quot; % entry).encode(&quot;utf-8&quot;)
 sock.sendall(new_data)
 
 # wait for two seconds
 time.sleep(2)

# close connection
sock.close()
</code></pre>
<h3 id="runningtheserverandclient">Running the Server and Client</h3>
To run both the server and the client program, open two terminal windows and issue the following commands - one per terminal window and in the following order:
<pre><code class="hljs">$ python3 echo-server.py
</code></pre>
and
<pre><code class="hljs">$ python3 echo-client.py
</code></pre>
The two figures below show the corresponding output of the example program:
<img class="img-responsive" alt="Server socket communication in Python" src="https://s3.amazonaws.com/block-sender/github/mockups/bs-plugin-browser/basic-socket-programming-python-1.png">
_Figure 1_
<img class="img-responsive" alt="Client socket communication in Python" src="https://s3.amazonaws.com/block-sender/github/mockups/bs-plugin-browser/basic-socket-programming-python-2.png">
_Figure 2_
<h2 id="conclusion">Conclusion</h2>
Writing Python programs that use IPC with sockets is rather simple. The example given above can certainly be extended to handle something more complex.