Mihajlo Pavloski

Python Modules: Creating, Importing, and Sharing

In this tutorial we are going to learn about using regular expressions in Python, including their syntax, and how to construct them using built-in Python modules. To do this we’ll cover the different operations in Python's <a rel="nofollow" target="_blank" href="https://docs.python.org/2/library/re.html">re module</a>, and how to use it in your Python applications.
<h2 id="whatareregularexpressions">What are Regular Expressions?</h2>
Regular expressions are basically just a sequence of characters that can be used to define a search pattern for finding text. This &quot;search engine&quot; is embedded within the Python programming language (and many other languages as well) and made available through the <code>re</code> module.
To use regular expressions (or &quot;regex&quot; for short) you usually specify the rules for the set of possible strings that you want to match and then ask yourself questions such as &quot;Does this string match the pattern?&quot;, or &quot;Is there a match for the pattern anywhere in this string?&quot;.
You can also use regexes to modify a string or to split it apart in various ways. These &quot;higher order&quot; operations all start by first matching text with the regex string, and then the string can be manipulated (like being split) once the match is found. All this is made possible by the <code>re</code> module available in Python, which we'll look at further in some later sections.
<h2 id="regularexpressionsyntax">Regular Expression Syntax</h2>
A regular expression specifies a pattern that aims to match the input string. In this section we'll show some of the special characters and patterns you can use to match strings.
<h2 id="matchingcharacters">Matching Characters</h2>
Regular expressions can contain both special and ordinary characters. Most ordinary characters, like 'A', 'a', or '0', are the simplest regular expressions; they simply match themselves. There are also other special characters which can't match themselves, i.e. <code>^</code>, <code>$</code>, <code>*</code>, <code>+</code>, <code>?</code>, <code>{</code>, <code>}</code>, <code>[</code>, <code>]</code>, <code>\</code>, <code>|</code>, <code>(</code>, and <code>)</code>. This is because they are used for higher-order matching functionality, which is described further in this table:
<table class="table table-bordered table-striped">
 <colgroup>
 <col class="span1">
 <col class="span3">
 <col class="span4">
 </colgroup>
 <thead>
 <tr>
 <th>Metacharacter</th>
 <th>Description</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>
 <code>*</code>
 </td>
 <td> Matches the preceding element zero or more times. For example, <code>ab*c</code> matches "ac", "abc", "abbbc", etc. <code>[xyz]*</code> matches "", "x", "y", "z", "zx", "zyx", "xyzzy", and so on. <code>(ab)*</code> matches "", "ab", "abab", "ababab", and so on. </td>
 </tr>
 <tr>
 <td>
 <code>+</code>
 </td>
 <td> Matches the preceding element one or more times. For example, <code>ab+c</code> matches "abc", "abbc", "abbbc", and so on, but not "ac". </td>
 </tr>
 <tr>
 <td>
 <code>?</code>
 </td>
 <td> Matches the preceding element zero or one time. For example, <code>ab?c</code> matches only "ac" or "abc". </td>
 </tr>
 <tr>
 <td>
 <code>|</code>
 </td>
 <td> The choice (also known as alternation or set union) operator matches either the expression before or the expression after this operator. For example, <code>abc|def</code> can match either "abc" or "def". </td>
 </tr>
 <tr>
 <td>
 <code>.</code>
 </td>
 <td> Matches any single character (many applications exclude newlines, and exactly which characters are considered newlines is flavor-, character-encoding-, and platform-specific, but it is safe to assume that the line feed character is included). Within POSIX bracket expressions, the dot character matches a literal dot. For example, <code>a.c</code> matches "abc", etc., but <code>[a.c]</code> matches only "a", ".", or "c". </td>
 </tr>
 <tr>
 <td>
 <code>^</code>
 </td>
 <td> Matches the starting position in the string, like the <code>startsWith()</code> function. In line-based tools, it matches the starting position of any line. </td>
 </tr>
 <tr>
 <td>
 <code>?</code>
 </td>
 <td> Matches the ending position of the string or the position just before a string-ending newline, like the <code>endsWith()</code> function. In line-based tools, it matches the ending position of any line. </td>
 </tr>
 </tbody>
</table>
Credit to Wikipedia for some of the regex descriptions.
<h2 id="regularexpressionsmethodsinpython">Regular Expressions Methods in Python</h2>
There are several methods available to use regular expressions. Here we are going to discuss some of the most commonly used methods and also give a few examples of how they are used. These methods include:
<ol>
<li>re.match()</li>
<li>re.search()</li>
<li>re.findall()</li>
<li>re.split()</li>
<li>re.sub()</li>
<li>re.compile()</li>
</ol>
<h3 id="rematchpatternstringflags0">re.match(pattern, string, flags=0)</h3>
This expression is used to match a character or set of characters at the beginning of a string. It's also important to note that this expression will only match at the beginning of the string and not at the beginning of each line if the given string has multiple lines.
The expression below will return <code>None</code> because Python does not appear at the beginning of the string.
<pre><code class="hljs"># match.py

import re
result = re.match(r&#x27;Python&#x27;, &#x27;It\&#x27;s easy to learn Python. Python also has elegant syntax&#x27;)

print(result)
</code></pre>
<pre><code class="hljs">$ python match.py
None
</code></pre>
<h3 id="researchpatternstring">re.search(pattern, string)</h3>
This module will checks for a match anywhere in the given string and will return the results if found, and <code>None</code> if not found.
In the following code we are simply trying to find if the word &quot;puppy&quot; appears in the string &quot;Daisy found a puppy&quot;.
<pre><code class="hljs"># search.py

import re

if re.search(&quot;puppy&quot;, &quot;Daisy found a puppy.&quot;):
 print(&quot;Puppy found&quot;)
else:
 print(&quot;No puppy&quot;)
</code></pre>
Here we first import the <code>re</code> module and use it to search the occurrence of the substring &quot;puppy&quot; in the string &quot;Daisy found a puppy&quot;. If it does exist in the string, a <a rel="nofollow" target="_blank" href="https://docs.python.org/2/library/re.html#re.MatchObject">re.MatchObject</a> is returned, which is considered &quot;truthy&quot; when evalutated in an if-statement.
<pre><code class="hljs">$ python search.py 
Puppy found
</code></pre>
<h3 id="recompilepatternflags0">re.compile(pattern, flags=0)</h3>
This method is used to compile a regular expression pattern into a regular expression object, which can be used for matching using its <code>match()</code> and <code>search()</code> methods, which we have discussed above. This can also save time since parsing/handling regex strings can be computationally expensive to run.
<pre><code class="hljs"># compile.py

import re

pattern = re.compile(&#x27;Python&#x27;)
result = pattern.findall(&#x27;Pythonistas are programmers that use Python, which is an easy-to-learn and powerful language.&#x27;)

print(result)

find = pattern.findall(&#x27;Python is easy to learn&#x27;)

print(find)
</code></pre>
<pre><code class="hljs">$ python compile.py 
[&#x27;Python&#x27;, &#x27;Python&#x27;]
[&#x27;Python&#x27;]
</code></pre>
Notice that only the matched string is returned, as opposed to the entire word in the case of &quot;Pythonistas&quot;. This is more useful when using a regex string that has special match characters in it.
<h3 id="resubpatternreplstring">re.sub(pattern, repl, string)</h3>
Like the name suggests, this expression is used to search and substitute for a new string if the pattern occurs.
<pre><code class="hljs"># sub.py

import re
result = re.sub(r&#x27;python&#x27;, &#x27;ruby&#x27;, &#x27;python is a very easy language&#x27;)

print(result)
</code></pre>
<pre><code class="hljs">$ python sub.py 
ruby is a very easy language
</code></pre>
<h3 id="refindallpatternstring">re.findall(pattern, string)</h3>
As you've seen prior to this section, this method finds and retrieves a list of all occurrences in the given string. It combines both the functions and properties of <code>re.search()</code> and <code>re.match()</code>. The following example will retrieve all the occurrences of &quot;Python&quot; from the string.
<pre><code class="hljs"># findall.py

import re

result = re.findall(r&#x27;Python&#x27;, &#x27;Python is an easy to learn, powerful programming language. Python also has elegant syntax&#x27;)
print(result)
</code></pre>
<pre><code class="hljs">$ python findall.py 
[&#x27;Python&#x27;, &#x27;Python&#x27;]
</code></pre>
Again, using an exact match string like this (&quot;Python&quot;) is really only useful for finding if the regex string occurs in the given string, or how many times it occurs.
<h3 id="resplitpatternstringmaxsplit0flags0">re.split(pattern, string, maxsplit=0, flags=0)</h3>
This expression will split a string at the location in which the specified pattern occurs in the string. It will also return the text of all groups in the pattern if an advanced feature like <a rel="nofollow" target="_blank" href="https://www.regular-expressions.info/brackets.html">capturing parentheses</a> are used in the pattern.
<pre><code class="hljs"># split.py

import re

result = re.split(r&quot;y&quot;, &quot;Daisy found a puppy&quot;)

if result:
 print(result)
else:
 print(&quot;No puppy&quot;)
</code></pre>
As you can see above, the character pattern &quot;y&quot; occurs three times and the expression has split in all instances where it occurs.
<pre><code class="hljs">$ python split.py 
[&#x27;Dais&#x27;, &#x27; found a pupp&#x27;, &#x27;&#x27;]
</code></pre>
<h2 id="practicalusesofregularexpressions">Practical uses of Regular Expressions</h2>
Whether you know it or not, we use regular expressions almost daily in our applications. Since regular expressions are available in just about every programming language, it's not easy to escape their usage. Let's look at some of the ways regular expressions can be used in your applications.
<h3 id="constructingurls">Constructing URLs</h3>
Every web page has a URL. Now imagine you have a Django website with an address like &quot;<a rel="nofollow" target="_blank" href="http://www.example.com/products/27/">http://www.example.com/products/27/</a>&quot;, where 27 is the ID of a product. It would be very cumbersome to write separate views to match every single product.
However, with regular expressions, we can create a pattern that will match the URL and extract the ID for us:
An expression that will match and extract any numerical ID could be <code>^products/(\d+)/$</code>.
<ul>
<li><code>^products/</code> tells Django to match a string that has &quot;products/&quot; at the beginning of the URL (where &quot;beginning&quot; of the string is specified by <code>^</code>)</li>
<li><code>(\d+)</code> means that there will be a number (specified by <code>\d+</code>) and we want it captured and extracted (specified by the parenthases)</li>
<li><code>/</code> tells Django that another &quot;/&quot; character should follow</li>
<li><code>$</code> indicates the end of the URL, meaning that only strings ending with the / will match this pattern</li>
</ul>
<h3 id="validatingemailaddresses">Validating Email Addresses</h3>
Every authentication system requires users to sign up and log in before they can be allowed access to the system. We can use regular expression to check if an email address supplied is in a valid format.
<pre><code class="hljs"># validate_email.py

import re

email = &quot;example@gmail.com&quot;

if not re.match(re.compile(r&#x27;^.+@[^.].*\.[a-z]{2,10}$&#x27;, flags=re.IGNORECASE), email):
 print(&quot;Enter a valid email address&quot;)
else:
 print(&quot;Email address is valid&quot;)
</code></pre>
As you can see, this is a pretty complicated regex string. Let's break it down a bit using the example email address in the code above. It basically means the following:
<ul>
<li><code>^.+@</code>: Match every character from the beginning of the string up until the '@' character
<ul>
<li>Matches &quot;example@&quot; from &quot;<a href="mailto:example@gmail.com">example@gmail.com</a>&quot;</li>
</ul>
</li>
<li><code>[^.].*</code>: Match every character except &quot;.&quot;
<ul>
<li>Matches &quot;gmail&quot; from &quot;<a href="mailto:example@gmail.com">example@gmail.com</a>&quot;</li>
</ul>
</li>
<li><code>\.[a-z]{2,10}$</code>: Match the domain TLD characters (with max length 10 characters) until the end of the string
<ul>
<li>Matches &quot;.com&quot; from &quot;<a href="mailto:example@gmail.com">example@gmail.com</a>&quot;</li>
</ul>
</li>
</ul>
So, as you'd expect, the code matches our example address:
<pre><code class="hljs">$ python validate_email.py 
Email address is valid
</code></pre>
<h3 id="validatingphonenumbers">Validating Phone Numbers</h3>
The following example is used to validate a list of prefixed Canadian numbers:
<pre><code class="hljs"># validate_numbers.py

import re

numbers = [&quot;+18009592809&quot;, &quot;=18009592809&quot;]

for number in numbers:
 if not re.match(re.compile(r&quot;^(\+1?[-. ]?(\d+))$&quot;), number):
 print(&quot;Number is not valid&quot;)
 else:
 print(&quot;Number is valid&quot;)
</code></pre>
<pre><code class="hljs">$ python validate_numbers.py 
Number is valid
Number is not valid
</code></pre>
As you can see, because the second number uses a &quot;=&quot; character instead of &quot;+&quot;, it is deemed invalid.
<h3 id="filteringunwantedcontent">Filtering Unwanted Content</h3>
Regular expressions can also be used to filter certain words out of post comments, which is particularly useful in blog posts and social media. The following example shows how you can filter out pre-selected words that users should not use in their comments.
<pre><code class="hljs"># filter.py

import re

curse_words = [&quot;foo&quot;, &quot;bar&quot;, &quot;baz&quot;]
comment = &quot;This string contains a foo word.&quot;
curse_count = 0

for word in curse_words:
 if re.search(word, comment):
 curse_count += 1

print(&quot;Comment has &quot; + str(curse_count) + &quot; curse word(s).&quot;)
</code></pre>
<pre><code class="hljs">$ python filter.py 
Comment has 1 curse word(s).
</code></pre>
<h2 id="conclusion">Conclusion</h2>
This tutorial has covered what is needed to be able to use regular expressions in any application. Feel free to consult the documentation for the <a rel="nofollow" target="_blank" href="https://docs.python.org/2/library/re.html">re module</a>, which has a ton of resources to help you accomplish your application's goals.

Esther Vaati

Introduction to Regular Expressions in Python

Adam McQuistan

Using Machine Learning to Predict the Weather: Part 2

<h2 id="understandingthepathvariable">Understanding the $PATH Variable</h2>
In this tutorial, we will show you how to permanently set your PATH on Linux.
First off, why should you care?
The $PATH variable, or just PATH, without the $ indicating variables, specifies a list of directories that impacts your computing platform's functionality in a critical way. This is because the $PATH is the list of directories in which the system searches for executable programs, scripts, or files.
Imagine trying to run the <code>ssh</code> command, for example, to connect to a server. What happens if the system cannot find the <code>ssh</code> program? You are unable to connect to servers and run computations. You have a plethora of useful programs, examples like <code>python</code>, <code>javac</code>, <code>npm</code>, <code>make</code>, <code>chmod</code>, <code>apt-get</code> and so on, that your computer needs to be able to find when you invoke them at the command line.
The $PATH variable is the key that makes it possible to find the correct program and execute it at your command without needing the executable's full directory path. When your PATH is set incorrectly, your shell will be unable to find programs, and certain commands will fail.
<h2 id="usingbash_profiletosetyourpath">Using bash_profile to Set your PATH</h2>
A common mistake with the $PATH variable is to set it in the current shell only, without persisting the change. When you open a new shell, the changes are lost, and you are once again unable to execute certain commands because those programs are not found in the PATH.
The first way of setting your $PATH permanently is to modify the $PATH variable in your Bash profile file, located at <code>/home/&lt;user&gt;/.bash_profile</code>.
For example, let's say I want to add a new directory <code>/home/tomahawk/tools/jdk1.8.0_92/bin</code> to my PATH. You might recognize this as a Java Development Kit installation. However, that is beside the point. Whatever the directory contains, I can add it to our path and make the programs that this <code>bin</code> directory contains accessible from the command line by adding the following line to the end of the file <code>~/.bash_profile</code>:
<pre><code class="hljs">...
export PATH=&quot;$PATH:/home/tomahawk/tools/jdk1.8.0_92/bin&quot;
</code></pre>
A good way to edit the file is to use <code>nano</code>, <code>vi</code>, <code>vim</code> or <code>emacs</code>. You can use the command <code>sudo &lt;editor&gt; ~/.bash_profile</code>, and enter your admin password when prompted, then add that line to the end and save the file.
To activate the changes in the current shell, you have to &quot;source&quot; the updated <code>bash_profile</code> file. You do this with the command:
<pre><code class="hljs">$ source ~/.bash_profile
</code></pre>
This simply imports the file's settings into the current shell. Now every time you open your shell, your <code>bash_profile</code> will automatically be &quot;sourced&quot; and you won't need to run this command every time.
Now we can invoke commands or programs in the new directory <code>/home/tomahawk/tools/jdk1.8.0_92/bin</code>, such as <code>javac</code>, located at <code>/home/tomahawk/tools/jdk1.8.0_92/bin/javac</code> by just typing at the command prompt the name of the program. We can do this now from any directory, because the PATH has been updated to look for executable programs in our new directory.
The command:
<pre><code class="hljs">$ javac --version
</code></pre>
Now runs the correct <code>javac</code> program, printing out something like the following:
<pre><code class="hljs">Java Compiler, version 1.8.0_92
</code></pre>
<code>bash_profile</code> is appropriate when you want to set a PATH variable customized for a single user of the system. If you wanted to set PATH for all users of a system, there are better ways to do so, which we cover further on in this guide.
<h2 id="usingbashrctosetyourpath">Using bashrc to Set your PATH</h2>
Instead of setting the PATH in <code>~/.bash_profile</code>, we can also add the directories we want to the PATH in <code>~/.bashrc</code> instead. Setting the PATH in <code>bashrc</code> looks identical to how we set it in <code>bash_profile</code>.
For example, to include the directory <code>/home/tomahawk/.rbenv/bin</code> in my path, I edit or create the file <code>/home/tomahawk/.bashrc</code>, adding the following line:
<pre><code class="hljs">...
export PATH=&quot;$PATH:/home/tomahawk/.rbenv/bin&quot;
</code></pre>
Notice that, like last time, the first thing in our new PATH export is the inclusion of the existing $PATH variable. This ensures that we preserve the current value of PATH, and just add any additional directories on to the PATH, after the $PATH variable. If you do not do this, you will overwrite the PATH variable entirely, and miss critical directories the system needs to be on the PATH. As a result, your system can become unusable.
The difference between using <code>bashrc</code> and <code>bash_profile</code> is that <code>bash_profile</code> is used for login shells. These run when you login via the console, or log in using <code>ssh</code>. In contrast, once you are logged in, and you open a command shell or run the <code>bash</code> command, the <code>bashrc</code> file will run. Your PATH settings from <code>bashrc</code> will then be available.
The effect of setting the PATH is similar. In addition, we must activate any changes in the <code>bashrc</code> file into the current shell the first time we make this change, just as we did for the <code>bash_profile</code> file. This time, we run <code>source ~/.bashrc</code>. We can now access the new PATH at the command line. It has been set permanently and will stay the same between multiple logins into the system.
<h2 id="usingaprofilefiletosetyourpath">Using a Profile File to Set your PATH</h2>
We can also set the PATH permanently using a user's profile file. This is different from <code>~/.bash_profile</code> in that it is set not for shells only, but for all programs.
User profiles are loaded at login. The PATH variable can be set in the <code>~/.profile</code> file.
To set my PATH to include everything already in $PATH, as well as a new directory <code>/home/tomahawk/.exenv/bin</code>, I edit the file at <code>~/.profile</code> and set the PATH as follows
<pre><code class="hljs">export PATH=&quot;$PATH:/home/tomahawk/.exenv/bin&quot;
</code></pre>
As in all prior examples, we will need to source these changes to make them active for the current shell, but subsequent logins will persist the changes.
Once that's done, I can run the <code>exenv</code> command, which is one of the programs available in the folder I just added to the PATH, and I get back the output of my exenv version:
<pre><code class="hljs">$ exenv version
1.3.4 (set by /home/tomahawk/.exenv/version)
</code></pre>
You can read more about the exenv program <a rel="nofollow noopener" target="_blank" href="https://github.com/mururu/exenv">here</a>, but you can use this process to set PATH permanently to include any program or directory you want.
<h2 id="permanentlysettingyourpathforothershellslikezshandksh">Permanently Setting your PATH for Other Shells like ZSH and KSH</h2>
If you use alternative shells such as zsh, ksh and others, you can set the PATH permanently using those shells' configuration.
<a href="/zsh-vs-bash/">Like bash</a>, both <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Z_shell">zsh</a> and <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/KornShell">ksh</a> use a zshrc and khsrc file, respectively, to set the path for non-login shells. For login shells, they use the analogous shell profile files zprofile and kprofile.
You can therefore set the PATH permanently for these shells in a similar way to what we did for Bash. For zsh, you can find these files, or create them if they do not exist, at <code>~/.zshrc</code> and <code>~/.zprofile</code>.
Similarly, you can set PATH permanently for ksh in the configuration files located at <code>~/.kshrc</code> and <code>~/.kprofile</code>.
There are plenty of other shells you can use, such as the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/C_shell">C Shell</a> and the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Tcsh">tcsh</a> shell. Setting the PATH permanently for them will generally follow the pattern we have seen here.
<h2 id="permanentlysettingsystemwidepathforallusers">Permanently Setting System-Wide PATH for all Users</h2>
System-wide settings for all users can be set in <code>/etc/profile</code>. There is considerable flexibility and multiple options for setting the PATH permanently system-wide.
Your Linux system will execute all script files ending in .sh in <code>/etc/profile.d</code> whenever a bash shell is entered, as well as when the desktop session loads.
You can therefore add a new file, such as <code>env.sh</code> inside the directory <code>/etc/profile.d</code>. In this file, we can export the PATH variable, setting it permanently to our choice of path directories, for example:
<pre><code class="hljs">export PATH=&quot;$PATH:/home/tomahawk/.rbenv/bin:/home/tomahawk/bin:/home/tomahawk/.local/bin&quot;
</code></pre>
Files in /etc/profile.d are sourced by /etc/profile, thus activating our system-wide PATH whenever a user logs in.
We can also set PATH for all users in <code>/etc/environment</code>, which takes key-value pairs of the form:
<pre><code class="hljs">PATH=&quot;/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games&quot;
</code></pre>
<h2 id="troubleshootingpathproblems">Troubleshooting PATH Problems</h2>
As we saw, setting the PATH permanently in Linux has many options. You can set the PATH for only a certain user, for all users, or for only certain types of command shells. However, it's a good idea to not fiddle with system-wide PATH settings unless you really know what you are doing.
If you encounter problems, a good starting point is to find out the current value of $PATH, by running the command:
<pre><code class="hljs">$ echo $PATH
</code></pre>
For more troubleshooting tips, check out <a rel="nofollow" target="_blank" href="https://docs.oracle.com/cd/E23823_01/html/817-0403/tsfileaccess-95658.html">this resource</a>.

Tendai Mutunhire

How to Permanently Set $PATH in Linux

Using Machine Learning to Predict the Weather: Part 1

<h2 id="introduction">Introduction</h2>
Python offers several options to run external processes and interact with the operating system. However, the methods are different for Python 2 and 3. Python 2 has several methods in the <code>os</code> module, which are now deprecated and replaced by the <code>subprocess</code> module, which is the preferred option in Python 3.
Throughout this article we'll talk about the various <code>os</code> and <code>subprocess</code> methods, how to use them, how they're different from each other, on what version of Python they should be used, and even how to convert the older commands to the newer ones.
Hopefully by the end of this article you'll have a better understanding of how to call external commands from Python code and which method you should use to do it.
First up is the older <code>os.popen*</code> methods.
<h2 id="theospopenmethods">The os.popen* Methods</h2>
The <code>os</code> module offers four different methods that allows us to interact with the operating system (just like you would with the command line) and create a <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Pipeline_(Unix)">pipe</a> to other commands. These methods I'm referring to are: <code>popen</code>, <code>popen2</code>, <code>popen3</code>, and <code>popen4</code>, all of which are described in the following sections.
The goal of each of these methods is to be able to call other programs from your Python code. This could be calling another executable, like your own compiled C++ program, or a shell command like <code>ls</code> or <code>mkdir</code>.
<h3 id="ospopen">os.popen</h3>
The <code>os.popen</code> method opens a pipe from a command. This pipe allows the command to send its output to another command. The output is an open file that can be accessed by other programs.
The syntax is as follows:
<pre><code class="hljs">os.popen(command[, mode[, bufsize]])
</code></pre>
Here the <code>command</code> parameter is what you'll be executing, and its output will be available via an open file. The argument <code>mode</code> defines whether or not this output file is readable ('r') or writable ('w'). Appending a 'b' to the <code>mode</code> will open the file in binary mode. Thus, for example &quot;rb&quot; will produce a readable binary file object.
In order to retrieve the exit code of the command executed, you must use the <code>close()</code> method of the file object.
The <code>bufsize</code> parameter tells <code>popen</code> how much data to buffer, and can assume one of the following values:
<ul>
<li>0 = unbuffered (default value)</li>
<li>1 = line buffered</li>
<li>N = approximate buffer size, when N &gt; 0; and default value, when N &lt; 0</li>
</ul>
This method is available for Unix and Windows platforms, and has been deprecated since Python version 2.6. If you're currently using this method and want to switch to the Python 3 version, here is the equivalent <code>subprocess</code> version for Python 3:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Method</th>
 <th>Replaced by</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>pipe = os.popen('cmd', 'r', bufsize)</td>
 <td>pipe = Popen('cmd', shell=True, bufsize=bufsize, stdout=PIPE).stdout</td>
 </tr>
 <tr>
 <td>pipe = os.popen('cmd', 'w', bufsize)</td>
 <td>pipe = Popen('cmd', shell=True, bufsize=bufsize, stdin=PIPE).stdin</td>
 </tr>
 </tbody>
</table>
The code below shows an example of how to use the <code>os.popen</code> method:
<pre><code class="hljs">import os

p = os.popen(&#x27;ls -la&#x27;)
print(p.read())
</code></pre>
The code above will ask the operating system to list all files in the current directory. The output of our method, which is stored in <code>p</code>, is an open file, which is read and printed in the last line of the code. The of this code (in the context of my current directory) result is as follows:
<pre><code class="hljs">$ python popen_test.py 
total 32
drwxr-xr-x 7 scott staff 238 Nov 9 09:13 .
drwxr-xr-x 29 scott staff 986 Nov 9 09:08 ..
-rw-r--r-- 1 scott staff 52 Nov 9 09:13 popen2_test.py
-rw-r--r-- 1 scott staff 55 Nov 9 09:14 popen3_test.py
-rw-r--r-- 1 scott staff 53 Nov 9 09:14 popen4_test.py
-rw-r--r-- 1 scott staff 49 Nov 9 09:13 popen_test.py
-rw-r--r-- 1 scott staff 0 Nov 9 09:13 subprocess_popen_test.py
</code></pre>
<h3 id="ospopen2">os.popen2</h3>
This method is very similar to the previous one. The main difference is what the method outputs. In this case it returns two file objects, one for the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Standard_streams#Standard_input_.28stdin.29">stdin</a> and another file for the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Standard_streams#Standard_output_.28stdout.29">stdout</a>.
The syntax is as follows:
<pre><code class="hljs">popen2(cmd[, mode[, bufsize]])
</code></pre>
These arguments have the same meaning as in the previous method, <code>os.popen</code>.
The <code>popen2</code> method is available for both the Unix and Windows platforms. However, it is found only in Python 2. Again, if you want to use the <code>subprocess</code> version instead (shown in more detail below), use the following instead:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Method</th>
 <th>Replaced by</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>(child_stdin, child_stdout) = os.popen2('cmd', mode, bufsize)</td>
 <td>
 p = Popen('cmd', shell=True, bufsize=bufsize, stdin=PIPE, stdout=PIPE, close_fds=True) 
 (child_stdin, child_stdout) = (p.stdin, p.stdout)
 </td>
 </tr>
 </tbody>
</table>
The code below shows an example on how to use this method:
<pre><code class="hljs">import os

in, out = os.popen2(&#x27;ls -la&#x27;)
print(out.read())
</code></pre>
This code will produce the same results as shown in the first code output above. The difference here is that the output of the <code>popen2</code> method consists of two files. Thus, the 2nd line of code defines two variables: <code>in</code> and <code>out</code>. In the last line, we read the output file <code>out</code> and print it to the console.
<h3 id="ospopen3">os.popen3</h3>
This method is very similar to the previous ones. However, the difference is that the output of the command is a set of three files: stdin, stdout, and <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Standard_streams#Standard_error_.28stderr.29">stderr</a>.
The syntax is:
<pre><code class="hljs">os.popen3(cmd[, mode[, bufsize]])
</code></pre>
where the arguments <code>cmd</code>, <code>mode</code>, and <code>bufsize</code> have the same specifications as in the previous methods. The method is available for Unix and Windows platforms.
Note that this method has been deprecated and the Python documentation advises us to replace the <code>popen3</code> method as follows:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Method</th>
 <th>Replaced by</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>
 (child_stdin, 
 child_stdout, 
 child_stderr) = os.popen3('cmd', mode, bufsize)
 </td>
 <td>
 p = Popen('cmd', shell=True, bufsize=bufsize, 
 stdin=PIPE, stdout=PIPE, stderr=PIPE, close_fds=True) 
 (child_stdin, 
 child_stdout, 
 child_stderr) = (p.stdin, p.stdout, p.stderr)
 </td>
 </tr>
 </tbody>
</table>
As in the previous examples, the code below will produce the same result as seen in our first example.
<pre><code class="hljs">import os

in, out, err = os.popen3(&#x27;ls -la&#x27;)
print(out.read())
</code></pre>
However, in this case, we have to define three files: stdin, stdout, and stderr. The list of files from our <code>ls -la</code> command is saved in the <code>out</code> file.
<h3 id="ospopen4">os.popen4</h3>
As you probably guessed, the <code>os.popen4</code> method is similar to the previous methods. However, in this case, it returns only two files, one for the stdin, and another one for the stdout and the stderr.
This method is available for the Unix and Windows platforms and (surprise!) has also been deprecated since version 2.6. To replace it with the corresponding <code>subprocess</code> <code>Popen</code> call, do the following:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Method</th>
 <th>Replaced by</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>
 (child_stdin, child_stdout_and_stderr) = os.popen4('cmd', mode, bufsize)
 </td>
 <td>
 p = Popen('cmd', shell=True, bufsize=bufsize, 
 stdin=PIPE, stdout=PIPE, stderr=STDOUT, close_fds=True) 
 (child_stdin, child_stdout_and_stderr) = (p.stdin, p.stdout)
 </td>
 </tr>
 </tbody>
</table>
The following code will produce the same result as in the previous examples, which is shown in the first code output above.
<pre><code class="hljs">import os

in, out = os.popen4(&#x27;ls -la&#x27;)
print(we.read())
</code></pre>
As we can see from the code above, the method looks very similar to <code>popen2</code>. However, the <code>out</code> file in the program will show the combined results of both the stdout and the stderr streams.
<h3 id="summaryofdifferences">Summary of differences</h3>
The differences between the different <code>popen*</code> commands all have to do with their output, which is summarized in the table below:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Method</th>
 <th>Arguments</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>popen</td>
 <td>stdout</td>
 </tr>
 <tr>
 <td>popen2</td>
 <td>stdin, stdout</td>
 </tr>
 <tr>
 <td>popen3</td>
 <td>stdin, stdout, stderr</td>
 </tr>
 <tr>
 <td>popen4</td>
 <td>stdin, stdout and stderr</td>
 </tr>
 </tbody>
</table>
In addition the <code>popen2</code>, <code>popen3</code>, and <code>popen4</code> are only available in Python 2 but not in Python 3. Python 3 has available the <code>popen</code> method, but it is recommended to use the <code>subprocess</code> module instead, which we'll describe in more detail in the following section.
<h2 id="thesusbprocesspopenmethod">The susbprocess.Popen Method</h2>
The <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/subprocess.html">subprocess module</a> was created with the intention of replacing several methods available in the <code>os</code> module, which were not considered to be very efficient. Within this module, we find the new <code>Popen</code> class.
The Python documentation recommends the use of <code>Popen</code> in advanced cases, when other methods such like <code>subprocess.call</code> cannot fulfill our needs. This method allows for the execution of a program as a child process. Because this is executed by the operating system as a separate process, the results are platform dependent.
The available parameters are as follows:
<pre><code class="hljs">subprocess.Popen(args, bufsize=0, executable=None, stdin=None, stdout=None, stderr=None, preexec_fn=None, close_fds=False, shell=False, cwd=None, env=None, universal_newlines=False, startupinfo=None, creationflags=0)
</code></pre>
One main difference of <code>Popen</code> is that it is a class and not just a method. Thus, when we call <code>subprocess.Popen</code>, we're actually calling the constructor of the class <code>Popen</code>.
There are quite a few arguments in the constructor. The most important to understand is <code>args</code>, which contains the command for the process we want to run. It can be specified as a sequence of parameters (via an array) or as a single command string.
The second argument that is important to understand is <code>shell</code>, which is defaults to <code>False</code>. On Unix, when we need to run a command that belongs to the shell, like <code>ls -la</code>, we need to set <code>shell=True</code>.
For example, the following code will call the Unix command <code>ls -la</code> via a shell.
<pre><code class="hljs">import subprocess
subprocess.Popen(&#x27;ls -la&#x27;, shell=True)
</code></pre>
The results can be seen in the output below:
<pre><code class="hljs">$ python subprocess_popen_test.py 
total 40
drwxr-xr-x 7 scott staff 238 Nov 9 09:13 .
drwxr-xr-x 29 scott staff 986 Nov 9 09:08 ..
-rw-r--r-- 1 scott staff 52 Nov 9 09:13 popen2_test.py
-rw-r--r-- 1 scott staff 55 Nov 9 09:14 popen3_test.py
-rw-r--r-- 1 scott staff 53 Nov 9 09:14 popen4_test.py
-rw-r--r-- 1 scott staff 49 Nov 9 09:13 popen_test.py
-rw-r--r-- 1 scott staff 56 Nov 9 09:16 subprocess_popen_test.py
</code></pre>
Using the following example from a Windows machine, we can see the differences of using the <code>shell</code> parameter more easily. Here we're opening Microsoft Excel from the shell, or as an executable program. From the shell, it is just like if we were opening Excel from a command window.
The following code will open Excel from the shell (note that we have to specify <code>shell=True</code>):
<pre><code class="hljs">import subprocess
subprocess.Popen(&quot;start excel&quot;, shell=True)
</code></pre>
However, we can get the same results by calling the Excel executable. In this case we are not using the shell, so we leave it with its default value (<code>False</code>); but we have to specify the full path to the executable.
<pre><code class="hljs">import subprocess
subprocess.Popen(&quot;C:\Program Files (x86)\Microsoft Office\Office15\excel.exe&quot;)
</code></pre>
In addition, when we instantiate the <code>Popen</code> class, we have access to several useful methods:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Method</th>
 <th>Description</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td><code>Popen.poll()</code></td>
 <td>Checks if the child process has terminated.</td>
 </tr>
 <tr>
 <td><code>Popen.wait()</code></td>
 <td>Wait for the child process to terminate.</td>
 </tr>
 <tr>
 <td><code>Popen.communicate()</code></td>
 <td>Allows to interact with the process.</td>
 </tr>
 <tr>
 <td><code>Popen.send_signal()</code></td>
 <td>Sends a signal to the child process.</td>
 </tr>
 <tr>
 <td><code>Popen.terminate()</code></td>
 <td>Stops the child process.</td>
 </tr>
 <tr>
 <td><code>Popen.kill()</code></td>
 <td>Kills a child process.</td>
 </tr>
 </tbody>
</table>
The full list can be found at the <a rel="nofollow" target="_blank" href="https://docs.python.org/2/library/subprocess.html">subprocess documentation</a>. The most commonly used method here is <code>communicate</code>.
The <code>communicate</code> method allows us to read data from the standard input, and it also allows us to send data to the standard output. It returns a tuple defined as <code>(stdoutdata, stderrdata)</code>.
For example, the following code will combine the Windows <code>dir</code> and <code>sort</code> commands.
<pre><code class="hljs">import subprocess

p1 = subprocess.Popen(&#x27;dir&#x27;, shell=True, stdin=None, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p2 = subprocess.Popen(&#x27;sort /R&#x27;, shell=True, stdin=p1.stdout)

p1.stdout.close()
out, err = p2.communicate() 
</code></pre>
In order to combine both commands, we create two subprocesses, one for the <code>dir</code> command and another for the <code>sort</code> command. Since we want to sort in reverse order, we add <code>/R</code> option to the <code>sort</code> call.
We define the stdout of process 1 as PIPE, which allows us to use the output of process 1 as the input for process 2. Then we need to close the stdout of process 1, so it can be used as input by process 2. The communication between process is achieved via the <code>communicate</code> method.
Running this from a Windows command shell produces the following:
<pre><code class="hljs">&gt; python subprocess_pipe_test.py
11/09/2017 08:52 PM 234 subprocess_pipe_test.py
11/09/2017 07:13 PM 99 subprocess_pipe_test2.py
11/09/2017 07:08 PM 66 subprocess_pipe_test3.py
11/09/2017 07:01 PM 56 subprocess_pipe_test4.py
11/09/2017 06:48 PM &lt;DIR&gt; ..
11/09/2017 06:48 PM &lt;DIR&gt; .
 Volume Serial Number is 2E4E-56A3
 Volume in drive D is ECA
 Directory of D:\MyPopen
 4 File(s) 455 bytes
 2 Dir(s) 18,634,326,016 bytes free
</code></pre>
<h2 id="wrappingup">Wrapping up</h2>
The <code>os</code> methods presented a good option in the past, however, at present the <code>subprocess</code> module has several methods which are more powerful and efficient to use. Among the tools available is the <code>Popen</code> class, which can be used in more complex cases. This class also contains the <code>communicate</code> method, which helps us pipe together different commands for more complex functionality.
What do you use the <code>popen*</code> methods for, and which do you prefer? Let us know in the comments!

Scott Robinson

Python's os and subprocess Popen Commands

TensorFlow Neural Network Tutorial

Frank Hofmann

Dimitrije Stamenic

Python Linked Lists

In this tutorial we will show you the various ways of how to exit Node.js programs. You need to understand first that Node.js works on a single thread or main process. You can spawn additional child processes to handle extra work. Exiting the main process lets us exit from Node.
While there are many ways to exit from Node, some ways are better than others for certain situations, like if you're running a REPL or not. We'll explain this in more detail throughout the article.
<h2 id="lettingascriptexitimplicitly">Letting a Script Exit Implicitly</h2>
Exiting from a Node.js program started at the command line can be as simple as waiting for the script to finish executing. Implicitly, the Node.js process will exit when it reaches the end of the script.
You can see this by running the following scipt:
<pre><code class="hljs">// batman.js
console.log(&#x27;Batman begins&#x27;);

process.on(&#x27;exit&#x27;, function(code) {
 return console.log(`About to exit with code ${code}`);
});
</code></pre>
Run the program with the command <code>node batman.js</code>, and you will see it output the first statement about Batman. In addition, the &quot;exit&quot; callback fires, resulting in a print out of the message about exiting, and an exit code. You should see something similar to the following:
<pre><code class="hljs">$ node batman.js 
Batman begins
About to exit with code 0
</code></pre>
Note that pending events and loops will block program exit. Add this repeating function at the end of the above script.
<pre><code class="hljs">// batman.js

// ...

setInterval((function() {
 return console.log(&#x27;I\&#x27;m Batman!&#x27;);
}), 1000);
</code></pre>
Run it again. This time, the program does not exit, because the repeating <code>setInterval</code> function blocks Node.js from exiting. Your output will look similar to this:
<pre><code class="hljs">$ node batman.js 
Batman begins
I&#x27;m Batman!
I&#x27;m Batman!
I&#x27;m Batman!
...
</code></pre>
Where the &quot;I'm Batman!&quot; phrase is continuously printed until you forcefully exit, like with <code>Ctrl + C</code>, or close your terminal.
<h2 id="usingprocessexit">Using process.exit()</h2>
We can exit Node.js programs using the explicit <code>process.exit</code> function call. The <code>process.exit</code> function exits from the current Node.js process. It takes an exit code, which is an integer.
The <code>process</code> object is a <a href="/using-global-variables-in-node-js/">global variable</a> that lets us manage the current Node.js process. Since it is a global, we can access it from anywhere inside a Node.js program without using <code>require</code> to import it.
Let us update the last program which does not exit because of the <code>setInterval</code> function. This time, we will force it to exit using a <code>timeout</code> after 5 seconds have elapsed. Add the following function to the same 'batman.js' script from before.
<pre><code class="hljs">// batman.js

// ...

setTimeout((function() {
 return process.exit(22);
}), 5000);
</code></pre>
When you run <code>node batman.js</code>, you will notice that this time, the program runs for a limited time and then exits with an exit code of 22.
<pre><code class="hljs">node batman.js 
Batman begins
I&#x27;m Batman!
I&#x27;m Batman!
I&#x27;m Batman!
I&#x27;m Batman!
About to exit with code 22
</code></pre>
Using <code>process.exit</code> works for exiting from the REPL (which we'll see later) as well as for terminating running Node.js programs or scripts.
Node.js interprets non-zero codes as failure, and an exit code of 0 as success.
<h2 id="exitingnodejsusingprocesskill">Exiting Node.js using process.kill()</h2>
We can also exit Node.js using <code>process.kill</code> to kill the running Node.js process. The difference between this and <code>process.exit</code> is that <code>process.kill</code> takes the pid, or process id, of the process we want to kill, as well as an optional signal that we want to send to the process. This means we can send signals to kill processes other than the main Node.js process. This is useful in highly concurrent applications with many running processes.
To kill the main Node process, we just pass the pid of the main process.
To see this in operation, replace the <code>setTimeout</code> function in our previous code example with this version that uses <code>process.kill</code>.
<pre><code class="hljs">// batman.js

// ...

setTimeout((function() {
 return process.kill(process.pid);
}), 5000);
</code></pre>
The program exits on schedule as before, after some of the same statements are printed. This time the pid is also printed to the console (yours may be different than 15):
<pre><code class="hljs">$ node batman.js 
Batman begins
I&#x27;m Batman!
I&#x27;m Batman!
I&#x27;m Batman!
I&#x27;m Batman!
Terminated: 15
</code></pre>
This method also works in the REPL as well as in Node.js programs.
<h2 id="howtoexitnodejsusingprocessabort">How to Exit Node.js using process.abort</h2>
Likewise, we can use <code>process.abort</code> to exit Node.js. This method works in the REPL as well as scripts and applications.
The difference between <code>process.abort</code>, <code>process.kill</code>, and <code>process.exit</code> is that <code>process.abort</code> always exits Node.js immediately and generates a core file. In addition, no event callbacks will run.
Note: The core file just mentioned is not created if <code>ulimit -c</code> is set to <code>0</code>. See <a rel="nofollow" target="_blank" href="https://www.cyberciti.biz/tips/linux-core-dumps.html">this tutorial</a> for more details.
To see it in action, replace the <code>process.kill</code> call in our <code>batman.js</code> file with a call to <code>process.abort</code> and run <code>node batman.js</code>.
<pre><code class="hljs">// batman.js

// ...

setTimeout((function() {
 return process.abort();
}), 5000);
</code></pre>
When you run the program with <code>node batman.js</code>, the program prints out the lines as before, but exits after the timeout has run, printing out some information about the state of the program when it aborted.
<pre><code class="hljs">$ node batman.js 
Batman begins
I&#x27;m Batman!
I&#x27;m Batman!
I&#x27;m Batman!
I&#x27;m Batman!
Abort trap: 6
</code></pre>
<h2 id="therepl">The REPL</h2>
Since the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop">REPL</a> (Read–eval–print loop) is a different type of execution environment from your typical Node.js script, these exit strategies deserve their own special sections. This is because the REPL is an interactive environment and does not just implicitly exit like a script would.
As we've alredy mentioned in some of the previous sections, some of the above strategies will work in the REPL as well. However, it's not typical to use them in this case, and you should instead use the following methods.
<h3 id="exitingusingakeycombination">Exiting using a Key Combination</h3>
Unlike vim, exiting from the REPL is really simple. In a running REPL, you can exit it by using the key combination <code>Ctrl + C</code>, and keying it in twice. This sends the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/SIGINT_(POSIX)">SIGINT</a>, or interruption signal, to the REPL. This is commonly used for quitting programs in POSIX systems.
<h3 id="usingtheexitcommand">Using the .exit Command</h3>
We can also exit a Node.js REPL by using the command &quot;.exit&quot;. When you enter this in a running Node REPL, the current REPL will exit. This invocation works similar to the <code>Ctrl + C</code> method discussed above.
<h2 id="conclusion">Conclusion</h2>
In many cases, using <code>process.exit</code> will suffice for exiting Node.js. However, as we saw, there are numerous alternatives. The various methods give you flexibility to exit from any particular place in your code. You need to be careful with this extra power, however, not to insert dangerous code that can result in unplanned exits, crashing your programs.

How to Exit in Node.js