Scott Robinson

Scheduling Jobs with python-crontab

Adam McQuistan

A SQLite Tutorial with Python

<h2 id="publicandprivateipaddresses">Public and Private IP Addresses</h2>
IP addresses are an essential part of modern networked communications. In this guide, we will show you how to find your own IP address. These instructions will work for most of the various <a rel="nofollow" target="_blank" href="http://stackabuse.com/tag/linux/">Linux</a> distributions like Ubuntu, Debian and Linux Mint, among others. BSD systems like FreeBSD and NetBSD, as well as Mac OS, might also be able to read their IP addresses using these instructions.
<h2 id="whatisanipaddress">What is an IP Address?</h2>
<h3 id="understandingthetcpipprotocol">Understanding the TCP/IP Protocol</h3>
IP addresses arose as part of what's known as the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Internet_protocol_suite">TCP/IP protocol</a>. This is a networking standard that defines how data is packaged and transferred between computers that are joined together in a computer network.
The TCP/IP protocol was invented in 1978 by Bob Kahn, Vint Cerf, and others.
TCP stands for Transmission Control Protocol, which is the protocol responsible for the reliable transmission of data over networks. In particular, TCP checks integrity of transmissions, and provides for re-transmission if packets have not been delivered correctly.
The IP protocol, on the other hand, deals with the routing of packets (like those sent by TCP). IP stands for Internet Protocol, which defines rules that let packets be sent from an origin to a destination computer. Under the IP protocol, packets are routed by a path of computers that gets progressively closer to the packet's intended recipient.
Each computer or device on the network is identified by an IP address, which is a numeric identifier unique to that device on the network.
<h3 id="differencesbetweenpublicandprivateaddresses">Differences Between Public and Private Addresses</h3>
There is a <a rel="nofollow" target="_blank" href="https://www.iplocation.net/public-vs-private-ip-address">difference between IP addresses</a> on the public internet and private IP addresses inside a local computer network. Internet Service Providers (ISPs) assign public IP addresses, which is akin to your public home address. When computers connect to the private network, they each get a private IP address to be used within just that network. Private IP addresses are assigned by your router or another device within the network. In turn, the router can route incoming packets to the correct device on the network using <a rel="nofollow" target="_blank" href="https://www.howtogeek.com/195323/do-two-computers-on-the-same-wi-fi-network-have-the-same-ip-address/">network address translation</a> (NAT).
<h2 id="howtofindyourpublicipaddress">How to Find your Public IP Address</h2>
You can discover your public IP address using a variety of commands that connect to the internet to run queries for you.
<h3 id="gettingyourpublicipaddresswiththedigcommand">Getting your Public IP Address with the dig Command</h3>
The <code>dig</code> command is a DNS lookup utility for Linux. Using <code>dig</code>, you can look up your public IP address by connecting to OpenDNS servers. OpenDNS hosts DNS servers that help discover the IP addresses of networks on the internet. Run the following command inside your bash, sh, or any other terminal.
<pre><code class="hljs">$ dig +short myip.opendns.com @resolver1.opendns.com
</code></pre>
As output you should get back your public IP address from OpenDNS resolvers. Your IP address will be a series of groups of digits of the format 216.58.216.164 (which is one of the many IP addresses for Google), or a variation of this sort.
<h3 id="usingcurltoreturnyourpublicipaddress">Using Curl to Return your Public IP Address</h3>
The <code>curl</code> command is another networking utility that allows you to interact with servers on the internet. You can query servers to return your public IP using the following options:
<pre><code class="hljs">$ curl http://ifconfig.me/ip
</code></pre>
OR
<pre><code class="hljs">$ curl http://icanhazip.com
</code></pre>
The output of each command will be your public IP as seen by these servers.
There are many other servers that can return your public IP address besides these two given above.
<h3 id="usingwgetinsteadofcurl">Using <code>wget</code> Instead of curl</h3>
<code>wget</code> is a commonly available utility for Linux that you can use instead of curl to download data and interact with servers. You can use <code>wget</code> to retrieve your public IP using the command:
<pre><code class="hljs">$ wget -qO- icanhazip.com
</code></pre>
<h3 id="usingabrowser">Using a Browser</h3>
If you have a browser available, you can visit one of the many IP websites out there:
<ul>
<li><a rel="nofollow" target="_blank" href="http://ifconfig.me/ip">http://ifconfig.me/ip</a></li>
<li><a rel="nofollow" target="_blank" href="https://checkmyip.org/">Check My IP</a></li>
<li><a rel="nofollow" target="_blank" href="http://ipecho.net/plain">http://ipecho.net/plain</a></li>
</ul>
or you can also check the other websites we reached with curl to see your IP in the browser. There are cases, such as when you are logged on to a Linux server, where you do not have access to a graphical user interface. In such cases, use the shell commands.
<h2 id="getyourprivateipaddresswiththeifconfigcommand">Get your Private IP Address with the <code>ifconfig</code> Command</h2>
You have multiple ways to get your private IP address. One way is to use the <code>ifconfig</code> command. <a rel="nofollow" target="_blank" href="https://linux.die.net/man/8/ifconfig">ifconfig</a> is a command line program that configures network interfaces on Linux.
You can retrieve your IP address using the <code>ifconfig</code> command coupled with various flags that filter for your private IP address. Run the following command inside your shell:
<pre><code class="hljs">$ ifconfig | grep -Eo &#x27;inet (addr:)?([0-9]*\.){3}[0-9]*&#x27; | grep -Eo &#x27;([0-9]*\.){3}[0-9]*&#x27; | grep -v &#x27;127.0.0.1&#x27;
</code></pre>
The above command checks all the active network interfaces, then filters for the TCP/IP interface, and finally filters the output for the local IP address. The final output is your private IP address. This will look similar to 192.168.1.2 or some variant.
We can filter the <code>ifconfig</code> output using <a href="https://www.gnu.org/software/sed/manual/sed.html">sed</a> instead, which is a utility for parsing and transforming streams of text. Run the following command to get your private IP address:
<pre><code class="hljs">$ ifconfig | sed -En &#x27;s/127.0.0.1//;s/.*inet (addr:)?(([0-9]*\.){3}[0-9]*).*/\2/p&#x27;
</code></pre>
In this example, we are filtering <code>ifconfig</code> output to get your IP address under the TCP/IP protocol. The above commands may fail if you do not have <code>sed</code> or <code>ifconfig</code> installed. In case the command fails, try the <code>hostname</code> command and follow the method below.
<h2 id="howtofindyourprivateipaddressusinghostnamecommand">How to Find your Private IP Address using Hostname Command</h2>
The <a rel="nofollow" target="_blank" href="hostname">https://linux.die.net/man/1/hostname</a> command returns the DNS information of the machine. You can find your private IP address by executing the following command in your shell:
<pre><code class="hljs">$ hostname -I
</code></pre>
The above command enumerates all your configured addresses on all network interfaces, including your private IP address.
<h2 id="getyourprivateipaddresswiththeipcommand">Get your Private IP Address with the IP Command</h2>
We can also get the private IP address of a Linux machine using the ip command. The <code>ip</code> command shows and manipulates routing, devices, policy routing and tunnels.
We can use the following variations of the <code>ip</code> command with flags to return our private IP address.
<pre><code class="hljs">$ ip route get 1 | awk &#x27;{print $NF;exit}&#x27;
</code></pre>
<pre><code class="hljs">$ ip route get 8.8.8.8 | head -1 | cut -d&#x27; &#x27; -f8
</code></pre>
<pre><code class="hljs">$ ip route get 8.8.8.8 | head -1 | awk '{print $7}'
</code></pre>
These commands print out the routing table entries for sending a request to alternate servers. These involve our private IP address as the source of the requests. We are filtering for that source using the Linux commands head, awk and cut to extract our private IP address.
<h2 id="findingyourprivateipaddressfromnetworkconnectionsettings">Finding your Private IP Address from Network Connection Settings</h2>
Besides the command line and the browser, we can also use Linux administrative applets. You may need to adjust these instructions to suit your exact Linux Distro.
<ul>
<li>Navigate to your &quot;Menu&quot;</li>
<li>Find the &quot;Preferences&quot; tab</li>
<li>Navigate to &quot;Network&quot; or the equivalent for managing your network information</li>
<li>Select the network type, (i.e. Wi-Fi, Ethernet, or other)</li>
<li>You can view your private IP address under the printed information about your IP</li>
</ul>
This will not work in a strictly command-line environment but will work on desktop Linux systems. If you are on a command-line only login, then you will need to try the other command-based methods given above.
<h2 id="numerouswaystogetipaddressonlinux">Numerous ways to get IP Address on Linux</h2>
As we saw, there are multiple ways to get your IP address on a Linux system. There are in fact two types of IP addresses, a public IP address and a private IP address. The public IP address identifies your computer or network to the outside world. Your private IP address identifies your machine inside your private network. To get your IP addresses, you can use a mix of commands such as <code>ifconfig</code>, <code>ip</code> or <code>hostname</code>, or make use of graphical environment apps.

Tendai Mutunhire

How to get your IP Address on Linux

What is Natural Language Processing?

<h2 id="whatispythonzlib">What is Python <code>zlib</code></h2>
The Python <code>zlib</code> library provides a Python interface to the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Zlib">zlib</a> C library, which is a higher-level abstraction for the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/DEFLATE">DEFLATE</a> lossless compression algorithm. The data format used by the library is specified in the RFC 1950 to 1952, which is available at <a rel="nofollow" target="_blank" href="http://www.ietf.org/rfc/rfc1950.txt">http://www.ietf.org/rfc/rfc1950.txt</a>.
The <code>zlib</code> compression format is free to use, and is not covered by any patent, so you can safely use it in commercial products as well. It is a lossless compression format (which means you don't lose any data between compression and decompression), and has the advantage of being portable across different platforms. Another important benefit of this compression mechanism is that it doesn't expand the data.
The main use of the <code>zlib</code> library is in applications that require compression and decompression of arbitrary data, whether it be a string, structured in-memory content, or files.
The most important functionalities included in this library are compression and decompression. Compression and decompression can both be done as a one-off operation, or by splitting the data into chunks like you'd seen from a stream of data. Both modes of operation are explained in this article.
One of the best things, in my opinion, about the <code>zlib</code> library is that it is compatible with the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Gzip">gzip</a> file format/tool (which is also based on DEFLATE), which is one of the most widely used compression applications on Unix systems.
<h2 id="compression">Compression</h2>
<h3 id="compressingastringofdata">Compressing a String of Data</h3>
The <code>zlib</code> library provides us with the <code>compress</code> function, which can be used to compress a string of data. The syntax of this function is very simple, taking only two arguments:
<pre><code class="hljs">compress(data, level=-1)
</code></pre>
Here the argument <code>data</code> contains the bytes to be compressed, and <code>level</code> is an integer value that can take the values -1 or 0 to 9. This parameter determines the level of compression, where level 1 is the fastest and yields the lowest level of compression. Level 9 is the slowest, yet it yields the highest level of compression. The value -1 represents the default, which is level 6. The default value has a balance between speed and compression. Level 0 yields no compression.
An example of using the <code>compress</code> method on a simple string is shown below:
<pre><code class="hljs">import zlib
import binascii

data = &#x27;Hello world&#x27;

compressed_data = zlib.compress(data, 2)

print(&#x27;Original data: &#x27; + data)
print(&#x27;Compressed data: &#x27; + binascii.hexlify(compressed_data))
</code></pre>
And the result is as follows:
<pre><code class="hljs">$ python compress_str.py 
Original data: Hello world
Compressed data: 785ef348cdc9c95728cf2fca49010018ab043d
</code></pre>
Figure 1
If we change the level to 0 (no compression), then line 5 becomes:
<pre><code class="hljs">compressed_data = zlib.compress(data, 0)
</code></pre>
And the new result is:
<pre><code class="hljs">$ python compress_str.py 
Original data: Hello world
Compressed data: 7801010b00f4ff48656c6c6f20776f726c6418ab043d
</code></pre>
Figure 2
You may notice a few differences comparing the outputs when using <code>0</code> or <code>2</code> for the compression level. Using a level of <code>2</code> we get a string (formatted in hexadecimal) of length 38, whereas with a level of <code>0</code> we get a hex string with length 44. This difference in length is due to the lack of compression in using level <code>0</code>.
If you don't format the string as hexadecimal, as I've done in this example, and view the output data you'll probably notice that the input string is still readable even after being &quot;compressed&quot;, although it has a few extra formatting characters around it.
<h3 id="compressinglargedatastreams">Compressing Large Data Streams</h3>
Large data streams can be managed with the <code>compressobj()</code> function, which returns a compression object. The syntax is as follows:
<pre><code class="hljs">compressobj(level=-1, method=DEFLATED, wbits=15, memLevel=8, strategy=Z_DEFAULT_STRATEGY[, zdict])
</code></pre>
The main difference between the arguments of this function and the <code>compress()</code> function is (aside from the <code>data</code> parameter) the <code>wbits</code> argument, which controls the window size, and whether or not the header and trailer are included in the output.
The possible values for <code>wbits</code> are:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Value</th>
 <th>Window size logarithm</th>
 <th>Output</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>+9 to +15</td>
 <td>Base 2</td>
 <td>Includes zlib header and trailer</td>
 </tr>
 <tr>
 <td>-9 to -15</td>
 <td>Absolute value of wbits</td>
 <td>No header and trailer</td>
 </tr>
 <tr>
 <td>+25 to +31</td>
 <td>Low 4 bits of the value</td>
 <td>Includes gzip header and trailing checksum</td>
 </tr>
 </tbody>
</table>
Table 1
The <code>method</code> argument represents the compression algorithm used. Currently the only possible value is <code>DEFLATED</code>, which is the only method defined in the RFC 1950. The <code>strategy</code> argument relates to compression tuning. Unless you really know what you're doing I'd recommend not to use it and just use the default value.
The following code shows how to use the <code>compressobj()</code> function:
<pre><code class="hljs">import zlib
import binascii

data = &#x27;Hello world&#x27;

compress = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, zlib.DEFLATED, -15)
compressed_data = compress.compress(data)
compressed_data += compress.flush()

print(&#x27;Original: &#x27; + data)
print(&#x27;Compressed data: &#x27; + binascii.hexlify(compressed_data))
</code></pre>
After running this code, the result is:
<pre><code class="hljs">$ python compress_obj.py 
Original: Hello world
Compressed data: f348cdc9c95728cf2fca490100
</code></pre>
Figure 3
As we can see from the figure above, the phrase &quot;Hello world&quot; has been compressed. Typically this method is used for compressing data streams that won't fit into memory at once. Although this example does not have a very large stream of data, it serves the purpose of showing the mechanics of the <code>compressobj()</code> function.
You may also be able to see how it would be useful in a larger application in which you can configure the compression and then pass around the compression object to other methods/modules. This can then be used to compress chunks of data in series.
You may also be able to see how it would be useful in a scenario where you have a data stream to compress. Instead of having to accumulate all of the data in memory, you can just call <code>compress.compress(data)</code> and <code>compress.flush()</code> on your data chunk and then move on to the next chunk while leaving the previous one to be cleaned up by garbage collection.
<h3 id="compressingafile">Compressing a File</h3>
We can also use the <code>compress()</code> function to compress the data in a file. The syntax is the same as in the first example.
In the example below we will compress a PNG image file named <code>logo.png</code> (which, I should note, is already a compressed version of the original raw image).
The example code is as follows:
<pre><code class="hljs">import zlib

original_data = open(&#x27;logo.png&#x27;, &#x27;rb&#x27;).read()
compressed_data = zlib.compress(original_data, zlib.Z_BEST_COMPRESSION)

compress_ratio = (float(len(original_data)) - float(len(compressed_data))) / float(len(original_data))

print(&#x27;Compressed: %d%%&#x27; % (100.0 * compress_ratio))
</code></pre>
In the above code, the <code>zlib.compress(...)</code> line uses the constant <code>Z_BEST_COMPRESSION</code>, which, as the name suggests, gives us the best compression level this algorithm has to offer. The next line then calculates the level of compression based on the ratio of length of compressed data over length of original data.
The result is as follows:
<pre><code class="hljs">$ python compress_file.py 
Compressed: 13%
</code></pre>
Figure 4
As we can see, the file was compressed by 13%.
The only difference between this example and our first one is the source of the data. However, I think it is important to show so you can get an idea of what kind of data can be compressed, whether it be just an ASCII string or binary image data. Simply read in your data from the file like you normally would and call the <code>compress</code> method.
<h3 id="savingcompresseddatatoafile">Saving Compressed Data to a File</h3>
The compressed data can also be saved to a file for later use. The example below shows how to save some compressed text into a file:
<pre><code class="hljs">import zlib

my_data = &#x27;Hello world&#x27;

compressed_data = zlib.compress(my_data, 2)

f = open(&#x27;outfile.txt&#x27;, &#x27;w&#x27;)
f.write(compressed_data)
f.close()
</code></pre>
The above example compresses our simple &quot;Hello world&quot; string and saves the compressed data into a file named <code>outfile.txt</code>. The <code>outfile.txt</code> file, when opened with our text editor, looks as follows:
<img alt="Compressed String File" src="https://s3.amazonaws.com/stackabuse/media/the-python-zlib-library-1.png" class="img-responsive">
Figure 5
<h2 id="decompression">Decompression</h2>
<h3 id="decompressingastringofdata">Decompressing a String of Data</h3>
A compressed string of data can be easily de-compressed by using the <code>decompress()</code> function. The syntax is as follows:
<pre><code class="hljs">decompress(data, wbits=MAX_WBITS, bufsize=DEF_BUF_SIZE)
</code></pre>
This function decompresses the bytes in the <code>data</code> argument. The <code>wbits</code> argument can be used to manage the size of the history buffer. The default value matches the largest window size. It also asks for the inclusion of the header and trailer of the compressed file. The possible values are:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Value</th>
 <th>Window size logarithm</th>
 <th>Input</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>+8 to +15</td>
 <td>Base 2</td>
 <td>Includes zlib header and trailer</td>
 </tr>
 <tr>
 <td>-8 to -15</td>
 <td>Absolute value of wbits</td>
 <td>Raw stream with no header and trailer</td>
 </tr>
 <tr>
 <td>+24 to +31 = 16 + (8 to 15)</td>
 <td>Low 4 bits of the value</td>
 <td>Includes gzip header and trailer</td>
 </tr>
 <tr>
 <td>+40 to +47 = 32 + (8 to 15)</td>
 <td>Low 4 bits of the value</td>
 <td>zlib or gzip format</td>
 </tr>
 </tbody>
</table>
Table 2
The initial value of the buffer size is indicated in the <code>bufsize</code> argument. However, the important aspect about this parameter is that it doesn't need to be exact, because if extra buffer size is needed, it will automatically be increased.
The following example shows how to decompress the string of data compressed in our previous example:
<pre><code class="hljs">import zlib

data = &#x27;Hello world&#x27;

compressed_data = zlib.compress(data, 2)
decompressed_data = zlib.decompress(compressed_data)

print(&#x27;Decompressed data: &#x27; + decompressed_data)
</code></pre>
The result is as follows:
<pre><code class="hljs">$ python decompress_str.py 
Decompressed data: Hello world
</code></pre>
Figure 5
<h3 id="decompressinglargedatastreams">Decompressing Large Data Streams</h3>
Decompressing big data streams may require memory management due to the size or source of your data. It's possible that you may not be able to use all of the available memory for this task (or you don't have enough memory), so the <code>decompressobj()</code> method allows you to divide up a stream of data into several chunks which you can decompress separately.
The syntax of the <code>decompressobj()</code> function is as follows:
<pre><code class="hljs">decompressobj(wbits=15[, zdict])
</code></pre>
This function returns a decompression object, which you use to decompress the individual data. The <code>wbits</code> argument has the same characteristics as in the <code>decompress()</code> function previously explained.
The following code shows how to decompress a big stream of data that is stored in a file. Firstly, the program creates a file named <code>outfile.txt</code>, which contains the compressed data. Note that the data is compressed using a value of <code>wbits</code> equal to +15. This ensures the creation of a header and a trailer in the data.
The file is then decompressed using chunks of data. Again, in this example the file doesn't contain a massive amount of data, but nevertheless, it serves the purpose of explaining the buffer concept.
The code is as follows:
<pre><code class="hljs">import zlib

data = &#x27;Hello world&#x27;

compress = zlib.compressobj(zlib.Z_DEFAULT_COMPRESSION, zlib.DEFLATED, +15)
compressed_data = compress.compress(data)
compressed_data += compress.flush()

print(&#x27;Original: &#x27; + data)
print(&#x27;Compressed data: &#x27; + compressed_data)

f = open(&#x27;compressed.dat&#x27;, &#x27;w&#x27;)
f.write(compressed_data)
f.close()

CHUNKSIZE = 1024

data2 = zlib.decompressobj()
my_file = open(&#x27;compressed.dat&#x27;, &#x27;rb&#x27;) 
buf = my_file.read(CHUNKSIZE)

# Decompress stream chunks
while buf:
 decompressed_data = data2.decompress(buf)
 buf = my_file.read(CHUNKSIZE)

decompressed_data += data2.flush()

print(&#x27;Decompressed data: &#x27; + decompressed_data)

my_file.close()
</code></pre>
After running the above code, we obtain the following results:
<pre><code class="hljs">$ python decompress_data.py 
Original: Hello world
Compressed data: x??H???W(?/?I?=
Decompressed data: Hello world
</code></pre>
Figure 6
<h3 id="decompressingdatafromafile">Decompressing Data from a File</h3>
The compressed data contained in a file can be easily decompressed, as you've seen in previous examples. This example is very similar to the previous one in that we're decompressing data that originates from a file, except that in this case we're going back to using the one-off <code>decompress</code> method, which decompresses the data in a single method call. This is useful for when your data is small enough to easily fit in memory.
This can be seen from the following example:
<pre><code class="hljs">import zlib

compressed_data = open(&#x27;compressed.dat&#x27;, &#x27;rb&#x27;).read()
decompressed_data = zlib.decompress(compressed_data)
print(decompressed_data)
</code></pre>
The above program opens the file <code>compressed.dat</code> created in a previous example, which contains the compressed &quot;Hello world&quot; string.
In this example, once the compressed data is retrieved and stored in the variable <code>compressed_data</code>, the program decompresses the stream and shows the result on the screen. As the file contains a small amount of data, the example uses the <code>decompress()</code> function. However, as the previous example shows, we could also decompress the data using the <code>decompressobj()</code> function.
After running the program we get the following result:
<pre><code class="hljs">$ python decompress_file.py 
Hello world
</code></pre>
Figure 7
<h2 id="wrappingup">Wrapping Up</h2>
The Python library <code>zlib</code> provides us with a useful set of functions for file compression using the <code>zlib</code> format. The functions <code>compress()</code> and <code>decompress()</code> are normally used. However, when there are memory constraints, the functions <code>compressobj()</code> and <code>decompressobj()</code> are available to provide more flexibility by supporting compression/decompression of data streams. These functions help split the data into smaller and more manageable chunks, which can be compressed or decompressed using the <code>compress()</code> and <code>decompress()</code> functions respectively.
Keep in mind that the <code>zlib</code> library also has quite a few more features than what we were able to cover in this article. For example you can use <code>zlib</code> to compute the checksum of some data to verify its integrity when decompressed. For more information on additional features like this, check out the <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/zlib.html">official documentation</a>.

Python zlib Library Tutorial

Formatting Strings with Python

Hapi vs Express: Comparing Node.js Web Frameworks

Using Machine Learning to Predict the Weather: Part 3

The <code>yield</code> keyword in Python is used to create generators. A <a href="/python-generators/">generator</a> is a type of collection that produces items on-the-fly and can only be iterated once. By using generators you can improve your application's performance and consume less memory as compared to normal collections, so it provides a nice boost in performance.
In this article we'll explain how to use the <code>yield</code> keyword in Python and what it does exactly. But first, let's study the difference between a simple list collection and generator, and then we will see how <code>yield</code> can be used to create more complex generators.
<h2 id="differencesbetweenalistandgenerator">Differences Between a List and Generator</h2>
In the following script we will create both a list and a generator and will try to see where they differ. First we'll create a simple list and check its type:
<pre><code class="hljs"># Creating a list using list comprehension
squared_list = [x**2 for x in range(5)]

# Check the type
type(squared_list)
</code></pre>
When running this code you should see that the type displayed will be &quot;list&quot;.
Now let's iterate over all the items in the <code>squared_list</code>.
<pre><code class="hljs"># Iterate over items and print them
for number in squared_list:
 print(number)
</code></pre>
The above script will produce following results:
<pre><code class="hljs">$ python squared_list.py 
0
1
4
9
16
</code></pre>
Now let's create a generator and perform the same exact task:
<pre><code class="hljs"># Creating a generator
squared_gen = (x**2 for x in range(5))

# Check the type
type(squared_gen)
</code></pre>
To create a generator, you start exactly as you would with list comprehension, but instead you have to use parentheses instead of square brackets. The above script will display &quot;generator&quot; as the type for <code>squared_gen</code> variable. Now let's iterate over the generator using a for-loop.
<pre><code class="hljs">for number in squared_gen:
 print(number)
</code></pre>
The output will be:
<pre><code class="hljs">$ python squared_gen.py 
0
1
4
9
16
</code></pre>
The output is the same as that of the list. So what is the difference? One of the main differences lies in the way the list and generators store elements in the memory. Lists store all of the elements in memory at once, whereas generators &quot;create&quot; each item on-the-fly, displays it, and then moves to the next element, discarding the previous element from the memory.
One way to verify this is to check the length of both the list and generator that we just created. The <code>len(squared_list)</code> will return 5 while <code>len(squared_gen)</code> will throw an error that a generator has no length. Also, you can iterate over a list as many times as you want but you can iterate over a generator only once. To iterate again, you must create the generator again.
<h2 id="usingtheyieldkeyword">Using the Yield Keyword</h2>
Now we know the difference between simple collections and generators, let us see how <code>yield</code> can help us define a generator.
In the previous examples, we created a generator implicitly using the list comprehension style. However in more complex scenarios we can instead create functions that return a generator. The <code>yield</code> keyword, unlike the <code>return</code> statement, is used to turn a regular Python function in to a generator. This is used as an alternative to returning an entire list at once. This will be again explained with the help of some simple examples.
Again, let's first see what our function returns if we do not use the <code>yield</code> keyword. Execute the following script:
<pre><code class="hljs">def cube_numbers(nums):
 cube_list =[]
 for i in nums:
 cube_list.append(i**3)
 return cube_list

cubes = cube_numbers([1, 2, 3, 4, 5])

print(cubes)
</code></pre>
In this script a function <code>cube_numbers</code> is created that accepts a list of numbers, take their cubes and returns the entire list to the caller. When this function is called, a list of cubes is returned and stored in the <code>cubes</code> variable. You can see from the output that the returned data is in-fact a full list:
<pre><code class="hljs">$ python cubes_list.py 
[1, 8, 27, 64, 125]
</code></pre>
Now, instead of returning a list, let's modify the above script so that it returns a generator.
<pre><code class="hljs">def cube_numbers(nums):
 for i in nums:
 yield(i**3)

cubes = cube_numbers([1, 2, 3, 4, 5])

print(cubes)
</code></pre>
In the above script, the <code>cube_numbers</code> function returns a generator instead of list of cubed number. It's very simple to create a generator using the <code>yield</code> keyword. Here we do not need the temporary <code>cube_list</code> variable to store cubed number, so even our <code>cube_numbers</code> method is simpler. Also, no <code>return</code> statement is needed, but instead the <code>yield</code> keyword is used to return the cubed number inside of the for-loop.
Now, when <code>cube_number</code> function is called, a generator is returned, which we can verify by running the code:
<pre><code class="hljs">$ python cubes_gen.py 
&lt;generator object cube_numbers at 0x1087f1230&gt;
</code></pre>
Even though we called the <code>cube_numbers</code> function, it doesn't actually execute at this point in time, and there are not yet any items stored in memory.
To get the function to execute, and therefore the next item from generator, we use the built-in <code>next</code> method. When you call the <code>next</code> iterator on the generator for the first time, the function is executed until the <code>yield</code> keyword is encountered. Once <code>yield</code> is found the value passed to it is returned to the calling function and the generator function is paused in its current state.
Here is how you get a value from your generator:
<pre><code class="hljs">next(cubes)
</code></pre>
The above function will return &quot;1&quot;. Now when you call <code>next</code> again on the generator, the <code>cube_numbers</code> function will resume executing from where it stopped previously at <code>yield</code>. The function will continue to execute until it finds <code>yield</code> again. The <code>next</code> function will keep returning cubed value one by one until all the values in the list are iterated.
Once all the values are iterated the <code>next</code> function throws a <a rel="nofollow" target="_blank" href="https://docs.python.org/2/library/exceptions.html#exceptions.StopIteration">StopIteration</a> exception. It is important to mention that the <code>cubes</code> generator doesn't store any of these items in memory, rather the cubed values are computed at runtime, returned, and forgotten. The only extra memory used is the state data for the generator itself, which is usually much less than a large list. This makes generators ideal for memory-intensive tasks.
Instead of always having to use the <code>next</code> iterator, you can instead use a &quot;for&quot; loop to iterate over a generators values. When using a &quot;for&quot; loop, behind the scenes the <code>next</code> iterator is called until all the items in the generator are iterated over.
<h2 id="optimizedperformance">Optimized Performance</h2>
As mentioned earlier, generators are very handy when it comes to memory-intensive tasks since they do not need to store all of the collection items in memory, rather they generate items on the fly and discards it as soon as the iterator moves to the next item.
In the previous examples the performance difference of a simple list and generator was not visible since the list sizes were so small. In this section we'll check out some examples where we can distinguish between the performance of lists and generators.
In the code below we will write a function that returns a list that contains 1 million dummy <code>car</code> objects. We will calculate the memory occupied by the process before and after calling the function (which creates the list).
Take a look at the following code:
<pre><code class="hljs">import time
import random
import os
import psutil

car_names = [&#x27;Audi&#x27;, &#x27;Toyota&#x27;, &#x27;Renault&#x27;, &#x27;Nissan&#x27;, &#x27;Honda&#x27;, &#x27;Suzuki&#x27;]
colors = [&#x27;Black&#x27;, &#x27;Blue&#x27;, &#x27;Red&#x27;, &#x27;White&#x27;, &#x27;Yellow&#x27;]

def car_list(cars):
 all_cars = []
 for i in range(cars):
 car = {
 &#x27;id&#x27;: i,
 &#x27;name&#x27;: random.choice(car_names),
 &#x27;color&#x27;: random.choice(colors)
 }
 all_cars.append(car)
 return all_cars

# Get used memory
process = psutil.Process(os.getpid())
print(&#x27;Memory before list is created: &#x27; + str(process.memory_info().rss/1000000))

# Call the car_list function and time how long it takes
t1 = time.clock()
cars = car_list(1000000)
t2 = time.clock()

# Get used memory
process = psutil.Process(os.getpid())
print(&#x27;Memory after list is created: &#x27; + str(process.memory_info().rss/1000000))

print(&#x27;Took {} seconds&#x27;.format(t2-t1))
</code></pre>
Note: You may have to <code>pip install psutil</code> to get this code to work on your machine.
In the machine on which the code was run, following results were obtained (yours may look slightly different):
<pre><code class="hljs">$ python perf_list.py 
Memory before list is created: 8
Memory after list is created: 334
Took 1.584018 seconds
</code></pre>
Before the list was created the process memory was 8 MB, and after the creation of list with 1 million items, the occupied memory jumped to 334 MB. Also, the time it took to create the list was 1.58 seconds.
Now, let's repeat the above process but replace the list with generator. Execute the following script:
<pre><code class="hljs">import time
import random
import os
import psutil

car_names = [&#x27;Audi&#x27;, &#x27;Toyota&#x27;, &#x27;Renault&#x27;, &#x27;Nissan&#x27;, &#x27;Honda&#x27;, &#x27;Suzuki&#x27;]
colors = [&#x27;Black&#x27;, &#x27;Blue&#x27;, &#x27;Red&#x27;, &#x27;White&#x27;, &#x27;Yellow&#x27;]

def car_list_gen(cars):
 for i in range(cars):
 car = {
 &#x27;id&#x27;:i,
 &#x27;name&#x27;:random.choice(car_names),
 &#x27;color&#x27;:random.choice(colors)
 }
 yield car

# Get used memory
process = psutil.Process(os.getpid())
print(&#x27;Memory before list is created: &#x27; + str(process.memory_info().rss/1000000))

# Call the car_list_gen function and time how long it takes
t1 = time.clock()
for car in car_list_gen(1000000):
 pass
t2 = time.clock()

# Get used memory
process = psutil.Process(os.getpid())
print(&#x27;Memory after list is created: &#x27; + str(process.memory_info().rss/1000000))

print(&#x27;Took {} seconds&#x27;.format(t2-t1))
</code></pre>
Here we have to use the <code>for car in car_list_gen(1000000)</code> loop to ensure that all 1000000 cars are actually generated.
Following results were obtained by executing the above script:
<pre><code class="hljs">$ python perf_gen.py 
Memory before list is created: 8
Memory after list is created: 40
Took 1.365244 seconds
</code></pre>
From the output, you can see that by using generators the memory difference is much smaller than before (from 8 MB to 40 MB) since the generators do not store the items in memory. Furthermore, the time taken to call the generator function was a bit faster as well at 1.37 seconds, which is about 14% faster than the list creation.
<h2 id="conclusion">Conclusion</h2>
Hopefully from this article you have a better understanding of the <code>yield</code> keyword, including how it's used, what it's used for, and why you'd want to use it. <a href="/python-generators/">Python generators</a> are a great way to improve the performance of your programs and they're very simple to use, but understanding when to use them is the challenge for many novice programmers.

Understanding Python's "yield" Keyword