<h2 id="introduction">Introduction</h2>
In this article I cover the basics of creating, interacting with, inspecting, and extracting zip archive files using Java (OpenJDK 11 to be specific). The code sample used in this article is in the form of a Gradle project and hosted in <a rel="nofollow" target="_blank" href="https://github.com/amcquistan/zip-files-in-java">this GitHub repo</a> for you to run and experiment with. Please exercise caution when changing code that deletes files.
As mentioned already, the code examples here are written in using Java 11 and utilizes the <code>var</code> keyword which was introduced in Java 10 and functional programming paradigms in Java 8, so a minimum version of Java 10 is required to run them as is.
<h3 id="contents">Contents</h3>
<ul>
<li><a href="#keyjavaclassesforworkingwithziparchives">Key Java Classes for Working with Zip Archives</a></li>
<li><a href="#commonfilepathsforthecodeexamples">Common File Paths for the Code Examples</a></li>
<li><a href="#inspectingthecontentsofaziparchive">Inspecting the Contents of a Zip Archive</a></li>
<li><a href="#extractingaziparchive">Extracting a Zip Archive</a></li>
<li><a href="#writingfilesdirectlyintoanewziparchive">Writing Files Directly into a New Zip Archive</a></li>
<li><a href="#zippinganexistingFileintoanewziparchive">Zipping an Existing File into a New Zip Archive</a></li>
<li><a href="#zippingafolderintoanewziparchive">Zipping a Folder into a New Zip Archive</a></li>
</ul>
<h2 id="keyjavaclassesforworkingwithziparchives">Key Java Classes for Working with Zip Archives</h2>
I feel it's a good idea to start things off by identifying some of the prominent classes that are commonly used when dealing with zip archives in Java. These classes live in either the <code>java.util.zip</code> or <code>java.nio.file</code> packages.
<ul>
<li><a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/api/index.html?java/util/zip/ZipFile.html">java.util.zip.ZipFile</a> is used to read in and interact with items (<code>ZipEntry</code> instances) in a zip archive</li>
<li><a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/api/java/util/zip/ZipEntry.html">java.util.zip.ZipEntry</a> is an abstraction representing a item such as a file or directory in a zip archive (ie, <code>ZipFile</code> instance)</li>
<li><a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/api/index.html?java/util/zip/ZipOutputStream.html">java.util.zip.ZipOutputStream</a> is an implementation of the abstract <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/api/java/io/OutputStream.html">OutputStream</a> class and used to write items to a Zip file</li>
<li><a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html">java.nio.file.Files</a> is very handy utilities class for streaming and copying file data into ZipOutputStream instances or out of ZipFile instances</li>
<li><a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/api/java/nio/file/Paths.html">java.nio.file.Path</a> another handy utilities class for effectively working with file paths</li>
</ul>
<h2 id="commonfilepathsforthecodeexamples">Common File Paths for the Code Examples</h2>
For the example code I use two common directories to write and read data to/from which are both relative to the root of the Gradle project. Take a look at the linked Repo in the introduction, or better yet, run the samples. Just keep these two Path variables in mind as they are used often as the starting directory for inputs and outputs.
<pre><code class="hljs">public class App {

 static final Path zippedDir = Path.of(&quot;ZippedData&quot;);
 static final Path inputDataDir = Path.of(&quot;InputData&quot;);
 
 // ... other stuff 
}
</code></pre>
<h2 id="inspectingthecontentsofaziparchive">Inspecting the Contents of a Zip Archive</h2>
You can instantiate a <code>ZipFile</code> class and pass it the path to an existing zip archive, which essentially opens it like any other file, then inspect the contents by querying the <code>ZipEntry</code> enumeration contained inside it. Note that <code>ZipFile</code> implements the <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/api/java/lang/AutoCloseable.html">AutoCloseable</a> interface, making it a great candidate for the <a target="_blank" href="/the-try-with-resources-statement-in-java/">try-with-resources</a> Java programming construct shown below and throughout the examples here.
<pre><code class="hljs">static void showZipContents() {
 try (var zf = new ZipFile(&quot;ZipToInspect.zip&quot;)) {
 
 System.out.println(String.format(&quot;Inspecting contents of: %s\n&quot;, zf.getName()));
 
 Enumeration&lt;? extends ZipEntry&gt; zipEntries = zf.entries();
 zipEntries.asIterator().forEachRemaining(entry -&gt; {
 System.out.println(String.format(
 &quot;Item: %s \nType: %s \nSize: %d\n&quot;,
 entry.getName(),
 entry.isDirectory() ? &quot;directory&quot; : &quot;file&quot;,
 entry.getSize()
 ));
 });
 } catch (IOException e) {
 e.printStackTrace();
 }
}
</code></pre>
Running the Gradle project using the following:
<pre><code class="hljs">$ ./gradlew run
</code></pre>
This yields output for the <code>App.showZipContents</code> method of:
<pre><code class="hljs">&gt; Task :run
Inspecting contents of: ZipToInspect.zip

Item: ZipToInspect/ 
Type: directory 
Size: 0

Item: ZipToInspect/greetings.txt 
Type: file 
Size: 160

Item: ZipToInspect/InnerFolder/ 
Type: directory 
Size: 0

Item: ZipToInspect/InnerFolder/About.txt 
Type: file 
Size: 39
</code></pre>
Here you can see that this prints out all files and directories in the zip archive, even the files within directories.
<h2 id="extractingaziparchive">Extracting a Zip Archive</h2>
Extracting the contents of a zip archive onto disk requires nothing more than replicating the same directory structure as what is inside the <code>ZipFile</code>, which can be determined via <code>ZipEntry.isDirectory</code> and then copying the files represented in the <code>ZipEntry</code> instances onto disk.
<pre><code class="hljs">static void unzipAZip() {
 var outputPath = Path.of(&quot;UnzippedContents&quot;);

 try (var zf = new ZipFile(&quot;ZipToInspect.zip&quot;)) {
 
 // Delete if exists, then create a fresh empty directory to put the zip archive contents
 initialize(outputPath);

 Enumeration&lt;? extends ZipEntry&gt; zipEntries = zf.entries();
 zipEntries.asIterator().forEachRemaining(entry -&gt; {
 try {
 if (entry.isDirectory()) {
 var dirToCreate = outputPath.resolve(entry.getName());
 Files.createDirectories(dirToCreate);
 } else {
 var fileToCreate = outputPath.resolve(entry.getName());
 Files.copy(zf.getInputStream(entry), fileToCreate);
 }
 } catch(IOException ei) {
 ei.printStackTrace();
 }
 });
 } catch(IOException e) {
 e.printStackTrace();
 }
}
</code></pre>
<h2 id="writingfilesdirectlyintoanewziparchive">Writing Files Directly into a New Zip Archive</h2>
Since writing a zip archive is really nothing more than writing a stream of data to some destination (a Zip file in this case) then writing data, like String data, to a zip archive is only different in that you need to match the data being written to <code>ZipEntry</code> instances added to the <code>ZipOutputStream</code>.
Again, <code>ZipOutputStream</code> implements the <code>AutoCloseable</code> interface, so it is best to use with a try-with-resources statement. The only real catch is to remember to close your <code>ZipEntry</code> when you are done with each one to make it clear when it should no longer receive data.
<pre><code class="hljs">static void zipSomeStrings() {
 Map&lt;String, String&gt; stringsToZip = Map.ofEntries(
 entry(&quot;file1&quot;, &quot;This is the first file&quot;),
 entry(&quot;file2&quot;, &quot;This is the second file&quot;),
 entry(&quot;file3&quot;, &quot;This is the third file&quot;)
 );
 var zipPath = zippedDir.resolve(&quot;ZipOfStringData.zip&quot;);
 try (var zos = new ZipOutputStream(
 new BufferedOutputStream(Files.newOutputStream(zipPath)))) {
 for (var entry : stringsToZip.entrySet()) {
 zos.putNextEntry(new ZipEntry(entry.getKey()));
 zos.write(entry.getValue().getBytes());
 zos.closeEntry();
 }
 } catch (IOException e) {
 e.printStackTrace();
 }
}
</code></pre>
<h2 id="zippinganexistingfileintoanewziparchive">Zipping an Existing File into a New Zip Archive</h2>
If you've <a target="_blank" href="/how-to-copy-a-file-in-java/">copied a File in Java</a> before then you are essentially already a PRO at creating a zip archive from an existing file (or directory for that matter). Again, the only real difference is that you need to take a little extra caution to be sure you are matching files up to the appropriate <code>ZipEntry</code> instances.
In this example I create an input file &quot;FileToZip.txt&quot; and write some data to it &quot;Howdy There Java Friends!&quot; and then use the <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#copy(java.nio.file.Path,%20java.io.OutputStream)">Files.copy(Path, OutputStream)</a> to associate the <code>ZipEntry</code> with the FileToZip.txt file inside the ZippedFile.zip zip archive I'm creating with a <code>ZipOutoutStream</code> instance.
<pre><code class="hljs">static void zipAFile() {
 var inputPath = inputDataDir.resolve(&quot;FileToZip.txt&quot;);
 var zipPath = zippedDir.resolve(&quot;ZippedFile.zip&quot;);
 
 try (var zos = new ZipOutputStream(
 new BufferedOutputStream(Files.newOutputStream(zipPath)))) {
 
 Files.writeString(inputPath, &quot;Howdy There Java Friends!\n&quot;);

 zos.putNextEntry(new ZipEntry(inputPath.toString()));
 Files.copy(inputPath, zos);
 zos.closeEntry();
 } catch (IOException e) {
 e.printStackTrace();
 }
}
</code></pre>
<h2 id="zippingafolderintoanewziparchive">Zipping a Folder into a New Zip Archive</h2>
Zipping a non-empty directory becomes a little more involved, especially if you want to maintain empty directories within the parent directory. To maintain the presence of an empty directory within a zip archive you need to be sure to create an entry that is suffixed with the file system directory separator when creating it's <code>ZipEntry</code>, and then immediately close it.
In this example I create a directory named <code>foldertozip</code> containing the structure shown below, then zip it into a zip archive.
<pre><code class="hljs">tree .
.
└── foldertozip
 ├── emptydir
 ├── file1.txt
 └── file2.txt
</code></pre>
In the following code notice that I use the <code>Files.walk(Path)</code> method to traverse the directory tree of <code>foldertozip</code> and look for empty directories (<code>emptydir</code> in this example) and if / when found I concatenate the directory separator to the name within the <code>ZipEntry</code>. After this I close it as soon as I add it to the <code>ZipOutputStream</code> instance.
I also use a slightly different approach to injecting the non-directory files into the <code>ZipOutputStream</code> compared to the last example, but I am just using this different approach for the sake of variety in the examples.
<pre><code class="hljs">static void zipADirectoryWithFiles() {
 var foldertozip = inputDataDir.resolve(&quot;foldertozip&quot;); 
 var dirFile1 = foldertozip.resolve(&quot;file1.txt&quot;);
 var dirFile2 = foldertozip.resolve(&quot;file2.txt&quot;); 

 var zipPath = zippedDir.resolve(&quot;ZippedDirectory.zip&quot;);
 try (var zos = new ZipOutputStream(
 new BufferedOutputStream(Files.newOutputStream(zipPath)))) {
 
 Files.createDirectory(foldertozip);
 Files.createDirectory(foldertozip.resolve(&quot;emptydir&quot;));
 Files.writeString(dirFile1, &quot;Does this Java get you rev&#x27;d up or what?&quot;);
 Files.writeString(dirFile2, &quot;Java Java Java ... Buz Buz Buz!&quot;);

 Files.walk(foldertozip).forEach(path -&gt; {
 try {
 var reliativePath = inputDataDir.relativize(path);
 var file = path.toFile();
 if (file.isDirectory()) {
 var files = file.listFiles();
 if (files == null || files.length == 0) {
 zos.putNextEntry(new ZipEntry(
 reliativePath.toString() + File.separator));
 zos.closeEntry();
 }
 } else {
 zos.putNextEntry(new ZipEntry(reliativePath.toString()));
 zos.write(Files.readAllBytes(path));
 zos.closeEntry();
 }
 } catch(IOException e) {
 e.printStackTrace();
 }
 });
 } catch(IOException e) {
 e.printStackTrace();
 }
}
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this article I have discussed and demonstrated a modern approach to working with zip archives in Java using pure Java and no third party libraries. You may also notice I use a few more modern Java language features, such as functional programming paradigms and the <code>var</code> keyword for type inferred variables, so please make sure you are using at least Java 10 when running these examples.
As always, thanks for reading and don't be shy about commenting or critiquing below.

Adam McQuistan

Working with Zip Files in Java

Muhammad Junaid Khalid

Debugging Python Applications with the PDB Module

<h2 id="introduction">Introduction</h2>
Memory management is the process of efficiently allocating, de-allocating, and coordinating memory so that all the different processes run smoothly and can optimally access different system resources. Memory management also involves cleaning memory of objects that are no longer being accessed.
In Python, the memory manager is responsible for these kinds of tasks by periodically running to clean up, allocate, and manage the memory. Unlike C, Java, and other programming languages, Python manages objects by using reference counting. This means that the memory manager keeps track of the number of references to each object in the program. When an object's reference count drops to zero, which means the object is no longer being used, the garbage collector (part of the memory manager) automatically frees the memory from that particular object.
The user need not to worry about memory management as the process of allocation and deallocation of memory is fully automatic. The reclaimed memory can be used by other objects.
<h2 id="pythongarbagecollection">Python Garbage Collection</h2>
As explained earlier, Python deletes objects that are no longer referenced in the program to free up memory space. This process in which Python frees blocks of memory that are no longer used is called Garbage Collection. The Python <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/gc.html">Garbage Collector</a> (GC) runs during the program execution and is triggered if the reference count reduces to zero. The reference count increases if an object is assigned a new name or is placed in a container, like tuple or dictionary. Similarly, the reference count decreases when the reference to an object is reassigned, when the object's reference goes <a target="_blank" href="/local-and-global-variables-in-python/">out of scope</a>, or when an object is deleted.
The memory is a heap that contains objects and other data structures used in the program. The allocation and deallocation of this heap space is controlled by the Python Memory manager through the use of API functions.
<h2 id="pythonobjectsinmemory">Python Objects in Memory</h2>
Each variable in Python acts as an object. Objects can either be simple (containing numbers, strings, etc.) or containers (dictionaries, lists, or user defined classes). Furthermore, Python is a dynamically typed language which means that we do not need to declare the variables or their types before using them in a program.
For example:
<pre><code class="hljs">&gt;&gt;&gt; x = 5
&gt;&gt;&gt; print(x)
5
&gt;&gt;&gt; del x
&gt;&gt;&gt; print(x)
Traceback (most recent call last):
 File &quot;&lt;mem_manage&gt;&quot;, line 1, in &lt;module&gt;
 print(x)
NameError : name &#x27;x&#x27; is not defined
</code></pre>
If you look at the first 2 lines of the above program, object <code>x</code> is known. When we delete the object <code>x</code> and try to use it, we get an error stating that the variable <code>x</code> is not defined.
You can see that the garbage collection in Python is fully automated and the programmer does not need to worry about it, unlike languages like C.
<h2 id="modifyingthegarbagecollector">Modifying the Garbage Collector</h2>
The Python garbage collector has three generations in which objects are classified. A new object at the starting point of its life cycle is the first generation of the garbage collector. As the object survives garbage collection, it will be moved up to the next generations. Each of the 3 generations of the garbage collector has a threshold. Specifically, when the threshold of number of allocations minus the number of deallocations is exceeded, that generation will run garbage collection.
Earlier generations are also garbage collected more often than the higher generations. This is because newer objects are more likely to be discarded than old objects.
The <code>gc</code> module includes functions to change the threshold value, trigger a garbage collection process manually, disable the garbage collection process, etc. We can check the threshold values of different generations of the garbage collector using the <code>get_threshold()</code> method:
<pre><code class="hljs">import gc
print(gc.get_threshold())
</code></pre>
Sample Output:
<pre><code class="hljs">(700, 10, 10)
</code></pre>
As you see, here we have a threshold of 700 for the first generation, and 10 for each of the other two generations.
We can alter the threshold value for triggering the garbage collection process using the <code>set_threshold()</code> method of the <code>gc</code> module:
<pre><code class="hljs">gc.set_threshold(900, 15, 15)
</code></pre>
In the above example, we have increased the threshold value for all the 3 generations. Increasing the threshold value will decrease the frequency of running the garbage collector. Normally, we need not think too much about Python's garbage collection as a developer, but this may be useful when optimizing the Python runtime for your target system. One of the key benefits is that Python's garbage collection mechanism handles a lot of low-level details for the developer automatically.
<h2 id="whyperformmanualgarbagecollection">Why Perform Manual Garbage Collection?</h2>
We know that the Python interpreter keeps a track of references to objects used in a program. In earlier versions of Python (until version 1.6), the Python interpreter used only the reference counting mechanism to handle memory. When the reference count drops to zero, the Python interpreter automatically frees the memory. This classical reference counting mechanism is very effective, except that it fails to work when the program has reference cycles. A reference cycle happens if one or more objects are referenced to each other, and hence the reference count never reaches zero.
Let's consider an example.
<pre><code class="hljs">&gt;&gt;&gt; def create_cycle():
... list = [8, 9, 10]
... list.append(list)
... return list
... 
&gt;&gt;&gt; create_cycle()
[8, 9, 10, [...]]
</code></pre>
The above code creates a reference cycle, where the object <code>list</code> refers to itself. Hence, the memory for the object <code>list</code> will not be freed automatically when the function returns. The reference cycle problem can't be solved by reference counting. However, this reference cycle problem can be solved by changing the behavior of the garbage collector in your Python application.
To do so, we can use the <code>gc.collect()</code> function of the <code>gc</code> module.
<pre><code class="hljs">import gc
n = gc.collect()
print(&quot;Number of unreachable objects collected by GC:&quot;, n)
</code></pre>
The <code>gc.collect()</code> returns the number of objects it has collected and de-allocated.
There are two ways to perform manual garbage collection: time-based or event-based garbage collection.
Time-based garbage collection is pretty simple: the <code>gc.collect()</code> function is called after a fixed time interval.
Event-based garbage collection calls the <code>gc.collect()</code> function after an event occurs (i.e. when the application is exited or the application remains idle for a specific time period).
Let's understand the manual garbage collection work by creating a few reference cycles.
<pre><code class="hljs">import sys, gc

def create_cycle():
 list = [8, 9, 10]
 list.append(list)

def main():
 print(&quot;Creating garbage...&quot;)
 for i in range(8):
 create_cycle()

 print(&quot;Collecting...&quot;)
 n = gc.collect()
 print(&quot;Number of unreachable objects collected by GC:&quot;, n)
 print(&quot;Uncollectable garbage:&quot;, gc.garbage)

if __name__ == &quot;__main__&quot;:
 main()
 sys.exit()
</code></pre>
The output is as below:
<pre><code class="hljs">Creating garbage...
Collecting...
Number of unreachable objects collected by GC: 8
Uncollectable garbage: []
</code></pre>
The script above creates a list object that is referred to by a variable, creatively named <code>list</code>. The first element of the list object refers to itself. The reference count of the list object is always greater than zero even if it is deleted or out of scope in the program. Hence, the <code>list</code> object is not garbage collected due to the circular reference. The garbage collector mechanism in Python will automatically check for, and collect, circular references periodically.
In the above code, as the reference count is at least 1 and can never reach 0, we have forcefully garbage collected the objects by calling <code>gc.collect()</code>. However, remember not to force garbage collection frequently. The reason is that even after freeing the memory, the GC takes time to evaluate the object's eligibility to be garbage collected, taking up processor time and resources. Also, remember to manually manage the garbage collector only after your app has started completely.
<h2 id="conclusion">Conclusion</h2>
In this article, we discussed how memory management in Python is handled automatically by using reference counting and garbage collection strategies. Without garbage collection, implementing a successful memory management mechanism in Python is impossible. Also, programmers need not worry about deleting allocated memory, as it is taken care of by the Python memory manager. This leads to fewer memory leaks and better performance.

Guest Contributor

Basics of Memory Management in Python

<h2 id="introduction">Introduction</h2>
In this tutorial, we'll explore one of the powerful features of the ES6 specification of JavaScript - the Spread Operator. Although the syntax is simple, sometimes the implementation is confusing if you do not understand it properly. In this tutorial, we'll demystify those three dots <code>...</code> of JavaScript that does amazing things with iterables.
<h2 id="usagesofthespreadoperator">Usages of the Spread Operator</h2>
There are different usages of the spread operator and each usage target to solve a different problem statement.
<h3 id="expandingarrays">Expanding Arrays</h3>
We can use the spread operator on iterables like a String or an array and it'll put the contents of the iterable into individual elements.
For an example:
<pre><code class="hljs">let greet = [&#x27;Hello&#x27;, &#x27;World&#x27;];
console.log(greet); // Without spread operator
console.log(...greet); // Using spread operator
</code></pre>
If we run this code we'll see the following:
<pre><code class="hljs">['Hello', 'World']
Hello World
</code></pre>
You must have noticed that in the second case (with spread operator), the contents of the <code>greet</code> list were expanded and thrown out of the array.
Sometimes, we may feel the need to convert a <code>String</code> into a list of characters. We can use spread operator for this use-case:
<pre><code class="hljs">let greetings = &quot;hello&quot;;
let chars = [...greetings];
console.log(chars);
</code></pre>
If we run this code, we'll be greeted with:
<pre><code class="hljs">[ 'h', 'e', 'l', 'l', 'o' ]
</code></pre>
These examples might not convince you of the usefulness this operator offers. In that name, let's take some real-world problems that can be solved with the spread operators.
<h3 id="combiningarrays">Combining Arrays</h3>
Let us take advantage of the fact that we can now expand an array using the spread operator. Let's say we have subscriber lists from two different sources and we want to combine both these sources and make a single subscribers list:
<pre><code class="hljs">let blog1Subscribers = [&#x27;billy@example.com&#x27;, &#x27;sally@gmail.com&#x27;];
let blog2Subscribers = [&#x27;timmy@gmail.com&#x27;, &#x27;tammy@example.com&#x27;, &#x27;tommy@gmail.com&#x27;];
let subscribers = [...blog1Subscribers, ...blog2Subscribers];
console.log(subscribers);
</code></pre>
If we run the above code, we'll get a single list of iterables. This was made possible as both <code>...blog1Subscribers</code> and <code>...blog2Subscribers</code> were spread out and the <code>[]</code> acted as the &quot;receiver&quot;, which effectively combined the spread items into a single list of items.
Note: The spread operator needs the receiver to put the expanded value into. If you omit the receiver, it'll throw an error.
We can also use the spread operator inside the <code>Array.push()</code> method to push the contents of one array into another:
<pre><code class="hljs">let arr1 = [&#x27;John&#x27;, &#x27;Sofia&#x27;, &#x27;Bob&#x27;];
let arr2 = [&#x27;Julia&#x27;, &#x27;Sean&#x27;, &#x27;Anthony&#x27;];
arr1.push(...arr2);
console.log(arr1);
</code></pre>
If we run this code, we'll see the following output:
<pre><code class="hljs">[ &#x27;John&#x27;, &#x27;Sofia&#x27;, &#x27;Bob&#x27;, &#x27;Julia&#x27;, &#x27;Sean&#x27;, &#x27;Anthony&#x27; ]
</code></pre>
<h3 id="copyingarraysandobjects">Copying Arrays and Objects</h3>
In JavaScript every non-primitive entity is an <code>Object</code>, which means that arrays are also objects. You may know that objects are copied as a reference-type:
<pre><code class="hljs">let arr1 = [&#x27;John&#x27;, &#x27;Sofia&#x27;, &#x27;Bob&#x27;];
let arr2 = arr1;
console.log(arr2);
arr1.push(&#x27;Sally&#x27;); // Change arr1
console.log(arr2);
</code></pre>
<pre><code class="hljs">[ 'John', 'Sofia', 'Bob' ]
[ 'John', 'Sofia', 'Bob', 'Sally' ]
</code></pre>
As expected, the values of the items in the array weren't copied, only the reference to them. We can solve this problem easily with the spread operator:
<pre><code class="hljs">let arr1 = [&#x27;John&#x27;, &#x27;Sofia&#x27;, &#x27;Bob&#x27;];
let arr2 = [...arr1];
console.log(arr2);
arr1.push(&#x27;Sally&#x27;); // Change arr1
console.log(arr2);
</code></pre>
Running this code produces the following:
<pre><code class="hljs">[ 'John', 'Sofia', 'Bob' ]
[ 'John', 'Sofia', 'Bob' ]
</code></pre>
As we can see, the <code>arr2</code> wasn't passed a reference like before, but rather it was populated with the values of <code>arr1</code> as a whole new object. So even when <code>arr1</code> changes, <code>arr2</code> remains the same.
We can also use the spread operator to create a copy of an array and add new elements into it at the same time:
<pre><code class="hljs">let arr1 = [&#x27;John&#x27;, &#x27;Sofia&#x27;, &#x27;Bob&#x27;];
let arr2 = [...arr1, &#x27;Anthony&#x27;, &#x27;Sean&#x27;];
console.log(arr2);
</code></pre>
<pre><code class="hljs">['John', 'Sofia', 'Bob', 'Anthony', 'Sean']
</code></pre>
Note: The spread operator works with all iterables, including Objects.
Previously this would have taken an extra line of code to add the new items to the new array.
Similarly we can copy objects by using the spread operator:
<pre><code class="hljs">let o1 = { a: 1, b: 2 };
let o2 = { c: 3, d: 4, ...o1 };
console.log(o2);
</code></pre>
<pre><code class="hljs">{ c: 3, d: 4, a: 1, b: 2 }
</code></pre>
As we can see we successfully copied the object <code>o1</code> into <code>o2</code>.
This feature has many real-world use-cases. For example, let us say we stored user registration information into an object. We can make a shallow copy of that object and add some more information into the copied object:
<pre><code class="hljs">let user = { name: &#x27;John&#x27;, email: &#x27;john@example.com&#x27; };
let _user = { ...user, ...getSession(user) };
console.log(_user);
</code></pre>
<pre><code class="hljs">{ name: 'John', email: 'john@example.com', 'token': 'abc123', 'expiresAt': 1565630480671 }
</code></pre>
We may also need to merge billing and shipping information into one:
<pre><code class="hljs">const billing = { billingContact: &#x27;0987654321&#x27;, billingAddress: &#x27;street no 123, xyz city&#x27; };
const shipping = { shippingContact: &#x27;123456789&#x27;, shippingAddress: &#x27;street no 999, abc city&#x27; };
const custInfo = { ...billing, ...shipping };
console.log(custInfo);
</code></pre>
If we run this code, we should be greeted with:
<pre><code class="hljs">{
 billingContact: '0987654321',
 billingAddress: 'street no 123, xyz city',
 shippingContact: '123456789',
 shippingAddress: 'street no 999, abc city'
}
</code></pre>
One question could be raised here. What if both objects have some of the same properties.
In case of clashing properties, the property of the last object wins. Let us see this in an example:
<pre><code class="hljs">const o1 = { a: 1, b: 2 };
const o2 = { b: 3, c: 4, ...o1};
console.log(o2);
</code></pre>
If you run this code, you should see the following:
<pre><code class="hljs">{ b: 2, c: 4, a: 1 }
</code></pre>
As we can see the properties of the second object <code>o2</code> wins. However, if we put the spread operator first:
<pre><code class="hljs">const o1 = { a: 1, b: 2 };
const o2 = { ...o1, b: 3, c: 4};
console.log(o2);
</code></pre>
<pre><code class="hljs">{ a: 1, b: 3, c: 4 }
</code></pre>
We can see that the property from <code>o1</code> wins, which makes sense since <code>o2</code> is the last object.
One use case of this feature could be to make default assignments:
<pre><code class="hljs">const userProvided = {
 name: &#x27;Bil Smith&#x27;,
 email: &#x27;bil@example.com&#x27;,
};
const defaultValues = {
 name: &#x27;Unknown&#x27;,
 address: &#x27;Alien&#x27;,
 phone: null,
 email: null
};
const userInfo = { ...defaultValues, ...userProvided };
</code></pre>
<h3 id="alternativetocallingfunctionswithapply">Alternative to Calling Functions with apply()</h3>
Let us say a function takes an argument - a list of marks of top 5 students in a class. We also have a list coming from an external source. Surely, we can avoid passing individual items and instead pass the entire list by using the <code>apply()</code> method:
<pre><code class="hljs">myFun(m1, m2, m3, m4, m5) {
 // Do something
}

let marks = [10, 23, 83, -1, 92];
myFun.apply(undefined, arr);
</code></pre>
We can get rid of the confusing <code>undefined</code> argument and make the code cleaner by calling the function directly with the spread operator:
<pre><code class="hljs">myFun(m1, m2, m3, m4, m5) {
 // Do something
}

let marks = [10, 23, 83, -1, 92];
myFun(...marks);
</code></pre>
<h3 id="usingwithmathfunctions">Using with Math Functions</h3>
JavaScript has a <code>Math</code> object which contains several methods to operate with a set of data, i.e. a list of data.
Let us say we want to get the maximum value from first three numbers of a list:
<pre><code class="hljs">let mylist = [10, 23, 83, -1, 92, -33, 76, 29, 76, 100, 644, -633];
Math.max(mylist[0], mylist[1], mylist[2]);
</code></pre>
What if we want to get the maximum of all numbers in a list? What if the list has n number of items? Surely we don't want <code>mylist[0], mylist[1]... mylist[1000]</code>.
The spread operator provides a cleaner solution:
<pre><code class="hljs">let mylist = [10, 23, 83, -1, 92, -33, 76, 29, 76, 100, 644, -633];
Math.max(...mylist);
</code></pre>
Note: Given that the spread operator works with both Arrays and Objects, you may sometimes be tempted to mix and match them. Don't do that! For example, the following action will result in an error:
<pre><code class="hljs">let user = {name:&#x27;John&#x27;, age:28, email:&#x27;john@example.com&#x27;};
let items = [...user];
</code></pre>
<pre><code class="hljs">TypeError: user is not iterable
</code></pre>
<h2 id="spreadoperatorvsrestparameter">Spread Operator vs Rest Parameter</h2>
Both the Spread Operator and Rest Parameter share the same syntax i.e. the three magical dots <code>...</code>. But they behave exactly opposite to each other. As a beginner, sometimes this may be confusing. The bottom-line to understand the behavior is to understand the context in which it is being used.
As we learned, the spread operator expands the contents of an iterable. In contrast, the rest operator collects all the remaining elements into an array.
<pre><code class="hljs">function doSum(...items) {
 let sum = 0;
 for (let item of items){
 sum += item;
 }
 return sum;
}

doSum(1);
doSum(1,2);
doSum(1, 2, 3, 4);
</code></pre>
If we run the above code, we'll be greeted with the following:
<pre><code class="hljs">1
3
6
10
</code></pre>
As we can see, every time the remaining elements were collected by the Rest Parameter.
We can also provide distinct variables for some of the elements and make the rest parameter collect the rest of the items. The only condition is that the rest parameter should always be the last parameter of the function:
<pre><code class="hljs">function doSum(times, ...items) {
 let sum = 0;
 for (let item of items){
 sum += item*times;
 }
 return sum;
}

doSum(1, 1);
doSum(2, 1, 2);
doSum(3, 1, 2, 3);
</code></pre>
If we run the above code, we'll see the following:
<pre><code class="hljs">1
6
18
</code></pre>
<h2 id="conclusion">Conclusion</h2>
As we can see the spread operator <code>...</code> is a really powerful feature of the ES6 specification of JavaScript. We can solve plenty of real-world problems easily by using this operator. As we learned from the various examples discussed in this article, it lets us write less code and do more.
In this article, we covered the common usages of the Spread Operator. We also discussed the similar-looking, but different, Rest Parameter. Please be aware that there could be dozens of other use-cases depending on the problem.

Shadab Ansari

Spread Operator in JavaScript

Robley Gori

Using Django Signals to Simplify and Decouple Code

<h2 id="introduction">Introduction</h2>
Developing symbols which have some value is a trait unique to humans. Recognizing these symbols and understanding the letters on an image is absolutely normal for us. We never really grasp letters like computers do, we completely base our ability to read them on our sight.
On the other hand, computers need something more concrete and organized to work with. They need a digital representation, not a graphical one.
Sometimes, this simply isn't possible. Sometimes, we wish to automate a task of rewriting text from an image with our own hands.
For these tasks, <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Optical_character_recognition">Optical Character Recognition</a> (OCR) was devised as a way to allow computers to &quot;read&quot; graphical content as text, similar to how humans do. Of course these systems, while relatively accurate, can still be off quite a bit. Even if they are, fixing up the mistakes of the system is still a lot easier and faster than doing everything from scratch by hand.
Like all systems, similar-in-nature, optical character recognition software trains on prepared datasets that feed it enough data to learn the difference between characters. It's also very important how these networks learn, if we want to make them accurate, though this is a topic for another article.
Instead of reinventing the wheel and coming up with a very complex (but useful) solution, let's settle down a bit and use what's already offered.
<h2 id="tesseract">Tesseract</h2>
The technology giant, Google, has been developing an OCR engine, <a rel="nofollow" target="_blank" href="https://github.com/tesseract-ocr/tesseract">Tesseract</a>, which has a decades-long history since its original inception. It offers an API for a bunch of languages, though we'll focus on the Tesseract Java API.
Tesseract is very easy to implement, and subsequently isn't overly powerful. It's mainly used for reading computer generated text on black and white images, which is done with decent accuracy. Although it's not really meant for real-world text.
For real-world, advanced Optical Character Recognition, we'd be better off using something like <a rel="nofollow" target="_blank" href="https://cloud.google.com/vision/">Google Vision</a>, which we'll go over in another article.
<h3 id="mavendependency">Maven Dependency</h3>
To import the engine into our project, we simply have to add a dependency:
<pre><code class="hljs">&lt;dependency&gt;
 &lt;groupId&gt;net.sourceforge.tess4j&lt;/groupId&gt;
 &lt;artifactId&gt;tess4j&lt;/artifactId&gt;
 &lt;version&gt;3.2.1&lt;/version&gt;
&lt;/dependency&gt;
</code></pre>
<h3 id="opticalcharacterrecognition">Optical Character Recognition</h3>
Using Tesseract is absolutely effortless:
<pre><code class="hljs">Tesseract tesseract = new Tesseract();
tesseract.setDatapath(&quot;E://DataScience//tessdata&quot;);
System.out.println(tesseract.doOCR(new File(&quot;...&quot;)));
</code></pre>
We firstly instantiate the <code>Tesseract</code> object and set the data path to the LSTM (Long Short-Term Memory) models pretrained for your use.
The data can be downloaded from the <a rel="nofollow" target="_blank" href="https://github.com/tesseract-ocr/tessdata">official GitHub</a> account.
Afterwards, we call the <code>doOCR()</code> method, which accepts a file and returns a String - the extracted content.
Let's feed it a picture with big, clear black letters upon a white background:
<img src="https://stackabuse.s3.amazonaws.com/media/tesseract-simple-java-optical-character-recognition-1.JPG" alt="">
Feeding it such a picture will produce a perfect result:
<pre><code class="hljs">Optical Character Recognition in Java is made easy with the help of Tesseract&#x27;
</code></pre>
However, this image is extremely easy to scan. It's normalized, high in resolution and the font is consistent.
Let's see what happens if I try to write something down myself, on a piece of paper, and we let it pass through the app:
<img src="https://stackabuse.s3.amazonaws.com/media/tesseract-simple-java-optical-character-recognition-2.jpg" alt="">
We can instantly see the difference it makes:
<pre><code class="hljs">A411“, written texz: is different {mm compatar generated but
</code></pre>
Some words are perfectly fine and you could easily make out &quot;written text is different from computer generated&quot;, but the first and last words are off by a lot.
Now, to make this a bit easier to use, let's transfer it into a very simple Spring Boot application to serve up the result in a more graphically pleasing way.
<h2 id="implementation">Implementation</h2>
<h3 id="springbootapplication">Spring Boot Application</h3>
First off, let's start by generating our project through <a rel="nofollow" target="_blank" href="https://start.spring.io/">Spring Initializr</a>. Include the <code>spring-boot-starter-web</code> and <code>spring-boot-starter-thymeleaf</code> dependencies. We'll import Tesseract manually:
<img src="https://stackabuse.s3.amazonaws.com/media/tesseract-simple-java-optical-character-recognition-5.JPG" alt="">
<h3 id="controller">Controller</h3>
The app doesn't need more than a single controller, which serves up our two views and handles the picture upload and optical character recognition:
<pre><code class="hljs">@Controller
public class FileUploadController {
 
 @RequestMapping(&quot;/&quot;)
 public String index() {
 return &quot;upload&quot;;
 }

 @RequestMapping(value = &quot;/upload&quot;, method = RequestMethod.POST)
 public RedirectView singleFileUpload(@RequestParam(&quot;file&quot;) MultipartFile file,
 RedirectAttributes redirectAttributes, Model model) throws IOException, TesseractException {

 byte[] bytes = file.getBytes();
 Path path = Paths.get(&quot;E://simpleocr//src//main//resources//static//&quot; + file.getOriginalFilename());
 Files.write(path, bytes);

 File convFile = convert(file);
 Tesseract tesseract = new Tesseract();
 tesseract.setDatapath(&quot;E://DataScience//tessdata&quot;);
 String text = tesseract.doOCR(file2);
 redirectAttributes.addFlashAttribute(&quot;file&quot;, file);
 redirectAttributes.addFlashAttribute(&quot;text&quot;, text);
 return new RedirectView(&quot;result&quot;);
 }

 @RequestMapping(&quot;/result&quot;)
 public String result() {
 return &quot;result&quot;;
 }

 public static File convert(MultipartFile file) throws IOException {
 File convFile = new File(file.getOriginalFilename());
 convFile.createNewFile();
 FileOutputStream fos = new FileOutputStream(convFile);
 fos.write(file.getBytes());
 fos.close();
 return convFile;
 }
}
</code></pre>
Tesseract works with Java's <code>File</code>s, but doesn't support <code>MultipartFile</code>, which we get by accepting a file through our form. To mitigate this, we've added a simple <code>convert()</code> method, which converts the <code>MultipartFile</code> into a regular <code>File</code>.
Once we've extracted the text using Tesseract, we simply add it to the model, alongside the scanned image and append it to the redirected view - <code>result</code>.
<h3 id="views">Views</h3>
Now, let's define a view which we can use to simply upload a file via a form:
<pre><code class="hljs">&lt;html&gt;
&lt;body&gt;
&lt;h1&gt;Upload a file for OCR:&lt;/h1&gt;

&lt;form method=&quot;POST&quot; action=&quot;/upload&quot; enctype=&quot;multipart/form-data&quot;&gt;
 &lt;input type=&quot;file&quot; name=&quot;file&quot; /&gt;&lt;br/&gt;&lt;br/&gt;
 &lt;input type=&quot;submit&quot; value=&quot;Submit&quot; /&gt;
&lt;/form&gt;

&lt;/body&gt;
&lt;/html&gt;
</code></pre>
And the resulting page:
<pre><code class="hljs">&lt;html xmlns:th=&quot;http://www.thymeleaf.org&quot;&gt;
&lt;body&gt;

&lt;h1&gt;Extracted Content:&lt;/h1&gt;
&lt;h2&gt;&gt;&lt;span th:text=&quot;${text}&quot;&gt;&lt;/span&gt;&lt;/h2&gt;

&lt;p&gt;From the image:&lt;/p&gt;
&lt;img th:src=&quot;&#x27;/&#x27; + ${file.getOriginalFilename()}&quot;/&gt;
&lt;/body&gt;
&lt;/html&gt;
</code></pre>
Running this app will greet us with a plain interface:
<img src="https://stackabuse.s3.amazonaws.com/media/tesseract-simple-java-optical-character-recognition-3.JPG" alt="">
Adding an image and submitting it will result in the text being extracted and the image shown on the screen:
<img src="https://stackabuse.s3.amazonaws.com/media/tesseract-simple-java-optical-character-recognition-4.JPG" alt="">
Success!
<h2 id="conclusion">Conclusion</h2>
Using Google's Tesseract engine, we built an extremely simple app that accepts an image through a form, extracts the textual contents from it and returns to us the submitted image.
This isn't a particularly useful application in and of itself, due to the limited power of Tesseract and the fact that the app is too simple for any other use besides demonstration purposes, but it should serve as a fun tool you could implement and test with.
Optical Character Recognition can come in handy when you wish to digitize content, especially when it comes to documents. These are easy to scan and are fairly accurate when it comes to extracting content. Of course, it's always wise to proofread the resulting document for potential errors.

David Landup

Tesseract: Simple Java Optical Character Recognition

As already pointed out in a previous article titled <a target="_blank" href="/commenting-python-code/">Commenting Python Code</a> you have learned that documentation is an essential, and a continuous step in the process of software development. The article mentioned above briefly introduced the concept of docstrings which is a way to create documentation for your Python code from within the code. This in-code documentation works for modules, classes, methods, and functions, and it is the preferred way to document all Python code.
There is a lot more to it, and that's why we will have a closer look at this subject in this article. We'll cover conventions on how to write docstrings correctly, as well as various docstring formats that are used in practice, followed by accessing a docstring from your Python script. And lastly, we'll present you a number of tools in order to use and evaluate docstrings.
<h2 id="divingintodocstrings">Diving into Docstrings</h2>
The term docstring is an abbreviation for documentation string, and describes your source code - i.e. what your function, module, or class does. It is added as a regular comment right below the head of a function, module, class, or method.
A typical example looks as follows, and is taken from a Python class for working with a measurement device like a mobile sensor to measure temperature, humidity, and wind velocity.
Listing 1: Python code with a single-line docstring
<pre><code class="hljs">class Device:
 def __init__(self, temp=0.0):
 &quot;Initialize the Device object with the given temperature value.&quot;
 
 self.set_temperature(temp)
 return
</code></pre>
In order to write a docstring correctly follow a number of conventions. These conventions are explained in more detail in the <a rel="nofollow" target="_blank" href="https://www.python.org/dev/peps/pep-0257/">PEP 257</a>, which stands for Python Enhancement Proposal.
<h2 id="singlelinedocstrings">Single-Line Docstrings</h2>
Due to simplicity the docstring used in Listing 1 comes as a single-line comment. Keep in mind to begin the text of a docstring with a capital letter, and end it with a period. Based on the fact that code is typically read more often than it is written, it is recommended to describe what the documented structure does as a kind of command instead of how it is done. Mentioning which kind of value is returned to the caller helps to understand the result of the function or method.
You may have noted that the method's docstring from Listing 1 is framed in single double-quotes. Well, as long as both the beginning and the end quotes are similar, Python is quite tolerant, and you are also allowed to use three single-quotes as well as three double-quotes, instead:
<pre><code class="hljs"> def get_temperature(self):
 &#x27;&#x27;&#x27;Return the stored temperature value as a float value.&#x27;&#x27;&#x27;

 return self.temperature
 
 def set_temperature(self, temp):
 &quot;&quot;&quot;Set the temperature value.&quot;&quot;&quot;

 self.temperature = float(temp)
 return
</code></pre>
Please make sure that the closing quotes are in the same line as with the opening quotes. Also, do not add any empty lines before, or after the text of the docstring.
<h2 id="multilinedocstrings">Multi-line Docstrings</h2>
Furthermore, a docstring can also be written as a multi-line comment. When using multi-line comments, two things change - the encapsulation of the docstring must be written in triple single- or double-quotes, and the structure of the docstring itself has a deeper meaning that is assigned to the entire text.
The first line of the docstring is interpreted as an abstract, or a short description, and is recommended to be written in the same way as a single-line docstring. An empty line that follows is interpreted as a separator between the abstract and the full description below. Listing 2 extends Listing 1, and does not use a specific format for the description, as mentioned below.
Listing 2: Multi-line docstring
<pre><code class="hljs">def set_temperature(self, temp):
 &quot;&quot;&quot;Set the temperature value.

 The value of the temp parameter is stored as a value in
 the class variable temperature. The given value is converted
 into a float value if not yet done.
 &quot;&quot;&quot;

 self.temperature = float(temp)
 return
</code></pre>
Following the docstring structure for multi-line strings is strongly recommended as automated indexing tools evaluate these texts, and therefore rely on the compliance with the block order.
<h2 id="docstringformats">Docstring Formats</h2>
You might expect that there is just one binding docstring format. Unfortunately, there is more than one, and all of these format variants work with multi-line docstrings.
<ul>
<li><a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/ReStructuredText">reStructured text</a> (reST) / <a rel="nofollow" target="_blank" href="http://www.sphinx-doc.org/en/master/">Sphinx</a>: This is the Official Python documentation standard. It uses the syntax of the lightweight markup language reStructured text (reST) which is similar in usage to <a target="_blank" href="/markdown-by-example/">Markdown</a>.</li>
<li><a rel="nofollow" target="_blank" href="https://sphinxcontrib-napoleon.readthedocs.io/en/latest/example_google.html">Google Docstrings</a>: Google's style of docstring</li>
<li>NumPy/<a rel="nofollow" target="_blank" href="https://www.scipy.org/">SciPy</a> Docstrings: A combination of reStructured text (reST) and Google Docstrings. Usable by Sphinx as well, and quite verbose.</li>
</ul>
Listing 3 shows how to write the docstring using reST. The keywords that you can use are the following:
<ul>
<li><code>param</code> and <code>type</code>: Parameter and its variable type</li>
<li><code>return</code> and <code>rtype</code>: Specify both the return value and type of the function or method</li>
<li><code>.. seealso::</code>: Further reading</li>
<li><code>.. notes::</code>: Add a note</li>
<li><code>.. warning::</code>: Add a warning</li>
</ul>
The order of the entries is not fixed, but stick to the same order throughout your entire project. The entries for <code>seealso</code>, <code>notes</code>, and <code>warning</code> are optional.
Listing 3: Multi-line docstring with reST data
<pre><code class="hljs">def set_temperature(self, temp):
 &quot;&quot;&quot;Set the temperature value.

 The value of the temp parameter is stored as a value in
 the class variable temperature. The given value is converted
 into a float value if not yet done.

 :param temp: the temperature value
 :type temp: float
 :return: no value
 :rtype: none
 &quot;&quot;&quot;

 self.temperature = float(temp)
 return
</code></pre>
In order to understand Google's docstrings you may have a look at Listing 4. The format is less dense, and uses more horizontal space.
Listing 4: Multi-line docstring (Google Format)
<pre><code class="hljs">def set_temperature(self, temp):
 &quot;&quot;&quot;Set the temperature value.

 The value of the temp parameter is stored as a value in
 the class variable temperature. The given value is converted
 into a float value if not yet done.

 Args:
 temp (float): the temperature value

 Returns:
 no value
 &quot;&quot;&quot;

 self.temperature = float(temp)
 return
</code></pre>
Finally, Listing 5 shows the same method in NumPy docstring format. It uses more vertical space, and looks easier to read than the original format.
Listing 5: Multi-line docstring (NumPy Format)
<pre><code class="hljs">def set_temperature(self, temp):
 &quot;&quot;&quot;Set the temperature value.

 The value of the temp parameter is stored as a value in
 the class variable temperature. The given value is converted
 into a float value if not yet done.

 Parameters
 ----------
 temp : float
 the temperature value

 Returns
 -------
 no value
 &quot;&quot;&quot;

 self.temperature = float(temp)
 return
</code></pre>
<h2 id="accessingdocstrings">Accessing Docstrings</h2>
In the Python interactive help system, the docstring is available via the <code>__doc__</code> attribute. Listing 6 shows how to use code to access the documentation string, which in our example is based on Listing 1.
Listing 6: Accessing the docstring value
<pre><code class="hljs">&gt;&gt;&gt; def set_temperature (self, temp):
... &quot;&quot;&quot;Set the temperature value.&quot;&quot;&quot;
... temperature = float(temp)
... return
... 
&gt;&gt;&gt; print(set_temperature.__doc__)
Set the temperature value.
</code></pre>
<h2 id="toolstousethedocstrings">Tools to Use the Docstrings</h2>
There is a number of tools that auto-generate documentation from docstrings, such as Sphinx, <a rel="nofollow" target="_blank" href="http://epydoc.sourceforge.net/">Epydoc</a>, <a rel="nofollow" target="_blank" href="http://www.doxygen.nl/">Doxygen</a>, <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/pydoc.html">PyDoc</a>, <a rel="nofollow" target="_blank" href="https://pdoc3.github.io/pdoc/">pdoc</a>, and the <a rel="nofollow" target="_blank" href="http://www.sphinx-doc.org/en/master/usage/extensions/autodoc.html">autodoc</a> extension for Sphinx. Most of them generate HTML documents for local use.
Pydoc is part of the Python distribution, and derives information about a module for the console, a web browser, or as an HTML document. Inside the Python shell use the <code>help()</code> function in order to learn more about a module, function, class, or method. Figure 1 shows the docstring from Listing 1 via the <a target="_blank" href="https://stackabuse.com/the-python-help-system/">Python help system</a>.
Figure 1: The extracted docstring
<img class="img-responsive" alt="pydoc python shell" src="https://stackabuse.s3.amazonaws.com/media/python-docstrings-1.png">
In order to see the built-in documentation for all of the Python modules that are installed locally, you may run pydoc as a local webserver. Using the parameter <code>-p</code> followed by the port number starts a small web server that is accessible using the given port. Listing 7 starts the pydoc server at port 1234, and Figure 2 shows the information extracted and made available by pydoc.
Listing 7: Running pydoc as a webserver
<pre><code class="hljs">$ pydoc3 -p 1234
Server ready at http://localhost:1234/
Server commands: [b]rowser, [q]uit
server&gt;
...
</code></pre>
Figure 2: The extracted docstring on a local webserver
<img class="img-responsive" alt="pydoc" src="https://stackabuse.s3.amazonaws.com/media/python-docstrings-2.png">
<h2 id="conclusion">Conclusion</h2>
Following the guidelines for documentation helps you and others to understand source code today and at a later time. Docstrings are used for more than that, for example for the generation of manuals. This idea in mind allows projects at a greater scale.

Frank Hofmann

Python Docstrings

Rounding Numbers in Python

<h2 id="overview">Overview</h2>
In this article, we'll introduce you to <a target="_blank" rel="nofollow" href="https://spring.io/projects/spring-cloud-sleuth">Spring Cloud Sleuth</a>, which is a distributed tracing framework for a microservice architecture in the Spring ecosystem.
In a typical microservice architecture we have many small applications deployed separately and they often need to communicate with each other. One of the challenges developers face is to trace a full request for logs to debug or to check for latency in downstream services.
To further add to the complexity, some services can have multiple instances running. It's difficult to track particular request logs in multiple services, especially if a particular service has many instances.
Spring Cloud Sleuth automatically adds some traces/metadata to your logs and inter service communication (via request headers) so it's easy to track a request via log aggregators like <a rel="nofollow" target="_blank" href="https://zipkin.io/">Zipkins</a>, ELK, etc.
This article assumes that you already have knowledge of Spring Cloud basic components. We have published several articles covering Spring Cloud if you want to read more:
<ul>
<li><a target="_blank" href="/spring-cloud-service-discovery-with-eureka/">Spring Cloud: Service Discovery with Eureka</a></li>
<li><a target="_blank" href="/spring-cloud-routing-with-zuul-and-gateway/">Spring Cloud: Routing with Zuul and Gateway</a></li>
<li><a target="_blank" href="/spring-cloud-hystrix/">Spring Cloud: Hystrix</a></li>
<li><a target="_blank" href="/spring-cloud-turbine/">Spring Cloud: Turbine</a></li>
</ul>
<h2 id="setup">Setup</h2>
In order to demonstrate the concept of tracing, we'll use a few services:
<ul>
<li>Eureka Server: Acts as a service registry and runs on port <code>8761</code>.</li>
<li>Address Service: A simple REST service that has a single endpoint of <code>/address/{customerId}</code> and runs on port <code>8070</code>.</li>
<li>Customer Service: A simple REST service that has a single endpoint of <code>/customer/{customerId}</code> and runs on port <code>8060</code>.</li>
<li>Portal Service: A simple REST service that has a single endpoint of <code>/fullDetails/{customerId}</code> and runs on port <code>8050</code>. This service internally calls <code>address-service</code> and <code>customer-service</code> to get data and combines them before the response.</li>
<li>Gateway: Single point of entry to our microservice architecture, built using Spring Cloud Gateway and running on port <code>8080</code>.</li>
</ul>
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-1.png" alt="spring-cloud-sleuth-request-flow">
And here is how the Eureka server looks like when all the service are running:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-2.png" alt="spring-cloud-sleuth-eureka">
Let's see what's written in each controller class, starting from <code>AddressController</code> of the <code>address-service</code>:
<pre><code class="hljs">@RestController 
@RequestMapping(value = &quot;/address&quot;) 
public class AddressController { 
 
 private static Logger log = LoggerFactory.getLogger(AddressController.class); 
 
 @GetMapping(value = &quot;/{customerId}&quot;) 
 public String address(@PathVariable(name = &quot;customerId&quot;, required = true) long customerId) { 
 log.info(&quot;GET /address/&quot;+customerId); 
 return &quot;Address of id=&quot;+customerId; 
 } 
}
</code></pre>
<code>CustomerController</code> of <code>customer-service</code>:
<pre><code class="hljs">@RestController 
@RequestMapping(value = &quot;/customer&quot;) 
public class CustomerController { 
 
 private static Logger log = LoggerFactory.getLogger(CustomerController.class); 
 
 @GetMapping(value = &quot;/{customerId}&quot;) 
 public String address(@PathVariable(name = &quot;customerId&quot;, required = true) long customerId){ 
 log.info(&quot;GET /customer/&quot;+customerId); 
 return &quot;Customer details of id=&quot;+customerId; 
 } 
}
</code></pre>
<code>PortalController</code> of <code>portal-service</code>:
<pre><code class="hljs">@RestController
public class PortalController {

 private static Logger log = LoggerFactory.getLogger(PortalController.class);

 @Autowired
 RestTemplate restTemplate;

 @GetMapping(value = &quot;/fullDetails/{customerId}&quot;)
 public String address(@PathVariable(name = &quot;customerId&quot;, required = true) long customerId) {
 log.info(&quot;GET /fullDetails/&quot; + customerId);

 String customerResponse = restTemplate.getForObject(&quot;http://customer-service/customer/&quot; + customerId, String.class);
 String addressResponse = restTemplate.getForObject(&quot;http://address-service/address/&quot; + customerId, String.class);

 return customerResponse + &quot;&lt;br&gt;&quot; + addressResponse;
 }
}
</code></pre>
To check let's run the <code>portal-service</code> endpoint via <code>gateway</code> by navigating your browser to <a rel="nofollow" target="_blank" href="http://localhost:8080/portal-service/fullDetails/12">http://localhost:8080/portal-service/fullDetails/12</a>. You should see something like this:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-3.png" alt="spring-cloud-sleuth-test-url">
Now, imagine tracing these logs on different servers. Also, even if you have these log files pushed to a common location and have a log aggregator, it would be difficult to find a full trace of a request between multiple services at some point of time.
<h2 id="addingspringcloudsleuth">Adding Spring Cloud Sleuth</h2>
Spring Cloud Sleuth adds unique IDs to your logs, which stay the same between many microservices and can be used by common log aggregators to see how a request flows.
To add this functionality, we need to add a dependency in the <code>pom.xml</code> file of each downstream service:
<pre><code class="hljs">&lt;dependency&gt; 
 &lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt; 
 &lt;artifactId&gt;spring-cloud-starter-sleuth&lt;/artifactId&gt; 
&lt;/dependency&gt;
</code></pre>
Restart all the application and hit the <a rel="nofollow" target="_blank" href="http://localhost:8080/portal-service/fullDetails/12">http://localhost:8080/portal-service/fullDetails/12</a> endpoint again and check the logs for each service.
<code>Portal service</code> logs:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-4.png" alt="spring-cloud-sleuth-logs1">
<code>Address service</code> logs:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-5.png" alt="spring-cloud-sleuth-logs2">
<code>Customer service</code> logs:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-6.png" alt="spring-cloud-sleuth-logs3">
Spring Cloud Sleuth adds two types of IDs to your logging:
<ul>
<li>Trace Id: A unique ID that remains the same throughout the request containing multiple microservices.</li>
<li>Span Id: A unique ID per microservice.</li>
</ul>
Basically, a Trace ID will contain multiple Span ID which can easily be used by log aggregation tools.
Sleuth not only adds these IDs to our logs but also propagates these to the next service calls (HTTP or MQ based). Also, it can send random sample logs to external applications like Zipkins out of the box.
<h2 id="logsaggregationwithzipkins">Logs Aggregation with Zipkins</h2>
Zipkins is a distributed tracing system usually used to troubleshoot latency problems in service architectures.
To run a Zipkin server, you can follow a quick and simple guide <a rel="nofollow" target="_blank" href="https://zipkin.io/pages/quickstart">here</a>.
I used the Java way to run it, by executing the commands:
<pre><code class="hljs">$ curl -sSL https://zipkin.io/quickstart.sh | bash -s
$ java -jar zipkin.jar
</code></pre>
Although you can also run it via Docker or straight from the source code.
By default, the Zipkin server will run on port <code>9411</code>. Navigate your browser to <a rel="nofollow" target="_blank" href="http://localhost:9411/zipkin/">http://localhost:9411/zipkin/</a>, to access its home page:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-7.png" alt="spring-cloud-sleuth-zipkin-home-page">
<h3 id="sleuthintegrationwithzipkins">Sleuth Integration with Zipkins</h3>
Now, we have to tell Sleuth to send data to the Zipkin server. First we need to add another dependency to the <code>pom.xml</code> file of each service:
<pre><code class="hljs">&lt;dependency&gt; 
 &lt;groupId&gt;org.springframework.cloud&lt;/groupId&gt; 
 &lt;artifactId&gt;spring-cloud-starter-zipkin&lt;/artifactId&gt; 
&lt;/dependency&gt;
</code></pre>
After this, we need to add following properties in the <code>application.properties</code> file of each service:
<pre><code class="hljs">spring.sleuth.sampler.probability=100 
spring.zipkin.baseUrl= http://localhost:9411/
</code></pre>
The <code>spring.zipkin.baseUrl</code> property tells Spring and Sleuth where to push data to. Also, by default, Spring Cloud Sleuth sets all spans to non-exportable. This means these traces (Trace Id and Span Id) appear in logs but are not exported to another remote store like Zipkin.
In order to export spans to the Zipkin server, we need to set a sampler rate using <code>spring.sleuth.sampler.probability</code>. A value of <code>100</code> means all the spans will be sent to the Zipkin server too.
Now, let's restart all applications again and hit the <a rel="nofollow" target="_blank" href="http://localhost:8080/portal-service/fullDetails/12">http://localhost:8080/portal-service/fullDetails/12</a> endpoint again.
Now, on the Zipkin home page at <a rel="nofollow" target="_blank" href="http://localhost:9411/zipkin/">http://localhost:9411/zipkin/</a>, click &quot;Find Traces&quot;:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-8.png" alt="spring-cloud-sleuth-zipkin-find-traces">
Upon clicking on a trace, we'll be navigated to its detail page:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-sleuth-9.png" alt="spring-cloud-sleuth-zipkin-find-traces-details">
Above we can see the request overall took around 16ms and a tree showing the time taken by each service.
Typically to visualize logs for debugging purposes we use the <a target="_blank" href="/introduction-to-the-elk-stack/">ELK stack</a>. To integrate it with Sleuth we can follow the explanation <a rel="nofollow" target="_blank" href="https://github.com/spring-cloud/spring-cloud-sleuth">here</a>.
<h2 id="conclusion">Conclusion</h2>
In this article, we've covered how to use Spring Cloud Sleuth in our existing spring-based microservice application. We saw how it is useful for log tracing of a single request that spans over multiple services. We also integrated it with a Zipkin server to see the latency of each sub-service in the overall request and response.
As always, the code for the examples used in this article can be found on <a rel="nofollow" target="_blank" href="https://github.com/StackAbuse/distributed-tracing/tree/master/Sleuth-Zipkin">GitHub</a>.