<h2 id="whatisanestedfunction">What is a Nested Function?</h2>
Functions are one of the &quot;first-class citizens&quot; of Python, which means that functions are at the same level as other Python objects like integers, strings, modules, etc. They can be created and destroyed dynamically, passed to other functions, returned as values, etc.
Python supports the concept of a &quot;nested function&quot; or &quot;inner function&quot;, which is simply a function defined inside another function. In the rest of the article, we will use the word &quot;inner function&quot; and &quot;nested function&quot; interchangeably.
There are various reasons as to why one would like to create a function inside another function. The inner function is able to access the variables within the enclosing scope. In this article, we will be exploring various aspects of inner functions in Python.
<h2 id="defininganinnerfunction">Defining an Inner Function</h2>
To define an inner function in Python, we simply create a function inside another function using the Python's <code>def</code> keyword. Here is an example:
<pre><code class="hljs">def function1(): # outer function
 print (&quot;Hello from outer function&quot;)
 def function2(): # inner function
 print (&quot;Hello from inner function&quot;)
 function2()

function1()
</code></pre>
Output
<pre><code class="hljs">Hello from outer function
Hello from inner function
</code></pre>
In the above example, <code>function2()</code> has been defined inside <code>function1()</code>, making it an inner function. To call <code>function2()</code>, we must first call <code>function1()</code>. The <code>function1()</code> will then go ahead and call <code>function2()</code> as it has been defined inside it.
It is important to mention that the outer function has to be called in order for the inner function to execute. If the outer function is not called, the inner function will never execute. To demonstrate this, modify the above code to the following and run it:
<pre><code class="hljs">def function1(): # outer function
 print (&quot;Hello from outer function&quot;)
 def function2(): # inner function
 print (&quot;Hello from inner function&quot;)
 function2()
</code></pre>
The code will return nothing when executed!
Here is another example:
<pre><code class="hljs">def num1(x):
 def num2(y):
 return x * y
 return num2
res = num1(10)

print(res(5))
</code></pre>
Output
<pre><code class="hljs">50
</code></pre>
The code returns the multiplication of the two numbers, that is, 10 and 5. The example shows that an inner function is able to access variables accessible in the outer function.
So far, you have seen that it is possible for us to access the variables of the outer function inside the inner function. What if we attempt to change the variables of the outer function from inside the inner function? Let us see what happens:
<pre><code class="hljs">def function1(): # outer function
 x = 2 # A variable defined within the outer function
 def function2(a): # inner function
 # Let&#x27;s define a new variable within the inner function
 # rather than changing the value of x of the outer function
 x = 6
 print (a+x)
 print (x) # to display the value of x of the outer function
 function2(3)

function1()
</code></pre>
Output
<pre><code class="hljs">2
9
</code></pre>
The output shows that it is possible for us to display the value of a variable defined within the outer function from the inner function, but not change it. The statement <code>x = 6</code> helped us create a new variable <code>x</code> inside the inner function <code>function2()</code> rather than changing the value of variable <code>x</code> defined in the outer function <code>function1()</code>.
In the next section, we will be discussing the main reasons as to why we use inner functions in Python.
<h2 id="whyuseinnerfunctions">Why use Inner Functions?</h2>
<h3 id="encapsulation">Encapsulation</h3>
A function can be created as an inner function in order to protect it from everything that is happening outside of the function. In that case, the function will be hidden from the global scope. Here is an example:
<pre><code class="hljs">def outer_function(x):
 # Hidden from the outer code
 def inner_increment(x):
 return x + 2
 y = inner_increment(x)
 print(x, y)

inner_increment(5)
#outer_function(5)
</code></pre>
Output
<pre><code class="hljs">Traceback (most recent call last):
 File &quot;C:/Users/admin/inner.py&quot;, line 7, in &lt;module&gt;
 inner_increment(5)
NameError: name 'inner_increment' is not defined
</code></pre>
In the above code, we are trying to call the <code>inner_increment()</code> function, but instead we got an error.
Now, comment out the call to <code>inner_increment()</code> and uncomment the call to <code>outer_function()</code> as shown below:
<pre><code class="hljs">def outer_function(x):
 # Hidden from the outer code
 def inner_increment(x):
 return x + 2
 y = inner_increment(x)
 print(x, y)

#inner_increment(5)
outer_function(5)
</code></pre>
Output
<pre><code class="hljs">5 7
</code></pre>
The script above shows that the inner function, that is, <code>inner_increment()</code> is protected from what is happening outside it since the variable <code>x</code> inside the <code>inner_increment</code> function is not affected by the value passed to the parameter <code>x</code> of the outer function. In other words, the variables inside the inner function is not accessible outside it. There is a great advantage with such a design pattern. After checking all arguments in the outer function, we can safely skip error checking within the inner function.
<h3 id="closuresandfactoryfunctions">Closures and Factory Functions</h3>
All the examples we have seen till now just contain ordinary functions that have been nested inside other functions. It is possible for us to write such functions in another way instead of nesting them inside other functions. We don't have a specific reason as to why we should nest them.
However, for the case of <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Closure_(computer_programming)">closures</a>, one must use the nested functions.
We can bind/pass data to a function without necessarily passing the data to the function via parameters. This is done using a closure. It is a function object that is able to remember values in the enclosing scopes even when they are not available in the memory. This means that we have a closure when a nested function references a value that is in its enclosing scope.
The purpose of a closure is to make the inner function remember the state of its environment when it is called, even if it is not in the memory. A closure is caused by an inner function, but it's not the inner function. The closure works by closing the local variable on the stack, which stays around after the creation of the stack has finished executing.
The following are the conditions that are required to be met in order to create a closure in Python:
<ul>
<li>There must be a nested function</li>
<li>The inner function has to refer to a value that is defined in the enclosing scope</li>
<li>The enclosing function has to return the nested function</li>
</ul>
Consider the following example:
<pre><code class="hljs">def function1(name):
 def function2():
 print(&#x27;Hello &#x27; + name)
 return function2

func = function1(&#x27;Nicholas&#x27;)
func()
</code></pre>
Output
<pre><code class="hljs">Hello Nicholas
</code></pre>
The above code demonstrates that with closures, we are able to generate and invoke a function from outside its scope via function passing. The scope of <code>function2()</code> is only inside <code>function1()</code>. However, with the use of closures, it was possible for us to extend this scope and invoke it from outside its scope.
Inner functions help us in defining <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Factory_method_pattern">factory functions</a>. A factory function is a function that creates another object. For example:
<pre><code class="hljs">def power_generator(num):

 # Create the inner function
 def power_n(power):
 return num ** power

 return power_n

power_two = power_generator(2)
power_three = power_generator(3)
print(power_two(8))
print(power_three(4))
</code></pre>
Output
<pre><code class="hljs">256
81
</code></pre>
In the script above, from the <code>power_n(power)</code> function, we have created two other objects, <code>power_two</code> and <code>power_three</code>. This makes <code>power_n(power)</code> a factory function since it generates the <code>power_two</code> and <code>power_three</code> functions for us using the parameter we pass it.
<h2 id="conclusion">Conclusion</h2>
An inner function is simply a function that is defined inside another function. The inner function is able to access the variables that have been defined within the scope of the outer function, but it cannot change them. There are a number of reasons as to why we may need to create an inner function. For instance, an inner function is protected from what happens outside it. Inner functions are also a good way of creating closures in Python.

Nicholas Samuel

Python Nested Functions

<h2 id="introduction">Introduction</h2>
Logging helps you keep track of events happening during the execution of your code, which can then be used in the future for debugging purposes. It provides a better picture of the flow of the application and helps developers trace the source of errors that happens during execution of your code, thereby enhancing the maintainability of the application.
In Python, most of the basic logging features are provided by the Python standard library. Hence, you can add logging to your application easily without any extra configurations. The standard logging module enables the developer to write status messages to a file or any other output stream.
<h2 id="theloggingmodule">The <code>logging</code> Module</h2>
The <code>logging</code> module is readily available in Python environments by default and provides a default logger named &quot;root&quot;. It defines the functions and classes that implement logging functionality.
The logging API provided by the standard library enables you to include your own messages in the application log, as well as integration with messages from third-party modules. It also provides a mechanism to annotate the log messages with the source, timestamp, severity and other metadata, which helps in log analysis.
<h2 id="loggingtypesloglevels">Logging Types (Log levels)</h2>
Each log message is associated with a severity level, which is an integer used to signal the criticality of the event(s) logged. The logging module has a helper function for each log level - these are named according to the log level. Below, is a list of logging levels along with their recommended usage.
<ul>
<li>
Debug (<code>logger.debug</code>): Provide very detailed output. Used for diagnosing problems.
</li>
<li>
Info (<code>logger.info</code>): Provides information on successful execution. Confirms if things are working as expected.
</li>
<li>
Warning (<code>logger.warn</code> or <code>logger.warning</code>): Issue a warning regarding a problem that might occur in the future or a recoverable fault.
</li>
<li>
Error (<code>logger.error</code>): Indicates a problem in the software as it is not executing as expected.
</li>
<li>
Critical (<code>logger.critical</code>): Indicates a serious error that might stop the program from running.
</li>
</ul>
By default, the root logger is set up to report all messages at or above the Warning level - any messages below this level are filtered. However, it is possible to configure the module explicitly to be more or less selective in filtering.
To add logging to a Python script, just import the module using <code>import logging</code>, and after a successful import the script can log messages using the <code>logging.*</code> methods, such as <code>logging.debug()</code>.
Here you can see a simple example of the logging module in action:
<pre><code class="hljs">import logging
logging.warning(&quot;Caution: This is the root logger!&quot;)
</code></pre>
Output:
<pre><code class="hljs">WARNING:root:Caution: This is the root logger!
</code></pre>
<h2 id="loggerobjects">Logger Objects</h2>
The <code>logging</code> module allows the user to create multiple logger objects. Different types of logger objects can be used to obtain fine-grained control over how different parts of a Python application log their messages - for example, the main Python app can use the <code>root</code> logger whereas the third-party libraries used from within this app can use their own logger objects with their own configurations.
While using the default <code>root</code> logger's functions, we can call the functions directly, e.g., <code>logging.debug()</code>. It is possible to configure your own logger by creating an object of the <code>Logger</code> class, and this can be useful if your application has multiple modules.
Let's have a look at some of the classes and functions in the <code>logging</code> module. The basic classes and their functions are as follows:
<ul>
<li>
Loggers – Exposes the interface that the app uses. The objects of this class are used directly to call the functions in the application.
</li>
<li>
Handlers – Sends log messages to the appropriate location in the software like standard output console, a file, via HTTP, or even email (via SMTP).
</li>
<li>
Filters – Gives fine-grained control over the choice of log records to display.
</li>
<li>
Formatters – Specifies the final layout of log records, specifying the attributes the output must contain.
</li>
</ul>
Of these, the objects of the <code>Logger</code> class are used most frequently.
To create a new logger, we can use the <code>logging.getLogger()</code> method. In the following script, we log errors using the <code>root</code> logger as well as our custom logger <code>my_logger</code>.
<pre><code class="hljs">import logging

my_logger = logging.getLogger(&quot;My Logger&quot;)
logging.error(&quot;Error: Root Log&quot;)
my_logger.error(&quot;Error: My Logger log&quot;)
</code></pre>
Output:
<pre><code class="hljs">ERROR:root:Error: Root Log
ERROR:My Logger:Error: My Logger log
</code></pre>
Each log message will not only indicate the source - the logger object through which it was logged - but will display a message based on the configuration of that logger object.
In the following sections, we will look at the various configuration options of logger objects.
<h2 id="loggingtoafilevsthestandardoutput">Logging to a File vs the Standard Output</h2>
By default logger objects output the logs to the standard output. You can use the <code>basicConfig()</code> method to change this and other parameters. Below is a list of parameters for the <code>basicConfig</code> method:
<ul>
<li>level: Set a severity level for the logger. Any messages below this severity level will not be logged.</li>
<li>filename: The name of the file where logs are written.</li>
<li>filemode: The mode in which the file specified, if any, is to be opened.</li>
<li>format: Specifies the format of the log message. This is a string with <code>LogRecord</code> attributes.</li>
</ul>
<code>LogRecord</code> object contains the information of the events that are logged, like the line number, time, name of the logger, etc. Discussing the LogRecord object is outside the scope of this article but more information is available <a rel="nofollow" target="_blank" href="https://docs.python.org/3/library/logging.html#logrecord-attributes">here</a>.
Below is a summary of the steps to be followed to record log events to a file:
<ol>
<li>Import the logging module.</li>
<li>Configure the logger using <code>basicConfig</code> method</li>
<li>Creating a logger object.</li>
<li>Setting the threshold value of logger.</li>
<li>Use the logging methods.</li>
</ol>
This can be better understood by an example:
<pre><code class="hljs"># Filename: test_logger.py

import logging

# Create a logger object
logger = logging.getLogger()

# Configure logger
logging.basicConfig(filename=&quot;test.log&quot;, format=&#x27;%(filename)s: %(message)s&#x27;, filemode=&#x27;w&#x27;)

# Setting threshold level
logger.setLevel(logging.DEBUG)

# Use the logging methods
logger.debug(&quot;This is a debug message&quot;)
logger.info(&quot;For your info&quot;)
logger.warning(&quot;This is a warning message&quot;)
logger.error(&quot;This is an error message&quot;)
logger.critical(&quot;This is a critical message&quot;)
</code></pre>
The script above will create a file <code>test.log</code>. The file will contain the following logging information:
<pre><code class="hljs">test_logger.py: This is a debug message
test_logger.py: For your info
test_logger.py: This is a warning message
test_logger.py: This is an error message
test_logger.py: This is a critical message
</code></pre>
<h2 id="datetimeinlogmessages">Date/Time in Log Messages</h2>
To display the date and time of the occurrence of an event, you can use <code>%(asctime)s</code> in your format string in the <code>basicConfig()</code> function. For example:
<pre><code class="hljs">import logging

logging.basicConfig(format=&#x27;%(asctime)s %(message)s&#x27;)
logging.warning(&#x27;is the time the Admin logged out.&#x27;)
</code></pre>
Output:
<pre><code class="hljs">2018-12-17 10:52:15,463 is the time the Admin logged out.
</code></pre>
If you want to change the way the date/time shows up, you can configure it using the <code>datefmt</code> parameter of the <code>basicConfig</code> method.
<h2 id="loggingvariables">Logging Variables</h2>
In real-world applications, we need to output logs according to the dynamic changes happening in our application. As seen in the above example, logging methods take a string as an argument. In addition, we can include variables and format the string with placeholders and then pass it to the log method. At run-time, the value of the variables will be output in the log messages.
Here's an example of that using string formatting:
<pre><code class="hljs">import logging

status = &quot;connection unavailable&quot;
logging.error(&quot;System reported: %s&quot;, status)
</code></pre>
Output:
<pre><code class="hljs">ERROR:root:System reported: connection unavailable
</code></pre>
Starting with Python 3.6 f-Strings can be used as an alternative to string format specifiers, thereby keeping the code easily readable when there are multiple parameters. Using f-strings you can specify any Python expression as part of the message and these are evaluated during run-time and the result will be embedded into the log messages.
The above example can be re-written using an f-string as:
<pre><code class="hljs">import logging

status = &quot;connection unavailable&quot;
logging.error(f&#x27;System reported: {status}&#x27;)
</code></pre>
<h2 id="loggingstacktraces">Logging Stack Traces</h2>
Capturing stack traces in your application is also supported by the logging module. Setting the <code>exc_info</code> parameter to <code>True</code> while calling the logging functions allows us to capture the exception information. By using this function, we can get information about the exception that is currently being handled. The information is specific to the current thread and the current stack frame.
<pre><code class="hljs">import logging

my_list = [1, 2]
try:
 print(my_list[3]) # Index out of range
except Exception as e:
 logging.error(&quot;Caught Exception!&quot;, exc_info=True)
</code></pre>
Output:
<pre><code class="hljs">ERROR:root:Caught Exception!
Traceback (most recent call last):
 File &quot;index.py&quot;, line 5, in &lt;module&gt;
 print(my_list[3]) # Index out of range
IndexError: list index out of range
</code></pre>
In case exceptions are not handled by the current stack frame, the information is then obtained from its caller (i.e. the calling stack frame) and so on until it finds a stack frame to handle the exception. The stack frame has the information of the most recently handled exception.
If the stack has no exceptions being handled, a tuple having <code>None</code> value is returned. Otherwise the function returns value of <code>type</code> (the type of exception that is handled), value (exception parameter) and traceback (the traceback object which encapsulates the call stack where the exception occurred originally).
<h2 id="conclusion">Conclusion</h2>
The design of the logging module is very practical, providing ready-to-use logging features that can add basic logging to a small project. It can be easily extended by using logger objects and their rich configuration options to support the needs of even the most demanding of applications. In addition to messages, the logging module can also be used to log exceptions and stack traces. This concludes the basic tutorial on implementing logging in Python.

Guest Contributor

Python Logging Basics

<h2 id="introduction">Introduction</h2>
The <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/technotes/guides/collections/index.html">Java Collections Framework</a> is a fundamental and essential framework that any strong Java developer should know like the back of their hand.
A Collection in Java is defined as a group or collection of individual objects that act as a single object.
There are many collection classes in Java and all of them extend the <code>java.util.Collection</code> and <code>java.util.Map</code> interfaces. These classes mostly offer different ways to formulate a collection of objects within a single object.
Java Collections is a framework that provides numerous operations over a collection - searching, sorting, insertion, manipulation, deletion etc.
This is the first part of a series of Java Collections articles:
<ul>
<li><a target="_blank" href="/java-collections-the-list-interface">The List Interface</a></li>
<li>The Set Interface (you are here)</li>
<li><a target="_blank" href="/java-collections-the-map-interface/">The Map Interface</a></li>
<li><a target="_blank" href="/java-collections-queue-and-deque-interfaces/">The Queue and Deque Interfaces</a></li>
</ul>
<h2 id="sets">Sets</h2>
The next common interface from the framework is <code>java.util.Set</code>.
Sets don't offer additional methods, other than the methods inherited from the <code>Collection</code> interface.
A Set models the mathematical set abstraction, and can't contain duplicate elements. That being said, it's also worth noting that these elements have no specific order within the set:
<pre><code class="hljs">List&lt;String&gt; names = Arrays.asList(&quot;David&quot;, &quot;Scott&quot;, &quot;Adam&quot;, &quot;Jane&quot;, &quot;Scott&quot;, &quot;David&quot;, &quot;Usman&quot;);
System.out.println(names);

Set&lt;String&gt; uniqueNames = new HashSet&lt;&gt;(names);
System.out.println(uniqueNames);
</code></pre>
Running this piece of code would yield:
<pre><code class="hljs">[David, Scott, Adam, Jane, Scott, David, Usman]
[Adam, David, Jane, Scott, Usman]
</code></pre>
As you can notice, the list <code>names</code> contains duplicate entries, and the set <code>uniqueNames</code> removes the duplicate ones and prints them out without a specific order.
<h3 id="addinganelement">Adding an Element</h3>
Using the <code>add()</code> method, similar as in Lists we can add objects to Set:
<pre><code class="hljs">Set&lt;String&gt; uniqueNames = new HashSet&lt;&gt;();
uniqueNames.add(&quot;David&quot;);
uniqueNames.add(&quot;Scott&quot;);
uniqueNames.add(&quot;Adam&quot;);
uniqueNames.add(&quot;Jane&quot;);
uniqueNames.add(&quot;Scott&quot;);
uniqueNames.add(&quot;David&quot;);
uniqueNames.add(&quot;Usman&quot;);

System.out.println(uniqueNames);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">[Adam, David, Jane, Scott, Usman]
</code></pre>
<h3 id="removingelements">Removing Elements</h3>
Using the boolean <code>remove()</code> method, we can remove the specified element from this set if it is present:
<pre><code class="hljs">System.out.println(uniqueNumbers.remove(2));

System.out.println(uniqueNumbers);
</code></pre>
Output:
<pre><code class="hljs">true
[1, 3]
</code></pre>
Another option is to use the <code>clear()</code> method to remove all elements of the Set:
<pre><code class="hljs">List&lt;String&gt; names = Arrays.asList(&quot;David&quot;, &quot;Scott&quot;, &quot;Adam&quot;, &quot;Jane&quot;, &quot;Scott&quot;, &quot;David&quot;, &quot;Usman&quot;);
Set&lt;String&gt; uniqueNames = new HashSet&lt;&gt;(names);

uniqueNames.clear();
System.out.println(uniqueNames);
</code></pre>
Running this piece of code would yield:
<pre><code class="hljs">[]
</code></pre>
Alternatively, we could rely on the <code>removeAll()</code> method:
<pre><code class="hljs">List&lt;String&gt; names = Arrays.asList(&quot;David&quot;, &quot;Scott&quot;, &quot;Adam&quot;, &quot;Jane&quot;, &quot;Scott&quot;, &quot;David&quot;, &quot;Usman&quot;);
List&lt;String&gt; newNames = Arrays.asList(&quot;David&quot;, &quot;Adam&quot;);
Set&lt;String&gt; uniqueNames = new HashSet&lt;&gt;(names);

uniqueNames.removeAll(newNames);
System.out.println(uniqueNames);
</code></pre>
Running this piece of code would yield:
<pre><code class="hljs">[Jane, Scott, Usman]
</code></pre>
It's important to notice that the <code>removeAll()</code> method accepts a <code>Collection</code> as an argument. This can be used to remove all common elements from two different collections, in this case a <code>List</code> and a <code>Set</code>.
Also keep in mind that you can use this method to remove all the elements from the <code>Collection</code> itself:
<pre><code class="hljs">uniqueName.removeAll(uniqueNames);
</code></pre>
This will, of course, end up with an empty set. However, this approach isn't recommended as calling the <code>removeAll()</code> method costs a lot more than the <code>clear()</code> method.
This is due to the <code>removeAll()</code> method comparing every single element from the argument collection with the collection that calls the method whereas <code>clear()</code> simply points them all to <code>null</code> and sets the size to 0.
<h3 id="containstheelement">Contains the Element</h3>
Using the boolean <code>contains()</code> method with the given object, we can check if this <code>Set</code> contains a specified element:
<pre><code class="hljs">List&lt;String&gt; names = Arrays.asList(&quot;David&quot;, &quot;Scott&quot;, &quot;Adam&quot;, &quot;Jane&quot;, &quot;Scott&quot;, &quot;David&quot;, &quot;Usman&quot;);
Set&lt;String&gt; uniqueNames = new HashSet&lt;&gt;(names);

System.out.println(uniqueNames.contains(&quot;David&quot;));
System.out.println(uniqueNames.contains(&quot;Scott&quot;));
System.out.println(uniqueNames.contains(&quot;Adam&quot;));
System.out.println(uniqueNames.contains(&quot;Andrew&quot;));
</code></pre>
Running this code would yield:
<pre><code class="hljs">true
true
true
false
</code></pre>
<h3 id="iteratingelements">Iterating Elements</h3>
The same as with lists, although possible to iterate with <code>for</code> and <code>enhanced-for</code> loops, it's better to use the Java Collections' <code>Iterator</code> for this task:
<pre><code class="hljs">Set&lt;E&gt; set = new TreeSet&lt;E&gt;();
...
for(Iterator&lt;E&gt; iterator = set.iterator(); iterator.hasNext()) {
 E element = iterator.next();
 element.someMethod();
 iterator.remove(element);
}
</code></pre>
Additionally, Java 8 introduces us with a really simple way to print out the elements using method references:
<pre><code class="hljs">set.forEach(System.out::println);
</code></pre>
<h3 id="retrievingsize">Retrieving Size</h3>
If you'd like to retrieve the size of a set:
<pre><code class="hljs">List&lt;String&gt; names = Arrays.asList(&quot;David&quot;, &quot;Scott&quot;, &quot;Adam&quot;, &quot;Jane&quot;, &quot;Scott&quot;, &quot;David&quot;, &quot;Usman&quot;);
Set&lt;String&gt; uniqueNames = new HashSet&lt;&gt;(names);

System.out.println(uniqueNames.size());
</code></pre>
Running this piece of code would yield:
<pre><code class="hljs">5
</code></pre>
<h3 id="checkingifempty">Checking if Empty</h3>
If you'd like to run a check to see whether a Set is empty or not before performing any operations on it:
<pre><code class="hljs">List&lt;String&gt; names = Arrays.asList(&quot;David&quot;, &quot;Scott&quot;, &quot;Adam&quot;, &quot;Jane&quot;, &quot;Scott&quot;, &quot;David&quot;, &quot;Usman&quot;);
Set&lt;String&gt; uniqueNames = new HashSet&lt;&gt;(names);

System.out.println(uniqueNames.isEmpty());
</code></pre>
Running this piece of code would yield:
<pre><code class="hljs">false
</code></pre>
<h3 id="implementationsanddifferences">Implementations and Differences</h3>
HashSet:
<ul>
<li>Based upon <code>HashMap</code> (Calls <a rel="nofollow" target="_blank" href="/javas-object-methods-hashcode/"><code>hashCode()</code></a> on the element and looks up the location)</li>
<li>Good general purpose implementation (Resizes when it runs out of space)</li>
</ul>
TreeSet:
<ul>
<li>Based upon <code>TreeMap</code> (Uses a Binary Tree with a required sort order)</li>
<li>Keeps elements in the given order</li>
</ul>
EnumSets:
<ul>
<li>Specialized implementation for enums (Uses a bitset based upon the ordinal of the enum)</li>
<li>Use when storing sets of enums</li>
</ul>
<h3 id="algorithmiccomparison">Algorithmic Comparison</h3>
<img src="https://s3.amazonaws.com/stackabuse/media/java-collections-3.png" alt="performance_comparison_hashset_treeset_enumset">
<h2 id="conclusion">Conclusion</h2>
The Java Collections framework is a fundamental framework that every Java developer should know how to use.
In the article, we've talked about the Set Interface and its implementations, their advantages, and disadvantages as well as the operations you'll most certainly use at one point or another.
If you're interested in reading more about the collection interfaces, continue reading - Java Collections: Queues, Dequeues and Stacks (coming soon).

Vuk Skobalj

Java Collections: The Set Interface

<h2 id="introduction">Introduction</h2>
The <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/technotes/guides/collections/index.html">Java Collections Framework</a> is a fundamental and essential framework that any <a target="_blank" rel="noopener" href="https://www.index.dev/hire/java-developer">strong Java developer</a> should know like the back of their hand.
A Collection in Java is defined as a group or collection of individual objects that act as a single object.
There are many collection classes in Java and all of them extend the <code>java.util.Collection</code> and <code>java.util.Map</code> interfaces. These classes mostly offer different ways to formulate a collection of objects within a single object.
Java Collections is a framework that provides numerous operations over a collection - searching, sorting, insertion, manipulation, deletion, etc. It's a high-level framework that provides a consistent API for working with collections.
This is the first part of a series of Java Collections articles:
<ul>
<li>The List Interface (you are here)</li>
<li><a target="_blank" href="/java-collections-the-set-interface/">The Set Interface</a></li>
<li><a target="_blank" href="/java-collections-the-map-interface/">The Map Interface</a></li>
<li><a target="_blank" href="/java-collections-queue-and-deque-interfaces/">The Queue and Deque Interfaces</a></li>
</ul>
<h2 id="problemswitharrays">Problems with Arrays</h2>
<a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/tutorial/java/nutsandbolts/arrays.html">Arrays</a> are one of the first things a new baked Java developer is introduced with. They're a simple and straightforward way to store multiple objects in a single object.
An array of objects, very similar to a collection, represents a group of objects as a single object.
Both an array and a collection are an object representing multiple other objects, so why is there a need for both?
Let's consider a collection of products:
<pre><code class="hljs">Product door = new Product(&quot;Wooden Door&quot;, 35);
Product floorPanel = new Product(&quot;Floor Panel&quot;, 25);
</code></pre>
We have a wooden door and a door panel, weighing 35 kg and 25 kg respectively. These are <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Plain_old_Java_object">POJOs</a>, which means that they only have a couple of getter and setter methods and a <code>toString()</code> method.
With this, it's quite simple to instantiate an array of these objects:
<pre><code class="hljs">Product[] products = { door, floorPanel };
</code></pre>
<h3 id="printingarrays">Printing Arrays</h3>
There are many reasons why someone would want to print an array, including debugging or returning the results:
<pre><code class="hljs">System.out.println(products);
</code></pre>
However, when we try to print it out, we're greeted with a result that isn't very human friendly:
<pre><code class="hljs">com.demo.collections.Product;@14ae5a5
</code></pre>
In fact, we need to rely on the helper class <code>java.util.Arrays</code> to get a sensible result:
<pre><code class="hljs">System.out.println(Arrays.toString(products));
</code></pre>
This time around, we're seeing something that makes more sense:
<pre><code class="hljs">[Product{name=&quot;Wooden Door&quot;, weight=35}, Product{name=&quot;Floor Panel&quot;, weight=25}]
</code></pre>
This is because the <code>Arrays.toString()</code> method is implemented to print out the elements of the array in a human-readable format, which isn't built into the array itself.
<h3 id="addingandremovingelements">Adding and Removing Elements</h3>
Our collection of products just got bigger, and we're supposed to add a window to the array:
<pre><code class="hljs">final Product window = new Product(&quot;Window&quot;, 15);
products = add(window, products);
System.out.println(Arrays.toString(products));


public static Object[] add(Object[] array, Object... elements) {
 Object[] tempArray = new Object[array.length + elements.length];
 System.arrayCopy(array, 0, tempArray, 0, array.length);

 for(int i = 0; i &lt; elements.length; i++) {
 tempArray[array.length+i] = elements[i];
 return tempArray;
 }
}
</code></pre>
This is exactly the type of situation where you'd probably rather shoot yourself in the leg - because arrays don't resize.
To add an element, we have to make a copy of the array into a new array, instantiate it with the new elements and assign the new array to our reference variable. This is a very inefficient way to add elements to an array. The same goes for removing elements.
Arrays are a more low-level construct and don't give us many features whereas Collections are made to combat that very problem and offer many features and great functionality.
<h2 id="collections">Collections</h2>
The Java Collections Framework ships with the JDK itself. It's worth remembering that in the old days, especially for people who wrote C code, developers weren't presented with data structures to choose from. In fact, people used to write their own data structures, which some do even today.
There are legitimate performance reasons why someone might find a custom data structure to be great for a specific project. But, for most developers, relying on the existing framework is a good choice.
Java is used to build big and complex systems and applications. That being said, almost every Java application will end up using the collections framework at one point or another.
All of the collection classes have an underlying data structure that they're implementing - Trees, HashTables, HashMaps, Queues, etc. Implementing these data structures yourself, while potentially fun, can be very hard - there are many corners you have to get right. There's no need to reinvent the wheel if it's already served to you unless you wish to practice and challenge yourself to come up with innovative and alternative solutions.
We'll be taking a look at a few different types of collections in Java:
<ul>
<li>Lists - A sequential (ordered) Collection. They keep track of the positions of all elements, like arrays and offer search, iteration and range-view operations of their elements. Lists can have duplicate elements.</li>
<li>Sets - Enforces uniqueness constraints - can't contain duplicate elements. It doesn't concern itself with the order of iteration within itself, as it models the mathematical set abstraction. Sets offer no additional functionality other than the inherited from Collections.</li>
<li>Queues - Introduce modification order, that's to say, if you add elements in a certain order, you have to follow a certain order. Queues offer additional insertion, removal and inspection operations upon its elements. It's unique for Queues to follow the FIFO (First in, first out) structure.</li>
<li>Deques - Similar to queues, double-ended queues (shortened to deques) additionally offer the possibility to perform operations on elements from both sides of the queue.</li>
<li>Maps - Although implementations of <code>java.util.Map</code> aren't considered &quot;true collections&quot;, they offer collection-view operations which practically enables collection-level manipulation. This collection isn't a collection of individual values, but pairs. These are associations between unique keys and values (Maps) that can be looked up from those keys. It's important to note that the keys are unique, and each key is associated with a value, but a value can be associated with more than one key.</li>
</ul>
<h3 id="interfacecollection">Interface <code>Collection</code></h3>
As mentioned above, all of the collection interfaces within the Java API extend a common interface - <code>java.util.Collection</code>. This main interface provides all of the common collections functionality.
Each sub-interface has several implementations, and some of these sub-interfaces offer additional operations:
<img src="https://s3.amazonaws.com/stackabuse/media/java-collections-1.png" alt="collection of collections">
The key point to understand is that each interface defines behavior and functional characteristics where we can utilize multiple data structures while implementations define performance characteristics, utilize a specific data structure and are instantiable. The interfaces are like the contracts, while the implementations are the concrete classes that fulfill those contracts.
The most commonly used methods in the <code>Collection</code> interface are:
<table style="border-collapse: collapse; width: 100%;">
<tr>
 <th>Method Name</th>
 <th>Method Description </th>
</tr>
<tr style="background-color: #f2f2f2;">
 <td>size()</td>
 <td>Get the number of elements in the Collection</td>
</tr>
<tr>
 <td>isEmpty()</td>
 <td>True if size() == 0, false otherwise </td>
</tr>
<tr style="background-color: #f2f2f2;">
 <td>add(element)</td>
 <td>Add the element at the beginning of this collection</td>
</td>
</tr>
<tr>
 <td>addAll(collection)</td>
 <td>Add all the elements of the argument collection to this collection </td>
</tr>
<tr style="background-color: #f2f2f2;">
 <td>remove(element)</td>
 <td>Remove the element from this collection </td>
</tr>
<tr>
 <td>removeAll(collection)</td>
 <td>Remove all the elements of the argument collection to this collection </td>
</tr>
<tr style="background-color: #f2f2f2;">
 <td>retainAll()</td>
 <td>Remove all the elements of this collection not in the argument collection </td>
</tr>
 <td>contains(element)</td>
 <td>True if the element is in this collection, false otherwise </td>
</tr>
<tr style="background-color: #f2f2f2;">
 <td>containsAll(collection)</td>
 <td>True if all the elements of the argument collection are in this collection </td>
</tr>
<tr>
 <td>clear()</td>
 <td>Remove all elements from this collection </td>
</tr>
</table>
<h2 id="lists">Lists</h2>
The first, and probably most commonly used interface - <code>java.util.List</code>.
Each element within the list has an index, an <code>int</code> value which defines their position. The indexing count starts at 0, the same as the indexing we can encounter with arrays.
The <code>java.util.List</code> interface also adds a couple of other operations beyond the regular common collection operations:
<ul>
<li><code>get(int index)</code></li>
<li><code>set(int index, Object object)</code></li>
</ul>
These operations are pretty self-explanatory and don't need further explanation. Though, let's take a look at some code examples.
<h3 id="addinganelement">Adding an element</h3>
Using the <code>add()</code> method, we can easily add objects to our list:
<pre><code class="hljs">List&lt;String&gt; products = new ArrayList&lt;&gt;();
products.add(&quot;Mug&quot;);
products.add(&quot;Wallet&quot;);
products.add(&quot;Phone&quot;);
System.out.println(products);
</code></pre>
Output:
<pre><code class="hljs">[Mug, Wallet, Phone]
</code></pre>
Note: We're instantiating the list as its concrete implementation <code>ArrayList</code>. In most cases, we'd use this implementation for a List.
Another note: You can specify the initial size of the ArrayList via the constructor to avoid resizing if you know a definitive size.
The interface also provides another version of the <code>add()</code> method, including an index. In this case, we add the element to the given index, and if the index is already taken by another element, all of the elements after the added one shift to the right by one:
<pre><code class="hljs">products.add(2, &quot;Pen&quot;);
System.out.println(products);
</code></pre>
Output:
<pre><code class="hljs">[Mug, Wallet, Pen, Phone]
</code></pre>
<h3 id="retrievingelements">Retrieving Elements</h3>
Using the <code>get()</code> method with the given index, we can retrieve a specific element in the list:
<pre><code class="hljs">System.out.println(products.get(0));
</code></pre>
Output:
<pre><code class="hljs">[Mug]
</code></pre>
<h3 id="removingelements">Removing Elements</h3>
Using the <code>remove()</code> method, we can remove an element from the list. Calling this method will return the element as well as shift the elements after it one index back, to fill in the now existent hole in the sequence:
<pre><code class="hljs">System.out.println(products.remove(1));
</code></pre>
Output:
<pre><code class="hljs">[Wallet]
</code></pre>
<h3 id="settingelements">Setting Elements</h3>
Using the <code>set()</code> method, we can replace an existing element given an index:
<pre><code class="hljs">products.set(1, &quot;Book&quot;);

System.out.println(products);
</code></pre>
Output:
<pre><code class="hljs">[Mug, Book, Phone]
</code></pre>
<h3 id="searchingforelements">Searching for Elements</h3>
Using the <code>indexOf()</code> method, we can also look up values, given an index. If the search fails, and no object exists with the given index, the list will return <code>-1</code>. In the case of multiple equal objects, the list will return only the first index.
Using the <code>lastIndexOf()</code> will return the last index of the given element.
<pre><code class="hljs">System.out.println(products.indexOf(5));
</code></pre>
Output:
<pre><code class="hljs">-1
</code></pre>
<h3 id="iteratingelements">Iterating Elements</h3>
Although possible to iterate with <code>for</code> and <code>enhanced-for</code> loops, the interface provides two new helper classes that allow us to iterate through lists - <code>Iterator</code> and <code>ListIterator</code>:
<pre><code class="hljs">for (Iterator&lt;E&gt; iterator = list.iterator(); iterator.hasNext(); ) {
 E element = iterator.next();
 element.someMethod();
 iterator.remove(element);
 //...
}

for (ListIterator&lt;E&gt; iterator = list.listIterator(); iterator.hasNext(); ) {
 E element = iterator.next();
 element.someMethod();
 iterator.remove(element);
 //...
}
</code></pre>
Note: The <code>ListIterator</code> offers more control over the list iteration as it allows for traversal in both directions, while <code>Iterator</code> only allows traversal in one direction.
Additionally, Java 8 introduces us with a really simple way to print out the elements using a method reference:
<pre><code class="hljs">list.forEach(System.out::println);
</code></pre>
<h3 id="implementationsanddifferences">Implementations and Differences</h3>
ArrayList: implements <code>java.util.List</code> as a dynamically resizing array:
<ul>
<li>Good general-purpose implementation</li>
<li>Used as default</li>
<li>More CPU cache sympathetic</li>
</ul>
LinkedList: implements <code>java.util.List</code> as a doubly-linked list:
<ul>
<li>Worse performance for many operations</li>
<li>Use when adding elements at the start</li>
<li>Use when adding/removing a lot</li>
</ul>
Generally speaking, <code>ArrayList</code> is much more commonly used than <code>LinkedList</code>. And to quote Joshua Bloch, the man who wrote <code>LinkedList</code>:
<blockquote>
&quot;Does anyone actually use LinkedList? I wrote it, and I never use it.&quot;
</blockquote>
<h3 id="performancecomparison">Performance Comparison</h3>
<img src="https://s3.amazonaws.com/stackabuse/media/java-collections-2.png" alt="performance comparison arraylist vs linkedlist">
Due to their different natures, these implementations have different approaches and method runtimes.
Depending on the requirements, you'll have to choose which to use. Generally speaking, due to its doubly-linked nature, <code>LinkedList</code> is good for frequent addition and removal whereas <code>ArrayList</code> is good for searching due to random access.
<h2 id="conclusion">Conclusion</h2>
The Java Collections framework is a fundamental framework that every Java developer should know how to use.
In the article, we've talked about collections in general, the problems with arrays and how the framework combats them. Afterwards, we jumped into the implementations of this interface, their advantages, and disadvantages, as well as the operations you'll most certainly use at one point or another.
If you're interested in reading more about the collection interfaces, continue reading - <a target="_blank" href="/java-collections-the-set-interface/">Java Collections: The Set Interface</a>.

Java Collections: The List Interface

Usman Malik

Pandas Library for Data Visualization in Python

<h2 id="introduction">Introduction</h2>
Web-scraping is an important technique, frequently employed in a lot of different contexts, especially data science and data mining. Python is largely considered the go-to language for web-scraping, the reason being the batteries-included nature of Python. With Python, you can create a simple scraping script in about 15 minutes and in under 100 lines of code. So regardless of usage, web-scraping is a skill that every Python programmer must have under his belt.
Before we start getting hands-on, we need to step back and consider what is web-scraping, when we should use it, and when to avoid using it.
As you already know, <a target="_blank" rel="noopener" href="https://www.scraperapi.com/web-scraping-learning-hub/">web-scraping is a technique employed to automatically extract data from websites</a>. What's important to understand is, web-scraping is a somewhat crude technique to extract data from various sources - typically web pages. If the developers of a website are generous enough to provide an API to extract data, that would be a much more stable and robust way to access the data. So, as a rule of thumb, if a website provides an API to programmatically retrieve their data, use that. If an API is not available, only then use web-scraping — understanding the key differences between <a target="_blank" rel="noopener noreferrer" href="https://blog.apify.com/web-scraping-vs-api/">web-scraping vs. API</a>.
Be sure to also comply with any rules or restrictions regarding web scraping for each website you use, as some do not allow it. With that being clear, let's jump right into the tutorial.
For this tutorial, we're going to scrape <code>http://quotes.toscrape.com/</code>, a site that lists famous quotes by renowned authors.
<h2 id="thewebscrapingpipeline">The Web Scraping Pipeline</h2>
We can understand web-scraping as a pipeline containing 3 components:
<ol>
<li>Downloading: Downloading the HTML web-page</li>
<li>Parsing: Parsing the HTML and retrieving data we're interested in</li>
<li>Storing: Storing the retrieved data in our local machine in a specific format</li>
</ol>
<h3 id="downloadinghtml">Downloading HTML</h3>
It only seems logical that to extract any data from a web-page, we first have to download it. There are two ways we can go about doing this:
1. Using Browser Automation Libraries
You can use browser automation libraries such as <a rel="nofollow" target="_blank" href="http://robotframework.org/SeleniumLibrary/">Selenium</a> to download the HTML from a web page. Selenium lets you open a browser, let's say Chrome, and control it however you want. You can open the web-page in a browser and then get the HTML code of that page, all automated using Selenium.
However, this method has a huge drawback — it is significantly slower. The reason being the overhead of running the browser and rendering the HTML in the browser. This method should only be used in exceptional cases — cases where the content we want to scrape uses JavaScript code in the browser, or requires us to click buttons/links to get to the data, which Selenium can do for us.
2. Using HTTP Libraries
HTTP libraries, such as the <a target="_blank" href="/the-python-requests-module/">Requests module</a> or <a rel="nofollow" target="_blank" href="https://docs.python.org/2/library/urllib.html">Urllib</a>, let you send the HTTP request, bypassing the need to open any browser at all, unlike the first method. This method should always be preferred, as it is much faster than Selenium.
Now let me show you how can we achieve this component of the pipeline using Selenium and Requests libraries:
Using Requests
Install the <code>requests</code> module with the following:
<pre><code class="hljs">$ pip install requests
</code></pre>
And now you can use it in your code, like this:
<pre><code class="hljs">import requests

result = requests.get(&#x27;http://quotes.toscrape.com/&#x27;)
page = result.text
</code></pre>
Here, an HTTP GET request is made to the URL, which is almost synonymous to downloading the webpage. Then, we can get the HTML source of the page by accessing the result object returned by the <code>requests.get()</code> method.
Using Selenium
You can install the <code>selenium</code> module via <code>pip</code>:
<pre><code class="hljs">$ pip install selenium
</code></pre>
<pre><code class="hljs">from selenium import webdriver

driver = webdriver.Chrome()
driver.get(&#x27;http://quotes.toscrape.com/&#x27;)
page = driver.page_source
</code></pre>
Here, we first start by creating a <code>webdriver</code> object, which represents the browser. Doing this will open the Chrome browser on the computer running the code. Then, by calling the <code>get</code> method of the <code>webdriver</code> object, we can open our URL. And finally, we get the source code by accessing the <code>page_source</code> property of the <code>webdriver</code> object.
In both the cases, the HTML source of the URL is stored in the page variable as a string.
<h3 id="parsinghtmlandextractingdata">Parsing HTML and Extracting Data</h3>
Without getting into theoretical computer science, we can define parsing as the process of analyzing a string so that we can understand its contents and thus access data within it easily.
In Python, there are two libraries that can help us with Parsing HTML: <a rel="nofollow" target="_blank" href="https://pypi.org/project/beautifulsoup4/">BeautifulSoup</a> and <a rel="nofollow" target="_blank" href="https://lxml.de/">Lxml</a>. Lxml is a more lower-level framework than BeautifulSoup, and we can use Lxml as a backend in BeautifulSoup, so for simple HTML parsing purposes, BeautifulSoup would be the preferred library.
But before we dive into parsing, we have to analyze the web-page's HTML and see how the data we want to scrape is structured and located. Only when we're armed with that information, we can get the information we want from the parsed HTML. But thankfully, we won't have to open the source code in an editor and manually understand and correlate each HTML element with the corresponding data in the rendered page. Most browsers offer an inspector, like <a rel="nofollow" target="_blank" href="https://developers.google.com/web/tools/chrome-devtools/">Chrome's Developer Tools</a>, which enables us to quickly look at the HTML code of any element just by clicking on them.
To do this in Chrome, open the web page in Chrome, then right-click on the data you want to scrape and select Inspect. In Firefox, this option is called Inspect Element - which is the same thing, but just a different name.
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/introduction-to-web-scraping-with-python-1.png">
You'll notice a pane opened at the bottom of the Chrome window, containing the source code of the element you clicked on. Browse through the source code a bit to get an idea of how the data which we want to scrape is structured in the HTML code.
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/introduction-to-web-scraping-with-python-2.png">
As you can understand after a little bit of inspection, each quote on <code>http://quotes.toscrape.com/</code> is contained in a <code>div</code> with the <code>class=&quot;quote&quot;</code> attribute. Within that <code>div</code>, the text of the quote is in a <code>span</code> with <code>class=&quot;text&quot;</code> and the author's name is in a <code>small</code> tag with <code>class=&quot;author&quot;</code>. This info will be required when we get to actually parsing the HTML and extracting our data.
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/introduction-to-web-scraping-with-python-3.png">
Now, let's start parsing the HTML page using BeautifulSoup. But first, we must install it:
<pre><code class="hljs">$ pip install beautifulsoup4
</code></pre>
Once installed, you can call it in your code like this:
<pre><code class="hljs">from bs4 import BeautifulSoup

soup = BeautifulSoup(page, &#x27;html.parser&#x27;)
</code></pre>
First of all, we create a parsed version of the page by passing it to the <code>BeautifulSoup</code> class constructor. As you can see, we also pass a second argument to the constructor, <code>html.parser</code>. That is the name of the parser that Beautiful Soup is going to use to parse the string you passed to it. You could've also used the parser <code>lxml</code>, which we previously talked about, given that you have the Lxml library installed.
<pre><code class="hljs">quotes = soup.find_all(&#x27;div&#x27;, class_=&#x27;quote&#x27;)
</code></pre>
Then, we extract all the <code>div</code> tags in the page containing <code>class=&quot;quote&quot;</code>, as we know those are the <code>div</code>s containing quotes. To do this, Beautiful Soup 4 offers a <code>find_all</code> function. We passed the tag name and the class name to the <code>find_all</code> function, and it returned all the tags satisfying the conditions, i.e. the tags containing our quotes.
An important thing to note here is, we're working with tree structures here. The variable <code>soup</code>, and also each element of <code>quotes</code>, are trees. In a way, the elements of <code>quotes</code> are parts of the larger <code>soup</code> tree. Anyway, without drifting off into a different discussion, let's carry on.
<pre><code class="hljs">scraped = []
for quote in quotes:
 text = quote.find(&#x27;span&#x27;, class_=&#x27;text&#x27;).text
 author = quote.find(&#x27;small&#x27;, class_=&#x27;author&#x27;).text
 scraped.append([text, author])
</code></pre>
We know that the text of the quote is in a <code>span</code> tag with <code>class=&quot;text&quot;</code> and the author is in a <code>small</code> tag with <code>class=&quot;author&quot;</code>. To extract them from the quote elements, we again employ a similar function, <code>find</code>. The <code>find</code> function takes the same arguments as the <code>find_all</code> function. The only difference is that it returns the first tag satisfying the conditions, whereas <code>find_all</code> returns a list of tags. Also, we want to access the <code>text</code> property of the returned object, which contains the text enclosed within that tag.
So, as you can see in the code, we loop through all the elements of the list <code>quotes</code>, and extract the quote text and author name, storing them as a list of lists with the name <code>scraped</code>. The <code>scraped</code> list, when printed on the console, looks like this:
<pre><code class="hljs">[[&#x27;“The world as we have created it is a process of our thinking. It cannot be changed without changing our thinking.”&#x27;,
 &#x27;Albert Einstein&#x27;],
 [&#x27;“It is our choices, Harry, that show what we truly are, far more than our abilities.”&#x27;,
 &#x27;J.K. Rowling&#x27;],
 [&#x27;“There are only two ways to live your life. One is as though nothing is a miracle. The other is as though everything is a miracle.”&#x27;,
 &#x27;Albert Einstein&#x27;],
 [&#x27;“The person, be it gentleman or lady, who has not pleasure in a good novel, must be intolerably stupid.”&#x27;,
 &#x27;Jane Austen&#x27;],
 [&quot;“Imperfection is beauty, madness is genius and it&#x27;s better to be absolutely ridiculous than absolutely boring.”&quot;,
 &#x27;Marilyn Monroe&#x27;],
 [&#x27;“Try not to become a man of success. Rather become a man of value.”&#x27;,
 &#x27;Albert Einstein&#x27;],
 [&#x27;“It is better to be hated for what you are than to be loved for what you are not.”&#x27;,
 &#x27;André Gide&#x27;],
 [&quot;“I have not failed. I&#x27;ve just found 10,000 ways that won&#x27;t work.”&quot;,
 &#x27;Thomas A. Edison&#x27;],
 [&quot;“A woman is like a tea bag; you never know how strong it is until it&#x27;s in hot water.”&quot;,
 &#x27;Eleanor Roosevelt&#x27;],
 [&#x27;“A day without sunshine is like, you know, night.”&#x27;,
 &#x27;Steve Martin&#x27;]]
</code></pre>
<h3 id="storingtheretrieveddata">Storing the Retrieved Data</h3>
Once we have acquired the data, we can store it in whatever format we want e.g, a CSV file, SQL database, or NoSQL database. To be strict, this step shouldn't count as a part of the scraping process, but still, I'll cover it briefly for the sake of completeness.
I'd say the most popular way of storing scraped data is storing them as CSV spreadsheets, so I'll show you how to do just that, very briefly. I won't go into the details, for that you should refer to the official Python documentation. So without further ado, let's jump into the code.
<pre><code class="hljs">import csv

with open(&#x27;quotes.csv&#x27;, &#x27;w&#x27;) as csv_file:
 writer = csv.writer(csv_file, delimiter=&#x27;,&#x27;)
 for quote in scraped:
 writer.writerow(quote)
</code></pre>
As we can see, the code is pretty self-explanatory. We are creating a CSV <code>writer</code> object from the opened quotes.csv file, and then writing the quotes one by one using the <code>writerow</code> function. As it's evident, the <code>writerow</code> function accepts a list as input and then writes that to the CSV as a row.
<h2 id="conclusionandnextsteps">Conclusion and Next Steps</h2>
This tutorial should help you understand what scraping is basically about while learning to implement a simple scraper yourself. This kind of scraper should suffice for simple automation or small-scale data retriever. But if you want to extract large amounts of data efficiently, you should look into scraping frameworks, especially <a rel="nofollow" target="_blank" href="https://scrapy.org/">Scrapy</a>. It'll help you write very fast, efficient scrapers using a few lines of code. Whatever framework you use, underneath that shiny surface that framework is also using these very basic scraping principles, so understanding this tutorial should help you build the foundational knowledge for your scrapping adventures.

Introduction to Web Scraping with Python

<h2 id="introduction">Introduction</h2>
Classification is a large domain in the field of statistics and machine learning. Generally, classification can be broken down into two areas:
<ol>
<li>
Binary classification, where we wish to group an outcome into one of two groups.
</li>
<li>
Multi-class classification, where we wish to group an outcome into one of multiple (more than two) groups.
</li>
</ol>
In this post, the main focus will be on using a variety of classification algorithms across both of these domains, less emphasis will be placed on the theory behind them.
We can use libraries in Python such as <a rel="nofollow" target="_blank" href="http://scikit-learn.org/stable/">Scikit-Learn</a> for machine learning models, and <a rel="nofollow" target="_blank" href="https://pandas.pydata.org/">Pandas</a> to import data as data frames.
These can easily be installed and imported into Python with <code>pip</code>:
<pre><code class="hljs">$ python3 -m pip install sklearn
$ python3 -m pip install pandas
</code></pre>
<pre><code class="hljs">import sklearn as sk
import pandas as pd
</code></pre>
<h2 id="binaryclassification">Binary Classification</h2>
For binary classification, we are interested in classifying data into one of two binary groups - these are usually represented as 0's and 1's in our data.
We will look at data regarding coronary heart disease (CHD) in South Africa. The goal is to use different variables such as tobacco usage, family history, ldl cholesterol levels, alcohol usage, obesity and more.
A full description of this dataset is available in the &quot;Data&quot; section of the <a rel="nofollow" target="_blank" href="https://web.stanford.edu/~hastie/ElemStatLearn/">Elements of Statistical Learning</a> website.
The code below reads the data into a Pandas data frame, and then separates the data frame into a <code>y</code> vector of the response and an <code>X</code> matrix of explanatory variables:
<pre><code class="hljs">import pandas as pd
import os

os.chdir(&#x27;/Users/stevenhurwitt/Documents/Blog/Classification&#x27;)
heart = pd.read_csv(&#x27;SAHeart.csv&#x27;, sep=&#x27;,&#x27;, header=0)
heart.head()

y = heart.iloc[:,9]
X = heart.iloc[:,:9]
</code></pre>
When running this code, just be sure to change the file system path on line 4 to suit your setup.
<table border="1" class="table table-striped">
 <thead>
 <tr style="text-align: right;">
 <th></th>
 <th>sbp</th>
 <th>tobacco</th>
 <th>ldl</th>
 <th>adiposity</th>
 <th>famhist</th>
 <th>typea</th>
 <th>obesity</th>
 <th>alcohol</th>
 <th>age</th>
 <th>chd</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <th>0</th>
 <td>160</td>
 <td>12.00</td>
 <td>5.73</td>
 <td>23.11</td>
 <td>1</td>
 <td>49</td>
 <td>25.30</td>
 <td>97.20</td>
 <td>52</td>
 <td>1</td>
 </tr>
 <tr>
 <th>1</th>
 <td>144</td>
 <td>0.01</td>
 <td>4.41</td>
 <td>28.61</td>
 <td>0</td>
 <td>55</td>
 <td>28.87</td>
 <td>2.06</td>
 <td>63</td>
 <td>1</td>
 </tr>
 <tr>
 <th>2</th>
 <td>118</td>
 <td>0.08</td>
 <td>3.48</td>
 <td>32.28</td>
 <td>1</td>
 <td>52</td>
 <td>29.14</td>
 <td>3.81</td>
 <td>46</td>
 <td>0</td>
 </tr>
 <tr>
 <th>3</th>
 <td>170</td>
 <td>7.50</td>
 <td>6.41</td>
 <td>38.03</td>
 <td>1</td>
 <td>51</td>
 <td>31.99</td>
 <td>24.26</td>
 <td>58</td>
 <td>1</td>
 </tr>
 <tr>
 <th>4</th>
 <td>134</td>
 <td>13.60</td>
 <td>3.50</td>
 <td>27.78</td>
 <td>1</td>
 <td>60</td>
 <td>25.99</td>
 <td>57.34</td>
 <td>49</td>
 <td>1</td>
 </tr>
 </tbody>
</table>
<h3 id="logisticregression">Logistic Regression</h3>
<a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Logistic_regression">Logistic Regression</a> is a type of <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Generalized_linear_model">Generalized Linear Model</a> (GLM) that uses a logistic function to model a binary variable based on any kind of independent variables.
To fit a binary logistic regression with <code>sklearn</code>, we use the <a rel="nofollow" target="_blank" href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html">LogisticRegression</a> module with <code>multi_class</code> set to <code>&quot;ovr&quot;</code> and fit <code>X</code> and <code>y</code>.
We can then use the <code>predict</code> method to predict probabilities of new data, as well as the <code>score</code> method to get the mean prediction accuracy:
<pre><code class="hljs">import sklearn as sk
from sklearn.linear_model import LogisticRegression
import pandas as pd
import os

os.chdir(&#x27;/Users/stevenhurwitt/Documents/Blog/Classification&#x27;)
heart = pd.read_csv(&#x27;SAHeart.csv&#x27;, sep=&#x27;,&#x27;,header=0)
heart.head()

y = heart.iloc[:,9]
X = heart.iloc[:,:9]

LR = LogisticRegression(random_state=0, solver=&#x27;lbfgs&#x27;, multi_class=&#x27;ovr&#x27;).fit(X, y)
LR.predict(X.iloc[460:,:])
round(LR.score(X,y), 4)
</code></pre>
<pre><code class="hljs">array([1, 1])
</code></pre>
<h3 id="supportvectormachines">Support Vector Machines</h3>
<a target="_blank" href="/implementing-svm-and-kernel-svm-with-pythons-scikit-learn/">Support Vector Machines</a> (SVMs) are a type of classification algorithm that are more flexible - they can do linear classification, but can use other non-linear basis functions. The following example uses a linear classifier to fit a hyperplane that separates the data into two classes:
<pre><code class="hljs">import sklearn as sk
from sklearn import svm
import pandas as pd
import os

os.chdir(&#x27;/Users/stevenhurwitt/Documents/Blog/Classification&#x27;)
heart = pd.read_csv(&#x27;SAHeart.csv&#x27;, sep=&#x27;,&#x27;,header=0)

y = heart.iloc[:,9]
X = heart.iloc[:,:9]

SVM = svm.LinearSVC()
SVM.fit(X, y)
SVM.predict(X.iloc[460:,:])
round(SVM.score(X,y), 4)
</code></pre>
<pre><code class="hljs">array([0, 1])
</code></pre>
<h3 id="randomforests">Random Forests</h3>
<a target="_blank" href="/random-forest-algorithm-with-python-and-scikit-learn/">Random Forests</a> are an ensemble learning method that fit multiple <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Decision_tree">Decision Trees</a> on subsets of the data and average the results. We can again fit them using <code>sklearn</code>, and use them to predict outcomes, as well as get mean prediction accuracy:
<pre><code class="hljs">import sklearn as sk
from sklearn.ensemble import RandomForestClassifier

RF = RandomForestClassifier(n_estimators=100, max_depth=2, random_state=0)
RF.fit(X, y)
RF.predict(X.iloc[460:,:])
round(RF.score(X,y), 4)
</code></pre>
<pre><code class="hljs">0.7338
</code></pre>
<h3 id="neuralnetworks">Neural Networks</h3>
<a target="_blank" href="/introduction-to-neural-networks-with-scikit-learn/">Neural Networks</a> are a machine learning algorithm that involves fitting many hidden layers used to represent neurons that are connected with synaptic activation functions. These essentially use a very simplified model of the brain to model and predict data.
We use <code>sklearn</code> for consistency in this post, however libraries such as <a rel="nofollow" target="_blank" href="https://www.tensorflow.org/">TensorFlow</a> and <a rel="nofollow" target="_blank" href="https://keras.io/">Keras</a> are more suited to fitting and customizing neural networks, of which there are a few varieties used for different purposes:
<pre><code class="hljs">import sklearn as sk
from sklearn.neural_network import MLPClassifier

NN = MLPClassifier(solver=&#x27;lbfgs&#x27;, alpha=1e-5, hidden_layer_sizes=(5, 2), random_state=1)
NN.fit(X, y)
NN.predict(X.iloc[460:,:])
round(NN.score(X,y), 4)
</code></pre>
<pre><code class="hljs">0.6537
</code></pre>
<h2 id="multiclassclassification">Multi-Class Classification</h2>
While binary classification alone is incredibly useful, there are times when we would like to model and predict data that has more than two classes. Many of the same algorithms can be used with slight modifications.
Additionally, it is common to split data into training and test sets. This means we use a certain portion of the data to fit the model (the training set) and save the remaining portion of it to evaluate to the predictive accuracy of the fitted model (the test set).
There's no official rule to follow when deciding on a split proportion, though in most cases you'd want about 70% to be dedicated for the training set and around 30% for the test set.
To explore both multi-class classifications, as well as training/test data, we will look at another <a rel="nofollow" target="_blank" href="https://web.stanford.edu/~hastie/ElemStatLearn/data.html">dataset from the Elements of Statistical Learning website</a>. This is data used to determine which one of eleven vowel sounds were spoken:
<pre><code class="hljs">import pandas as pd

vowel_train = pd.read_csv(&#x27;vowel.train.csv&#x27;, sep=&#x27;,&#x27;, header=0)
vowel_test = pd.read_csv(&#x27;vowel.test.csv&#x27;, sep=&#x27;,&#x27;, header=0)

vowel_train.head()

y_tr = vowel_train.iloc[:,0]
X_tr = vowel_train.iloc[:,1:]

y_test = vowel_test.iloc[:,0]
X_test = vowel_test.iloc[:,1:]
</code></pre>
<table border="1" class="table table-striped">
 <thead>
 <tr style="text-align: right;">
 <th></th>
 <th>y</th>
 <th>x.1</th>
 <th>x.2</th>
 <th>x.3</th>
 <th>x.4</th>
 <th>x.5</th>
 <th>x.6</th>
 <th>x.7</th>
 <th>x.8</th>
 <th>x.9</th>
 <th>x.10</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <th>0</th>
 <td>1</td>
 <td>-3.639</td>
 <td>0.418</td>
 <td>-0.670</td>
 <td>1.779</td>
 <td>-0.168</td>
 <td>1.627</td>
 <td>-0.388</td>
 <td>0.529</td>
 <td>-0.874</td>
 <td>-0.814</td>
 </tr>
 <tr>
 <th>1</th>
 <td>2</td>
 <td>-3.327</td>
 <td>0.496</td>
 <td>-0.694</td>
 <td>1.365</td>
 <td>-0.265</td>
 <td>1.933</td>
 <td>-0.363</td>
 <td>0.510</td>
 <td>-0.621</td>
 <td>-0.488</td>
 </tr>
 <tr>
 <th>2</th>
 <td>3</td>
 <td>-2.120</td>
 <td>0.894</td>
 <td>-1.576</td>
 <td>0.147</td>
 <td>-0.707</td>
 <td>1.559</td>
 <td>-0.579</td>
 <td>0.676</td>
 <td>-0.809</td>
 <td>-0.049</td>
 </tr>
 <tr>
 <th>3</th>
 <td>4</td>
 <td>-2.287</td>
 <td>1.809</td>
 <td>-1.498</td>
 <td>1.012</td>
 <td>-1.053</td>
 <td>1.060</td>
 <td>-0.567</td>
 <td>0.235</td>
 <td>-0.091</td>
 <td>-0.795</td>
 </tr>
 <tr>
 <th>4</th>
 <td>5</td>
 <td>-2.598</td>
 <td>1.938</td>
 <td>-0.846</td>
 <td>1.062</td>
 <td>-1.633</td>
 <td>0.764</td>
 <td>0.394</td>
 <td>-0.150</td>
 <td>0.277</td>
 <td>-0.396</td>
 </tr>
 </tbody>
</table>
We will now fit models and test them as is normally done in statistics/machine learning: by training them on the training set and evaluating them on the test set.
Additionally, since this is multi-class classification, some arguments will have to be changed within each algorithm:
<pre><code class="hljs">import pandas as pd
import sklearn as sk
from sklearn.linear_model import LogisticRegression
from sklearn import svm
from sklearn.ensemble import RandomForestClassifier
from sklearn.neural_network import MLPClassifier

vowel_train = pd.read_csv(&#x27;vowel.train.csv&#x27;, sep=&#x27;,&#x27;,header=0)
vowel_test = pd.read_csv(&#x27;vowel.test.csv&#x27;, sep=&#x27;,&#x27;,header=0)

y_tr = vowel_train.iloc[:,0]
X_tr = vowel_train.iloc[:,1:]

y_test = vowel_test.iloc[:,0]
X_test = vowel_test.iloc[:,1:]

LR = LogisticRegression(random_state=0, solver=&#x27;lbfgs&#x27;, multi_class=&#x27;multinomial&#x27;).fit(X_tr, y_tr)
LR.predict(X_test)
round(LR.score(X_test,y_test), 4)

SVM = svm.SVC(decision_function_shape=&quot;ovo&quot;).fit(X_tr, y_tr)
SVM.predict(X_test)
round(SVM.score(X_test, y_test), 4)

RF = RandomForestClassifier(n_estimators=1000, max_depth=10, random_state=0).fit(X_tr, y_tr)
RF.predict(X_test)
round(RF.score(X_test, y_test), 4)

NN = MLPClassifier(solver=&#x27;lbfgs&#x27;, alpha=1e-5, hidden_layer_sizes=(150, 10), random_state=1).fit(X_tr, y_tr)
NN.predict(X_test)
round(NN.score(X_test, y_test), 4)
</code></pre>
<pre><code class="hljs">0.5455
</code></pre>
Although the implementations of these models were rather naive (in practice there are a variety of parameters that can and should be varied for each model), we can still compare the predictive accuracy across the models. This will tell us which one is the most accurate for this specific training and test dataset:
<table class="table table-striped">
 <thead>
 <tr>
 <th>Model</th>
 <th>Predictive Accuracy</th>
 </tr>
 </thead>
 <tbody>
 <tr>
 <td>Logistic Regression</td>
 <td>46.1%</td>
 </tr>
 <tr>
 <td>Support Vector Machine</td>
 <td>64.07%</td>
 </tr>
 <tr>
 <td>Random Forest</td>
 <td>57.58%</td>
 </tr>
 <tr>
 <td>Neural Network</td>
 <td>54.55%</td>
 </tr>
 </tbody>
</table>
This shows us that for the vowel data, an SVM using the default radial basis function was the most accurate.
<h2 id="conclusion">Conclusion</h2>
To summarize this post, we began by exploring the simplest form of classification: binary. This helped us to model data where our response could take one of two states.
We then moved further into multi-class classification, when the response variable can take any number of states.
We also saw how to fit and evaluate models with training and test sets. Furthermore, we could explore additional ways to refine model fitting among various algorithms.

Steven Hurwitt

Classification in Python with Scikit-Learn and Pandas

<h2 id="introduction">Introduction</h2>
This article is the final tutorial of a series describing the often forgotten about methods of the Java language's base Object class. The following are the methods of the base Java Object which are present in all Java objects due to the implicit inheritance of Object.
<ul>
<li><a target="_blank" href="/javas-object-methods-tostring/">toString</a></li>
<li><a target="_blank" href="/javas-object-methods-getclass/">toClass</a></li>
<li><a target="_blank" href="/javas-object-methods-equals-object/">equals</a></li>
<li><a target="_blank" href="/javas-object-methods-hashcode/">hashCode</a></li>
<li><a target="_blank" href="/javas-object-methods-clone/">clone</a></li>
<li><a target="_blank" href="/javas-object-methods-finalize">finalize</a></li>
<li>wait &amp; notify (you are here)</li>
</ul>
The focus of this article are the <code>Object#wait()</code> and <code>Object#notify</code> methods (and their variations) which are used to communicate and coordinate control between threads of a multi-threaded application.
<h2 id="basicoverview">Basic Overview</h2>
The <code>Object#wait()</code> method is used within a synchronization block or member method and causes the thread it is called in to wait indefinitely until another thread calls <code>Object#notify()</code> (or it's variation <code>Object#notifyAll()</code>) on the same object that the original <code>Object#wait()</code> was called on.
Wait has three variations:
<ul>
<li><code>void wait()</code> - waits until either <code>Object#notify()</code> or <code>Object#noifyAll()</code> is called</li>
<li><code>void wait(long timeout)</code> - waits for either the milliseconds that are specified to elapse or notify is called</li>
<li><code>void wait(long timeout, int nanos)</code> - same as the one above but, with the extra precision of the nanoseconds supplied</li>
</ul>
The <code>Object#notify()</code> is used to wake up a single thread that is waiting on an object that <code>wait</code> was called on. Note that in the case of multiple threads waiting on the object the woken up thread is selected randomly by the operating system
Notify has three variations:
<ul>
<li><code>void notify()</code> - randomly selects and wakes up a thread waiting on the object <code>wait</code> was called on</li>
<li><code>void notifyAll()</code> - wakes up all threads waiting on the object</li>
</ul>
<h2 id="theclassicproducerconsumerproblem">The Classic Producer Consumer Problem</h2>
Like all things in programming, these concepts of using <code>Object#wait()</code> and <code>Object#notify()</code> are best understood through a carefully thought out example. In this example I am going to implement a multi-threaded producer / consumer application to demonstrate the use of <code>wait</code> and <code>notify</code>. This application will use a producer to generate a random integer that is to represent a number of even random numbers that consumer threads will need to randomly generate.
The class design and specifications for this example are as follows:
<code>NumberProducer</code>: produce a random integer between 1-100 that represents the number of random even numbers a consumer will need to generate. The random number is to be placed in a queue by the producer where a consumer can retrieve it and go to work producing random even numbers
<code>NumberQueue</code>: a queue that will enqueue a number from the producer and dequeue that number to a consumer eagerly awaiting the chance to generate a series of random even number
<code>NumberConsumer</code>: a consumer that will retrieve a number from the queue representing the number of random even integers to generate
The <code>NumberQueue</code>.
<pre><code class="hljs">import java.util.LinkedList;

public class NumberQueue {
 private LinkedList&lt;Integer&gt; numQueue = new LinkedList&lt;&gt;();

 public synchronized void pushNumber(int num) {
 numQueue.addLast(num);
 notifyAll();
 }

 public synchronized int pullNumber() {
 while(numQueue.size() == 0) {
 try {
 wait();
 } catch (InterruptedException e) {
 e.printStackTrace();
 }
 }
 return numQueue.removeFirst();
 }

 public synchronized int size() {
 return numQueue.size();
 }
}
</code></pre>
<code>NumberQueue</code> has a <code>LinkedList</code> that will contain the numbers data internally and provide access to it via three synchronized methods. Here the methods are synchronized so that a lock will be placed on the access to the <code>LinkedList</code> data structure guaranteeing that at most only one thread can have control over the method at a time. Furthermore, the <code>NumberQueue#pushNumber</code> method calls it's inherited <code>Object#notifyAll</code> method upon adding a new number letting the consumers know work is available. Similarly, the <code>NumberQueue#pullNumber</code> method uses a loop along with a call to it's inherited <code>Object#wait</code> method to suspend execution if it has no numbers in its list until it has data for consumers.
The <code>NumberProducer</code> class.
<pre><code class="hljs">import java.util.Random;

public class NumberProducer extends Thread {
 private int maxNumsInQueue;
 private NumberQueue numsQueue;

 public NumberProducer(int maxNumsInQueue, NumberQueue numsQueue) {
 this.maxNumsInQueue = maxNumsInQueue;
 this.numsQueue = numsQueue;
 }

 @Override
 public void run() {
 System.out.println(getName() + &quot; starting to produce ...&quot;);
 Random rand = new Random();
 // continuously produce numbers for queue
 while(true) {
 if (numsQueue.size() &lt; maxNumsInQueue) {
 // random numbers 1-100
 int evenNums = rand.nextInt(99) + 1;
 numsQueue.pushNumber(evenNums);
 System.out.println(getName() + &quot; adding &quot; + evenNums);
 }
 try {
 Thread.sleep(800);
 } catch (InterruptedException e) {
 e.printStackTrace();
 }
 }
 }
}
</code></pre>
<code>NumberProducer</code> inherits the <code>Thread</code> class and contains a field called <code>maxNumsInQueue</code> that puts a limit on the number of items the queue can hold, and it also has a reference to the <code>NumberQueue</code> instance via its <code>numsQueue</code> field, which it gains via a single constructor. It overrides the <code>Thread#run</code> method which contains an infinite loop that adds a random integer between 1-100 to the <code>NumberQueue</code> every 800 milliseconds. This happens as long as the queue is within its limit, thus populating the queue and governing work for the consumers.
The <code>NumberConsumer</code> class.
<pre><code class="hljs">import java.util.ArrayList;
import java.util.List;
import java.util.Random;
import java.util.StringJoiner;

public class NumberConsumer extends Thread {
 private NumberQueue numQueue;

 public NumberConsumer(NumberQueue numQueue) {
 this.numQueue = numQueue;
 }

 @Override
 public void run() {
 System.out.println(getName() + &quot; starting to consume ...&quot;);
 Random rand = new Random();
 // consume forever
 while(true) {
 int num = numQueue.pullNumber();
 List&lt;Integer&gt; evens = new ArrayList();
 while(evens.size() &lt; num) {
 int randInt = rand.nextInt(999) + 1;
 if (randInt % 2 == 0) {
 evens.add(randInt);
 }
 }
 String s = &quot; &quot; + getName() + &quot; found &quot; + num + &quot; evens [&quot;;
 StringJoiner nums = new StringJoiner(&quot;,&quot;);
 for (int randInt : evens) {
 nums.add(Integer.toString(randInt));
 }
 s += nums.toString() + &quot;]&quot;;
 System.out.println(s);
 }
 }
}

</code></pre>
<code>NumberConsumer</code> also inherits from <code>Thread</code> and maintains a reference to the <code>NumberQueue</code> via the <code>numQueue</code> reference field obtained via its constructor. Its over-ridden run method similarly contains an infinite loop, which inside it pulls a number off the queue as they are available. Once it receives the number it enters another loop which produces random integers from 1-1000, tests it for evenness and adds them to a list for later display.
Once it finds the required number of random even numbers specified by the <code>num</code> variable pulled off the queue it exits the inner loop and proclaims to the console its findings.
The <code>EvenNumberQueueRunner </code> class.
<pre><code class="hljs">public class EvenNumberQueueRunner {

 public static void main(String[] args) {
 final int MAX_QUEUE_SIZE = 5;

 NumberQueue queue = new NumberQueue();
 System.out.println(&quot; NumberProducer thread NumberConsumer threads&quot;);
 System.out.println(&quot;============================= =============================&quot;);

 NumberProducer producer = new NumberProducer(MAX_QUEUE_SIZE, queue);
 producer.start();

 // give producer a head start
 try {
 Thread.sleep(3000);
 } catch (InterruptedException e) {
 e.printStackTrace();
 }

 NumberConsumer consumer1 = new NumberConsumer(queue);
 consumer1.start();

 NumberConsumer consumer2 = new NumberConsumer(queue);
 consumer2.start();
 }
}
</code></pre>
<code>EvenNumberQueueRunner</code> is the main class in this application which begins by instantiating the <code>NumberProducer</code> class and launches it as a thread. Then it gives it a 3 second head start to fill its queue with the max number of even numbers to be generated. Finally the <code>NumberConsumer</code> class is instantiated twice and launched as threads which then go off pulling numbers off the queue and creating the indicated number of even integers.
Example output from the program is shown here. Note that no two runs are likely to produce the same output as this application is purely random in nature from the numbers produced to the randomness that the operating system switches between active threads on the CPU.
<pre><code class="hljs"> NumberProducer thread NumberConsumer threads
============================= =============================
Thread-0 starting to produce ...
Thread-0 adding 8
Thread-0 adding 52
Thread-0 adding 79
Thread-0 adding 62
Thread-1 starting to consume ...
Thread-2 starting to consume ...
 Thread-1 found 8 evens [890,764,366,20,656,614,86,884]
 Thread-2 found 52 evens [462,858,266,190,764,686,36,730,628,916,444,370,860,732,188,652,274,608,912,940,708,542,760,194,642,192,22,36,622,174,66,168,264,472,228,972,18,486,714,244,214,836,206,342,388,832,8,666,946,116,342,62]
 Thread-2 found 62 evens [404,378,276,308,470,156,96,174,160,704,44,12,934,426,616,318,942,320,798,696,494,484,856,496,886,828,386,80,350,920,142,686,118,240,398,488,976,512,642,108,542,122,536,482,734,430,564,200,844,462,12,124,368,764,496,728,802,836,478,986,292,486]
 Thread-1 found 79 evens [910,722,352,656,250,974,602,342,144,952,916,188,286,468,618,496,764,642,506,168,966,274,476,744,142,348,784,164,346,344,48,862,754,896,896,784,574,464,134,192,446,524,424,710,128,756,934,672,816,604,186,18,432,250,466,144,930,914,670,434,764,176,388,534,448,476,598,984,536,920,282,478,754,750,994,60,466,382,208]
Thread-0 adding 73
 Thread-2 found 73 evens [798,692,698,280,688,174,528,632,528,278,80,746,790,456,352,280,574,686,392,26,994,144,166,806,750,354,586,140,204,144,664,214,808,214,218,414,230,364,986,736,844,834,826,564,260,684,348,76,390,294,740,550,310,364,460,816,650,358,206,892,264,890,830,206,976,362,564,26,894,764,726,782,122]
Thread-0 adding 29
 Thread-1 found 29 evens [274,600,518,222,762,494,754,194,128,354,900,226,120,904,206,838,258,468,114,622,534,122,178,24,332,432,966,712,104]
Thread-0 adding 65

... and on and on ...
</code></pre>
I would like to take a moment to explain my usage of the <code>notifyAll()</code> method within <code>NumberQueue#pushNumber</code> because my choice was not random. By using the <code>notifyAll()</code> method I am giving the two consumer threads equal chance at pulling a number off the queue to do work on rather than leaving it up to the OS to pick one over the other. This is important because if I had simply used <code>notify()</code> then there is a good chance that the thread the OS selects to access the queue is not yet ready to do more work and is working on it's last set of even numbers (ok, it's a little far fetched that it would still be trying to find up to a max of 1000 even numbers after 800 milliseconds but, hopefully you understand what I'm getting at). Basically what I want to make clear here is that in nearly all cases you should prefer the <code>notifyAll()</code> method over the <code>notify()</code> variant.
<h2 id="conclusion">Conclusion</h2>
In this final article of the series of Java Object class methods I have covered the purpose and usage of the variations of <code>wait</code> and <code>notify</code>. It should be said that these methods are fairly primitive and the Java concurrency mechanisms have evolved since then but, in my opinion, <code>wait</code> and <code>notify</code> still are a valuable set of tools to have in your Java programming tool belt.
As always, thanks for reading and don't be shy about commenting or critiquing below.