Dean Shaff

Asynchronous vs Synchronous Python Performance Analysis

<h2 id="introduction">Introduction</h2>
In this article, we will look into few approaches of exception handling in Spring REST applications.
This tutorial assumes that you have a basic knowledge of Spring and can create simple REST APIs using it.
If you'd like to read more about exceptions and custom exceptions in Java, we've covered it in detail in <a target="blank" href="/exception-handling-in-java-a-complete-guide-with-best-and-worst-practices/">Exception Handling in Java: A Complete Guide with Best and Worst Practices</a> and <a target="blank" href="/how-to-make-custom-exceptions-in-java/">How to Make Custom Exceptions in Java</a>.
<h2 id="whydoit">Why Do It?</h2>
Suppose we have a simple user service where we can fetch and update registered users. We have a simple model defined for the users:
<pre><code class="hljs">public class User {
 private int id;
 private String name;
 private int age;

 // Constructors, getters, and setters
</code></pre>
Let's create a REST controller with a mapping that expects an <code>id</code> and returns the <code>User</code> with the given <code>id</code> if present:
<pre><code class="hljs">@RestController
public class UserController {

 private static List&lt;User&gt; userList = new ArrayList&lt;&gt;();
 static {
 userList.add(new User(1, &quot;John&quot;, 24));
 userList.add(new User(2, &quot;Jane&quot;, 22));
 userList.add(new User(3, &quot;Max&quot;, 27));
 }

 @GetMapping(value = &quot;/user/{id}&quot;)
 public ResponseEntity&lt;?&gt; getUser(@PathVariable int id) {
 if (id &lt; 0) {
 return ResponseEntity.status(HttpStatus.BAD_REQUEST).build();
 }
 User user = findUser(id);
 if (user == null) {
 return ResponseEntity.status(HttpStatus.NOT_FOUND).build();
 }

 return ResponseEntity.ok(user);
 }

 private User findUser(int id) {
 return userList.stream().filter(user -&gt; user.getId().equals(id)).findFirst().orElse(null);
 }
}
</code></pre>
Apart from just finding the user, we also have to perform additional checks, like the <code>id</code> that's passed should always be greater than 0, otherwise we have to return a <code>BAD_REQUEST</code> status code.
Similarly, if the user is not found then we have to return a <code>NOT_FOUND</code> status code. Additionally, we might have to add text for some detail about the error to the client.
For each check, we have to create a <a rel="nofollow" target="_blank" href="https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/http/ResponseEntity.html">ResponseEntity</a> object having response codes and text according to our requirements.
We can easily see that these checks will have to be made multiple times as our APIs grows. For an example, suppose we are adding a new PATCH request mapping to update our users, we need to again create these <code>ResponseEntity</code> objects. This creates the problem of maintaining consistencies within the app.
So the problem we are trying to solve is the separation of concerns. Of course, we have to perform these checks in each <code>RequestMapping</code> but instead of handling validation/error scenarios and what responses need to be returned in each of them, we can simply throw an exception after a violation and these exceptions will be then handled separately.
Now, you can use built-in exceptions already provided by Java and <a rel="nofollow" target="_blank" href="www.spring.io">Spring</a>, or if needed you can create your own exceptions and throw them. This will also centralize our validation/error handling logic.
Additionally, we can't return default server error messages to the client when serving an API. Neither can we return stack traces that are convoluted and hard to understand for our clients. Proper exception handling with Spring is a very important aspect of building a good REST API.
Alongisde exception handling, <a target="_blank" href="/java-rest-api-documentation-with-swagger2">REST API Documentation</a> is a must.
<h2 id="exceptionhandlingviaresponsestatus">Exception Handling via @ResponseStatus</h2>
The <a rel="nofollow" target="_blank" href="https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/bind/annotation/ResponseStatus.html">@ResponseStatus</a> annotation can be used on methods and exception classes. It can be configured with a status code which would be applied to the HTTP response.
Let's create a custom exception to handle the situation when the user is not found. This will be a runtime exception hence we have to extend the <code>java.lang.RuntimeException</code> class.
We'll also mark this class with <code>@ResponseStatus</code>:
<pre><code class="hljs">@ResponseStatus(value = HttpStatus.NOT_FOUND, reason = &quot;User Not found&quot;)
public class UserNotFoundException extends RuntimeException {

}
</code></pre>
When Spring catches this Exception, it uses the configuration provided in <code>@ResponseStatus</code>.
Changing our controller to use the same:
<pre><code class="hljs"> @GetMapping(value = &quot;/user/{id}&quot;)
 public ResponseEntity&lt;?&gt; getUser(@PathVariable int id) {
 if (id &lt; 0) {
 return ResponseEntity.status(HttpStatus.BAD_REQUEST).build();
 }
 User user = findUser(id);
 return ResponseEntity.ok(user);
 }

 private User findUser(int id) {
 return userList.stream().filter(user -&gt; user.getId().equals(id)).findFirst().orElseThrow(() -&gt; new UserNotFoundException());
 }
</code></pre>
As we can see, the code is cleaner now with separation of concerns.
<h2 id="restcontrolleradviceandexceptionhandler">@RestControllerAdvice and @ExceptionHandler</h2>
Let's create a custom exception to handle validation checks. This again will be a <code>RuntimeException</code>:
<pre><code class="hljs">public class ValidationException extends RuntimeException {
 private static final long serialVersionUID = 1L;
 private String msg;

 public ValidationException(String msg) {
 this.msg = msg;
 }

 public String getMsg() {
 return msg;
 }
}
</code></pre>
<a rel="nofollow" target="_blank" href="https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/bind/annotation/RestControllerAdvice.html">@RestControllerAdvice</a> is a new feature of Spring that can be used to write common code for exception handling.
This is usually used in conjunction with <a rel="nofollow" target="_blank" href="https://docs.spring.io/spring/docs/current/javadoc-api/org/springframework/web/bind/annotation/ExceptionHandler.html">@ExceptionHandler</a> that actually handles different exceptions:
<pre><code class="hljs">@RestControllerAdvice
public class AppExceptionHandler {

 @ResponseBody
 @ExceptionHandler(value = ValidationException.class)
 public ResponseEntity&lt;?&gt; handleException(ValidationException exception) {
 return ResponseEntity.status(HttpStatus.BAD_REQUEST).body(exception.getMsg());
 }
}
</code></pre>
You can think of <code>RestControllerAdvice</code> as a sort of <a rel="nofollow" target="_blank" href="https://docs.spring.io/spring/docs/5.1.0.RELEASE/spring-framework-reference/core.html#aop">Aspect</a> in your Spring code. Whenever your Spring code throws an exception which has a handler defined in this class, appropriate logic could be written according to business needs.
Notice that unlike <code>@ResponseStatus</code> we could do many things with this approach, like logging our exceptions, notifying etc.
What if we wanted to update the age of an existing user? We have 2 validation checks that need to be made:
<ul>
<li>The <code>id</code> must be greater than 0</li>
<li>The <code>age</code> must be between 20 to 60</li>
</ul>
With that in mind, let's make an endpoint for just that:
<pre><code class="hljs"> @PatchMapping(value = &quot;/user/{id}&quot;)
 public ResponseEntity&lt;?&gt; updateAge(@PathVariable int id, @RequestParam int age) {
 if (id &lt; 0) {
 throw new ValidationException(&quot;Id cannot be less than 0&quot;);
 }
 if (age &lt; 20 || age &gt; 60) {
 throw new ValidationException(&quot;Age must be between 20 to 60&quot;);
 }
 User user = findUser(id);
 user.setAge(age);

 return ResponseEntity.accepted().body(user);
 }
</code></pre>
By default <code>@RestControllerAdvice</code> is applicable to the whole application but you can restrict it to a specific package, class or annotation.
For package level restriction you can do something like:
<pre><code class="hljs">@RestControllerAdvice(basePackages = &quot;my.package&quot;)
</code></pre>
or
<pre><code class="hljs">@RestControllerAdvice(basePackageClasses = MyController.class)
</code></pre>
To apply to specific class:
<pre><code class="hljs">@RestControllerAdvice(assignableTypes = MyController.class)
</code></pre>
To apply it to controllers with certain annotations:
<pre><code class="hljs">@RestControllerAdvice(annotations = RestController.class)
</code></pre>
<h2 id="responseentityexceptionhandler">ResponseEntityExceptionHandler</h2>
<a rel="nofollow" target="_blank" href="https://docs.spring.io/spring-framework/docs/current/javadoc-api/org/springframework/web/servlet/mvc/method/annotation/ResponseEntityExceptionHandler.html">ResponseEntityExceptionHandler</a> provides some basic handling for Spring exceptions.
We can extend this class and override methods to customize them:
<pre><code class="hljs">@RestControllerAdvice
public class GlobalResponseEntityExceptionHandler extends ResponseEntityExceptionHandler {

 @Override
 protected ResponseEntity&lt;Object&gt; handleMissingServletRequestParameter(MissingServletRequestParameterException ex, HttpHeaders headers, HttpStatus status, WebRequest request) {
 return errorResponse(HttpStatus.BAD_REQUEST, &quot;Required request params missing&quot;);
 }

 private ResponseEntity&lt;Object&gt; errorResponse(HttpStatus status, String message) {
 return ResponseEntity.status(status).body(message);
 }
}
</code></pre>
To register this class for exception handling we have to annotate it with <code>@ResponseControllerAdvice</code>.
Again there are many things that can be done here and it depends on your requirements.
<h2 id="whichtousewhen">Which to Use When?</h2>
As you can see, Spring provides us with different options to do exception handling in our apps. You can use one or a combination of them based on your needs. Here is the rule of thumb:
<ul>
<li>For custom exceptions where your status code and message are fixed, consider adding <code>@ResponseStatus</code> to them.</li>
<li>For exceptions where you need to do some logging, use <code>@RestControllerAdvice</code> with <code>@ExceptionHandler</code>. You also have more controller over your response text here.</li>
<li>For changing the behavior of the default Spring exception responses, you can extend the <code>ResponseEntityExceptionHandler</code> class.</li>
</ul>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Note: Be careful in mixing these options in the same application. If the same thing is handled at more than one places, you might get a different behavior than expected.

 </div>
 </div>
 </div>
 <h2 id="conclusion">Conclusion</h2>
In this tutorial, we discussed several ways to implement an exception handling mechanism for a REST API in Spring.
As always, the code for the examples used in this article can be found on <a rel="nofollow" target="_blank" href="https://github.com/StackAbuse/learning-spring/tree/master/exception-handling">Github</a>.

Dhananjay Singh

Exception Handling in Spring

<h2 id="introduction">Introduction</h2>
Converting a primitive <code>int</code>, or its respective wrapper class <code>Integer</code>, to a <code>String</code> is a common and simple operation. The same goes for the other way around, <a target="_blank" href="/java-convert-string-to-integer">converting a String to Integer</a>.
<h2 id="convertingintegertostring">Converting Integer to String</h2>
When converting an int or Integer to a String, there are four approaches. The <code>String</code> class provides a couple of methods - <code>valueOf()</code> and <code>format()</code> for this purpose, though the already mentioned <code>Integer</code> class also offers a <code>toString()</code> method that takes care of this problem. Additionally, you could also rely on <code>StringBuilder</code>'s <code>append()</code> method, though this isn't a commonly used scenario:
<ul>
<li><code>String.valueOf(int i)</code> - This approach is considered best practice due to simplicity. This method accepts other types as well - <code>boolean</code>, <code>char</code>, <code>char[]</code>, <code>double</code>, <code>float</code>, <code>long</code>, <code>Object</code></li>
<li><code>Integer.toString(int i)</code> - This approach is older than <code>valueOf()</code> and simply utilizes the method every Java object has to return a String representing the given int. This approach, unlike the previous, can return a <code>NullPointerException</code> though.</li>
<li><code>String.format(String format, Object... args)</code> - Returns a String formatted according to the format specifier and the following arguments</li>
<li><code>StringBuilder.append(int i).toString()</code> - Same as the <code>valueOf()</code> method, this method accepts all primitive types with the addition of some other types like <code>String</code>, <code>StringBuffer</code> and <code>CharSequence</code>.</li>
</ul>
<h3 id="stringvalueof">String.valueOf()</h3>
The <code>valueOf()</code> method is the static method of String class that returns the String representation of the specified type.
There are many types allowed here:
<ul>
<li>Object</li>
<li>char[]</li>
<li>boolean</li>
<li>char</li>
<li>int</li>
<li>long</li>
<li>float</li>
<li>double</li>
</ul>
But we'll be focusing on <code>int</code> for this tutorial. The representation returned exactly matches the representation returned by <code>Integer.toString()</code> when passing the same argument:
<pre><code class="hljs">int i = 12345;

String string = String.valueOf(i);
String otherString = Integer.toString(i);

System.out.println(string.equals(otherString));
System.out.println(string == otherString);
System.out.println(string);
System.out.println(otherString);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">true
false
12345
12345
</code></pre>
This makes sense since both of these methods return a new String. The <code>equals()</code> method returns true, because their value is the same, whereas <code>==</code> returns false since their reference variables don't point to the same object in memory.
Using <code>valueOf()</code> keeps your code consistent across conversion of one data type to another. In the case of an Integer, the <code>valueOf()</code> method is also able to cache frequently used numbers e.g. from -127 to 128, in order to speed up the conversion and reduce memory.
Because of this, it's encouraged to use the <code>valueOf()</code> method for both String and Integer conversion.
<h3 id="integertostring">Integer.toString()</h3>
This approach utilizes one of the most common Java methods - <code>toString()</code> shared amongst all objects.
The method has many usages and warrants a detailed explanation. If you'd like to read more about it, we've got <a target="_blank" href="/javas-object-methods-tostring/">a great article</a> already covering it!
In this case, the method returns a String object representing the specified int. The argument is converted to signed decimal representation and returned as a String:
<pre><code class="hljs">int i = 12345;

String string = Integer.toString(i);
String otherString = new Integer(i).toString();

System.out.println(string.equals(otherString));
System.out.println(string == otherString);
System.out.println(string);
System.out.println(otherString);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">true
false
12345
12345
</code></pre>
If your variable is of primitive type (int), it is better to use <code>Integer.toString(int)</code> or <code>String.valueOf(int)</code>. But if your variable is already an instance of <code>Integer</code> (wrapper class of the primitive type int), it is better to just invoke its <code>toString()</code> method as shown above.
<h3 id="stringformat">String.format()</h3>
Returns a String formatted according to the format specifier and the following arguments. While the purpose of this method isn't exactly to convert, but rather format a String, it can also be used for conversion.
There are quite a few specifiers to choose from:
<ul>
<li><code>%a</code> - Hex output of a floating point number</li>
<li><code>%b</code> - true if not null, false if null</li>
<li><code>%c</code> - Unicode character</li>
<li><code>%d</code> - Decimal Integer</li>
<li><code>%e</code> - Scientific notation of a decimal Integer</li>
<li><code>%f</code> - Decimal floating point number</li>
<li><code>%g</code> - Same as above, but possibly scientific, depending on the value</li>
<li><code>%h</code> - Hex String of <a target="_blank" href="/javas-object-methods-hashcode/">hashCode()</a> return value</li>
<li><code>%n</code> - Line separator</li>
<li><code>%o</code> - Octal Integer</li>
<li><code>%s</code> - String</li>
<li><code>%t</code> - Date/Time conversion</li>
<li><code>%x</code> - Hex String</li>
</ul>
We'll be focusing on <code>%d</code> for this tutorial:
<pre><code class="hljs">int i = 12345;

String string = String.format(&quot;%d&quot;, i);

System.out.println(string);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">12345
</code></pre>
The <code>format()</code> method throws an <code>IllegalFormatConversionException</code> if you pass an incompatible type e.g. passing <code>Integer</code> for specifier <code>%s</code>, which expects a String object.
Some of the most common specifiers are <code>%s</code> for String, <code>%d</code> for decimal integer and <code>%n</code>.
<h3 id="stringbuilderandstringbuffer">StringBuilder and StringBuffer</h3>
Both <code>StringBuffer</code> and <code>StringBuilder</code> are classes used to concatenate multiple values into a single String.
<code>StringBuffer</code> is thread safe but slower, whereas <code>StringBuilder</code> isn't thread safe but is faster.
<pre><code class="hljs">int i = 12345;

String string = new StringBuilder().append(i).toString();
String otherString = new StringBuffer().append(i).toString();

System.out.println(string.equals(otherString));
System.out.println(string == otherString);
System.out.println(string);
System.out.println(otherString);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">true
false
12345
12345
</code></pre>
<h2 id="conclusion">Conclusion</h2>
We've covered one of the fundamental topics of Java and common problem developers face - Converting an int or Integer to a String using tools shipped with the JDK.

Vuk Skobalj

Java Convert Integer to String

<h2 id="introduction">Introduction</h2>
Converting a <code>String</code> to an <code>int</code>, or its respective wrapper class <code>Integer</code>, is a common and simple operation. The same goes for the other way around, <a target="_blank" href="/java-convert-integer-to-string">converting a Integer to String</a>.
There are multiple ways to achieve this simple conversion using methods built-in to the JDK.
<h2 id="convertingstringtointeger">Converting String to Integer</h2>
For converting String to Integer or int, there are four built-in approaches. The wrapper class <code>Integer</code> provides a few methods specifically for this use - <code>parseInt()</code>, <code>valueOf()</code> and <code>decode()</code>, although we can also use its constructor and pass a String into it.
These three methods have different return types:
<ul>
<li><code>parseInt()</code>- returns primitive <code>int</code>.</li>
<li><code>valueOf()</code> - returns a new or cached instance of <code>Integer</code></li>
<li><code>decode()</code> - returns a new or cached instance of <code>Integer</code></li>
</ul>
That being said, it's valid to raise a question:
<blockquote>
&quot;What's the difference between <code>valueOf()</code> and <code>decode()</code> then?
</blockquote>
The difference is that <code>decode()</code> also accepts other number representations, other than regular decimal - hex digits, octal digits, etc.
Note: It's better practice to instantiate an <code>Integer</code> with the help of the <code>valueOf()</code> method rather than relying on the constructor. This is because the <code>valueOf()</code> method will return a cached copy for any value between -128 and 127 inclusive, and by doing so will reduce the memory footprint.
<h3 id="integerparseint">Integer.parseInt()</h3>
The <code>parseInt()</code> method ships in two flavors:
<ul>
<li><code>parseInt(String s)</code> - Accepting the String we'd like to parse</li>
<li><code>parseInt(String s, int radix)</code> - Accepting the String as well as the base of the numeration system</li>
</ul>
The <code>parseInt()</code> method converts the input String into a primitive <code>int</code> and throws a <code>NumberFormatException</code> if the String cannot be converted:
<pre><code class="hljs">String string1 = &quot;100&quot;;
String string2 = &quot;100&quot;;
String string3 = &quot;Google&quot;;
String string4 = &quot;20&quot;;

int number1 = Integer.parseInt(string1);
int number2 = Integer.parseInt(string2, 16); 
int number3 = Integer.parseInt(string3, 32); 
int number4 = Integer.parseInt(string4, 8); 

System.out.println(&quot;Parsing String \&quot;&quot; + string1 + &quot;\&quot;: &quot; + number1);
System.out.println(&quot;Parsing String \&quot;&quot; + string2 + &quot;\&quot; in base 16: &quot; + number2);
System.out.println(&quot;Parsing String \&quot;&quot; + string3 + &quot;\&quot; in base 32: &quot; + number3);
System.out.println(&quot;Parsing String \&quot;&quot; + string4 + &quot;\&quot; in base 8: &quot; + number4);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">Parsing String &quot;100&quot;: 100
Parsing String &quot;100&quot; in base 16: 256
Parsing String &quot;Google&quot; in base 32: 562840238
Parsing String &quot;20&quot; in base 8: 16
</code></pre>
<h3 id="integervalueof">Integer.valueOf()</h3>
The <code>valueOf()</code> ships in three flavors:
<ul>
<li><code>valueOf(String s)</code> - Accepts a String and parses it into an Integer</li>
<li><code>valueOf(int i)</code> - Accepts an int and parses it into an Integer</li>
<li><code>valueOf(String s, int radix)</code> - Accepts a String and returns an Integer representing the value and then parses it with the given base</li>
</ul>
The <code>valueOf()</code> method, unlike the <code>parseInt()</code> method, returns an <code>Integer</code> instead of a primitive <code>int</code> and also throws a <code>NumberFormatException</code> if the String cannot be converted properly and only accepts decimal numbers:
<pre><code class="hljs">int i = 10;
String string1 = &quot;100&quot;;
String string2 = &quot;100&quot;;
String string3 = &quot;Google&quot;;
String string4 = &quot;20&quot;;

int number1 = Integer.valueOf(i);
int number2 = Integer.valueOf(string1); 
int number3 = Integer.valueOf(string3, 32); 
int number4 = Integer.valueOf(string4, 8); 

System.out.println(&quot;Parsing int &quot; + i + &quot;: &quot; + number1);
System.out.println(&quot;Parsing String \&quot;&quot; + string1 + &quot;\&quot;: &quot; + number2);
System.out.println(&quot;Parsing String \&quot;&quot; + string3 + &quot;\&quot; in base 32: &quot; + number3);
System.out.println(&quot;Parsing String \&quot;&quot; + string4 + &quot;\&quot; in base 8: &quot; + number4);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">Parsing int 10: 10
Parsing String &quot;100&quot;: 100
Parsing String &quot;Google&quot; in base 32: 562840238
Parsing String &quot;20&quot; in base 8: 16
</code></pre>
<h3 id="integerdecode">Integer.decode()</h3>
The <code>decode()</code> method accepts a single parameter and comes in one flavor:
<ul>
<li><code>decode(String s)</code> - Accepts a String and decodes it into an Integer</li>
</ul>
The <code>decode()</code> method accepts decimal, hexadecimal and octal numbers, but doesn't support binary:
<pre><code class="hljs">String string1 = &quot;100&quot;;
String string2 = &quot;50&quot;;
String string3 = &quot;20&quot;;

int number1 = Integer.decode(string1);
int number2 = Integer.decode(string2); 
int number3 = Integer.decode(string3); 

System.out.println(&quot;Parsing String \&quot;&quot; + string1 + &quot;\&quot;: &quot; + number2);
System.out.println(&quot;Parsing String \&quot;&quot; + string2 + &quot;\&quot;: &quot; + number2);
System.out.println(&quot;Parsing String \&quot;&quot; + string3 + &quot;\&quot;: &quot; + number3);
</code></pre>
Running this piece of code will yield:
<pre><code class="hljs">Parsing String &quot;100&quot;: 50
Parsing String &quot;50&quot;: 50
Parsing String &quot;20&quot;: 20
</code></pre>
<h2 id="conclusion">Conclusion</h2>
We've covered one of the fundamental topics of Java and common problem developers face - Converting a String to an Integer or int using tools shipped with the JDK.

Java Convert String to Integer

<h2 id="introduction">Introduction</h2>
By definition, web scraping means getting useful information from web pages. The process should remove the hassle of having to browse pages manually, be automated, and allow to gather and classify the information you're interested in programmatically.
<a rel="nofollow" target="_blank" href="https://nodejs.org/en/">Node.js</a> is a great tool to use for web scraping. It allows implementing web scraping routines in a couple of lines of code using the open source module provided by <code>npm</code> - the Node Package Manager.
<h2 id="themainstepsofwebscraping">The Main Steps of Web Scraping</h2>
As we have already defined - web scraping is nothing more than automating the manual browsing and collecting of information from specific websites in your preferred web browser.
This process consists of the 3 main steps:
<ul>
<li>Getting the HTML source code from the website</li>
<li>Making sense of the HTML content, finding the information we're interested in, and extracting it</li>
<li>Moving the discovered information to storage of your choice (text file, database, etc.)</li>
</ul>
The first and final steps are usually pretty much the same, depending on your application's requirements. However, making sense of the HTML content requires you to write specific code for every website you'd like to scrape.
<h2 id="caution">Caution</h2>
<blockquote>
Depending on your usage of these techniques and technologies, your application could be performing illegal actions
</blockquote>
There are a few cases where you would want to be cautious about:
<ul>
<li>DoS - A <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Denial-of-service_attack">Denial of Service attack</a> practically relies on sending so many requests to the server that it simply can't handle any more. All new incoming requests would then be denied. If you're scraping a website too often, it can be considered a DoS attack.</li>
<li>Terms of Service - Many websites, and almost all bigger websites clearly state that web scraping on their platforms is prohibited. If many people scraped these websites, it would pretty much end up being a DDoS attack, which by itself is illegal.</li>
<li>Abusive Software - Many tools and frameworks online offer a large variety of tools and functionalities. Some allow users to fill forms, submit data, upload and download files etc. <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/CAPTCHA">CAPTCHA</a> is used to combat this, however, even this can be surpassed with a bot.</li>
</ul>
<h2 id="manualwebscrapingalgorithm">Manual Web Scraping Algorithm</h2>
As an example, we'll be looking up the <a rel=nofollow href="https://www.yellowpages.com">Yellow Pages</a> for companies providing printing services in New York.
First of all, let's define our task and the desired result:
<blockquote>
Produce a list of the companies providing printing services in New York in the form of a &quot;.CSV&quot; file that should have the company name, email, phone, and link columns that would describe each company.
</blockquote>
Here's how we would do that manually:
<ol>
<li>Navigate our browser to the appropriate link and put in &quot;printing&quot; and &quot;New York&quot; into the search fields and execute the search</li>
<li>Select and store the name of the first company in the list</li>
<li>Store the link to the company page and follow it</li>
<li>Find an email address and store it</li>
<li>Write stored values to a &quot;.CSV&quot; file either by using a text editor or another tool for editing tables like Excel or Google Sheets</li>
</ol>
Repeating these steps multiple time will give us a table filled with company details.
<h2 id="automatingtheprocesswithwebscraping">Automating the Process with Web Scraping</h2>
To automate the process, we should follow the same steps programmatically.
<h3 id="settingupthedevelopmentenvironment">Setting up the Development Environment</h3>
We'll be using Node.js and <code>npm</code> to develop this sample project. So make sure those tools are installed on your machine and let's start by running the following command in an empty directory of your choice followed by creating an empty <code>index.js</code> page that will contain our code:
<pre><code class="hljs">$ npm init
</code></pre>
The next step is to install the required modules from the <code>npm</code>.
From the manual algorithm described above, we see that we'll need something to get the HTML source code, parse the content, and make sense of it and then write down the array of JavaScript objects into the &quot;.CSV&quot; file:
<pre><code class="hljs">$ npm install --save request request-promise cheerio objects-to-csv
</code></pre>
Executing the preceding line in the terminal will install the required modules in the <code>node_modules</code> directory and save them as dependencies in the <code>package.json</code> file.
<h3 id="retrievinginformation">Retrieving Information</h3>
Once all the preparation are done, you can open the <code>index.js</code> file in your favorite text editor and require the modules we've just installed:
<pre><code class="hljs">const rp = require(&#x27;request-promise&#x27;);
const otcsv = require(&#x27;objects-to-csv&#x27;);
const cheerio = require(&#x27;cheerio&#x27;);
</code></pre>

 <div class="alert alert-note">
 <div class="flex">
 
 <div class="flex-shrink-0 mr-3">
 <img src="/assets/images/icon-information-circle-solid.svg" class="icon" aria-hidden="true" />
 </div>
 
 <div class="w-full">
 Tip: When sending requests to websites, you can use <a href="https://oxylabs.io/" target="_blank" rel="noopener noreferrer">proxies</a> to rotate IPs, maintain anonymity, and reduce the risk of getting blocked. This is especially helpful if you plan to scrape many pages or repeat requests frequently.

 </div>
 </div>
 </div>
 Completing this step in the manual algorithm will give us the link, that we'll split up in two parts and add to the <code>index.js</code> right after the modules:
<pre><code class="hljs">const baseURL = &#x27;https://www.yellowpages.com&#x27;;
const searchURL = &#x27;/search?search_terms=printing&amp;geo_location_terms=New+York%2C+NY&#x27;;
</code></pre>
Then, we should write a function, that will return the array of JavaScript objects representing the companies from the task description.
In order to be able to translate the manual algorithm into code, we'll first have to do some manual work using the <code>inspector</code> tool of our web browser.
We'll need to find the specific HTML elements that hold the information we're interested in. In our case, the business name could be found at an element like:
<pre><code class="hljs">&lt;a class=&quot;business-name&quot; href=&quot;/new-york-ny/mip/global-copy-2988131?lid=1000106255864&quot; itemprop=&quot;name&quot;&gt;Global Copy&lt;/a&gt;
</code></pre>
<img src="https://s3.amazonaws.com/stackabuse/media/web-scraping-with-node-js-3.png" alt="web_scraping_with_node_js">
Which is an <code>&lt;a&gt;</code> tag with the class <code>business-name</code>, that holds the name of the company and an <code>href</code> property that holds a link to the individual company page. Both of them will be useful for us in the future.
Following the link to the company page, we'll need to find the remaining two pieces of data: the phone and the email. They are located under the <code>&lt;p&gt;</code> tags with the class <code>phone</code> and the <code>&lt;a&gt;</code> tag with the class <code>email-business</code>.
<img src="https://s3.amazonaws.com/stackabuse/media/web-scraping-with-node-js-5.png" alt="web_scraping_with_node_js">
<img src="https://s3.amazonaws.com/stackabuse/media/web-scraping-with-node-js-6.png" alt="web_scraping_with_node_js">
Take a note that to get the phone, we'll need the value stored inside of the tag and to get an email, we'll need an <code>href</code> property of the <code>&lt;a&gt;</code> tag.
Let's look what the manual algorithm will look like when we try to implement it programmatically:
<ol>
<li>Get the HTML source of the page that we're looking to scrape by using the <code>request-promise</code> module and feeding it a link that we got on the first step of our manual algorithm</li>
<li>Map the array of the company names in the original HTML into an array of objects with the properties name, link, phone, and email</li>
<li>Convert the resulting array into a CSV file</li>
</ol>
Here's how this algorithm will look like when implemented:
<pre><code class="hljs">const getCompanies = async () =&gt; {
 const html = await rp(baseURL + searchURL);
 const businessMap = cheerio(&#x27;a.business-name&#x27;, html).map(async (i, e) =&gt; {
 const link = baseURL + e.attribs.href;
 const innerHtml = await rp(link);
 const emailAddress = cheerio(&#x27;a.email-business&#x27;, innerHtml).prop(&#x27;href&#x27;);
 const name = e.children[0].data;
 const phone = cheerio(&#x27;p.phone&#x27;, innerHtml).text();

 return {
 emailAddress,
 link,
 name,
 phone,
 }
 }).get();
 return Promise.all(businessMap);
};
</code></pre>
It follows the rules we've set up earlier and returns a <code>Promise</code> that resolves into an array of JavaScript objects:
<pre><code class="hljs">{
 &quot;emailAddress&quot;: &quot;mailto:globalcopyny@yahoo.com&quot;,
 &quot;link&quot;: &quot;https://www.yellowpages.com/new-york-ny/mip/global-copy-2988131?lid=1000106255864&quot;,
 &quot;name&quot;: &quot;Global Copy&quot;,
 &quot;phone&quot;: &quot;(212) 222-2679&quot;
}
</code></pre>
<h2 id="promise">Promise</h2>
Just in case you're unfamiliar with the core concepts of the asynchronous programming in modern JavaScript, here's a short introduction to Promises.
A <code>Promise</code> is a special type that acts like a placeholder for the value. It could be in a couple of states:
<ul>
<li>pending</li>
<li>fulfilled</li>
<li>rejected</li>
</ul>
Without getting too detailed about it, you should just know that the functions that return promises, don't return the actual values. In order to get access to the result of the Promise or the resolution value, you should write another function, which should be passed into the <code>then</code> block and expect the value that the Promised will be resolved with. If any error occurs inside of the Promise while it's <code>pending</code>, then it will transition into the <code>rejected</code> state with the possibility to handle the error in the <code>catch</code> block.
<h2 id="storingdataandfinishingtouches">Storing Data and Finishing Touches</h2>
Although the result seems fairly good, taking a closer look at the array of the companies, shows us that the email has an unnecessary <code>mailto:</code> prefix and sometimes the emails and company names are missing:
<pre><code class="hljs">{
 &quot;emailAddress&quot;: undefined,
 &quot;link&quot;: &quot;https://www.yellowpages.com/new-york-ny/mip/apm-451909424?lid=451909424&quot;,
 &quot;name&quot;: undefined,
 &quot;phone&quot;: &quot;(212) 856-9800&quot;
}
</code></pre>
By getting back to the inspector tool, we see that the name of the company could be also found in the internal company page inside of an <code>&lt;h1&gt;</code> tag.
The emails are sometimes missing for certain companies and we can't do anything about it.
The <code>mailto:</code> prefix could be removed by using a <code>replace</code> function.
Here are the adjustments you should make to your code:
<pre><code class="hljs">...
 const name = e.children[0].data || cheerio(&#x27;h1&#x27;, innerHtml).text();
...
 emailAddress: emailAddress ? emailAddress.replace(&#x27;mailto:&#x27;, &#x27;&#x27;) : &#x27;&#x27;,
...
</code></pre>
Now, that we have extracted all the necessary data from the web page and have a clean array of JavaScript objects, we can prepare the &quot;.CSV&quot; file required by our task definition:
<pre><code class="hljs">getCompanies()
 .then(result =&gt; {
 const transformed = new otcsv(result);
 return transformed.toDisk(&#x27;./output.csv&#x27;);
 })
 .then(() =&gt; console.log(&#x27;SUCCESSFULLY COMPLETED THE WEB SCRAPING SAMPLE&#x27;));
</code></pre>
The <code>getCompanies</code> function returns a promise which resolves into an array of objects prepared to be written to the CSV file, which is done in the first <code>then</code> block. The second one resolves when the CSV file is successfully written to the file system, thus completing the task.
By adding this piece of code to our <code>index.js</code> file, and running it with:
<pre><code class="hljs">$ node index.js
</code></pre>
We should get an <code>output.csv</code> file right in our working directory. The table represented by this <code>.CSV</code> file contains 4 columns - <code>emailAddress</code>, <code>link</code>, <code>name</code>, <code>phone</code>, and each row describes a single company.
<h2 id="conclusion">Conclusion</h2>
In essence, web scraping is browsing web pages, picking up useful information according to the task, and storing it somewhere, all of which is done so programmatically. In order to be able to do this with code, this process should be first done manually using an <code>inspector</code> tool of the browser or by analyzing the raw HTML content of the target web page.
Node.js provides reliable and simple tools to make web scraping a straightforward task that could save you a lot of time when compared to processing the links manually.
Although it might seem tempting to automate all of your everyday tasks like this, but be careful when it comes to using these tools, because it's easy to violate the terms of service of a website if you haven't fully read it, or even just by generating a huge amount of traffic with your scraper.

Vasyl Lagutin

Web Scraping with Node.js

<h2 id="introduction">Introduction</h2>
Graph-related interview questions are very common, and many algorithms you'll be expected to know fall into this category.
It's important to be acquainted with all of these algorithms - the motivation behind them, their implementations and applications.
If you're interested in reading more about <a target="_blank" href="/programming-interview-questions">Programming Interview Questions</a> in general, we've compiled a lengthy list of these questions, including their explanations, implementations, visual representations and applications.
<h2 id="graphdatastructureinterviewquestions">Graph Data Structure Interview Questions</h2>
Plainly said - a Graph is a non-linear data structure made up of nodes/vertices and edges.
Nodes are entities in our graph, and the edges are the lines connecting them:
<img src="https://s3.amazonaws.com/stackabuse/media/programming-interview-questions-1.PNG" alt="what_are_graphs">
Representation of a graph
There are many kinds of graphs, undirected graphs, directed graphs, vertex labeled graphs, cyclic graphs, edge-labeled graphs, weighted graphs etc.
Nonetheless, they are most often used to represent networks be it a city network, city streets, a terrain for AI to pass or social network connections.
<h2 id="breadthfirstsearch">Breadth First Search</h2>
<h3 id="interviewquestionexample">Interview Question Example</h3>
Find the shortest path in a Maze. The path blocks are represented as <code>1</code> and the wall blocks are represented as <code>0</code>. You can move in four directions (up, down, left, right).
<pre><code class="hljs">
 0, 0, 0, 0, 0, 0, 0, 1, 1
 0, 1, 1, 1, 1, 0, 0, 1, 0
 0, 1, 0, 0, 1, 1, 1, 1, 0
 1, 1, 1, 0, 1, 0, 0, 0, 0
 1, 0, 1, 1, 1, 1, 1, 1, 1
 1, 1, 0, 0, 0, 0, 0, 0, 0
 0, 1, 0, 0, 1, 1, 1, 1, 0
 0, 1, 0, 0, 1, 0, 0, 1, 0
 0, 1, 1, 1, 1, 0, 0, 1, 1

</code></pre>
<h3 id="explanation">Explanation</h3>
Breadth First Search (BFS for short) is a graph-traversal algorithm, often used for finding the shortest path from the starting node to the goal node.
BFS visits &quot;layer-by-layer&quot;. This means that in a Graph like shown below, it first visits all the children of the starting node. These children are treated as the &quot;second layer&quot;.
<img src="https://s3.amazonaws.com/stackabuse/media/programming-interview-questions-2.gif" alt="bfs_search_gif">
After visiting all the children of the starting node, the algorithm then visits all the children of the already visited children, practically moving to the third layer.
It's important to note that BFS will reach the end of the graph and then calculate the shortest path possible. It guarantees to find the shortest possible path, though it pays the price of not being as memory friendly as some other algorithms.
To accomplish this, BFS uses a <code>Queue</code> and this is an important feature of the algorithm. The terminal condition of the algorithm is that the <code>Queue</code> is empty, so it's going to go until it visits every single node.
Let's take a look at the pseudo-code behind the implementation:
<pre><code class="hljs">Queue queue
node set visited true
queue.enqueue(node)

while queue not empty
 current = queue.dequeue()
 for v in current children
 if v is not visited
 v set visited true
 queue.enqueue(v)
</code></pre>
We start off by adding the starting node to the queue. Since the queue is not empty, we set the current node to be the root node, and remove it from the queue.
We then check all of the current node's children and visit them if we haven't visited them before. After that, we add the children to the queue.
Each of these children will become the next <code>current</code> node. And since the queue data structure follows the FIFO (First in, first out) structure, we visit the rest of the children from the second layer, before continuing to visit the children from the proceeding layers.
<h3 id="bfsapplications">BFS Applications</h3>
<blockquote>
&quot;Where can we use BFS?&quot;
</blockquote>
This algorithm is used in a lot of systems all around you! It's very often used for pathfinding, alongside Depth First Search. It's being used in your GPS Navigation system to find neighboring locations and even on social networking websites and search engines.
BFS found itself used a lot in artificial intelligence and machine learning, especially in robotics. And probably most noticeably, in your programming career, it's being used for garbage collection.
<h2 id="depthfirstsearch">Depth First Search</h2>
<h3 id="interviewquestionexample">Interview Question Example</h3>
Check if the given graph is strongly connected or not:
<img src="" alt="dfs_check">
<h3 id="explanation">Explanation</h3>
Depth First Search (DFS) is another graph traversal algorithm, similar to Breadth First Search.
DFS searches as far as possible along a branch and then backtracks to search as far as possible in the next branch. This means that in the proceeding Graph, it starts off with the first neighbor, and continues down the line as far as possible.
<img src="https://s3.amazonaws.com/stackabuse/media/programming-interview-questions-4.gif" alt="dfs_search">
Once it reaches the final node in that branch (1), it backtracks to the first node where it was faced with a possibility to change course (5) and visits that whole branch, in our case only (2).
Then it backtracks again to the node (5) and since it's already visited nodes (1) and (2), it backtracks to (3) and re-routes to the next branch (8).
The algorithm continues traversing in this fashion until all nodes have been visited and the stack is empty.
DFS doesn't guarantee to find the shortest possible path and can settle for a local optimum, unlike BFS. Thanks to this, it's more memory friendly, though not by a lot.
To accomplish this, DFS uses a <code>Stack</code>, which is the main difference between these two algorithms. A <code>Stack</code> follows a LIFO (Last-in, first-out) structure, which is the reason it goes as far as possible, before taking the other layers into consideration.
You can implement DFS with the help of recursion or iteration. The recursive approach is shorter and simpler, though both approaches are pretty much the same performance-wise.
Let's take a look at the recursive pseudo-code behind the implementation:
<pre><code class="hljs">dfs(node)
node set visited true

for n in node neighbors
 if n is not visited
 dfs(n)
</code></pre>
We start off by calling the algorithm on the given node and set that it's visited.
We then check all of the node's neighbors, and for each one, run the same algorithm if it isn't visited. This means that before advancing to the next neighbor, it visits all of the children of the first neighbor. Then goes back to the top, and visits the second neighbor - which advances the search for all of its children, and so on.
You might've noticed the lack of <code>Stack</code> in this example and I'd like to address it properly. Both approaches use a <code>Stack</code> structure in the background, though in the recursive approach, we don't explicitly define it like so.
Let's take a look at the iterative pseudo-code behind the implementation:
<pre><code class="hljs">dfs(node)
 Stack stack
 node set visited true
 stack.push(node)

 while stack is not empty
 current = stack.pop()
 for n in node neighbors
 if n is not visited
 n set visited true
 stack.push(n)
</code></pre>
Here, we add the starting node to the <code>Stack</code>. Since the stack is not empty, we set the <code>current</code> node to be the starting node and take it out of the stack.
We consider all of the neighbors, one by one. The first neighbor gets added to the stack, which makes it the new <code>current</code> node in the next iteration. This way, all of the children of this node will be checked before the node next to this one, from the original neighbor list.
<h3 id="dfsapplications">DFS Applications</h3>
<blockquote>
&quot;Where can we use DFS?&quot;
</blockquote>
Similar to BFS, DFS is often used in artificial intelligence and machine learning. These two algorithms are very common and often compared.
DFS is generally used for pathfinding between nodes, and especially for finding paths for mazes! DFS is also used for topological sorting and finding strongly connected components.
<h2 id="asearch">A* Search</h2>
Given the following graph, where the black node is the starting node, the red nodes are the walls and the green node is the goal node, find the shortest path to the green node:
<img src="https://s3.amazonaws.com/stackabuse/media/programming-interview-questions-5.PNG" alt="a_star_search_example">
<h3 id="explanation">Explanation</h3>
A* Search is quite different than the two we've previously talked about. It's also a graph traversal algorithm, however, it's used to handle real-life pathfinding scenarios.
A* search relies on a knowledge and heuristic cost function for the given node as a way to decide which node it should visit next. In most cases, this cost function is denoted as <code>f(x)</code>.
The cost function is calculated as a sum of two other functions:
<ul>
<li><code>g(x)</code> - The function that represents the cost of moving from the starting node to a given node.</li>
<li><code>h(x)</code> - The estimated cost from the given node to the goal node.</li>
</ul>
We obviously don't know the exact movement cost from the given note to the goal node, which is the reason why <code>h(x)</code> is also often referred to as the &quot;heuristic&quot;.
The algorithm always picks the node with the lowest <code>f(x)</code> value. This is logical, considering how the function is calculated.
The fewer steps we take from the starting point combined with how close we get to the goal makes the value of <code>f(x)</code> lower if we're going with the shortest path to the goal. Walking away from the goal, and making more steps than needed to get there increases the <code>f(x)</code> function.
<h3 id="asearchapplications">A* Search Applications</h3>
<blockquote>
&quot;Where can we use A* Search?
</blockquote>
This algorithm is generally used for most of the shortest path problems. It's often used for real-life searching scenarios as well as video games. Most NPCs and AI players rely on A* to intelligently search for a path, fast and efficient.
<h2 id="dijkstraalgorithm">Dijkstra Algorithm</h2>
Given the following weighted graph, find the shortest path between the vertices A and H.
<img src="https://s3.amazonaws.com/stackabuse/media/programming-interview-questions-6.PNG" alt="dijkstra_interview_graph">
<h3 id="explanation">Explanation</h3>
Dijkstra Algorithm is a notorious graph traversal algorithm for finding the shortest path from a given node/vertex to another.
There are several implementations of this algorithm and some even use different data structures and have different applications. We'll cover the classic one - finding the shortest path between two nodes.
Dijkstra works in a slightly different fashion to the previous algorithms. Instead of starting at the root node, it starts at the goal node and backtracks its way to the starting node.
The algorithm does this by building a set of notes with the minimum distance from the source, dictated by the &quot;weight&quot; of the edges.
It's built with a few variables:
<ul>
<li>dist - The distance from the goal node (most often referred to as the source node) to every other node in the graph. The <code>dist</code> to the source node is 0 since that's where we start. Additionally, the distance to all other nodes is set to infinity. As the algorithm traverses the graph, the distance to all other nodes gets recalculated according to the current node.</li>
<li>Q - Similar to BFS, we need to define a Queue data type and fill it with all of the nodes from the graph. The terminal condition for the algorithm is if the Queue is empty.</li>
<li>Set - We'll also need a set to contain all of the visited nodes, in order to not visit them again. In the end, the set will contain all nodes from the graph.</li>
</ul>
Let's take a look at the pseudo-code behind its implementation:
<pre><code class="hljs">dijkstra(Graph, source)
 source set distance 0

 for n in Graph
 if n is not source
 n set dist infinity
 n set parent none
 queue.add(n)

 while Q is not empty
 n = node in Q with min dist(n)
 remove n from Q

 for node in n neighbors
 i = dist(n) + length(node, goal)
 if i &lt; dist(goal)
 dist(goal) = i
</code></pre>
<h3 id="dijkstraapplications">Dijkstra Applications</h3>
<blockquote>
&quot;Where can we use Dijkstra?&quot;
</blockquote>
This algorithm is generally used for many of the shortest path problems, and it's most definitely a notable algorithm.
Google Maps uses Dijkstra to find the shortest path in navigation. IP Routing also relies heavily exactly on Dijkstra and it has applications in computer networking.
<h2 id="comparingbfsdfsaanddijkstra">Comparing BFS, DFS, A* and Dijkstra</h2>
This might not pop out as an interview question, but it's most certainly a logical one. We've covered 4 different algorithms that are used mostly for the same purpose.
<blockquote>
&quot;Which one should I use? Which one is the best?&quot;
</blockquote>
The answer to this common question is: all of them.
Each of these algorithms, although seemingly similar, offer advantages over the other, based on the situation you're implementing them in and your specific case.
Here are some cases where you could be able to choose which one you would use:
<ul>
<li>Finding currently alive members in a family tree.</li>
</ul>
In such a case, it would be best to use DFS since mostly the furthest levels are alive. BFS would waste too much time considering levels that shouldn't be considered. It wouldn't make sense to use Dijkstra or A* for this purpose.
<ul>
<li>Finding the deceased members in a family tree.</li>
</ul>
In such a case, using DFS would be less practical than BFS as most of the deceased members would be located near the top. BFS would traverse this graph layer by layer and add all relevant members to the list.
<ul>
<li>Finding the shortest path from point A to point B in a graph/map.</li>
</ul>
In such a case, although a bit controversial, using A* search over Dijkstra is common practice. Using BFS and DFS here can be very impractical. BFS assumes that the weight between all nodes is the same, whereas the actual weight/cost could vary. This is where Dijkstra comes into play!
Dijkstra takes the cost of moving on an edge into account. If the cost of going straight is bigger than going around it, it will go around. This may translate to congested roads on a map for an example.
A* focuses on reaching the goal, and only makes steps that seem promising and reasonable, according to its functions.
Dijkstra takes all other nodes into consideration, whereas A* only takes the reasonable ones. Because of this, A* is faster than Dijkstra and is often considered &quot;the intelligent searching algorithm&quot;.
Some also refer to A* as the &quot;informed&quot; Dijkstra algorithm. If you take away the heuristic of A* - which means that it chooses the next step only according to the cost so far, you practically get Dijkstra, in reverse though. This makes A* run not greedily towards the goal, but rather in all directions considering every single node in the graph, which again is the same as Dijkstra.
If you're looking for an optimized search and results looking exclusively at the goal - A* is the choice for you.
If you're looking for an algorithm to compute the shortest path between the starting point and every single node in the tree, which includes the goal node - Dijkstra is the choice for you.
<h2 id="resources">Resources</h2>
If you're looking for a better way to practice these kinds of programming interview questions, as well as others, then we'd recommend trying out a service like <a class="dcp-link" rel="nofollow" target="_blank" href="https://stackabu.se/daily-coding-problem">Daily Coding Problem</a>. They'll send you a new problem to solve every day, all of which come from top companies. You can also find out more <a target="_blank" href="/daily-coding-problem-practice-programming-puzzles-to-your-inbox/">here</a> if you want more info.
<h2 id="conclusion">Conclusion</h2>
In this article, we've covered the common interview questions related to the graph data structure.
Again, if you're interested in reading more about <a target="_blank" href="/programming-interview-questions">Programming Interview Questions</a> in general, we've compiled a lengthy list of these questions, including their explanations, implementations, visual representations and applications.

David Landup

Graph Data Structure Interview Questions

<h2 id="introduction">Introduction</h2>
If you're a programmer aspiring to work in a top-tier tech company like Google, Microsoft, Apple, or Facebook - you're probably concerned with the interview process.
These interviews can be daunting, especially if you're not familiar with the type of questions that you'll be expected to answer.
This is the reason why we've decided to compile this selection of programming interview questions you might face in an interview tomorrow, hopefully helping you both learn about them, understanding their nature, as well as reinforcing your knowledge.
Most of these questions revolve around Data Structures and Algorithms so the most attention is given to them. Although the focus is on Data Structures and Algorithms, we'll cover a few logical questions you might run into as well!
Most of these questions and examples will also contain some basic-level explanations, since, in my humble opinion, a lot of terminologies are being used without proper understanding of what they actually mean.
It's worth mentioning that a certain degree of mathematical knowledge here is important. Don't worry, you don't have to be a mathematical genius, but knowing <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Discrete_mathematics">Discrete Mathematics</a> is a huge advantage.
<h2 id="datastructureandalgorithms">Data Structure and Algorithms</h2>
A Data Structure is a fundamental and simple concept. It's a way of organizing data and their relationship to allow efficient operations to be performed on them.
There are many data structures that programmers and developers regularly use such as:
<ul>
<li>Arrays</li>
<li>Binary Trees</li>
<li>Graphs</li>
<li>Linked Lists</li>
<li>Matrixes</li>
<li>Stacks</li>
<li>Queues</li>
<li>Heaps</li>
<li>Hash Tables</li>
</ul>
If you'd like to see <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/List_of_data_structures">the whole list</a>, it's quite lengthy.
An algorithm is a piece of code that represents a certain set of instructions, usually selected to act as a finder to the solution of a specific problem.
They should be efficient and fast, which means that they both take the least possible time to complete as well as consume the least possible memory space, depending on the nature of the algorithm and the problem at hand.
Knowing your data structures and algorithms is important. It allows you to understand the underlying logic behind the tools you use every single day. Knowing which algorithms to use and which data structures to employ is a valuable thing in a production environment.
The ability to pick out an efficient solution compared to another is crucial. It also incites intuitive ways to solve problems that you might be faced with, and depending on what kind of person you are - it might be fun to refresh a bit on some high-school mathematics.
<h3 id="graphdatastructureinterviewquestions">Graph Data Structure Interview Questions</h3>
<ul>
<li><a target="_blank" href="/graph-data-structure-interview-questions#breadthfirstsearch">Breadth First Search</a></li>
<li><a target="_blank" href="/graph-data-structure-interview-questions#depthfirstsearch">Depth First Search</a></li>
<li><a target="_blank" href="/graph-data-structure-interview-questions#asearch">A* Search</a></li>
<li><a target="_blank" href="/graph-data-structure-interview-questions#dijkstraalgorithm">Dijkstra Algorithm</a></li>
<li><a target="_blank" href="/graph-data-structure-interview-questions#comparingbfsdfsaanddijkstra">Comparing BFS, DFS, A* and Dijkstra</a></li>
</ul>
<h3 id="linkedlistinterviewquestions">Linked List Interview Questions</h3>
<ul>
<li><a target="_blank" href="/linked-list-programming-interview-questions#insertingandremovingnodes">Inserting and Removing Nodes</a></li>
<li><a target="_blank" href="/linked-list-programming-interview-questions#comparingstringsrepresentedaslinkedlists">Comparing Strings Represented as Linked Lists</a></li>
<li><a target="_blank" href="/linked-list-programming-interview-questions#reversingalist">Reversing a List</a></li>
<li><a target="_blank" href="/linked-list-programming-interview-questions#selectingarandomnode">Selecting a Random Node</a></li>
<li><a target="_blank" href="/linked-list-programming-interview-questions#findingthemiddlenode">Finding the Middle Node</a></li>
<li><a target="_blank" href="/linked-list-programming-interview-questions#kthelementfromlastnode">Kth Element from Last Node</a></li>
<li><a target="_blank" href="/linked-list-programming-interview-questions#frequencyofagivennumber">Frequency of a Given Number</a></li>
<li><a target="_blank" href="/linked-list-programming-interview-questions#intersectionoftwolinkedlists">Intersection of Two Linked Lists</a></li>
</ul>
<h3 id="dynamicprogramminginterviewquestionscomingsoon">Dynamic Programming Interview Questions (coming soon)</h3>
<ul>
<li>Fibonacci Number Sequence</li>
<li>Longest Common Subsequence</li>
</ul>
<h3 id="sortingansearchinginterviewquestionscomingsoon">Sorting an Searching Interview Questions (coming soon)</h3>
<ul>
<li>Binary Search</li>
<li>Bubble Sort</li>
<li>Insertion Sort</li>
<li>Merge Sort</li>
<li>Heap Sort</li>
<li>Quick Sort</li>
<li>Interpolation</li>
<li>Tree/Binary Search Tree</li>
<li>Minimum Depth</li>
<li>Maximum Path Sum</li>
</ul>
<h3 id="numbertheoryinterviewquestionscomingsoon">Number Theory Interview Questions (coming soon)</h3>
<ul>
<li>Euclid's GCD Algorithm</li>
<li>Extending Euclid's GCD Algorithm</li>
<li>Diophantine Equation</li>
<li>Chinese Remainder Theorem</li>
<li>Modular Inverse</li>
<li>Semi-Perfect Numbers</li>
</ul>
<h3 id="stringinterviewquestionscomingsoon">String Interview Questions (coming soon)</h3>
<ul>
<li>Reversing a String</li>
<li>Checking if String contains only digits</li>
<li>Finding Duplicate Characters in a String</li>
<li>How to Convert a String to Integer</li>
<li>Removing Duplicate Characters in a String</li>
<li>Finding the Maximum Occuring Character in a String</li>
<li>Find the First Non-Repeating Character in a String</li>
<li>Checking if Two Strings are Anagrams of Each Other</li>
<li>Counting the Number of Words in a String</li>
</ul>
<h3 id="arrayinterviewquestionscomingsoon">Array Interview Questions (coming soon)</h3>
<ul>
<li>Finding the Missing Number from Array</li>
<li>Finding Duplicate Integers in an Array</li>
<li>Finding the Largest and Smallest Number in Unsorted Array</li>
<li>Removing Duplicates from an Array</li>
<li>Reversing an Array</li>
<li>Finding the k-th Smallest Integer in an Unsorted Array</li>
<li>Finding Common Elements Between Multiple Arrays</li>
</ul>
<h2 id="practiceandstrategies">Practice and Strategies</h2>
That being said, a service we most definitely recommend is - <a class="dcp-link" rel="nofollow" target="_blank" href="http://stackabu.se/daily-coding-problem">Daily Coding Problem</a>.
Daily Coding Problem is a simple and very useful platform that emails you one coding problem to solve every morning. This ensures that you practice consistently and often enough to stay in shape over a long period of time. We wrote up a more <a target="_blank" href="/daily-coding-problem-practice-programming-puzzles-to-your-inbox/">in-depth review of the DCP</a> if you want to find out more.
While solving these problems, you'll notice a lot of the aforementioned data structures and algorithms, as well as the importance of innovative thinking.

Programming Interview Questions

<h2 id="introduction">Introduction</h2>
This article is a continuation of a series of articles describing the often forgotten about methods of the Java language's base Object class. The following are the methods of the base Java Object which are present in all Java objects due to the implicit inheritance of Object.
<ul>
<li><a target="_blank" href="/javas-object-methods-tostring/">toString</a></li>
<li><a target="_blank" href="/javas-object-methods-getclass/">toClass</a></li>
<li><a target="_blank" href="/javas-object-methods-equals-object/">equals</a></li>
<li><a target="_blank" href="/javas-object-methods-hashcode/">hashCode</a></li>
<li><a target="_blank" href="/javas-object-methods-clone/">clone</a></li>
<li>finalize (you are here)</li>
<li><a target="_blank" href="/javas-object-methods-wait-notify/">wait &amp; notify</a></li>
</ul>
The focus of this article is the <code>Object#finalize()</code> method which is used during the garbage collection process internally by the Java Virtual Machine (JVM). Traditionally the method has been overriden by subclasses of Object when the class instance needs to close or purge system resources such as database connections and file handlers. However, Java language experts have long been making the argument that overriding <code>finalize()</code> in order to perform operations like resource destruction is not a good idea.
In fact, the offical Oracle Java <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/10/docs/api/java/lang/Object.html#finalize()">docs</a> state that the <code>finalize()</code> method itself has been deprecated, thus tagging it for removal in future releases of the language as the underlying mechanisms for object creation and garbage collection have been under reevaluation. I strongly recommend following the advice to leave the <code>finalize()</code> method unimplemented.
Furthermore, I want to make clear this article's main objective is to provide the guidance to migrate existing code that implements <code>finalize()</code> to the preferred construct of implementing the <code>AutoClosable</code> interface along with try-with-resource paired construct introduced in Java 7.
<h2 id="exampleofimplementingfinalize">Example of Implementing Finalize</h2>
Here is an example that you might see in some legacy code where finalize has been overriden to provide the functionality of cleaning up a database resource by closing a connection to a SQLite database within a class named <code>PersonDAO</code>. Note that this example uses SQLite which requires a third party Java Database Connectivity Connector (JDBC) Driver, which will need to be downloaded from <a rel="nofollow" target="_blank" href="https://bitbucket.org/xerial/sqlite-jdbc/downloads/">here</a> and added to the classpath if you want to follow along.
<pre><code class="hljs">import java.nio.file.Path;
import java.nio.file.Paths;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.SQLException;
import java.sql.Statement;
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;

public class MainFinalize {

 public static void main(String[] args) {
 try {
 PersonDAO dao = new PersonDAO();
 Person me = new Person(&quot;Adam&quot;, &quot;McQuistan&quot;, LocalDate.parse(&quot;1987-09-23&quot;));
 dao.create(me);
 } catch(SQLException e) {
 e.printStackTrace();
 }
 }
 
 /* PersonDAO implementing finalize() */
 static class PersonDAO {
 private Connection con;
 private final Path SQLITE_FILE = Paths.get(System.getProperty(&quot;user.home&quot;), &quot;finalize.sqlite3&quot;);
 private final String SQLITE_URL = &quot;jdbc:sqlite:&quot; + SQLITE_FILE.toString();
 
 public PersonDAO() throws SQLException {
 con = DriverManager.getConnection(SQLITE_URL);
 
 String sql = &quot;CREATE TABLE IF NOT EXISTS people (&quot;
 + &quot;id integer PRIMARY KEY,&quot;
 + &quot;first_name text,&quot;
 + &quot;last_name text,&quot;
 + &quot;dob text);&quot;;
 Statement stmt = con.createStatement();
 stmt.execute(sql);
 }
 
 void create(Person person) throws SQLException {
 String sql = &quot;INSERT INTO people (first_name, last_name, dob) VALUES (?, ?, ?)&quot;;
 PreparedStatement stmt = con.prepareStatement(sql);
 stmt.setString(1, person.getFirstName());
 stmt.setString(2, person.getLastName());
 DateTimeFormatter fmt = DateTimeFormatter.ofPattern(&quot;yyyy-MM-dd&quot;);
 stmt.setString(3, person.getDob().format(fmt));
 stmt.executeUpdate();
 }
 
 @Override
 public void finalize() {
 try {
 con.close();
 } catch(SQLException e) {
 System.out.println(&quot;Uh, oh ... could not close db connection&quot;);
 }
 }
 }
 
 /* Simple Person data class */
 static class Person {
 private final String firstName;
 private final String lastName;
 private final LocalDate dob;
 
 Person(String firstName, String lastName, LocalDate dob) {
 this.firstName = firstName;
 this.lastName = lastName;
 this.dob = dob;
 }
 
 String getFirstName() {
 return firstName;
 }

 String getLastName() {
 return lastName;
 }

 LocalDate getDob() {
 return dob;
 }
 }
}
</code></pre>
As I mentioned previously this is not the preferred method to close down a resource, and in fact, should be strongly discouraged. Instead one should implement code similar to that in the <code>PersonDAO#finalize</code> method within the <code>AutoClosable#close</code> method as shown below in the example that follows.
<h2 id="abettersolutiontrywithresourcesandautocloseable">A Better Solution: Try-with-Resources and AutoCloseable</h2>
Java 7 introduced the <code>AutoCloseable</code> interface along with a enhancement to the traditional try / catch construct which provides a superior solution to cleaning up resources which in an object. By using this combination of <code>AutoClosable</code> and try-with-resources the programmer has greater control over how and when a resource will be freed up which was often unpredictable when overriding the <code>Object#finalize()</code> method.
The example that follows takes the previous <code>PersonDAO</code> and implements the <code>AutoCloseable#close</code> interface to close the database connection. The main method then utilizes the try-with-resources construct in place of the previous try / catch to handle cleaning things up.
<pre><code class="hljs">public class MainFinalize {

 public static void main(String[] args) {
 try (PersonDAO dao = new PersonDAO()) {
 Person me = new Person(&quot;Adam&quot;, &quot;McQuistan&quot;, LocalDate.parse(&quot;1987-09-23&quot;));
 dao.create(me);
 } catch(SQLException e) {
 e.printStackTrace();
 }
 }
 
 /* PersonDAO implementing finalize() */
 static class PersonDAO implements AutoCloseable {
 private Connection con;
 private final Path SQLITE_FILE = Paths.get(System.getProperty(&quot;user.home&quot;), &quot;finalize.sqlite3&quot;);
 private final String SQLITE_URL = &quot;jdbc:sqlite:&quot; + SQLITE_FILE.toString();
 
 public PersonDAO() throws SQLException {
 con = DriverManager.getConnection(SQLITE_URL);
 
 String sql = &quot;CREATE TABLE IF NOT EXISTS people (&quot;
 + &quot;id integer PRIMARY KEY,&quot;
 + &quot;first_name text,&quot;
 + &quot;last_name text,&quot;
 + &quot;dob text);&quot;;
 Statement stmt = con.createStatement();
 stmt.execute(sql);
 }
 
 void create(Person person) throws SQLException {
 String sql = &quot;INSERT INTO people (first_name, last_name, dob) VALUES (?, ?, ?)&quot;;
 PreparedStatement stmt = con.prepareStatement(sql);
 stmt.setString(1, person.getFirstName());
 stmt.setString(2, person.getLastName());
 
 DateTimeFormatter fmt = DateTimeFormatter.ofPattern(&quot;yyyy-MM-dd&quot;);
 stmt.setString(3, person.getDob().format(fmt));
 stmt.executeUpdate();
 }
 
 @Override
 public void close() {
 System.out.println(&quot;Closing resource&quot;);
 try {
 con.close();
 } catch(SQLException e) {
 System.out.println(&quot;Uh, oh ... could not close db connection&quot;);
 }
 }
 }
 
 /* Simple Person data class */
 static class Person {
 // everything remains the same here ...
 }
}
</code></pre>
It is worth while to explain the try-with-resources contruct with a little more detail. Essentially the try-with-resource block translates to a full blown try / catch / finally block as shown below but, with the benefit of being cleaner in that everytime you use a class implementing <code>AutoCloseable#close</code> you do would use a try-with-resource rather than reimplementing the finally block in a try / catch / finally as shown below. Note that the <code>java.sql.Connection</code> class implements <code>AutoCloseable#close</code>.
<pre><code class="hljs">try {
 Connection con = DriverManager.getConnection(someUrl);
 // other stuff ...
} catch (SQLException e) {
 // logging ... 
} finally {
 try {
 con.close();
 } catch(Exception ex) {
 // logging ...
 }
}
</code></pre>
Would be best implemented like so:
<pre><code class="hljs">try (Connection con = DriverManager.getConnection(someUrl)) {
 // do stuff with con ...
} catch (SQLException e) {
 // logging ... 
}
</code></pre>
<h2 id="conclusion">Conclusion</h2>
In this article I purposefully described the <code>Object#finalize()</code> method in a fleeting manner due to the fact that it is not suggested that one should implement it. To contrast the lack of depth spent on the <code>finalize()</code> method I have described a preferrable approach to solving the problem of resource cleanup using the <code>AutoClosable</code> and try-with-resources duo.
As always, thanks for reading and don't be shy about commenting or critiquing below.

Adam McQuistan

Java's Object Methods: finalize()

<h2 id="introduction">Introduction</h2>
Time series analysis refers to the analysis of change in the trend of the data over a period of time. Time series analysis has a variety of applications. One such application is the prediction of the future value of an item based on its past values. Future stock price prediction is probably the best example of such an application. In this article, we will see how we can perform time series analysis with the help of a <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Recurrent_neural_network">recurrent neural network</a>. We will be predicting the future stock prices of the Apple Company (AAPL), based on its stock prices of the past 5 years.
<h2 id="dataset">Dataset</h2>
The data that we are going to use for this article can be downloaded from <a rel="nofollow" target="_blank" href="https://finance.yahoo.com/quote/AAPL/history?p=AAPL&.tsrc=fin-srch">Yahoo Finance</a>. For training our algorithm, we will be using the Apple stock prices from 1st January 2013 to 31 December 2017. For the sake of prediction, we will use the Apple stock prices for the month of January 2018. So in order to evaluate the performance of the algorithm, download the actual stock prices for the month of January 2018 as well.
Let's now see how our data looks. Open the Apple stock price training file that contains data for five years. You will see that it contains seven columns: Date, Open, High, Low, Close, Adj Close and Volume. We will be predicting the opening stock price, therefore we are not interested in the rest of the columns.
If you plot the opening stock prices against the date, you will see the following plot:
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/time-series-analysis-lstm-using-pythons-keras-library-1.png">
You can see that the trend is highly non-linear and it is very difficult to capture the trend using this information. This is where the power of <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Long_short-term_memory">LSTM</a> can be utilized. LSTM (Long Short-Term Memory network) is a type of recurrent neural network capable of remembering the past information and while predicting the future values, it takes this past information into account.
Enough of the preliminaries, let's see how LSTM can be used for time series analysis.
<h2 id="predictingfuturestockprices">Predicting Future Stock Prices</h2>
Stock price prediction is similar to any other machine learning problem where we are given a set of features and we have to predict a corresponding value. We will perform the same steps as we do perform in order to solve any machine learning problem. Follow these steps:
<h3 id="importlibraries">Import Libraries</h3>
The first step, as always is to import the required libraries. Execute the following script to do so:
<pre><code class="hljs">import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
</code></pre>
<h3 id="importdataset">Import Dataset</h3>
Execute the following script to import the data set. For the sake of this article, the data has been stored in the Datasets folder, inside the &quot;E&quot; drive. You can change the path accordingly.
<pre><code class="hljs">apple_training_complete = pd.read_csv(r&#x27;E:\Datasets\apple_training.csv&#x27;)
</code></pre>
As we said earlier, we are only interested in the opening price of the stock. Therefore, we will filter all the data from our training set and will retain only the values for the Open column. Execute the following script:
<pre><code class="hljs">apple_training_processed = apple_training_complete.iloc[:, 1:2].values
</code></pre>
<h3 id="datanormalization">Data Normalization</h3>
As a rule of thumb, whenever you use a neural network, you should normalize or scale your data. We will use <code>MinMaxScaler</code> class from the <code>sklear.preprocessing</code> library to scale our data between 0 and 1. The <code>feature_range</code> parameter is used to specify the range of the scaled data. Execute the following script:
<pre><code class="hljs">from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler(feature_range = (0, 1))

apple_training_scaled = scaler.fit_transform(apple_training_processed)
</code></pre>
<h3 id="converttrainingdatatorightshape">Convert Training Data to Right Shape</h3>
As I said earlier, in a time series problems, we have to predict a value at time T, based on the data from days T-N where N can be any number of steps. In this article, we are going to predict the opening stock price of the data based on the opening stock prices for the past 60 days. I have tried and tested different numbers and found that the best results are obtained when past 60 time steps are used. You can try different numbers and see how your algorithm performs.
Our feature set should contain the opening stock price values for the past 60 days while the label or dependent variable should be the stock price at the 61st day. Execute the following script to create feature and label set.
<pre><code class="hljs">features_set = []
labels = []
for i in range(60, 1260):
 features_set.append(apple_training_scaled[i-60:i, 0])
 labels.append(apple_training_scaled[i, 0])
</code></pre>
In the script above we create two lists: <code>feature_set</code> and <code>labels</code>. There are 1260 records in the training data. We execute a loop that starts from 61st record and stores all the previous 60 records to the <code>feature_set</code> list. The 61st record is stored in the <code>labels</code> list.
We need to convert both the <code>feature_set</code> and the <code>labels</code> list to the numpy array before we can use it for training. Execute the following script:
<pre><code class="hljs">features_set, labels = np.array(features_set), np.array(labels)
</code></pre>
In order to train LSTM on our data, we need to convert our data into the shape accepted by the LSTM. We need to convert our data into three-dimensional format. The first dimension is the number of records or rows in the dataset which is 1260 in our case. The second dimension is the number of time steps which is 60 while the last dimension is the number of indicators. Since we are only using one feature, i.e Open, the number of indicators will be one. Execute the following script:
<pre><code class="hljs">features_set = np.reshape(features_set, (features_set.shape[0], features_set.shape[1], 1))
</code></pre>
<h3 id="trainingthelstm">Training The LSTM</h3>
We have preprocessed our data and have converted it into the desired format. now is the time to create our LSTM. The LSTM model that we are going to create will be a sequential model with multiple layers. We will add four LSTM layers to our model followed by a dense layer that predicts the future stock price.
Let's first import the libraries that we are going to need in order to create our model:
<pre><code class="hljs">from keras.models import Sequential
from keras.layers import Dense
from keras.layers import LSTM
from keras.layers import Dropout
</code></pre>
In the script above we imported the <code>Sequential</code> class from <code>keras.models</code> library and <code>Dense</code>, <code>LSTM</code>, and <code>Dropout</code> classes from <code>keras.layers</code> library.
As a first step, we need to instantiate the <code>Sequential</code> class. This will be our model class and we will add LSTM, Dropout and Dense layers to this model. Execute the following script
<pre><code class="hljs">model = Sequential()
</code></pre>
<h4 id="creatinglstmanddropoutlayers">Creating LSTM and Dropout Layers</h4>
Let's add LSTM layer to the model that we just created. Execute the following script to do so:
<pre><code class="hljs">model.add(LSTM(units=50, return_sequences=True, input_shape=(features_set.shape[1], 1)))
</code></pre>
To add a layer to the sequential model, the <code>add</code> method is used. Inside the <code>add</code> method, we passed our LSTM layer. The first parameter to the LSTM layer is the number of neurons or nodes that we want in the layer. The second parameter is <code>return_sequences</code>, which is set to true since we will add more layers to the model. The first parameter to the <code>input_shape</code> is the number of time steps while the last parameter is the number of indicators.
Let's now add a dropout layer to our model. Dropout layer is added to avoid over-fitting, which is a phenomenon where a machine learning model performs better on the training data compared to the test data. Execute the following script to add dropout layer.
<pre><code class="hljs">model.add(Dropout(0.2))
</code></pre>
Let's add three more LSTM and dropout layers to our model. Run the following script.
<pre><code class="hljs">model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))

model.add(LSTM(units=50, return_sequences=True))
model.add(Dropout(0.2))

model.add(LSTM(units=50))
model.add(Dropout(0.2))
</code></pre>
<h4 id="creatingdenselayer">Creating Dense Layer</h4>
To make our model more robust, we add a dense layer at the end of the model. The number of neurons in the dense layer will be set to 1 since we want to predict a single value in the output.
<pre><code class="hljs">model.add(Dense(units = 1))
</code></pre>
<h4 id="modelcompilation">Model Compilation</h4>
Finally, we need to compile our LSTM before we can train it on the training data. The following script compiles the our model.
<pre><code class="hljs">model.compile(optimizer = &#x27;adam&#x27;, loss = &#x27;mean_squared_error&#x27;)
</code></pre>
We call the compile method on the Sequential model object which is &quot;model&quot; in our case. We use the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Mean_squared_error">mean squared error</a> as loss function and to reduce the loss or to optimize the algorithm, we use the <a rel="nofollow" target="_blank" href="https://machinelearningmastery.com/adam-optimization-algorithm-for-deep-learning/">adam</a> optimizer.
<h4 id="algorithmtraining">Algorithm Training</h4>
Now is the time to train the model that we defined in the previous few steps. To do so, we call the <code>fit</code> method on the <code>model</code> and pass it our training features and labels as shown below:
<pre><code class="hljs">model.fit(features_set, labels, epochs = 100, batch_size = 32)
</code></pre>
Depending upon your hardware, model training can take some time.
<h3 id="testingourlstm">Testing our LSTM</h3>
We have successfully trained our LSTM, now is the time to test the performance of our algorithm on the test set by predicting the opening stock prices for the month of January 2018. However, as we did with the training data, we need to convert our test data in the right format.
Let's first import our test data. Execute the following script:
<pre><code class="hljs">apple_testing_complete = pd.read_csv(r&#x27;E:\Datasets\apple_testing.csv&#x27;)
apple_testing_processed = apple_testing_complete.iloc[:, 1:2].values
</code></pre>
In the above script, we import our test data and as we did with the training data, we removed all the columns from the test data except the column that contains opening stock prices.
If the opening stock prices for the month of January 2018 are plotted against the dates, you should see the following graph.
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/time-series-analysis-lstm-using-pythons-keras-library-2.png">
You can see that the trend is highly non-linear. Overall, the stock prices see small rise at the start of the month followed by a downward trend at the end of the month, with a slight increase and decrease in the stock prices in-between. It is extremely difficult to forecast such a trend. Let's see if the LSTM we trained is actually able to predict such a trend.
<h4 id="convertingtestdatatorightformat">Converting Test Data to Right Format</h4>
For each day of January 2018, we want our feature set to contain the opening stock prices for the previous 60 days. For the 1st of January, we need the stock prices for the previous 60 days. To do so, we need to concatenate our training data and test data before preprocessing. Execute the following script to do so:
<pre><code class="hljs">apple_total = pd.concat((apple_training_complete[&#x27;Open&#x27;], apple_testing_complete[&#x27;Open&#x27;]), axis=0)
</code></pre>
Now let's prepare our test inputs. The input for each day should contain the opening stock prices for the previous 60 days. That means we need opening stock prices for the 20 test days for the month of January 2018 and the 60 stock prices from the last 60 days for the training set. Execute the following script to fetch those 80 values.
<pre><code class="hljs">test_inputs = apple_total[len(apple_total) - len(apple_testing_complete) - 60:].values
</code></pre>
As we did for the training set, we need to scale our test data. Execute the following script:
<pre><code class="hljs">test_inputs = test_inputs.reshape(-1,1)
test_inputs = scaler.transform(test_inputs)
</code></pre>
We scaled our data, now let's prepare our final test input set that will contain previous 60 stock prices for the month of January. Execute the following script:
<pre><code class="hljs">test_features = []
for i in range(60, 80):
 test_features.append(test_inputs[i-60:i, 0])
</code></pre>
Finally, we need to convert our data into the three-dimensional format which can be used as input to the LSTM. Execute the following script:
<pre><code class="hljs">test_features = np.array(test_features)
test_features = np.reshape(test_features, (test_features.shape[0], test_features.shape[1], 1))
</code></pre>
<h4 id="makingpredictions">Making Predictions</h4>
Now is the time to see the magic. We preprocessed our test data and now we can use it to make predictions. To do so, we simply need to call the <code>predict</code> method on the model that we trained. Execute the following script:
<pre><code class="hljs">predictions = model.predict(test_features)
</code></pre>
Since we scaled our data, the predictions made by the LSTM are also scaled. We need to reverse the scaled prediction back to their actual values. To do so, we can use the <code>ìnverse_transform</code> method of the scaler object we created during training. Take a look at the following script:
<pre><code class="hljs">predictions = scaler.inverse_transform(predictions)
</code></pre>
Finally, let's see how well did our algorithm predicted the future stock prices. Execute the following script:
<pre><code class="hljs">plt.figure(figsize=(10,6))
plt.plot(apple_testing_processed, color=&#x27;blue&#x27;, label=&#x27;Actual Apple Stock Price&#x27;)
plt.plot(predictions , color=&#x27;red&#x27;, label=&#x27;Predicted Apple Stock Price&#x27;)
plt.title(&#x27;Apple Stock Price Prediction&#x27;)
plt.xlabel(&#x27;Date&#x27;)
plt.ylabel(&#x27;Apple Stock Price&#x27;)
plt.legend()
plt.show()
</code></pre>
The output looks like this:
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/time-series-analysis-lstm-using-pythons-keras-library-3.png">
In the output, the blue line represents the actual stock prices for the month of January 2018, while the red line represents the predicted stock prices. You can clearly see that our algorithm has been able to capture the overall trend. The predicted prices also see a bullish trend at the beginning followed by a bearish or downwards trend at the end. Amazing, isn't it?
<h3 id="conclusion">Conclusion</h3>
A long short-term memory network (LSTM) is one of the most commonly used neural networks for time series analysis. The ability of LSTM to remember previous information makes it ideal for such tasks. In this article, we saw how we can use LSTM for the Apple stock price prediction. I would suggest that you download stocks of some other organization like Google or Microsoft from Yahoo Finance and see if your algorithm is able to capture the trends.