<h2 id="introduction">Introduction</h2>
There are many ways to go about <a target="_blank" href="/reading-and-writing-files-in-java/">Reading and Writing Files in Java</a>.
We typically have some data in memory, on which we perform operations, and then persist in a file. However, if we want to change that information, we need to put the contents of the file back into memory and perform operations.
If, for an example, our file contains a long list that we want to sort, we'll have to read it into an adequate data structure, perform operations, and then persist it once again - in this case an <code>ArrayList</code>.
This can be achieved with several different approaches:
<ul>
<li><code>Files.readAllLines()</code></li>
<li><code>FileReader</code></li>
<li><code>Scanner</code></li>
<li><code>BufferedReader</code></li>
<li><code>ObjectInputStream</code></li>
<li>Java Streams API</li>
</ul>
<h2 id="filesreadalllines">Files.readAllLines()</h2>
Since Java 7, it's possible to load all lines of a file into an <code>ArrayList</code> in a very simple way:
<pre><code class="hljs">try {
 ArrayList&lt;String&gt; lines = new ArrayList&lt;&gt;(Files.readAllLines(Paths.get(fileName)));
}
catch (IOException e) {
 // Handle a potential exception
}
</code></pre>
We can also specify a <code>charset</code> to handle different formats of text, if necessary:
<pre><code class="hljs">try {
 Charset charset = StandardCharsets.UTF_8;
 ArrayList&lt;String&gt; lines = new ArrayList&lt;&gt;(Files.readAllLines(Paths.get(fileName), charset));
}
catch (IOException e) {
 // Handle a potential exception
}
</code></pre>
<code>Files.readAllLines()</code> opens and closes the necessary resources automatically.
<h2 id="scanner">Scanner</h2>
As nice and simple as the previous method was, it's only useful for reading the file line by line. What would happen if all the data was stored in a single line?
<code>Scanner</code> is an easy to use tool for parsing primitive types and Strings. Using <code>Scanner</code> can be as simple or as hard as the developer wants to make it.
A simple example of when we'd prefer to use <code>Scanner</code> would be if our file had only one line, and the data needs to be parsed into something usable.
A delimiter is a sequence of characters that <code>Scanner</code> uses to separate values. By default, it uses a series of spaces/tabs as a delimiter (whitespace between values), but we can declare our own delimiter and use it to parse the data.
Let's take a look at an example file:
<pre><code class="hljs">some-2123-different-values- in - this -text-with a common-delimiter
</code></pre>
In such a case, it's easy to notice that all the values have a common delimiter. We can simply declare that &quot;-&quot; surrounded by any number of whitespaces is our delimiter.
<pre><code class="hljs">// We&#x27;ll use &quot;-&quot; as our delimiter
ArrayList&lt;String&gt; arrayList = new ArrayList&lt;&gt;();
try (Scanner s = new Scanner(new File(fileName)).useDelimiter(&quot;\\s*-\\s*&quot;)) {
 // \\s* in regular expressions means &quot;any number or whitespaces&quot;.
 // We could&#x27;ve said simply useDelimiter(&quot;-&quot;) and Scanner would have
 // included the whitespaces as part of the data it extracted.
 while (s.hasNext()) {
 arrayList.add(s.next());
 }
}
catch (FileNotFoundException e) {
 // Handle the potential exception
}
</code></pre>
Running this piece of code would yield us an <code>ArrayList</code> with these items:
<pre><code class="hljs">[some, 2, different, values, in, this, text, with a common, delimiter]
</code></pre>
On the other hand, if we had only used the default delimiter (whitespace), the <code>ArrayList</code> would look like:
<pre><code class="hljs">[some-2-different-values-, in, -, this, -text-with, a, common-delimiter]
</code></pre>
<code>Scanner</code> has some useful functions for parsing data, such as <code>nextInt()</code>, <code>nextDouble()</code>, etc.
Important: Calling <code>.nextInt()</code> will NOT return the next <code>int</code> value that can be found in the file! It will return an <code>int</code> value only if the next items the <code>Scanner</code> &quot;scans&quot; is a valid <code>int</code> value, otherwise an exception will be thrown. An easy way to make sure an exception doesn't arise is to perform a corresponding &quot;has&quot; check - like <code>.hasNextInt()</code> before actually using <code>.nextInt()</code>.
Even though we don't see that when we call functions like <code>scanner.nextInt()</code> or <code>scanner.hasNextDouble()</code>, <code>Scanner</code> uses regular expressions in the background.
Very Important: An extremely common mistake with using <code>Scanner</code> occurs when working with files that have multiple lines and using <code>.nextLine()</code> in conjunction with <code>.nextInt()</code>,<code>nextDouble()</code>, etc.
Let's take a look at another file:
<pre><code class="hljs">12
some data we want to read as a string in one line
10
</code></pre>
Oftentimes, newer developers who use <code>Scanner</code> would write code like:
<pre><code class="hljs">try (Scanner scanner = new Scanner(new File(&quot;example.txt&quot;))) {
 int a = scanner.nextInt();
 String s = scanner.nextLine();
 int b = scanner.nextInt();

 System.out.println(a + &quot;, &quot; + s + &quot;, &quot; + b);
}
catch (FileNotFoundException e) {
 // Handle a potential exception
}
//catch (InputMismatchException e) {
// // This will occur in the code above
//}
</code></pre>
This code seems to be logically sound - we read an integer from the file, then the following line, then the second integer. If you try to run this code, the <code>InputMismatchException</code> will be thrown without an obvious reason.
If you start debugging and printing what you've scanned, you'll see that <code>int a</code> loaded well, but that <code>String s</code> is empty.
Why is that? The first important thing to note is that once <code>Scanner</code> reads something from the file, it continues scanning the file from the first character after the data it previously scanned.
For example, if we had &quot;12 13 14&quot; in a file and called <code>.nextInt()</code> once, the scanner would afterward pretend as if there was only &quot; 13 14&quot; in the file. Notice that the space between &quot;12&quot; and &quot;13&quot; is still present.
The second important thing to note - the first line in our <code>example.txt</code> file doesn't only contain the number <code>12</code>, it contains what it called a &quot;newline character&quot;, and it is actually <code>12\n</code> instead of just <code>12</code>.
Our file, in reality, looks like this:
<pre><code class="hljs">12\n
some data we want to read as a string in one line\n
10
</code></pre>
When we first call <code>.nextInt()</code>, <code>Scanner</code> reads only the number 12, and leaves the first <code>\n</code> unread.
<code>.nextLine()</code> then reads all the characters that scanner hasn't read yet until it reaches the first <code>\n</code> character, which it skips over and then returns the characters it read. This is exactly what the problem is in our case - we have a leftover <code>\n</code> character after reading the <code>12</code>.
So when we call <code>.nextLine()</code> we get an empty string as a result since <code>Scanner</code> doesn't add the <code>\n</code> character to the string it returns.
Now the <code>Scanner</code> is at the beginning of the second line in our file, and when we try to call <code>.nextInt()</code>, <code>Scanner</code> encounters something that can't be parsed to an <code>int</code> and throws the aforementioned <code>InputMismatchException</code>.
<h3 id="solutions">Solutions</h3>
<ul>
<li>Since we know what exactly is wrong in this code, we can hardcode a workaround. We'll simply &quot;consume&quot; the newline character between <code>.nextInt()</code> and <code>.nextLine()</code>:</li>
</ul>
<pre><code class="hljs">...
int a = scanner.nextInt();
scanner.nextLine(); // Simply consumes the bothersome \n
String s = scanner.nextLine();
...
</code></pre>
<ul>
<li>Given that we know how <code>example.txt</code> is formatted we can read the entire file line by line and parse the necessary lines using <code>Integer.parseInt()</code>:</li>
</ul>
<pre><code class="hljs">...
int a = Integer.parseInt(scanner.nextLine());
String s = scanner.nextLine();
int b = Integer.parseInt(scanner.nextLine());
...
</code></pre>
<h2 id="bufferedreader">BufferedReader</h2>
<code>BufferedReader</code> reads text from a character-input stream, but it does so by buffering characters in order to provide efficient <code>.read()</code> operations. Since accessing an HDD is a very time-consuming operation, <code>BufferedReader</code> gathers more data than we ask for, and stores it in a buffer.
The idea is that when we call <code>.read()</code> (or similar operation) we are likely to read again soon from the same block of data from which we've just read, and so &quot;surrounding&quot; data is stored in a buffer. In case we wanted to read it, we'd read it directly from the buffer instead of from the disk, which is much more efficient.
This brings us to what <code>BufferedReader</code> is good for - reading large files. <code>BufferedReader</code> has a significantly larger buffer memory than <code>Scanner</code> (8192 characters by default vs 1024 characters by default, respectively).
<code>BufferedReader</code> is used as a wrapper for other Readers, and so constructors for <code>BufferedReader</code> take a Reader object as a parameter, such as a <code>FileReader</code>.
We're using <a target="_blank" href="/the-try-with-resources-statement-in-java/">try-with-resources</a> so we don't have to close the reader manually:
<pre><code class="hljs">ArrayList&lt;String&gt; arrayList = new ArrayList&lt;&gt;();

try (BufferedReader reader = new BufferedReader(new FileReader(fileName))) {
 while (reader.ready()) {
 arrayList.add(reader.readLine());
 }
}
catch (IOException e) {
 // Handle a potential exception
}
</code></pre>
It's advised to wrap a <code>FileReader</code> with a <code>BufferedReader</code>, exactly due to the performance benefits.
<h2 id="objectinputstream">ObjectInputStream</h2>
<code>ObjectInputStream</code> should only be used alongside <code>ObjectOutputStream</code>. What these two classes help us accomplish is to store an object (or array of objects) into a file, and then easily read from that file.
This can only be done with classes that implement the <code>Serializable</code> interface. The <code>Serializable</code> interface has no methods or fields and serves only to identify the semantics of being serializable:
<pre><code class="hljs">public static class MyClass implements Serializable {
 int someInt;
 String someString;

 public MyClass(int someInt, String someString) {
 this.someInt = someInt;
 this.someString = someString;
 }
}

public static void main(String[] args) throws IOException, ClassNotFoundException {
 // The file extension doesn&#x27;t matter in this case, since they&#x27;re only there to tell
 // the OS with what program to associate a particular file
 ObjectOutputStream objectOutputStream =
 new ObjectOutputStream(new FileOutputStream(&quot;data.olivera&quot;));

 MyClass first = new MyClass(1, &quot;abc&quot;);
 MyClass second = new MyClass(2, &quot;abc&quot;);

 objectOutputStream.writeObject(first);
 objectOutputStream.writeObject(second);
 objectOutputStream.close();

 ObjectInputStream objectInputStream =
 new ObjectInputStream(new FileInputStream(&quot;data.olivera&quot;));

 ArrayList&lt;MyClass&gt; arrayList = new ArrayList&lt;&gt;();

 try (objectInputStream) {
 while (true) {
 Object read = objectInputStream.readObject();
 if (read == null)
 break;

 // We should always cast explicitly
 MyClass myClassRead = (MyClass) read;
 arrayList.add(myClassRead);
 }
 }
 catch (EOFException e) {
 // This exception is expected
 }

 for (MyClass m : arrayList) {
 System.out.println(m.someInt + &quot; &quot; + m.someString);
 }
}
</code></pre>
<h2 id="javastreamsapi">Java Streams API</h2>
Since Java 8, another quick and easy way to load the contents of a file into an <code>ArrayList</code> would be using the <a target="_blank" href="/introduction-to-java-8-streams/">Java Streams API</a>:
<pre><code class="hljs">// Using try-with-resources so the stream closes automatically
try (Stream&lt;String&gt; stream = Files.lines(Paths.get(fileName))) {
 ArrayList&lt;String&gt; arrayList = stream.collect(Collectors.toCollection(ArrayList::new));
}
catch (IOException e) {
 // Handle a potential exception
}
</code></pre>
Though, keep in mind that this approach, just like <code>Files.readAllLines()</code> would only work if the data is stored in lines.
The code above doesn't do anything special, and we'd rarely use streams this way. However, since we're loading this data into an <code>ArrayList</code> so that we can process it in the first place - streams provide an excellent way of doing this.
We can easily sort/filter/map the data before storing it in an <code>ArrayList</code>:
<pre><code class="hljs">try (Stream&lt;String&gt; stream = Files.lines(Paths.get(fileName))) {
 ArrayList&lt;String&gt; arrayList = stream.map(String::toLowerCase)
 .filter(line -&gt; !line.startsWith(&quot;a&quot;))
 .sorted(Comparator.comparing(String::length))
 .collect(Collectors.toCollection(ArrayList::new));
}
catch (IOException e) {
 // Handle a potential exception
}
</code></pre>
<h2 id="conclusion">Conclusion</h2>
There are several different ways in which you can read data from a file into an <code>ArrayList</code>. When you only need to read the lines as elements use <code>Files.readAllLines</code>; when you have data that can be easily parsed use <code>Scanner</code>; when working with large files use <code>FileReader</code> wrapped with <code>BufferedReader</code>; when dealing with an array of objects use <code>ObjectInputStream</code> (but make sure the data was written using <code>ObjectOutputStream</code>).

Olivera Popović

Java: Read a File into an ArrayList

Guest Contributor

Introduction to the Python lxml Library

Usman Malik

Python for NLP: Topic Modeling

Jacob Stopak

Securing Spring Boot Web Applications

<h2 id="introduction">Introduction</h2>
try-with-resources is one of the several <code>try</code> statements in Java, aimed to relieve developers of the obligation to release resources used in a <code>try</code> block.
It was initially introduced in Java 7 and the whole idea behind it was that the developer doesn't need to worry about resource management for resources they use only in one try-catch-finally block. This is achieved by removing the need for <code>finally</code> blocks, which developers only used to close resources in practice.
Additionally, code using try-with-resources is often cleaner and more readable, therefore, making the code easier to manage, especially when we're dealing with many <code>try</code> blocks.
<h2 id="syntax">Syntax</h2>
The syntax for try-with-resources is almost identical to the usual try-catch-finally syntax. The only difference are the parentheses after <code>try</code> in which we declare which resources we'll be using:
<pre><code class="hljs">BufferedWriter writer = null;
try {
 writer = new BufferedWriter(new FileWriter(fileName));
 writer.write(str); // do something with the file we&#x27;ve opened
} catch (IOException e) {
 // handle the exception
} finally {
 try {
 if (writer != null)
 writer.close();
 } catch (IOException e) {
 // handle the exception
 }
}
</code></pre>
The same code written using try-with-resources would look like this:
<pre><code class="hljs">try(BufferedWriter writer = new BufferedWriter(new FileWriter(fileName))){
 writer.write(str); // do something with the file we&#x27;ve opened
}
catch(IOException e){
 // handle the exception
}
</code></pre>
The way Java understands this code:
<blockquote>
The resources opened in the parentheses after the try statement will only be needed here and now. I'll call their <code>.close()</code> methods as soon as I'm done with the work in the try block. If an exception is thrown while in the try block, I'll close those resources anyway.
</blockquote>
Before this approach was introduced, closing resources was done manually, as seen in the code before. This was essentially boilerplate code, and codebases were littered with them, reducing readability and making them harder to maintain.
The <code>catch</code> and <code>finally</code> part of try-with-resources work as expected, with <code>catch</code> blocks handling their respective exceptions and the <code>finally</code> block executed regardless of whether there was an exception or not. The only difference are suppressed exceptions, which are explained at the end of this article.
Note: Since Java 9, it isn't necessary to declare the resources within the try-with-resources statement. We can do something like this instead:
<pre><code class="hljs">BufferedWriter writer = new BufferedWriter(new FileWriter(fileName));
try (writer) {
 writer.write(str); // do something with the file we&#x27;ve opened
}
catch(IOException e) {
 // handle the exception
}
</code></pre>
<h3 id="multipleresources">Multiple Resources</h3>
Another good aspect of try-with-resources is the ease of adding/removing resources we're using while having the assurance that they will be closed after we've finished.
If we wanted to work with several files, we'd open the files in the <code>try()</code> statement and separate them with a semi-colon:
<pre><code class="hljs">try (BufferedWriter writer = new BufferedWriter(new FileWriter(fileName));
 Scanner scanner = new Scanner(System.in)) {
if (scanner.hasNextLine())
 writer.write(scanner.nextLine());
}
catch(IOException e) {
 // handle the exception
}
</code></pre>
Java then takes care to call <code>.close()</code> on all resources we've opened in <code>try()</code>.
Note: They're closed in reverse declaration order, meaning that, in our example, <code>scanner</code> will be closed before the <code>writer</code>.
<h2 id="supportedclasses">Supported Classes</h2>
All the resources declared in <code>try()</code> must implement the <code>AutoCloseable</code> interface. These are usually various types of writers, readers, sockets, output or input streams, etc. Anything you'd need to write <code>resource.close()</code> after you've finished working with it.
This, of course, includes user-defined objects that implement the <code>AutoClosable</code> interface. Though, you'll rarely come across a situation where you wish to write your own resources.
In case that happens, you need to implement the <code>AutoCloseable</code> or <code>Closeable</code> (only there to preserve backward compatibility, prefer <code>AutoCloseable</code>) interface and override the <code>.close()</code> method:
<pre><code class="hljs">public class MyResource implements AutoCloseable {
 @Override
 public void close() throws Exception {
 // close your resource in the appropriate way
 }
}
</code></pre>
<h2 id="exceptionhandling">Exception Handling</h2>
If an exception is thrown from within a Java try-with-resources block, any resource opened inside the parentheses of the <code>try</code> block will still get closed automatically.
As mentioned previously, try-with-resources works the same as try-catch-finally, except with one small addition. The addition is called suppressed exceptions. It is not necessary to understand suppressed exceptions in order to use try-with-resources, but reading about them might be useful for debugging when nothing else seems to work.
Imagine a situation:
<ul>
<li>For some reason, an exception occurs in the try-with-resources block</li>
<li>Java stops execution in the try-with-resources block, and calls <code>.close()</code> on all the resources declared in <code>try()</code></li>
<li>One of the <code>.close()</code> methods throws an exception</li>
<li>Which exception would the <code>catch</code> block &quot;catch&quot;?</li>
</ul>
This situation introduces us to the aforementioned suppressed exceptions. A suppressed exception is an exception which is in a way ignored when it's thrown within the implicit finally block of a try-with-resources block, in the case where an exception is thrown from the <code>try</code> block as well.
Those exceptions are exceptions that occur in the <code>.close()</code> methods and they are accessed differently than &quot;regular&quot; exceptions.
It's important to understand that the order of execution is:
<ol>
<li>try-with-resources block</li>
<li>implicit finally</li>
<li>catch block (if an exception was thrown in [1] and/or [2])</li>
<li>(explicit) finally</li>
</ol>
For example, here's a resource that does nothing but throw exceptions:
<pre><code class="hljs">public static class MyResource implements AutoCloseable {
 // method throws RuntimeException
 public void doSomething() {
 throw new RuntimeException(&quot;From the doSomething method&quot;);
 }

 // we&#x27;ll override close so that it throws an exception in the implicit finally block of try-with-resources (when it attempts to close our resource)
 @Override
 public void close() throws Exception {
 throw new ArithmeticException(&quot;I can throw whatever I want, you can&#x27;t stop me.&quot;);
 }
}

public static void main(String[] arguments) throws Exception {
 // declare our resource in try
 try (MyResource resource = new MyResource()) {
 resource.doSomething();
 }
 catch (Exception e) {
 System.out.println(&quot;Regular exception: &quot; + e.toString());

 // getting the array of suppressed exceptions, and its length
 Throwable[] suppressedExceptions = e.getSuppressed();
 int n = suppressedExceptions.length;

 if (n &gt; 0) {
 System.out.println(&quot;We&#x27;ve found &quot; + n + &quot; suppressed exceptions:&quot;);
 for (Throwable exception : suppressedExceptions) {
 System.out.println(exception.toString());
 }
 }
 }
}
</code></pre>
This code is runnable. You can use it to experiment with using multiple <code>MyResource</code> objects or seeing what happens when try-with-resources doesn't throw an exception, but <code>.close()</code> does.
Hint: Suddenly, exceptions thrown while closing the resources start being important.
It's important to note that in case a resource throws an exception when you try to close it, any other resources opened within the same try-with-resources block will still get closed.
Another fact to note is that in a situation where the <code>try</code> block doesn't throw an exception, and where there are multiple exceptions thrown while attempting to <code>.close()</code> the resources used, the first exception will be propagated up the call stack, while the others will be suppressed.
As you can see in the code, you can get the list of all suppressed exceptions by accessing the <code>Throwable</code> array returned by <code>Throwable.getSuppressed()</code>.
Remember, only a single exception can be thrown inside the try block. As soon as an exception is thrown, the try block code is exited, and Java tries to close the resources.
<h2 id="conclusion">Conclusion</h2>
try-with-resources should be used instead of the regular try-catch-finally whenever possible. It's easy to forget to close one of your resources after coding for hours or forgetting to close a resource you've just added to your program after a random burst of inspiration.
The code is more readable, easier to change and maintain, and usually shorter.

The try-with-resources Statement in Java

<h2 id="overview">Overview</h2>
In this article, we'll introduce you to routing your applications via <a rel="nofollow" target="_blank" href="https://cloud.spring.io/spring-cloud-netflix/multi/multi__router_and_filter_zuul.html">Netflix's Zuul</a> and <a rel="nofollow" target="_blank" href="https://spring.io/projects/spring-cloud-gateway">Spring Cloud Gateway</a>.
In a typical microservice architecture we have many small applications running on different hosts and ports. The problem in this type of architecture is how clients (Web Applications in browsers, Mobile apps, Third-party apps making a web service call, etc.) can access these end microservices without knowing their hosts and ports. For example, browser restricts calls to different domains (also known as <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Cross-origin_resource_sharing">CORS</a>).
What we need is a common entry point to our microservices. Using this, we not only free the clients from knowing deployment details about all of the backend services, but also reduce development effort on the server side. At the same time, if an end microservice has multiple instances running, we can do load balancing at this entry point.
Furthermore, we can also write all of the authentication and authorization mechanisms at this level. This reduces significant development on the end microservices side.
<h2 id="edgeservers">Edge Servers</h2>
To solve this problem, Netflix created Zuul server and later open-sourced it. Spring provided a nice wrapper around it for easily incorporating it to the Spring stack.
Note: Netflix recently released <a rel="nofollow" target="_blank" href="https://github.com/Netflix/zuul/wiki/Getting-Started-2.0">Zuul 2</a>, but Spring hasn't added it to its ecosystem yet. Because of this we'll be using Zuul 1 in this article.
Spring also released its own router called Spring Cloud Gateway. It has non-blocking APIs and supports long-lived connections like WebSockets.
We will look into both of these solutions in this article. The architecture diagram looks like:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-0.png" alt="spring-cloud-routing-with-zuul-gateway">
<h2 id="setup">Setup</h2>
This article assumes that you already have knowledge of <a target="_blank" href="/spring-cloud-service-discovery-with-eureka/">Netflix's Eureka project</a>, which is used as a service registry and for load balancing. We have the following setup up for the backend service:
<ul>
<li>Eureka Server: Acting as a service registry and running on port <code>8761</code>.</li>
<li>User Service: A simple REST service that has a single endpoint of <code>/getPublicAddress</code> and is running on port <code>8100</code>.</li>
<li>Product Service: A simple REST service that has a single endpoint of <code>/categories</code> and is running on port <code>8200</code>.</li>
</ul>
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-1.png" alt="spring-cloud-routing-with-zuul-gateway-eureka-setup">
<h2 id="netflixzuul">Netflix Zuul</h2>
The best way to start with a skeleton project is to use <a rel="nofollow" target="_blank" href="https://start.spring.io/">Spring Initializr</a>. Select your preferred version of Spring Boot and add the &quot;Zuul&quot; and &quot;Eureka Discovery&quot; dependencies, and generate as a Maven project:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-3.png" alt="spring-zuul-server-setup">
To make it a Zuul proxy server, all we need to do is add the <code>@EnableZuulProxy</code> annotation to our main class:
<pre><code class="hljs">@SpringBootApplication
@EnableZuulProxy
public class ZuulApplication {

 public static void main(String[] args) {
 SpringApplication.run(ZuulApplication.class, args);
 }
}
</code></pre>
We will be running the Zuul server on port <code>8050</code> and it also needs to register itself to the Eureka server. So in <code>application.properties</code> we'll add the following:
<pre><code class="hljs">server.port=8050

spring.application.name=zuul-edge-server
 
eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka/
</code></pre>
Let's start this server and navigate your browser to the Eureka server at <a rel="nofollow" target="_blank" href="http://localhost:8761/">http://localhost:8761/</a>:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-3.png" alt="spring-zuul-server-display">
Now that Zuul has been registered with Eureka, let's test routing to our user-service through it by navigating the browser to the <a rel="nofollow" target="_blank" href="http://localhost:8050/user-service/getPublicAddress">http://localhost:8050/user-service/getPublicAddress</a> endpoint:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-4.png" alt="spring-zuul-route-user-service">
Similarly, for the product-service navigate your browser to <a rel="nofollow" target="_blank" href="http://localhost:8050/product-service/categories">http://localhost:8050/product-service/categories</a>:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-5.png" alt="spring-zuul-route-product-service">
As you can see, we are calling the backend services through Zuul. By default, Eureka client IDs become part of the URIs. For example, here we made a call to Zuul using <code>/product-service/categories</code>. Zuul will check if there is any service registered as <code>product-service</code> in Eureka. If it's there, it will get the URL for the service and append the remaining original URL part, <code>/categories</code> to it and make the call.
Also, Zuul is <a rel="nofollow" target="_blank" href="https://spring.io/guides/gs/client-side-load-balancing/">Ribbon</a> aware, so it will automatically load balance the call if there are multiple instance of the backend service running.
The defaults can, of course, be changed by tweaking the properties file, which can be found <a rel="nofollow" target="_blank" href="https://cloud.spring.io/spring-cloud-netflix/multi/multi__router_and_filter_zuul.html">here</a>. It's also not necessary for all backend services to be registered on Eureka. We can also route to other domains too.
<h2 id="springcloudgateway">Spring Cloud Gateway</h2>
Let's see another popular edge server called Spring Cloud Gateway, which is built on Spring Framework 5, <a target="_blank" href="/spring-reactor-tutorial/">Project Reactor</a> and Spring Boot 2.0. Once again let's create a new project with Spring Initializr. Select your preferred version of Spring Boot and add the &quot;Gateway&quot; and &quot;Eureka Discovery&quot; dependencies, and generate as a Maven project:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-6.png" alt="spring-gateway-server-setup">
We will be running the Zuul server on port <code>8060</code> and it also needs to register itself to the Eureka server. So in <code>application.properties</code> we'll add:
<pre><code class="hljs">server.port=8060

spring.application.name=gateway-edge-server
 
eureka.client.serviceUrl.defaultZone=http://localhost:8761/eureka/

spring.cloud.gateway.discovery.locator.enabled=true
spring.cloud.gateway.discovery.locator.lowerCaseServiceId=true
</code></pre>
Unlike Zuul, Spring Cloud Gateway doesn't automatically look in Eureka for routing calls. So we enabled it by adding a couple of additional properties.
Let's start this server and navigate your browser to the Eureka server at <a rel="nofollow" target="_blank" href="http://localhost:8761/">http://localhost:8761/</a>:
<img src="https://s3.amazonaws.com/stackabuse/media/spring-cloud-routing-with-zuul-gateway-7.png" alt="spring-zuul-server-display">
Similar to the previous example, we can test our routing to user-service and product-service by navigating our browser to <a rel="nofollow" target="_blank" href="http://localhost:8060/user-service/getPublicAddress">http://localhost:8060/user-service/getPublicAddress</a> and <a rel="nofollow" target="_blank" href="http://localhost:8060/product-service/categories">http://localhost:8060/product-service/categories</a>, respectively.
Just like Zuul, Spring Cloud Gateway checks for a service in Eureka by the first path variable. Other ways to change default are in its documentation, which can be found <a rel="nofollow" target="_blank" href="https://cloud.spring.io/spring-cloud-gateway/single/spring-cloud-gateway.html">here</a>.
<h2 id="conclusion">Conclusion</h2>
In this article, we've covered how to use Spring Cloud Zuul and Gateway for routing traffic to backend microservices. We created two simple REST services that registered with Eureka server. We then created the Zuul server that also registered with Eureka and then routes traffic based on it. We then saw an alternate approach with Spring Cloud Gateway.
As always, the code for the examples used in this article can be found on <a rel="nofollow" target="_blank" href="https://github.com/StackAbuse/spring-cloud/tree/master/spring-routing">GitHub</a>.

Dhananjay Singh

Spring Cloud: Routing with Zuul and Gateway

Python for NLP: Sentiment Analysis with Scikit-Learn

<h2 id="introduction">Introduction</h2>
Web Browser Automation is gaining popularity, and many frameworks/tools have arose to offer automation services to developers.
Web Browser Automation is often used for testing purposes in development and production environments, though it's also often used for web scraping data from public sources, analysis, and data processing.
Really, what you do with automation is up to you, though, just make sure that what you're doing is legal, as &quot;bots&quot; created with automation tools can often infringe laws or a site's terms of service.
<a rel="nofollow" target="_blank" href="https://www.seleniumhq.org/">Selenium</a> is one of the widely used tools used for Web Browser Automation, and offers a lot of functionality and power over a browser.
It supports many languages such as C#, Java, Perl, PHP, and Ruby, though for the sake of this tutorial, we'll be using it with Python on Windows.
<h2 id="whatisselenium">What is Selenium?</h2>
Selenium is a great tool that allows developers to simulate end-users with only a few lines of code. Using the tools it offers, it's very easy to use web pages and simulate a human, though it's hard to really replicate human behavior.
To combat &quot;bots&quot;, which are meant to replicate humans, many sophisticated systems are used to recognize human-like behavior, which is border-line impossible to replicate using programming tools.
If you're building an application with Selenium, make sure that you adhere to all laws associated with Web Browser Automation, or simply use it for testing purposes in your own production environment.
Some of the most popular tasks accomplished with Selenium include, but are not limited to:
<ul>
<li>Clicking buttons</li>
<li>Inputting text</li>
<li>Extracting text</li>
<li>Accessing Cookies</li>
<li>Pressing keys</li>
</ul>
<h2 id="prerequisites">Prerequisites</h2>
Before we get started, we'll need to do a few things to get set up:
<ul>
<li><a rel="nofollow" target="_blank" href="https://www.google.com/chrome/">Install Google Chrome</a> on your computer - we will be simulating a user on Google Chrome, although you could simulate other browsers, but for this article I'll be using Google Chrome.</li>
<li>Get <a rel="nofollow" target="_blank" href="https://sites.google.com/a/chromium.org/chromedriver/downloads">chromedriver.exe</a>, because in order to actually simulate the browser you'll need a path to this executable on your computer.</li>
<li>Install the <a rel="nofollow" target="_blank" href="https://pypi.org/project/selenium/">selenium package</a> using <code>pip install selenium</code> on the command line.</li>
</ul>
<h2 id="thebasics">The Basics</h2>
Alright, now we're all set to start working with Selenium. The first thing you'll need to do is start the browser:
<pre><code class="hljs">from selenium import webdriver

EXE_PATH = r&#x27;path\to\chromedriver.exe&#x27;
driver = webdriver.Chrome(executable_path=EXE_PATH)
driver.get(&#x27;https://google.com&#x27;)
</code></pre>
Running this will open Google Chrome and navigate it to <code>https://google.com</code>.
Here, it's important to note that the connection to the web page is made with the <code>get(URL)</code> function of the <code>driver</code> object.
As you might have noticed, the <code>driver</code> is the Selenium <code>WebDriver</code> object, you use it to access the browser programmatically, for example:
<pre><code class="hljs">print(driver.page_source)
</code></pre>
The code above prints the source HTML code of the entire page. If you need to collect data, this is very useful.
<h2 id="locatingelements">Locating Elements</h2>
Usually, you don't need the contents of an entire page, but rather specific elements.
In order to do so, you'll first need to detect your target on the page, and for that you can use the Inspect Element tool in Google Chrome.
That means that if you need the contents of a certain element in the page, to get the tags ID you can do the following (in a regular session of Google Chrome):
<ul>
<li>Right click on the element</li>
<li>Choose &quot;Inspect&quot;</li>
<li>And in the new window, you can take a look at the HTML of the element and the ID will be after <code>id=</code>.</li>
</ul>
Upon getting the elements we need, we can perform different kinds of operations on them.
<h3 id="gettingelementsbyid">Getting Elements by ID</h3>
If you have an exact ID of the element you're looking for, it's easy to retrieve it:
<pre><code class="hljs">element = driver.find_element_by_id(&#x27;element_id&#x27;)
</code></pre>
<h3 id="gettingelementsbyname">Getting Elements by Name</h3>
Similar to the previous approach:
<pre><code class="hljs">element = driver.find_element_by_name(&#x27;element_name&#x27;)
</code></pre>
<h3 id="gettingelementsbyclass">Getting Elements by Class</h3>
And again, similar to the previous approach:
<pre><code class="hljs">element = driver.find_element_by_class_name(&#x27;element_class_name&#x27;)
</code></pre>
<h3 id="gettingelementsbyhtmltag">Getting Elements by HTML Tag</h3>
In some cases, you might want to get all elements by a certain tag:
<pre><code class="hljs">links = driver.find_elements_by_tag_name(&#x27;a&#x27;)
</code></pre>
In this case, <code>links</code> is populated with all <code>a</code> tags, which now contains each link in the page. This can be useful for web-crawling purposes.
<h3 id="gettingelementsbyxpath">Getting Elements by XPath</h3>
Not all elements have an ID, or maybe you don't want to access every <code>a</code> HTML tag. There are other ways to retrieve a very specific element, like <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/XPath">XPath</a>, which is another way to retrieve elements. With XPath, you can find elements more easily and efficiently:
<pre><code class="hljs">tag_list = driver.find_elements_by_xpath(&quot;//tag[@attr=&#x27;val&#x27;]&quot;)
</code></pre>
<code>tag_list</code> now contains each <code>tag</code> that has and attribute <code>attr</code> set to <code>val</code>:
<pre><code class="hljs">&lt;tag attr=&#x27;val&#x27;&gt;Foo&lt;/tag&gt;
</code></pre>
You can now iterate <code>tag_list</code>, and interact with each Selenium <code>WebElement</code> in it.
You can read more about the XPath system in Selenium <a rel="nofollow" target="_blank" href="https://www.guru99.com/xpath-selenium.html">here</a>.
<h2 id="seleniumwebelement">Selenium WebElement</h2>
A Selenium <code>WebElement</code> practically represents an HTML element. You can perform operations on these elements similar to how you'd do it as an end-user.
These operations include:
<ul>
<li>Accessing simple properties of the element, like the text inside (<code>element.text</code>)</li>
<li>Accessing parent elements, which are also of type <code>WebElement</code> (<code>element.parent</code>)</li>
<li>Accessing specific attributes, like the <code>href</code> of an <code>a</code> tag (<code>element.get_attribute('href')</code>)</li>
<li>Searching within it (the same way you'd search in <code>driver</code>)</li>
<li>Clicking it (<code>element.click()</code>)</li>
<li>Inputting text if possible (<code>element.send_keys(‘Input Text')</code>)</li>
</ul>
<h2 id="seleniumwebdriver">Selenium WebDriver</h2>
<code>WebDriver</code> is similar to <code>WebElement</code>, however, the main difference is their scope. The latter's scope is the element itself, whereas the former's scope is the whole page.
You can do plenty of things with a Selenium <code>WebDriver</code> object as well, practically anything you could do as a human with a normal browser.
Some other very useful things are:
<ul>
<li>Executing JavaScript: <code>driver.execute_script(&quot;script&quot;)</code></li>
<li>Saving a screenshot: <code>driver.save_screenshot('image.png')</code></li>
<li>Initiate in &quot;headless mode&quot;, where the browser saves time by not rendering the page:</li>
</ul>
<pre><code class="hljs">from selenium.webdriver.chrome.options import Options

options = Options()
options.headless = True
driver = webdriver.Chrome(executable_path=EXE_PATH, chrome_options=options)
driver.set_window_size(1440, 900)
</code></pre>
Note that the window size is set to <code>(1440, 900)</code>, that is to prevent all sorts of bugs regarding some elements not loading properly because of the headless mode.
You could change the resolution to any other reasonably large resolution, but you have to make sure the resolution is changed from the defaults when going in headless mode.
<h2 id="navigatingthepage">Navigating the Page</h2>
<h3 id="accessingcookies">Accessing Cookies</h3>
You might find yourself in need to add or remove <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/HTTP_cookie">browser cookies</a>:
<pre><code class="hljs">ck = {&#x27;some_attr&#x27;: &#x27;foo&#x27;, &#x27;some_other_attr&#x27;: &#x27;bar&#x27;}
driver.add_cookie(ck)
</code></pre>
This adds a cookie to the browser, which can be helpful if you need to add authentication or preference cookies, for example. It's important to make sure that the cookie is in <code>dict</code> format.
It's also very easy to retrieve the cookies from the browser:
<pre><code class="hljs">cookies = driver.get_cookies()
for ck in cookies:
 print(ck)
</code></pre>
The code above prints each cookie in the browser.
<h3 id="alteringthehtml">Altering the HTML</h3>
Sometimes you might find yourself in need of changing a certain element's property.
As mentioned before, you can use a Selenium <code>WebDriver</code> to execute JavaScript, and changing properties of elements just so happens to be very easy to do with JavaScript:
<pre><code class="hljs">driver.execute_script(&quot;arguments[0].setAttribute(&#x27;attr&#x27;,&#x27;value&#x27;)&quot;, element)
</code></pre>
Here <code>element</code> is the element to alter, <code>attr</code> is the attribute to change and <code>value</code> is the new value.
<h3 id="downloadingfilesusingdownloadlinks">Downloading Files using Download Links</h3>
Sometimes you might need to download a file from a website:
<pre><code class="hljs">from selenium import webdriver
from selenium.webdriver.chrome.options import Options

options = Options()
options.add_experimental_option(&quot;prefs&quot;, {
 &quot;download.default_directory&quot;: r&quot;path\to\directory&quot;,
 &quot;download.prompt_for_download&quot;: False,
 &quot;download.directory_upgrade&quot;: True,
 &quot;safebrowsing.enabled&quot;: True
})

driver = webdriver.Chrome(executable_path=EXE_PATH, chrome_options=options)
</code></pre>
You can specify the path of the save location by defining <code>download.default_directory</code>, such as <code>path\to\directory</code>.
<h3 id="pressingkeys">Pressing Keys</h3>
<pre><code class="hljs">import time
from selenium.webdriver import ActionChains
from selenium.webdriver.common.keys import Keys

action = ActionChains(driver)
for _ in range(3):
 action.send_keys(Keys.ARROW_DOWN)
 time.sleep(.1)
action.perform()
</code></pre>
This code presses the down arrow (<code>Keys.ARROW_DOWN</code>) 3 times. After each <code>action.send_keys(Keys.ARROW_DOWN)</code> the program waits a little bit. This is recommended to make sure all the keys register.
If we simply fired off several commands, they might get lost in the process and won't actually register.
<code>Keys</code> contains all of the keys on the keyboard, meaning that you can also use this method to tab (<code>Keys.TAB</code>) between elements on the page making it easier to interact with it (<code>Keys.RETURN</code> and <code>Keys.SPACE</code> are very important as well).
<h3 id="clickingbuttons">Clicking Buttons</h3>
Note that you can use key presses to navigate between elements in a page, for example you can use Tabs and Spaces to fill in checkboxes, and use the arrow keys to navigate between dropdown menu items.
Of course, a more natural way to select checkboxes and dropdown items would be to simply retrieve the element using the driver and click it:
<pre><code class="hljs">checkbox = driver.find_element_by_id(&#x27;checkbox&#x27;)
checkbox.click()
</code></pre>
<h3 id="inputtingforms">Inputting Forms</h3>
You can also simulate key presses within elements themselves:
<pre><code class="hljs">element.send_keys(Keys.CONTROL, &#x27;a&#x27;)
element.send_keys(value)
</code></pre>
This way, the keys register inside the <code>element</code>, so that if you would like to fill in a <code>textarea</code>, you could do it like so.
By the way, this code uses a keyboard shortcut (CTRL + A) to select all text inside the element. The next line replaces the selected text with the <code>value</code> entered.
To register keyboard shortcuts, pass all of the desired keys in the parameters to <code>send_keys</code>.
<h3 id="scrolling">Scrolling</h3>
Sometimes parts of the page load only after you scroll down (like an Instagram feed or any other infinite scrolling page). This can easily be done via executing a simple JavaScript script:
<pre><code class="hljs">driver.execute_script(&quot;window.scrollTo(0, document.body.scrollHeight);&quot;)
</code></pre>
The code above uses a JavaScript command to scroll to the bottom of the page, now you can use <code>driver.page_source</code> again and get the new content.
<h2 id="conclusion">Conclusion</h2>
Selenium is one of the widely used tools used for Web Browser Automation, and offers a lot of functionality and power over a human-controlled browser.
It's mainly used for production or integration environment testing/automatization, though it can also be used as a web scraper for research purposes, etc. Be sure to adhere to all laws when you practice web scraping of public content in order to not infringe on any laws.