<h2 id="introduction">Introduction</h2>
It is an unwritten consensus that Python is one of the <a target="_blank" rel="nofollow" href="/why-beginners-should-learn-python/">best starting programming languages</a> to learn as a novice. It is extremely versatile, easy to read/analyze, and quite pleasant to the eye. The Python programming language is highly scalable and is widely considered as one of the best toolboxes to build tools and utilities that you may want to use for diverse reasons.
This article will briefly covers how Python handles one of the most important components of any operating system: its files and directories. Fortunately, Python has built-in functions to create and manipulate files, either <a rel="nofollow" target="_blank" href="https://github.com/upalr/Python-camp/wiki/1.-Introduction-and-flat-files">flat files</a> or <a rel="nofollow" target="_blank" href="https://ncoghlan-devs-python-notes.readthedocs.io/en/latest/python3/text_file_processing.html">text files</a>. The <code>io</code> module is the default module for accessing files, therefore we will not need to import any external library for general IO operations.
The key functions used for file handling in Python are: <code>open()</code>, <code>close()</code>, <code>read()</code>, <code>write()</code> and <code>append()</code>.
<h2 id="openingfileswithopen">Opening Files with <code>open()</code></h2>
This function returns a file object called &quot;handle&quot;, which is used to read from and write to a file. The arguments that the function can receive are as follows:
<pre><code class="hljs">open(filename, mode=&#x27;r&#x27;, buffering=-1, encoding=None, errors=None, newline=None, closefd=True, opener=None)
</code></pre>
Normally, only the <code>filename</code> and <code>mode</code> parameters are needed, while the others are used implicitly set with their default values.
The following code snippet shows how this function can be used:
<pre><code class="hljs">file_example = open (&quot;TestingText.txt&quot;) 
</code></pre>
This will open a text file called &quot;TestingText&quot; in read-only mode. Take note that only the filename parameter was specified, this is due to the &quot;read&quot; mode being the default mode for the open function.
The access modes available for the <code>open()</code> function are as follows:
<ul>
<li><code>r</code>: Opens the file in read-only mode. Starts reading from the beginning of the file and is the default mode for the <code>open()</code> function.</li>
<li><code>rb</code>: Opens the file as read-only in binary format and starts reading from the beginning of the file. While binary format can be used for different purposes, it is usually used when dealing with things like images, videos, etc.</li>
<li><code>r+</code>: Opens a file for reading and writing, placing the pointer at the beginning of the file.</li>
<li><code>w</code>: Opens in write-only mode. The pointer is placed at the beginning of the file and this will overwrite any existing file with the same name. It will create a new file if one with the same name doesn't exist.</li>
<li><code>wb</code>: Opens a write-only file in binary mode.</li>
<li><code>w+</code>: Opens a file for writing and reading.</li>
<li><code>wb+</code>: Opens a file for writing and reading in binary mode.</li>
<li><code>a</code>: Opens a file for appending new information to it. The pointer is placed at the end of the file. A new file is created if one with the same name doesn't exist.</li>
<li><code>ab</code>: Opens a file for appending in binary mode.</li>
<li><code>a+</code>: Opens a file for both appending and reading.</li>
<li><code>ab+</code>: Opens a file for both appending and reading in binary mode.</li>
</ul>
If both the Python file being executed and the target file to read doesn't exist in the same directory, we need to pass the full path of the file to read, to the <code>open()</code> function as shown in the following code snippet:
<pre><code class="hljs">file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;)
</code></pre>
Note: Something to keep in mind is to always make sure that both the file name and the path given are correct. If either is incorrect or doesn't exist, the error <code>FileNotFoundError</code> will be thrown, which needs to then be caught and handled by your program to prevent it from crashing.
To avoid this issue, as a best practice, errors can be caught with a try-finally block to handle the exception as shown below.
<pre><code class="hljs">try:
 file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;)
except IOError:
 print(&quot;An error was found. Either path is incorrect or file doesn&#x27;t exist!&quot;)

finally:
 print(&quot;Terminating process!&quot;)
</code></pre>
<h2 id="readingfromfileswithread">Reading from Files with <code>read()</code></h2>
Python contains 3 functions to read files: <code>read()</code>, <code>readline()</code>, and <code>readlines()</code>. The last two functions are merely helper functions that make reading certain types of files easier.
For the examples that will be used, &quot;TestingText.txt&quot; contains the following text:
<pre><code class="hljs">Hello, world! Python is the way to coding awesomeness.

If you don't believe me, try it on your own.

Come, you will enjoy the Dark Side. We have cookies!
</code></pre>
The read method is used as follows:
<pre><code class="hljs">file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;, &quot;r&quot;)

print(file_example.read())
</code></pre>
The output will be as follows:
<pre><code class="hljs">Hello, world! Python is the way to coding awesomeness.

If you don't believe me, try it on your own.

Come, you will enjoy the Dark Side. We have cookies!
</code></pre>
Note: Special characters may not be read correctly using the <code>read</code> function. To read special characters correctly, you can pass the encoding parameter to <code>read()</code> function and set its value to <code>utf8</code> as shown below:
<pre><code class="hljs">file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;, &quot;r&quot;, encoding=&quot;utf8&quot; )
</code></pre>
Also, the function <code>read()</code>, as well as the helper-function <code>readline()</code>, can receive a number as a parameter that will determine the number of bytes to read from the file. In the case of a text file, this will be the number of characters returned.
<pre><code class="hljs">file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;, &quot;r&quot;)

print(file_example.read(8))
</code></pre>
The output will be as follows:
<pre><code class="hljs">Hello, w
</code></pre>
The helper-function <code>readline()</code> behaves in a similar manner, but instead of returning the whole text, it will return a single line.
<pre><code class="hljs">file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;, &quot;r&quot;)

print(file_example.readline())
print(file_example.readline(5))
</code></pre>
In the script above, the first <code>print()</code> statement returns the first line and inserts a blank line in the output console. The next <code>print()</code> statement is separated from the previous line with a blank line and starts on a new line as shown in the output:
<pre><code class="hljs">Hello world! Python is the way to coding awesomeness.

If yo
</code></pre>
Finally, the helper-function <code>readlines()</code> reads all of the text and splits them in to lines for easy reading. Take a look at the following example:
<pre><code class="hljs">print(file_example.readlines())
</code></pre>
The output from this code will be:
<pre><code class="hljs">Hello world! Python is the way to coding awesomeness. If you don't believe me, try it on your own. Come, you will enjoy the Dark Side. We have cookies!
</code></pre>
Keep in mind that the <code>readlines()</code> function is considered to be <a rel="nofollow" target="_blank" href="https://stackoverflow.com/a/17246300">much slower and more inefficient</a> than the <code>read()</code> function, without not many benefits. One good alternative to this is using a loop that will function much smoother and faster:
<pre><code class="hljs">file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;, &quot;r&quot;)

for lines in file_example:
 print(lines) 
</code></pre>
Note that if the line is not printed, it will be replaced in the buffer by the next statement
<h2 id="writingtofileswithwrite">Writing to Files with <code>write()</code></h2>
When using this function, any information inside the file with the same name will be overwritten. Its behavior is similar to the <code>read()</code> function, but inserting information rather than reading it.
<pre><code class="hljs">file_example2 = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingTextTwo.txt&quot;, &quot;w&quot;)

file_example2.write(&quot;This is a test. Enjoy it!\n&quot;) #&#x27;\n&#x27; is for change line.

file_example2.write(&quot;Another thing to know is doing it slowly.\n&quot;)

file_example2.write(&quot;One by one. Yay!&quot;)
</code></pre>
If several lines need to be written, the sub-function <code>writelines()</code> could be used instead:
<pre><code class="hljs">listOfThingsToSay = [&quot;I like things like: \n&quot;, &quot;Ice Cream\n&quot;, &quot;Fruits\n&quot;, &quot;Movies\n&quot;, &quot;Anime\n&quot;, &quot;Naps\n&quot;, &quot;Jerky\n&quot;]

file_example2 = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingTextTwo.txt&quot;, &quot;w&quot;)

file_example2.writelines(listOfThingsToSay)
</code></pre>
Note: to be able to use <code>print()</code> function, the mode needs to be set as <code>w+</code>, which allows reading as well as writing.
<h2 id="addingtofileswithappend">Adding to Files with <code>append()</code></h2>
This function acts similar to the <code>write()</code> function, however, instead of overwriting the file, the <code>append()</code> function appends content to the existing file.
If a text file named &quot;TestingTextThree&quot; contains the following information:
<pre><code class="hljs">Some essential things are missing in life and should not be avoided.
</code></pre>
In order to append new text, the following code could be used:
<pre><code class="hljs">listOfThingsDo = [&quot;You need at least to: \n&quot;, &quot;Eat fried Ice Cream\n&quot;, &quot;Go to Disney\n&quot;, &quot;Travel to the moon\n&quot;, &quot;Cook Pineapple Pizza\n&quot;, &quot;Dance Salsa\n&quot;]

file_example3 = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingTextThree.txt&quot;, &quot;a+&quot;)

file_example3.writelines(listOfThingsToDo)

for newline in file_example3
 print(newlines)
</code></pre>
The output will be as follows:
<pre><code class="hljs">Some essential things are missing in life and should not be avoided.

You need at least to:

Eat fried Ice Cream

Go to Disney

Travel to the moon

Cook Pineapple Pizza

Dance Salsa
</code></pre>
<h2 id="closingopenedfileswithclose">Closing Opened Files with <code>close()</code></h2>
The <code>close()</code> function clears the memory buffer and closes the file. This means that we'll no longer be able to read from the file, and we'll have to re-open it if we want to read from it again. Also, some operating systems, such as Windows, treat opened files as locked, so it is important to clean up after yourself within your code.
Using the previously used sample code, this function is used as follows:
<pre><code class="hljs">file_example = open (&quot;F:\\Directory\\AnotherDirectory\\tests\\TestingText.txt&quot;, &quot;r&quot;)

print(file_example.read())

file_example.close()
</code></pre>
<h2 id="conclusion">Conclusion</h2>
Python is one of the most robust programming languages and is also one of the most used. It is easy to implement as well as analyze, making it a perfect tool for beginners in programming. Furthermore, its versatility makes it a perfect starting point for programming novices.
In regards to file handling, Python has easy-to-use functions with rapid response time and relatively resilient error handling methods, so the development and debug processes are much more pain-free than in other languages when it comes to working with files.

Guest Contributor

File Handling in Python

Usman Malik

Text Summarization with NLTK in Python

Beginner's Tutorial on the Pandas Python Library

The products we build often rely on multiple web servers and/or multiple database servers. In such cases, we often don’t have centralized tools for analyzing and storing logs. Under such circumstances, identifying different types of events and correlating them with other types of events is an almost impossible mission.
A single exception condition, somewhere in the middle of the system, can be disastrous for both the end user and the development team. A user may end up looking at a blank page after submitting payment for your service, or a large packet-loss can occur within your network, resulting in stretching a simple 10-minute job into a 10-hour headache.
The logical thing would be to know exactly what happened and how to prevent it from happening again. However, this means pulling logs from individual machines, aggregating them to take care of the clock skew and joining them through some kind of transaction id before you can even start asking questions.
This is all under the condition that the logs you should be checking stand in the computer's working memory or when <code>cat + grep + xargs</code> is not sufficient for analysis.
Nowadays, most IT infrastructures have moved to public clouds such as <a rel="nofollow" target="_blank" href="https://aws.amazon.com/">Amazon Web Services</a>, <a rel="nofollow" target="_blank" href="https://azure.microsoft.com/en-us/">Microsoft Azure</a>, <a rel="nofollow" target="_blank" href="https://www.digitalocean.com/">Digital Ocean</a>, or <a rel="nofollow" target="_blank" href="https://cloud.google.com">Google Cloud</a>.
Public cloud security tools and logging platforms both became necessary companions to these services. Even though they are separate tools, they were built to work together.
<a rel="nofollow" target="_blank" href="https://www.elastic.co/elk-stack">ELK Stack</a> is created exactly for such situations.
<h2 id="whatistheelkstack">What is the ELK Stack?</h2>
ELK stands for:
<ul>
<li><a rel="nofollow" target="_blank" href="https://www.elastic.co/products/elasticsearch">ElasticSearch</a> - Used for deep search and data analytics. It's an open source distributed NoSQL database, built in Java and based on Apache Lucene. Lucene takes care of storing disk data, indexing, and document scanning while ElasticSearch keeps document updates, APIs, and document distribution between ElasticSearch instances in the same cluster.</li>
<li><a rel="nofollow" target="_blank" href="https://www.elastic.co/products/logstash">Logstash</a> - Used for centralized logging, log enrichment, and parsing. It's an ETL (Extract, Transfer, Load) tool that transforms and stores logs within ElasticSearch.</li>
<li><a rel="nofollow" target="_blank" href="https://www.elastic.co/products/kibana">Kibana</a> - Used for powerful and beautiful data visualizations. It's a web-based tool through which ElasticSearch databases visualize and analyze previously stored data.</li>
</ul>
These three tools are necessary for analysis and visualization of events in the system.
<h2 id="loganalysis">Log Analysis</h2>
Performance isolation is hard to reach, particularly whenever systems are heavily loaded. Every relevant event within your system should be logged. It is better to overdo it with logging than to not have information about a potentially problematic event.
A good &quot;target&quot; for logging at the application level is everything related to any user interaction with the system - be it authorization of the user to the system or a user request to some URL, an email sent to the user, etc. Since you log a variety of information, this log is not structured at all, but it should have some basic information so that you can access it more easily.
The best advice regarding logging in distributed systems is to &quot;stamp&quot; any relevant event at the source, that is propagated in some way through the distributed system, whether it touches more parts of the system or not. In the case of a request for a web page, for an example, a load balancer or web server would be put on such a &quot;stamp&quot;. This sealed event is transmitted further until the end of its life. These &quot;stamps&quot; are often realized as UUIDs.
This ensures that the linearity of the event is not lost and that you can group and tie together events - for example, you can link that page request later with the query to the database that is necessary for the page to appear.
<h2 id="logstash">Logstash</h2>
When the event is logged in the log file, Logstash comes into play. Logstash ensures that your entries are transformed into one of the supported target formats that will later be submitted to the ElasticSearch server.
It collects data from different sources and then streams the data in the form of a data processing pipeline to the ElasticSearch instance.
It's safe to imagine ElasticSearch as a database and LogStash as the streaming component which pushes the logs or files on to it.
Logstash processing takes place in three stages:
<ul>
<li>Input - The entry, besides a file, can also be a syslog, <a rel="nofollow" target="_blank" href="https://redis.io/">Redis</a>, or <a rel="nofollow" target="_blank" href="https://github.com/elastic/logstash-forwarder/blob/master/PROTOCOL.md">Lumberjack</a> (now logstash-forwarder). Redis is often used as a pub/sub broker in a centralized Logstash infrastructure, where Logstash producers send their messages, and one of the instances processes them further.</li>
<li>Transformation and Filtering - This stage takes place according to the set of previously defined rules, where besides, for an example, a date format transformation, the IP address information (city/country of the IP) can be registered as well. The free <a rel="nofollow" target="_blank" href="https://dev.maxmind.com/geoip/geoip2/geolite2/">MaxMind</a> database is used for this purpose.</li>
<li>Output - The output can be transformed through plugins to almost any format, In our case, it will be ElasticSearch.</li>
</ul>
Let's have a look at a Logstash configuration file and an example of an application log:
logstash.conf:
<pre><code class="hljs">input {
 file {
 path =&gt; [&quot;/var/log/myapp/*.log&quot;]
 }
}

filter {
 // Collect or process some data (e.g. 'time') as timestamp and
 // store them into @timestamp data.
 // That data will later be used by Kibana.
 
 date {
 match =&gt; [ &quot;time&quot; ]
 }

 // Adds geolocation data based on the IP address.
 geoip {
 source =&gt; &quot;ip&quot;
 }
}

output {
 elasticsearch {
 hosts =&gt; [&quot;localhost:9200&quot;]
 }
 stdout { codec =&gt; rubydebug }
}
</code></pre>
Application log::
<pre><code class="hljs">{
 &quot;action&quot;: &quot;action log&quot;,
 &quot;user&quot;: &quot;John&quot;,
 &quot;time&quot;: 12:12:2012,
 &quot;ip&quot;: &quot;192.168....&quot;,
 &quot;transaction-id&quot;: &quot;r5e1244-32432-1465-q346-6ahsms57081x4&quot;
}
</code></pre>
<h2 id="elasticsearch">ElasticSearch</h2>
The moment Logstash finishes its job and forwards the logs, ElasticSearch can already process the data.
With the simple <code>curl</code> command <code>curl -XGET 'http://192.168.(host ip):9200/_search'</code> you can see &quot;documents&quot; in the database.
<code>192.168.(host ip)</code> is, in this case, the address of <a rel="nofollow" target="_blank" href="http://boot2docker.io/">boot2docker VM</a>, and <code>9200</code> is the default exposed port.
Since the result is in JSON format, your results are ready for further processing. <code>curl -XGET 'http: //192.168.(host ip):9200/_search?hello'</code> returns a more readable version of the document for quick inspection. Depending on the development, it's common practice to log the payload request/response, correlation ID, etc, so that you could track the error through Kibana's UI dashboard later.
<h2 id="kibana">Kibana</h2>
Kibana is basically just a static UI (HTML+ CSS + JS) client which displays the data in the way you wanted over the ElasticSearch instance where you see different reports and analytics. Except that through Kibana you can easily make queries over ElasticSearch indexes, the main advantage is that for that queries, no matter how complicated they can be, you can &quot;arrange&quot; them in a very transparent &quot;dashboard&quot; and that &quot;dashboard&quot; to save and share with others.
Kibana comes with a very powerful set of visualization tools and so, for example, you can see how often a similar event is repeated over time, you can aggregate events according to various criteria (for example, how many requests came from a particular IP address in the last hour, whether the number of errors on a particular server has increased, or even notice an increase in the number of requests for a particular page). Also, this is a great tool for detecting anomalies that are caused by changes in your system.
<img class="img-responsive" alt="Kibana" src="https://s3.amazonaws.com/stackabuse/media/introduction-elk-stack-1.png">
<h2 id="tryingouttheelkstack">Trying out the ELK Stack</h2>
To be able to try the ELK stack and try some of the commands we are about to cover, you'll need to install ELK and Docker, or boot2docker, locally.
<a rel="nofollow" target="_blank" href="http://www.docker.com/">Docker</a> is a tool designed to make it easier to create, deploy, and run applications by using containers.
Containers allow a developer to package up an application with all of the parts it needs, such as libraries and other dependencies, and ship it all out as one package. This article will not cover Docker and boot2docker in detail - the official resources are more than excellent and it's very easy to set up this combination.
For the purpose of this test, we will use <a rel="nofollow" target="_blank" href="https://registry.hub.docker.com/u/sebp/elk/">ELK Docker Image</a>:
<pre><code class="hljs">$ docker pull sebp/elk
$ docker run -p 5601:5601 -p 9200:9200 -p 5000:5000 -it --name elk sebp/elk
$ docker exec -it elk /bin/bash
$ /logstash/bin/logstash -e &#x27;input { stdin { } } output { elasticsearch { host =&gt; localhost } }&#x27;
# Now write the message e.g.
This is a test message.
CTRL-C
</code></pre>
From the host machine, we'll use <code>curl</code>:
<pre><code class="hljs">$ curl -s -XGET &#x27;http://192.168.(host ip):9200/_search?hello&amp;q=message&#x27;
</code></pre>
The result will look like this:
<pre><code class="hljs">{
 &quot;took&quot; : 3,
 &quot;timed_out&quot; : false,
 &quot;_shards&quot; : {
 &quot;total&quot; : 6,
 &quot;successful&quot; : 6,
 &quot;failed&quot; : 0
 },
 &quot;hits&quot; : {
 &quot;total&quot; : 1,
 &quot;max_score&quot; : 0.4790727,
 &quot;hits&quot; : [ {
 &quot;_index&quot; : &quot;logstash-2018.08.26&quot;,
 &quot;_type&quot; : &quot;logs&quot;,
 &quot;_id&quot; : &quot;Z8kCtFYHOWWinjAN6pHUqE&quot;,
 &quot;_score&quot; : 0.5720636,
 &quot;_source&quot;:{&quot;message&quot;:&quot;This is a test message.&quot;,&quot;@version&quot;:&quot;1&quot;,&quot;@timestamp&quot;:&quot;2018-08-26T09:41:17.634Z&quot;,&quot;host&quot;:&quot;5136b631e113&quot;}
 } ]
 }
}
</code></pre>
As you can see, our search resulted in 1 log instance to be returned to us in JSON format.
<h2 id="elasticsearchanddatalossonnetworkpartitions">ElasticSearch and Data Loss on Network Partitions</h2>
As with all other tools, ELK Stack comes with its own set of problems. During the previous year and this year, numerous articles have been written about how ElasticSearch loses data due to network partition, even if the network partition conditions last only a few seconds.
<a rel="nofollow" target="_blank" href="https://aphyr.com/posts/323-call-me-maybe-elasticsearch-1-5-0">Here</a> you can find more information on how ElasticSearch behaves during a transitive, non-transit network partition, as well as in the singular node partition.
What is interesting is that even micro-partitions can cause a lag of 90 seconds within which the new leader within the cluster works and everything that was inserted during that cluster can be potentially discarded, so it's a good idea to keep your logs in the original format and not rely on 100% on ElasticSearch, even for &quot;mission-critical&quot; logs.
<h2 id="conclusion">Conclusion</h2>
With this, we only scratched the surface and presented a very small set of possibilities of the ELK Stack.
The problems solved in this article are quick and simple analysis of logs and visualization of events in the system, using a monolithic interface that can serve as a basis for further instrumentation.
Happy logging!

Vuk Skobalj

Introduction to the ELK Stack

<h2 id="copyingfilesinjava">Copying Files in Java</h2>
Copying a file or directory used to be a typical development task. With the introduction of Docker containers and a desire for maximum immutability, we see it less and less often.
Still, it's a fundamental concept, and it might be useful to know what options does the developer have when they need to copy a file.
<h3 id="iostreams">I/O Streams</h3>
Before Java 1.5, the standard way to copy a file was by utilizing <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/tutorial/essential/io/streams.html">I/O Streams</a>:
<pre><code class="hljs">private void copyFile(File src, File dest) throws IOException {
 InputStream inputStream = null;
 OutputStream outputStream = null;
 try {
 inputStream = new FileInputStream(source);
 outputStream = new FileOutputStream(dest);

 // the size of the buffer doesn&#x27;t have to be exactly 1024 bytes, try playing around with this number and see what effect it will have on the performance
 byte[] buffer = new byte[1024];
 int length = 0;
 while ((length = is.read(buffer)) &gt; 0) {
 os.write(buffer, 0, length);
 }
 } finally {
 is.close();
 os.close();
 }
}
</code></pre>
<h3 id="channelsandjavanio">Channels and java.nio</h3>
With the major refactoring of the Java platform and Java 1.4 release in 2000, the package <code>java.nio</code> was introduced.
The <code>File</code> copying process started to look a bit more elegant and lean:
<pre><code class="hljs">private void copyFileUsingChannel(File src, File dest) throws IOException {
 FileChannel sourceChannel = null;
 FileChannel destinationChannel = null;
 try {
 sourceChannel = new FileInputStream(src).getChannel();
 destinationChannel = new FileOutputStream(dest).getChannel();
 destinationChannel.transferFrom(sourceChannel, 0, sourceChannel.size());
 } finally {
 sourceChannel.close();
 destinationChannel.close();
 }
}
</code></pre>
The most notable difference is that we got rid of the while loop and we don't control the buffer size directly anymore.
<h3 id="fileshelperclass">Files Helper Class</h3>
Along with Java 7, released in 2011, a helper class was added that lets us copy files in a single line of code:
<pre><code class="hljs">private static void copyFile(File src, File dest) throws IOException {
 Files.copy(src.toPath(), dest.toPath());
}
</code></pre>
Taking a look at the <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/tutorial/essential/io/copy.html">API documentation</a>, we see that it's possible to provide an additional option to the <code>copy()</code> method.
The following options from the <code>StandardCopyOption</code> and <code>LinkOption</code> enums are supported:
<ul>
<li>
REPLACE_EXISTING – Performs the copy even when the target file already exists. If the target is a symbolic link, the link itself is copied (and not the target of the link). If the target is a non-empty directory, the copy fails with the FileAlreadyExistsException exception.
</li>
<li>
COPY_ATTRIBUTES – Copies the file attributes associated with the file to the target file. The exact file attributes supported are file system and platform dependent, but last-modified-time is supported across platforms and is copied to the target file.
</li>
<li>
NOFOLLOW_LINKS – Indicates that symbolic links should not be followed. If the file to be copied is a symbolic link, the link is copied (and not the target of the link).
</li>
</ul>
<h3 id="apachecommonsioandgoogleguava">Apache Commons IO and Google Guava</h3>
These two very handy helper libraries provide a very similar API for copying files. The only downside of using any of those is having to append another heavy <code>.jar</code> file into the project.
But if it's already there, then why not?
<pre><code class="hljs">private void copyFileApacheCommons(File src, File dest) throws IOException {
 FileUtils.copyFile(src, dest);
}
</code></pre>
<pre><code class="hljs">private void copyFileGoogleGuava(File src, File dest) throws IOException {
 Files.copy(src, dest);
}
</code></pre>
<h3 id="bonus">Bonus</h3>
<blockquote>
Wait, but what I have a String representing the filename and not the actual File object?
</blockquote>
No worries, we have it covered!
<pre><code class="hljs">private File filenameToFile(String filename) throws IOException {
 return new File(filename);
}
</code></pre>
<h3 id="whichoneshouldyouuse">Which one should you use?</h3>
<a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/tutorial/essential/io/streams.html">I/O Streams</a> and <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/7/docs/api/java/nio/channels/package-summary.html">Channels</a> are a viable option to play around and understand how does the process look from the inside.
Another one would be if you're working on a legacy project with an outdated version of Java, and updating it isn't an option. In all other cases, you should check whether <a rel="nofollow" target="_blank" href="https://commons.apache.org/">Apache Commons or <a rel="nofollow" target="_blank" href="https://github.com/google/guava">Google Guava</a> is already in the list of dependencies and choose between one of those.
If they're not, then you shouldn't add them for the sake of this single utility method and use Java 7 <code>Files</code> helper class.

Vasyl Lagutin

How to Copy a File in Java

Text Classification with Python and Scikit-Learn

David Landup

Structural Design Patterns in Java

<h2 id="introduction">Introduction</h2>
In this guide, we'll take a look at how to integrate the H2 database with Spring Boot projects.
<h2 id="whatistheh2database">What is the H2 Database?</h2>
<a rel="nofollow" target="_blank" href="http://www.h2database.com/">H2</a> is an open-source, Java-based, embedded database. It's very fast and very lightweight. Typically, it's used as an in-memory database which means it stores the data in memory and will not persist data on disk. Although if we need to persist the data, at the flick of a switch - you can persist data as well.
<blockquote>
The H2 database is not recommended for production environments and is rather used for proofs of concept, tests, prototypes and similar applications.
</blockquote>
<h2 id="differencebetweenpersistentandinmemorydatabases">Difference Between Persistent and In-Memory Databases</h2>
It is worth understanding the difference between persistent and in-memory databases. Persistent databases persist the data in physical memory - that's the entire point of databases. In the case of in-memory databases, data is stored in system memory and data will be lost when the program is closed. In essence - it's just like using RAM to store your objects.
Once the program is closed, the data is lost.
<blockquote>
Why use in-memory databases at all then?
</blockquote>
In-memory databases are helpful for proofs of concept, as you can easily substitute an in-memory database with a persistent one. It mocks the functionality and backgone of an actual persistent database, while giving you the ease of use and flexibility of performing queries and &quot;persisting&quot; objects, while still being lightweight.
Although H2 does allow you to persist data outside of memory - don't be tempted to replace a more robust database, such as PostgreSQL, MySQL, OracleDB, etc. with it.
<h2 id="settingupthespringbootproject">Setting up the Spring Boot Project</h2>
Let's start out with a skeleton application. The easiest way is to use <a rel="nofollow" target="_blank" href="https://start.spring.io/">Spring Initializr</a> and generate a Spring Boot project.
<blockquote>
Make sure to add Web, H2 and JPA (Java Persistence API) dependencies.
</blockquote>
Again, even though H2 typically isn't used as a persistent database - it mocks classical databases, and it interfaces with the Persistence API.
Clicking on the &quot;Generate Project&quot; button will download the <code>SpringBootH2Integration.zip</code> file, which you can import into your IDE of choice:
<img class="img-responsive" alt="Spring Initializr" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-1.png">
<h2 id="understandingh2sdefaultsconfigurations">Understanding H2's Defaults Configurations</h2>
Spring Boot is an opinionated framework and does all the default configurations based on the dependencies available in the classpath.
Since we added H2 as a dependency, Spring Boot knows that in this project we will be connecting to the H2 database, so it auto-configures H2-related properties like the database <code>URL</code>, <code>username</code>, <code>password</code>, etc:
<pre><code class="hljs">spring.datasource.url=jdbc:h2:mem:testdb
spring.datasource.driverClassName=org.h2.Driver
spring.datasource.username=sa
spring.datasource.password=
spring.h2.console.enabled=false
</code></pre>
If we look at the <code>datasource.url</code> property, the name of the schema is <code>testdb</code> and is an in-memory database (refered to as &quot;mem&quot;). We will see how we can override the default values in upcoming sections.
<h2 id="overridingh2sdefaultconfigurations">Overriding H2's Default Configurations</h2>
To override any property, just redefine the property with the updated value in the <code>application.properties</code> file available under the <code>src/main/resources</code> folder.
For example, by default H2 console is disabled, so let's enable it by adding below property in the <code>application.properties</code> file:
<pre><code class="hljs">spring.h2.console.enabled=true
</code></pre>
<h2 id="performingcrudoperationswithh2inspringboot">Performing CRUD Operations with H2 in Spring Boot</h2>
In the following sections, we are going to create a Person API, which is a simple REST service in which we will interact with H2 using Spring Boot.
<blockquote>
If you're not yet familiar with the creation of REST APIs, read our <a target="_blank" href="https://stackabuse.com/build-a-spring-boot-rest-api-with-java-full-guide/">Complete Guide to Building REST APIs with Spring Boot</a>!
</blockquote>
The Person API will perform CRUD operations using different endpoints. Below are the details of endpoints:
<ol>
<li><code>GET</code> call to <code>/persons</code> will display all person objects available in H2 database.</li>
<li><code>GET</code> call to <code>/persons/{id}</code> will display the person object with the given id.</li>
<li><code>POST</code> call to <code>/persons</code> will insert a new person object in the database.</li>
<li><code>DELETE</code> call to <code>/persons/{id}</code> will delete the person object with the given id from the database.</li>
</ol>
Before moving further, it makes sense to verify that the project we have created in the previous section is working fine and there are no errors in it.
To verify, run the <code>SpringBootH2IntegrationApplication.java</code> file the way we run any Java program. Doing so, the application will start and we can see the server startup message in the console.
From the console, we can find the URL where we can open the H2 console:
<pre><code class="hljs">2018-08-18 22:43:30.578 INFO 3792 --- [ost-startStop-1] o.s.b.w.servlet.ServletRegistrationBean : Servlet webServlet mapped to [/h2-console/*]
</code></pre>
Recall that we did enable the H2 console in the <code>application.properties</code> file.
Let's open the H2 console by typing in <a rel="nofollow" target="_blank" href="http://localhost:8080/h2-console">http://localhost:8080/h2-console</a> in the browser address bar:
<img class="img-responsive" alt="H2 console 1" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-6.png">
Click connect to log in and see the various options:
<img class="img-responsive" alt="H2 console 2" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-7.png">
You may see different value in the JDBC URL so change the Database URL to <code>jdbc:h2:mem:testdb</code> in the login screen as this is the default URL configured by Spring Boot.
<h2 id="implementingarestapiwithcrudfunctionalities">Implementing a REST API with CRUD Functionalities</h2>
Finally, let's spin up a simple REST API that performs CRUD operations on a <code>Person</code> entity.
<h3 id="defininganentity">Defining an Entity</h3>
Naturally, we'll be starting out with our domain model and a definition of the <code>Person</code> model:
<pre><code class="hljs">@Entity
public class Person {

 @Id
 @GeneratedValue
 private int id;
 private String name;
 private int age;
 private String emailId;

 // Getters and setters
}
</code></pre>
<h3 id="definingapersonrepository">Defining a PersonRepository</h3>
Utilizing the best of Spring Data JPA - we'll create a really simple, auto-generated <code>CrudRepository</code> for our entities, named <code>PersonRepository</code>:
<pre><code class="hljs">public interface PersonRepository extends CrudRepository&lt;Person, Integer&gt; {}
</code></pre>
Without any further implementation, the <code>PersonRepository</code> has CRUD functionality through <code>save()</code>, <code>findById()</code>, <code>deleteById()</code> and <code>update()</code> methods, which accept any valid <code>Person</code> entity.
<h3 id="definingapersonservice">Defining a PersonService</h3>
Once the repository interface is ready - let's create a concrete service to use the repository to actually perform the reading, writing and updating:
<pre><code class="hljs">@Service
public class PersonService {

 @Autowired
 PersonRepository personRepository;

 public List&lt;Person&gt; findAll() {
 return personRepository.findAll();
 }

 public Person getPersonById(int id) {
 return personRepository.findById(id).get();
 }

 public void saveOrUpdate(Person person) {
 personRepository.save(person);
 }

 public void delete(int id) {
 personRepository.deleteById(id);
 }
}
</code></pre>
<h3 id="definingapersoncontroller">Defining a PersonController</h3>
Finally - to expose the service to end-users, we'll create a <code>@Controller</code>-annotated class, <code>PersonController</code> and expose a few endpoints:
<pre><code class="hljs">@RestController
public class PersonController {

 @Autowired
 PersonService personService;

 @GetMapping(&quot;/persons&quot;)
 private List&lt;Person&gt; getAllPersons() {
 return personService.findAll();
 }

 @GetMapping(&quot;/persons/{id}&quot;)
 private Person getPerson(@PathVariable(&quot;id&quot;) int id) {
 return personService.getPersonById(id);
 }

 @DeleteMapping(&quot;/persons/{id}&quot;)
 private void deletePerson(@PathVariable(&quot;id&quot;) int id) {
 personService.delete(id);
 }

 @PostMapping(&quot;/persons&quot;)
 private int savePerson(@RequestBody Person person) {
 personService.saveOrUpdate(person);
 return person.getId();
 }
}
</code></pre>
Great! We can now test out the functionality of the API and the responses we get.
<h2 id="testingthepersonapi">Testing the Person API</h2>
To test the REST service, we'll be using <a target="_blank" href="https://stackabuse.com/getting-started-with-postman/">Postman</a>, which can be integrated into the Chrome browser easily using their <a rel="nofollow" target="_blank" href="https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en">browser extension</a> or used as a standalone tool.
<h3 id="testcase1creatinganewperson">Test Case 1: Creating a New Person</h3>
Using postman, we'll configure a valid <code>POST</code> request for the creation of users:
<ol>
<li>Set Request Type: POST</li>
<li>Set URL: <a href="http://localhost:8080/persons">http://localhost:8080/persons</a></li>
<li>Set Request Header: Content-Type:application/json</li>
<li>Set Body as: {&quot;name&quot;:&quot;person A&quot;, &quot;age&quot;:23, &quot;emailId&quot;: &quot;<a href="mailto:person.a@tutorial.com">person.a@tutorial.com</a>&quot;}</li>
<li>Click &quot;Send&quot;</li>
<li>In the response we will get a <code>personId</code>.</li>
</ol>
<img class="img-responsive" alt="Test 1 Postman" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-8.png">
To validate the data inserted in the H2 database:
<ol>
<li>Open the <a rel="nofollow" target="_blank" href="http://localhost:8080/h2-console">H2 console</a></li>
<li>Validate the <code>PERSON</code> table. It will show us the data we have inserted in POST request.</li>
</ol>
<img class="img-responsive" alt="Test 1 H2" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-9.png">
Repeat Test Case 1 with a different person and you'll have two person objects inserted in the database:
<img class="img-responsive" alt="Test 11 H2" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-10.png">
<h3 id="testcase2retrievingthenewpersonobject">Test Case 2: Retrieving the new Person Object</h3>
Now, let's try to retrieve one of these users, via their ID. To do this, we'll construct a new request in postman:
<ol>
<li>Request Type: GET</li>
<li>URL: <a href="http://localhost:8080/persons/1">http://localhost:8080/persons/1</a></li>
<li>Request Header: Content-Type:application/json</li>
<li>Click &quot;Send&quot;</li>
<li>In response, we will get person data with id 1.</li>
</ol>
<img class="img-responsive" alt="Test 2 Postman" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-11.png">
<h3 id="testcase3deletingapersonobject">Test Case 3: Deleting a Person Object</h3>
Finally, let's try to delete a person from the database:
<ol>
<li>Request Type: DELETE</li>
<li>URL: <a href="http://localhost:8080/persons/1">http://localhost:8080/persons/1</a></li>
<li>Request Header: Content-Type:application/json</li>
<li>Click &quot;Send&quot;</li>
</ol>
<img class="img-responsive" alt="Test 3 Postman" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-12.png">
After running this, validate that data has been deleted from H2 database:
<img class="img-responsive" alt="Test 3 H2" src="https://s3.amazonaws.com/stackabuse/media/integrating-h2-database-spring-boot-13.png">
<h2 id="conclusion">Conclusion</h2>
In this short guide, we've taken a look at how to integrate the in-memory H2 database with Spring Boot, and built a simple REST API around it to showcase the usage.
H2 is a lightweight, embedded database written in Java, typically used for prototyping.