In the current age where Data Science / AI is booming, it is important to understand how Machine Learning is used in the industry to solve complex business problems. In order to select which Machine Learning model should be used in production, a selection metric is chosen upon which different machine learning models are scored.
One of the most commonly used metrics nowadays is <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic">AUC-ROC</a> (Area Under Curve - Receiver Operating Characteristics) curve. ROC curves are pretty easy to understand and evaluate once there is a good understanding of the confusion matrix and different kinds of errors.
In this article, I will explain the following topics:
<ul>
<li>Introduction to confusion matrix and different statistic computed on it</li>
<li>Definitions of TP, FN, TN, FP</li>
<li>Type 1 and Type 2 errors</li>
<li>Statistics computed from Recall, Precision, F-Score</li>
<li>Introduction to AUC ROC Curve</li>
<li>Different scenarios with ROC Curve and Model Selection</li>
<li>Example of ROC Curve with Python</li>
</ul>
<h2 id="introductiontoconfusionmatrix">Introduction to Confusion Matrix</h2>
In order to showcase the predicted and actual class labels from the Machine Learning models, the <a rel="nofollow" target="_blank" href="https://en.wikipedia.org/wiki/Confusion_matrix">confusion matrix</a> is used. Let us take an example of a binary class classification problem.
<img class="img-responsive" alt="Confusion Matrix" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-1.jpg">
The class labeled 1 is the positive class in our example. The class labeled as 0 is the negative class here. As we can see, the Positive and Negative Actual Values are represented as columns, while the Predicted Values are shown as the rows.
<h3 id="definitionsoftpfptnandfn">Definitions of TP, FP, TN, and FN</h3>
Let us understand the terminologies, which we are going to use very often in the understanding of ROC Curves as well:
<ul>
<li>TP = True Positive – The model predicted the positive class correctly, to be a positive class.</li>
<li>FP = False Positive – The model predicted the negative class incorrectly, to be a positive class.</li>
<li>FN = False Negative – The model predicted the positive class incorrectly, to be the negative class.</li>
<li>TN = True Negative – The model predicted the negative class correctly, to be the negative class.</li>
</ul>
<h3 id="type1andtype2errors">Type 1 and Type 2 Errors</h3>
There are two types of errors that can be identified here:
<ul>
<li>
Type 1 Error: The model predicted the instance to be a Positive class, but it is incorrect. This is False Positive (FP).
</li>
<li>
Type 2 Error: The model predicted the instance to be the Negative class, but is it incorrect? This is False Negative (FN).
</li>
</ul>
<h3 id="statisticscomputedfromconfusionmatrix">Statistics Computed from Confusion Matrix</h3>
In order to evaluate the model, some basic facts/statistics from the representation of the confusion matrix are calculated.
<img class="img-responsive" alt="Precision - Recall" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-2.png">
Source: <a rel="nofollow" target="_blank" href="https://commons.wikimedia.org/wiki/File:Precisionrecall.svg">https://commons.wikimedia.org/wiki/File:Precisionrecall.svg</a>
Recall: Out of all the positive classes, how many instances were identified correctly.
<pre><code class="hljs">Recall = TP / (TP + FN)
</code></pre>
Precision: Out of all the predicted positive instances, how many were predicted correctly.
<pre><code class="hljs">Precision = TP / (TP + FP)
</code></pre>
F-Score: From Precision and Recall, F-Measure is computed and used as metrics sometimes. F – Measure is nothing but the harmonic mean of Precision and Recall.
<pre><code class="hljs">F-Score = (2 * Recall * Precision) / (Recall + Precision)
</code></pre>
<h2 id="introductiontoaucroccurve">Introduction to AUC - ROC Curve</h2>
AUC–ROC curve is the model selection metric for binary/multi class classification problems. ROC is a probability curve for different classes. ROC tells us how good the model is for distinguishing the given classes, in terms of the predicted probability.
A typical ROC curve has False Positive Rate (FPR) on the X-axis and True Positive Rate (TPR) on the Y-axis.
<img class="img-responsive" alt="ROC" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-3.png">
The area covered by the curve is the area between the orange line (ROC) and the axis. This area covered is AUC. The bigger the area covered, the better the machine learning models are at distinguishing the given classes. Ideal value for AUC is 1.
<h3 id="differentscenarioswithroccurveandmodelselection">Different Scenarios with ROC Curve and Model Selection</h3>
<h4 id="scenario1bestcasescenario">Scenario #1 (Best Case Scenario)</h4>
For any classification model, the best scenario is when there is a clear distinction between the two / all the classes.
<img class="img-responsive" alt="S1" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-4.png">
The graph above shows the Predicted Class Probability for both classes 0 and 1. The threshold is 0.5 which means, if the predicted probability of the class for an instance is less than 0.5, that instance is predicted to be an instance of class 0. If the probability of the class for an instance is equal or greater than 0.5, the instance is classified as the instance of class 1.
The AUC-ROC curve for this case is as below.
<img class="img-responsive" alt="S1 ROC" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-5.png">
As we can see here, we have a clear distinction between the two classes as a result, we have the AUC of 1. The maximum area between ROC curve and base line is achieved here.
<h4 id="scenario2randomguess">Scenario #2 (Random Guess)</h4>
In the event where both the class distributions simply mimic each other, AUC is 0.5. In other words, our model is 50% accurate for instances and their classification. The model has no discrimination capabilities at all in this case.
<img class="img-responsive" alt="S2 ROC" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-6.png">
We can see there is no clear discrimination between the two classes.
<img class="img-responsive" alt="S2" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-7.png">
It is evident from the ROC AUC curve diagram, that the area between ROC and the axis is 0.5. This is still not the worst model but it makes a random guess, much like a human would do.
<h4 id="scenario3worstcasescenario">Scenario #3 (Worst Case Scenario)</h4>
If the model completely mis-classifies the classes, it is the worst case.
<img class="img-responsive" alt="S3" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-8.png">
Completely opposite of the best case scenario (scenario #1), in this case, all the instances of class 1 are misclassified as class 0 and all the instances of class 0 are misclassified as class 1.
<img class="img-responsive" alt="S3 ROC" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-9.png">
As a result, we get AUC to be 0, which is the worst case scenario.
<h4 id="scenario4industrynormscenario">Scenario #4 (Industry / Norm Scenario)</h4>
In a usual industry scenario, best cases are never observed. We never get a clear distinction between the two classes.
<img class="img-responsive" alt="S4" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-10.png">
In this case, as observed, we have some overlapping and that introduces Type 1 and Type 2 errors to the model prediction. In this case we get AUC to be somewhere between 0.5 and 1.
<h2 id="examplewithpython">Example with Python</h2>
Let us see an example of ROC Curves with some data and a classifier in action!
Step 1: Import libraries
<pre><code class="hljs">import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
# roc curve and auc score
from sklearn.datasets import make_classification
from sklearn.neighbors import KNeighborsClassifier
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score
</code></pre>
Step 2: Defining a python function to plot the ROC curves.
<pre><code class="hljs">def plot_roc_curve(fpr, tpr):
 plt.plot(fpr, tpr, color=&#x27;orange&#x27;, label=&#x27;ROC&#x27;)
 plt.plot([0, 1], [0, 1], color=&#x27;darkblue&#x27;, linestyle=&#x27;--&#x27;)
 plt.xlabel(&#x27;False Positive Rate&#x27;)
 plt.ylabel(&#x27;True Positive Rate&#x27;)
 plt.title(&#x27;Receiver Operating Characteristic (ROC) Curve&#x27;)
 plt.legend()
 plt.show()
</code></pre>
Step 3: Generate sample data.
<pre><code class="hljs">data_X, class_label = make_classification(n_samples=1000, n_classes=2, weights=[1,1], random_state=1)
</code></pre>
Step 4: Split the data into train and test sub-datasets.
<pre><code class="hljs">trainX, testX, trainy, testy = train_test_split(data_X, class_label, test_size=0.3, random_state=1)
</code></pre>
Step 5: Fit a model on the train data.
<pre><code class="hljs">model = RandomForestClassifier()
model.fit(trainX, trainy)
</code></pre>
Step 6: Predict probabilities for the test data.
<pre><code class="hljs">probs = model.predict_proba(testX)
</code></pre>
Step 7: Keep Probabilities of the positive class only.
<pre><code class="hljs">probs = probs[:, 1]
</code></pre>
Step 8: Compute the AUC Score.
<pre><code class="hljs">auc = roc_auc_score(testy, probs)
print(&#x27;AUC: %.2f&#x27; % auc)
</code></pre>
Output:
<pre><code class="hljs">AUC: 0.95
</code></pre>
Step 9: Get the ROC Curve.
<pre><code class="hljs">fpr, tpr, thresholds = roc_curve(testy, probs)
</code></pre>
Step 10: Plot ROC Curve using our defined function
<pre><code class="hljs">plot_roc_curve(fpr, tpr)
</code></pre>
Output:
<img class="img-responsive" alt="Output" src="https://s3.amazonaws.com/stackabuse/media/understanding-roc-curves-python-11.png">
<h2 id="conclusion">Conclusion</h2>
AUC-ROC curve is one of the most commonly used metrics to evaluate the performance of machine learning algorithms particularly in the cases where we have imbalanced datasets. In this article we see ROC curves and its associated concepts in detail. Finally, we demonstrated how ROC curves can be plotted using Python.

Guest Contributor

Understanding ROC Curves with Python

In order to checkout a branch from a remote repository, you will have to perform two steps. First, you need to fetch the actual branch data, which includes the commits, files, references, etc. Second, you'll want to actually check it out so your working directory contains the branch files.
This can be done fairly simply in Git. The following commands assume you only have one remote repo for your repository:
<pre><code class="hljs">$ git fetch
$ git checkout &lt;branch&gt;
</code></pre>
Using <code>git fetch</code> without any parameters like this will retrieve all branches from the remote repo, but if you have multiple remote repos then you should specify which one to retrieve from:
<pre><code class="hljs">$ git fetch &lt;remote-repo&gt;
</code></pre>
But of course, <code>fetch</code> is only needed if you haven't retrieved updates from the remote recently. If you have, then you can simply use one of the <code>checkout</code> commands detailed here.
Then to checkout the branch you want, and to tell Git to track it to the remote branch via the <code>-t</code> argument, use the following command:
<pre><code class="hljs">$ git checkout -t &lt;remote-repo&gt;/&lt;remote-branch&gt;
</code></pre>
Not specifying a local branch name will use the same name as the remote branch. Of course, you can also specify a different local branch name with like this:
<pre><code class="hljs">$ git checkout -b &lt;local-branch&gt; &lt;remote-repo&gt;/&lt;remote-branch&gt;
</code></pre>
There are quite a few ways to do the same thing in Git, but in some cases it depends on what version of Git you're currently using. For example, the first set of commands shown in this article are not available in Git versions &lt; 1.6.6. So be mindful of this in case it doesn't work for you.

Scott Robinson

Git: Checkout a Remote Branch

If you merge a branch in to another, and for whatever reason decide you want to undo the merge, there are some ways to do this with Git.
The solution to this is simpler if you haven't yet pushed the changes to a remote repo, and if you have then you'll likely have to rely on something like <code>git revert</code> instead of the solution proposed below. For this short article I'll explain how you can undo a Git merge that hasn't been pushed yet.
The first step would be to use <code>reflog</code> to find the commit right before the merge:
<pre><code class="hljs">$ git reflog
</code></pre>
Executing this on your repo will return a list of recent commits, including their abbreviated hashes, distance from <code>HEAD</code>, and description. It will look something like this:
<pre><code class="hljs">$ git reflog
8135d07 HEAD@{0}: commit (merge): Merge branch local/bug-34 into local/master
03979c8 HEAD@{1}: commit: Added support for query params
9f7a993 HEAD@{2}: commit (initial): Initial commit
</code></pre>
Once you find the commit that you want to revert back to, use the <code>reset</code> command, similar to what we did when <a href="/git-revert-to-a-previous-commit/">reverting to a previous commit</a>, which is essentially what we're doing here:
<pre><code class="hljs">$ git reset --hard &lt;commit-hash&gt;
</code></pre>
So in the above example we might want to revert to the commit with the message &quot;Added support for query params&quot;, which we'd do like this:
<pre><code class="hljs">$ git reset --hard 03979c8
</code></pre>
The <code>--hard</code> option will revert the code in the tree, staging, and working copies. If you don't want to lose any local changes you've made then <code>--merge</code> or <code>--mixed</code> may be better options for you.
If the commit you want to get back to is only one behind <code>HEAD</code>, then you can instead use <code>ORIG_HEAD</code> as a shortcut in the <code>reset</code> command:
<pre><code class="hljs">$ git reset --hard ORIG_HEAD
</code></pre>
This will be easier in some cases since you won't have to use <code>reflog</code> to find the commit hash. <code>ORIG_HEAD</code> is roughly equivalent to <code>HEAD@{1}</code>, so only use it if you're wanting to revert back a single commit.
Note: Once you have pushed changes to a remote repo, it is not recommended to revert commits in this way since you'd be erasing history. In that case I'd recommend using the <code>git revert</code> command, which will undo the unwanted changes as a separate commit.

Git: Revert a Merge

Usman Malik

Dimitrije Stamenic

Sorting and Merging Single Linked List

<h2 id="introduction">Introduction</h2>
In this tutorial, we will explore the conversion of Python scripts to Windows executable files in four simple steps. Although there are many ways to do it, we'll be covering, according to popular opinion, the simplest one so far.
This tutorial has been designed after reviewing many common errors that people face while performing this task, and hence contains detailed information to install and set up all the dependencies as well. Feel free to skip any step, if you already have those dependencies installed. Without any further ado, let's start.
<h2 id="step1installcurl">Step 1: Install cURL</h2>
<a rel="nofollow" target="_blank" href="https://curl.haxx.se/">cURL</a> provides a library and command line tool for transferring data using various protocols. We need it to download the <a rel="nofollow" target="_blank" href="https://pip.pypa.io/en/stable/">pip package manager</a> in the next step. Many of you would already have it set up, which you can check by running the following command:
<pre><code class="hljs">$ curl --version
</code></pre>
If the command above returns a curl version, you can skip the next instructions in this step. As for the rest of you, you can install curl by following these three steps:
<ol>
<li>Go to <a rel="nofollow" target="_blank" href="https://curl.haxx.se/dlwiz/?type=bin&os=Win64&flav=-&ver=*&cpu=x86_64">https://curl.haxx.se/dlwiz/?type=bin&amp;os=Win64&amp;flav=-&amp;ver=*&amp;cpu=x86_64</a></li>
<li>Download the curl package which matches your system's specifications (32-bit/64-bit)</li>
<li>Unzip the file and go to the bin folder, you can find the curl.exe file there</li>
</ol>
However, this means that you can only use the curl command in that particular folder. In order to be able to use the curl command from anywhere on your machine, right-click on curl.exe, click on &quot;Properties&quot; and copy the &quot;Location&quot; value. After that, right-click on &quot;My PC&quot; and click on &quot;Properties&quot;. In the option panel on the left, select the option &quot;Advanced System Settings&quot;. It has been highlighted in the screenshot below.
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/converting-python-scripts-to-executable-files-1.png">
In the window that appears, click &quot;Environment Variables&quot; near the bottom right. It has been highlighted in the screenshot below.
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/converting-python-scripts-to-executable-files-2.png">
In the next window, find and double click on the user variable named &quot;Path&quot;, then click on &quot;New&quot;. A new text box will be created in that window; paste the &quot;Location&quot; value of the &quot;curl.exe&quot; file that you copied earlier, and then click on 'OK'.
cURL should now be accessible from anywhere in your system. Confirm your installation by running the command below:
<pre><code class="hljs">$ curl --version
</code></pre>
Let's go to the next step.
<h2 id="step2installpip">Step 2: Install pip</h2>
In this step, we will install pip, which is basically a package manager for Python packages. We need it in the next step to install the <a rel="nofollow" target="_blank" href="https://www.pyinstaller.org/">pyinstaller library</a>. Most of you would already have it set up, to check run the following command:
<pre><code class="hljs">$ pip --version
</code></pre>
If the command above returned a pip version, you can skip the next instructions in this step.
As for the rest, you can install pip by running the following two commands in the command prompt:
<pre><code class="hljs">$ curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
</code></pre>
<pre><code class="hljs">$ python get-pip.py
</code></pre>
That's it. Pip has now been installed to your local machine! You can run the following command for confirmation:
<pre><code class="hljs">$ pip --version
</code></pre>
Before moving to the next step, you need to repeat what we did for curl.exe so that you can access the pip command from anywhere in your machine, but this time we'll be doing so for &quot;pip.exe&quot;.
Hit the Windows key and search for &quot;pip.exe&quot;, then right-click on the first search result and click on &quot;Open File Location&quot;, it will take you to the folder in which that file is located. Right-click on the &quot;pip.exe&quot; file and then select &quot;Properties&quot;. After that, copy the &quot;Location&quot; value and paste it in the Path variable just like we did in Step 1.
<h2 id="step3installpyinstaller">Step 3: Install PyInstaller</h2>
In this step, we'll install pyinstaller using pip. We need pyinstaller to convert our Python scripts into executable (.exe) files. You just need to copy paste the command below into your command prompt and run it:
<pre><code class="hljs">$ pip install pyinstaller
</code></pre>
Again, to confirm your installation, run the following command:
<pre><code class="hljs">$ pyinstaller --version
</code></pre>
Note: If you've Anaconda installed in your system, then you're probably using <code>conda</code> package manager instead. In that case, run the following commands below, in sequence:
<pre><code class="hljs">$ conda install -c conda-forge pyinstaller
$ conda install -c anaconda pywin32
</code></pre>
This step marks the end of all installations. In the next step, we'll be converting our Python files to an executable with just a single command.
<h2 id="step4convertpythonfilestoexecutables">Step 4: Convert Python Files to Executables</h2>
This is the last step. We'll use pyinstaller to convert our <code>.py</code> files to <code>.exe</code> with a single command. So, let's do it!
Open up the command prompt and navigate to the directory that your Python file/script is located in. Alternatively, you can open that directory using File Explorer, right-click + shift and then select &quot;Open Command Prompt in this folder&quot;. Before converting your file, you should check that your file works as expected. For that purpose, I have written a basic Python script which prints the number 10 when executed.
Let's run the script and see if it works fine before converting it to an executable file. Run the following command on your command prompt:
<pre><code class="hljs">$ python name_of_your_file.py
</code></pre>
In my case, the filename was 'sum.py'.
To create a standalone executable file in the same directory as your Python file, run the following command:
<pre><code class="hljs">$ pyinstaller --onefile &lt;file_name&gt;.py
</code></pre>
This instruction might take some time to complete. Upon completion, it will generate three folders. You can find the executable file in the 'dist' folder. Please note that the &quot;onefile&quot; argument tells pyinstaller to create a single executable file only.
Let's now run our executable file to see if the procedure worked!
<img class="img-responsive" src="https://s3.amazonaws.com/stackabuse/media/converting-python-scripts-to-executable-files-3.png">
Ta-da! It worked just as expected.
A little tip, if your executable file closes too fast for you to notice the output, you can add an <code>input()</code> line at the end of your Python file, which keeps the prompt open while waiting for using input. That is how I was able to take a screenshot of my output as well.
Also note that if your executable depends on any other executable files, like <code>phantomjs</code>, you need to keep them in the same directory as your Python file's directory so that pyinstaller can include it in the executable.
<h2 id="conclusion">Conclusion</h2>
In this tutorial, we discussed in detail the conversion of Python scripts to executable files using Python's pyinstaller library in four steps. We started by installing cURL, followed by pip and pyinstaller. Lastly, we converted a sample Python file to executable to ensure that the procedure works on Windows.

Converting Python Scripts to Executable Files

Robley Gori

We will optimize common patterns and procedures in Python programming in an effort to boost the performance and enhance the utilization of the available computing resources.

Python Performance Optimization

<h2 id="introduction">Introduction</h2>
This is the final article in a short series dedicated to <a target="_blank" href="/libraries-for-reading-and-writing-csvs-in-java">Libraries for Reading and Writing CSVs in Java</a>, and a direct continuation from the previous article - <a target="_blank" href="/reading-and-writing-csvs-in-java-with-apache-commons-csv">Reading and Writing CSVs in Java with Apache Commons CSV</a>.
<h2 id="opencsv">OpenCSV</h2>
<a rel="nofollow" target="_blank" href="http://opencsv.sourceforge.net/">OpenCSV</a> is one of the simplest and easiest CSV parsers to understand, using standard <code>Reader</code>/<code>Writer</code> classes and offering a <code>CSVReader</code> implementation on top.
Just like Apache Commons CSV, OpenCSV operates with an Apache 2.0 license. Before downloading and deciding whether to use OpenCSVs parsers, you can browse through the <a rel="nofollow" target="_blank" href="https://sourceforge.net/p/opencsv/source/ci/master/tree/">source code and Java docs</a>, and even check out their JUnit test suite, which is included in their git repository.
OpenCSV is also included in the <a rel="nofollow" target="_blank" href="https://mvnrepository.com/artifact/com.opencsv/opencsv/4.4">MVNRepository</a>, making dependency management straightforward.
The <code>CSVReader</code> allows fetching a single record at a time, multiple records as a list or as an iterator, making it flexible in terms of usability of the read data. The library also includes handy features such as reading, writing to and from beans, and direct mapping from a CSV to a Java Map using the header row.
OpenCSV does not have as wide a variety of pre-defined formats as Apache Commons CSV. It relies on two parsers:
<ul>
<li>CSVParser - The original parser defined in OpenCSV. This works for most simple parsing instances but fails if there are escape characters defined as part of the record itself.</li>
<li>RFC4180Parser - similar to the <code>CSVFormat.RFC4180</code> parser in Apache Commons CSV. Works on CSV files which are formatted according to the specifications of RFC 4180. This version of the parser considers all characters between the opening and closing quotation marks as content, except for the double quote character, which needs to be escaped with another double quote.</li>
</ul>
<h3 id="readingcsvswithopencsv">Reading CSVs with OpenCSV</h3>
Reading CSVs with OpenCSV is faster than with Apache Commons CSV because the <code>CSVWriter</code> is implemented to be multi-threaded, when using the <code>CSVToBean.parse()</code> method.
The <code>CSVReader</code> is also implemented using Java Iterable, so it is possible to manage both memory and time constraints based on the implementation method you choose.
OpenCSV has two objects types for reading CSVs - CSVReader, and its sub class CSVReaderHeaderAware.
<code>CSVReader</code> is similar to its Apache Commons CSV <code>CSVParser</code> counterpart and can be used for both simple and complicated parsing scenarios.
To iterate through each record in a CSV file, where <code>record</code> will be a string array with the comma separated values split into individual fields:
<pre><code class="hljs">CSVReader csvReader = new CSVReader (new InputStreamReader(csvFile.getInputStream()));
while ((record = csvReader.readNext()) != null) {
 // do something
}
</code></pre>
If your CSV is delimited by a character other than a comma, you can use the two-parameter constructor instead, and specify the delimiter you want the <code>CSVReader</code> to use.
For example if your CSV contains tab separated values, you can initialize the <code>CSVReader</code> as follows:
<pre><code class="hljs">CSVReader csvReader = new CSVReader(new InputStreamReader(csvFile.getInputStream()), &#x27;\t&#x27;);
</code></pre>
OpenCSV also has a more complicated way of parsing CSV files which involves implementing beans to map the fields in a CSV, and then use annotations for identifying the types of records with either header-based, or position-based annotations.
This helps because it allows the records of a CSV to be processed as a common dataset, instead of as a collection of individual fields.
If the header names of the file being processed are consistent, you can annotate the columns using the <code>@CSVBindByName</code> annotation and allow OpenCSV to take care of the mapping and copying side of processing the parsed data.
For example with our tree dataset:
<pre><code class="hljs">public class Trees {
 @CSVBindByName
 private int index;

 @CSVBindByName
 private int girth;

 @CSVBindByName
 private int height;

 @CSVBindByName
 private int volume;

 public int getIndex() {
 return this.index;
 }

 public void setIndex(int newIndex) {
 this.index = newIndex;
 }
 ...
}
</code></pre>
As long as your CSV file contains a header named with the variable names in our class declaration, OpenCSV can parse and read data into the corresponding element, with type conversions automatically handled:
<pre><code class="hljs">List&lt;Trees&gt; treeParser = new CSVToBeanBuilder(FileReader(&quot;somefile.csv&quot;)).withType(Trees.class).build().parse();
</code></pre>
Validations can be added to the getter and setter methods where needed, and mandatory fields can be specified by setting the <code>required</code> flag on the annotation.
If the header name is slightly different from the name of the variable, the String can be set in the annotation as well. The ability to map the header name when the column name is different is useful in our example since our actual dataset contains the unit of measure of the field, along with a space and punctuation characters which are not allowed in standard Java variable names.
The flag and mapping can be specified with the annotation in this case:
<pre><code class="hljs">...
 @CSVBindByName (column = &quot;Girth (in)&quot;, required = true)
 private int girth;
...
</code></pre>
If your CSV file does not have a header, you can map by column position along with the <code>@CSVBindByPosition</code> annotation.
Keep in mind the OpenCSV positions are 0-based:
<pre><code class="hljs">public class Trees{
 @CSVBindByPosition(position = 0, required = true)
 private int index;

 @CSVBindByPosition(position = 1, required = true)
 private int girth;

 @CSVBindByPosition(position = 2)
 private int height;

 @CSVBindByPosition(position = 3)
 private int volume;
}
</code></pre>
If you want to handle more complicated scenarios, you can implement a class with the <a rel="nofollow" target="_blank" href="http://opencsv.sourceforge.net/apidocs/com/opencsv/bean/MappingStrategy.html">MappingStrategy Interface</a> and define the translation or mapping schema that suits your parsing scenario.
<h3 id="writingcsvswithopencsv">Writing CSVs with OpenCSV</h3>
OpenCSV has more options than Apache Commons CSV when it comes to writing data to CSV files. It allows you to either write from an array of Strings, or write from a list of objects.
Writing from a list of objects requires that the objects be initialized and declared beforehand. So to keep things simple, let's consider working with an array of strings.
To generate a CSV file with data from an array of strings:
<pre><code class="hljs">CSVWriter csvWriter = new CSVWriter(new FileWriter(&quot;new.csv&quot;), &#x27;,&#x27;);
String[] records = &quot;Index.Girth.Height.Volume&quot;.split(&quot;.&quot;);
csvWriter.writeNext(records);
csvWriter.close();
</code></pre>
OpenCSV works with the concept that CSV is not just comma-separated values; it allows you to define which delimiter you want to use in the file as a parameter in the <code>CSVWriter</code> constructor.
Similarly, when defining a String array, you may find it useful to declare a String and then separate it into values based on a delimiter. This is especially useful when you need to copy a selected subset of data rows from one CSV or database file to another.
When initializing the <code>CSVWriter</code>, the <code>FileWriter</code> or <code>Writer</code> is mandatory. Initializing the writer using just one parameter results in a default comma separated file.
There are some additional parameters for specific use cases:
<ul>
<li><code>Char separator</code> - the delimiter. If undeclared the default delimiter will be a comma.</li>
<li><code>Char quotechar</code> - the quotation character. This will be used in case your dataset contains a value with a comma as part of the dataset, and you need to generate a comma separated file. Generally either double quotes, single quotes or slashes are used as quote characters.</li>
<li><code>Char escapechar</code> - This is generally used to escape the <code>quotechar</code>.</li>
<li><code>String lineend</code> - the string or character that determines the end of a line of data.</li>
</ul>
You could construct the <code>CSVWriter</code> including all optional parameters:
<pre><code class="hljs">CSVWriter csvWriter = new CSVWriter(new FileWriter(&quot;new.csv&quot;), &quot;,&quot;, &quot;&#x27;&quot;,&quot;/&quot;, &quot;\n&quot;);
</code></pre>
<code>CSVWriter</code> also has some fields that you can pass as parameters to the constructor. You can define these values as constants and reuse the characters and strings across your codebase to preserve consistency.
For example after declaring:
<pre><code class="hljs">CSVWriter.DEFAULT_SEPARATOR = &quot;,&quot;;
CSVWriter.DEFAULT_QUOTE_CHARACTER = &quot;&#x27;&quot;;
CSVWriter.DEFAULT_ESCAPE_CHARACTER = &quot;/&quot;;
CSVWriter.DEFAULT_LINE_END = &quot;\n&quot;;
</code></pre>
You could use:
<pre><code class="hljs">CSVWriter csvWriter = new CSVWriter(new FileWriter(&quot;new.csv&quot;), CSVWriter.DEFAULT_SEPARATOR, CSVWriter.DEFAULT_QUOTE_CHARACTER, CSVWriter.DEFAULT_ESCAPE_CHARACTER, CSVWriter.DEFAULT_LINE_END);
</code></pre>
Or make use of OpenCSV using the default values if values are not explicitly defined in the constructor and simply call:
<pre><code class="hljs">CSVWriter csvWriter = new CSVWriter(new FileWriter(&quot;new.csv&quot;));
</code></pre>
So if your data includes a line with a username and an address, for example: JohnDoe, 19/2, ABC Street, Someplace, the actual string format that you'd need it to be in is &quot;JohnDoe&quot;, &quot;19//2/, ABC Street/, Someplace&quot;.
<h2 id="conclusion">Conclusion</h2>
<a rel="nofollow" target="_blank" href="http://opencsv.sourceforge.net/">OpenCSV</a> is one of the simplest and easiest CSV parsers to understand, using standard <code>Reader</code>/<code>Writer</code> classes and offering a <code>CSVReader</code> implementation on top.
<ul>
<li><a target="_blank" href="/reading-and-writing-csvs-in-java">Reading and Writing CSVs in Core Java</a></li>
<li><a target="_blank" href="/reading-and-writing-csvs-in-java-with-apache-commons-csv">Reading and Writing CSVs in Java with Apache Commons CSV</a></li>
</ul>

Jean Fernando

Reading and Writing CSVs in Java with OpenCSV

<h2 id="introduction">Introduction</h2>
This is the second article in a short series dedicated to <a target="_blank" href="/libraries-for-reading-and-writing-csvs-in-java">Libraries for Reading and Writing CSVs in Java</a>, and a direct continuation from the previous article - <a target="_blank" href="/reading-and-writing-csvs-in-java">Reading and Writing CSVs in Core Java</a>.
<h2 id="apachecommonscsv">Apache Commons CSV</h2>
The <a rel="nofollow" target="_blank" href="http://commons.apache.org/proper/commons-csv/download_csv.cgi">Apache Commons CSV</a> library is the Apache Software Foundation's version of a Java CSV parser. According to the <a rel="nofollow" target="_blank" href="https://commons.apache.org/proper/commons-csv/summary.html">project summary</a>, it attempts to &quot;provide a simple interface for reading and writing CSV files of various types&quot;.
As with all libraries associated with Apache, it operates with an <a rel="nofollow" target="_blank" href="https://www.apache.org/licenses/LICENSE-2.0">Apache license</a>, meaning it can be used, distributed and modified freely.
Apache Commons allows developers to define their own formats, but offers a predefined set of formats with its <a rel="nofollow" target="_blank" href="http://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVFormat.html">CSVFormat class</a>.
These predefined formats are:
<ul>
<li>RFC4180 - comma separated format defined by RFC 4180.</li>
<li>DEFAULT - Similar to RFC4180 format, but allows empty lines in between rows of data. This format is used if not otherwise specified when you define a parser with the Apache Commons CSV library.</li>
<li>EXCEL - Similar to RFC 4180, but allows missing column names, and ignores empty lines.</li>
<li>TDF - Predefined format for CSV files which are delimited with tabs (<code>\t</code>) instead of commas.</li>
<li>MYSQL - The format supported by MySql's <code>SELECT INTO OUTFILE</code> and <code>LOAD DATA INFILE</code> operations.</li>
<li>ORACLE - CSV format used by the Oracle SQL loader.</li>
<li>INFORMIX_UNLOAD and INFORMIX_UNLOAD_CSV - specialized formats defined for use with the <a rel="nofollow" target="_blank" href="https://www.ibm.com/analytics/informix">IBM Informix</a> embeddable database.</li>
<li>MONGODB_CSV and MONGODB_TSV - Works with the poplar NoSQL <a rel="nofollow" target="_blank" href="https://www.mongodb.com/">MongoDB</a> database, for comma separated and tab separated values respectively.</li>
<li>POSTGRESQL_CSV and POSTGRESQL_TEXT - Format supported by PostgreSQL databases.</li>
</ul>
If you need a very simple solution, Apache Commons CSV might not be suitable. The implementation is designed to offer maximum flexibility, which makes the source code equate to around 30,000 lines, and prove quite complex to understand in some cases.
However, if you do need to cover a wide variety of formats, Apache Commons is a trusted library, well maintained and updated regularly, with extensive Java Docs and developer documentation to support beginners.
It is included in the Maven Central repository and has no external dependencies.
<h3 id="readingcsvswithapachecommonscsv">Reading CSVs with Apache Commons CSV</h3>
The Apache Commons library offers several methods to access individual fields in a CSV file. If you're working with apps like Excel, then your Excel formatted CSV is likely to have a header.
However, if you're using CSV as a basic comma-separated set of text, to transfer data between systems or to feed into another processing application, the file may contain data starting from the first row itself, without a header. The Apache Commons CSV library takes both these scenarios into consideration.
If your CSV file does not contain a header, or if you're unsure whether it does, you can use the index to access a record. Since <code>CSVRecord</code> implements the Java Iterable Interface, the index is 0-based, even though CSV indexes, when opened with Excel and most other applications, start with 1:
<pre><code class="hljs">CSVParser csvParser = CSVFormat.DEFAULT.parse(new InputStreamReader(csvFile.getInputStream()));
for (CSVRecord record : csvParser) {
 String field_1 = record.get(0);
 String field_2 = record.get(1);
 ...
}
</code></pre>
If you're sure that the CSV file you need to parse has a header, and you know the header format before processing, you can use the header column string to fetch records.
Let's consider <a rel="nofollow" target="_blank" href="https://people.sc.fsu.edu/~jburkardt/data/csv/trees.csv">a sample CSV file</a> with tree data, and call it the 'tree data CSV' for future reference in this article:
<table class="table">
 <tr>
 <th>Index</th>
 <th>Girth (in)</th>
 <th>Height (ft)</th>
 <th>Volume (ft)</th>
 </tr>
 <tr>
 <td>1</td>
 <td>8.3</td>
 <td>70</td>
 <td>10.3</td>
 </tr>
</table>
To refer to each row of data we could use indexing as with the previous example, or the column header:
<pre><code class="hljs">InputStreamReader input = new InputStreamReader(csvFile.getInputStream());
CSVParser csvParser = CSVFormat.EXCEL.withFirstRecordAsHeader().parse(input);
for (CSVRecord record : csvParser) {
 String field_1 = record.get(&quot;Index&quot;);
 String field_2 = record.get(&quot;Girth (in)&quot;);
 String field_3 = record.get(&quot;Height (ft)&quot;);
 String field_4 = record.get(&quot;Volume (ft)&quot;);
}
</code></pre>
If you want to read a file that does not contain a header row, want to define your own header, or find indexing confusing, Apache Commons also allows the definition of a header for parsing.
Instead of using the <code>.withFirstRecordAsHeader()</code> method when defining the format of the CSV file, you can define a header manually. For example, if you want to avoid referring to the units of measure in the header in our tree data file, you can redefine the header to use your own string values:
<pre><code class="hljs">CSVParser csvParser = CSVFormat.REF4180.withHeader(&quot;Index&quot;, &quot;Girth&quot;, &quot;Height&quot;, &quot;Volume&quot;);
for (CSVRecord record : csvParser) {
 String field_2 = record.get(&quot;Girth&quot;);
}
</code></pre>
If your CSV file contains a header, but you want to define your own header and skip reading the header in the file, use <code>.readNext()</code> to skip the first row:
<pre><code class="hljs">CSVRecord header = csvParser.readNext();
// read the other rows in a loop as usual
</code></pre>
You can also use an enumeration to define a header if you need to use your definition across multiple classes as in this example:
<pre><code class="hljs">public enum treeHeader {
 Index, Girth, Height, Volume
}
...
CSVParser csvParser = CSVFormat.DEFAULT.withHeader(treeHeader.class).parse(input);
// read rows
</code></pre>
The Apache Commons CSV library has some <a rel="nofollow" target="_blank" href="https://commons.apache.org/proper/commons-csv/apidocs/org/apache/commons/csv/CSVRecord.html">additional methods</a> to make parsing easier, including:
<ul>
<li><code>.getRecordNumber()</code> - returns the number assigned to the record in the CSV file.</li>
<li><code>.isConsistent()</code> - can be used for error handling, returns True or False based on whether the size of the current record matches the size of the header row.</li>
<li><code>.size()</code> - can be used to determine the number of values in the record.</li>
<li><code>.toString()</code> - returns the record as a String. Useful when you need to store the entire row as a String for later processing, hashing or comparison.</li>
</ul>
<h3 id="writingcsvswithapachecommonscsv">Writing CSVs with Apache Commons CSV</h3>
The Apache Commons CSV library is mostly focused on reading data from CSV files. When it comes to writing, the methods recommended in the <a rel="nofollow" target="_blank" href="http://commons.apache.org/proper/commons-csv/user-guide.html">user guide</a> are quite familiar to our core Java implementation.
However, there are some useful additional features, such as the ability to define a header when printing, and printing directly from a JDBC <code>ResultSet</code> object.
The <code>CSVPrinter</code> class implements the <code>Flushable</code> and <code>Closeable</code> interfaces, making its behavior similar to an extension of a usual Java <code>Writer</code> or <code>StringWriter</code>.
For example to generate our tree data CSV:
<pre><code class="hljs">CSVPrinter csvPrinter = new CSVPrinter(stringWriter, CSVFormat.DEFAULT.withHeader(&quot;Index&quot;, &quot;Girth&quot;, &quot;Height&quot;, &quot;Volume&quot;));
csvPrinter.printRecord(&quot;1&quot;, &quot;8.3&quot;, &quot;70&quot;, &quot;10.3&quot;);
csvPrinter.flush();
</code></pre>
The method <code>.withHeader()</code> accepts the String values of the header as parameters.
If you want to generate a CSV without a header you can simply use <code>new CSVPrinter(stringWriter, CSVFormat.DEFAULT);</code>.
If you want to use the column headers from an existing JDBC <code>ResultSet</code> for printing, Apache Commons CSV accepts a <code>ResultSet</code> as a parameter for its <code>.withHeader()</code> method:
<pre><code class="hljs">CSVPrinter csvPrinter = CSVFormat.RFC4180.withHeader(treeDataResultSet).print(out);
csvPrinter.printRecords(treeDataResultSet);
</code></pre>
If the <code>CSVFormat</code> you use allows empty lines, you can use <code>csvPrinter.println()</code> to print a blank line in between data rows.
In addition to a <code>ResultSet</code>, the <code>.printRecords()</code> method is also able to work with an Iterable Array, or a Collection of Strings passed to the method using <a rel="nofollow" target="_blank" href="https://docs.oracle.com/javase/8/docs/technotes/guides/language/varargs.html">varArgs</a>.
<h2 id="conclusion">Conclusion</h2>
Apache Commons CSV attempts to provide a simple interface for reading and writing CSV files of various types.
The implementation is designed to offer maximum flexibility, which makes the source code quite complex to understand in some cases.
However, if you do need to cover a wide variety of formats, Apache Commons is a trusted library, well maintained and updated regularly, with extensive Java Docs and developer documentation to support beginners.
<ul>
<li><a target="_blank" href="/reading-and-writing-csvs-in-java">Reading and Writing CSVs in Core Java</a></li>
<li><a target="_blank" href="/reading-and-writing-csvs-in-java-with-opencsv/">Reading and Writing CSVs in Java with OpenCSV</a></li>
</ul>

Reading and Writing CSVs in Java with Apache Commons CSV

When collaborating with colleagues, or even when you're just using an open source library, you'll often need to fetch a branch from a remote repository using Git.
The &quot;base case&quot; to fetch a branch is fairly simple, but like with many other Git operations, it can become quite confusing when other constraints are introduced and you need to start using one of the many options available. In this article I'll try and shed some light on the commands that need to be run and options that are commonly used.
Generally, you'll want to run this sequence:
<pre><code class="hljs">$ git fetch &lt;remote-repo&gt; &lt;remote-branch&gt;:&lt;local-branch&gt;
$ git checkout &lt;local-branch&gt;
</code></pre>
The <code>fetch</code> command will retrieve the remote branch you're interested in and all related objects and references, storing it in a new local branch that you specified by the argument <code>&lt;local-branch&gt;</code>.
Once everything has been downloaded from the remote repo you can then check it out to actually inspect and play around with the code.
If you only have one remote repo then you can omit all of the arguments to <code>git fetch</code>, which will retrieve all branches and updates, and then run <code>git checkout &lt;branch-name&gt;</code> since all remote branches are already on your system.
Given <a target="_blank" href="/git-difference-between-git-fetch-and-git-pull/">how fetch works</a>, the example command above will retrieve all of the code in the branch you're interested in but it won't affect any of your local branches since nothing is merged with <code>fetch</code>.
Often times you'll want your new local branch to track the remote one, which is helpful for easily pulling and pushing changes. To do this, you should use the <code>--track</code> option with the <code>checkout</code> command, which simultaneously checks out the branch and tracks it with the remote branch. Here is what that would look like:
<pre><code class="hljs">$ git checkout --track &lt;remote-repo&gt;/&lt;remote-branch&gt;
</code></pre>
This will create a local branch of the same name as the remote one.
If you want to checkout the remote branch to a local one, but with a different name, then you need to include the <code>-b</code> option to create the new local branch:
<pre><code class="hljs">$ git checkout --track -b &lt;local-branch&gt; &lt;remote-repo&gt;/&lt;remote-branch&gt;
</code></pre>
In practice, it'll look something like this:
<pre><code class="hljs">$ git checkout --track -b fix-144 origin/bug-144
Branch fix-144 set up to track remote branch bug-144 from origin.
Switched to a new branch &#x27;fix-144&#x27;
</code></pre>
To verify your new branch is tracking the remote branch, run the <code>branch</code> command with the <code>-vv</code> option:
<pre><code class="hljs">$ git branch -vv
* fix-144 0774548 [origin/bug-144] Fix #144
 master dc538f6 [origin/master] 4.16.4
</code></pre>
If you are interested in learning about how Git's code works, check out the <a rel="nofollow" target="_blank" href="https://initialcommit.io/">Initial Commit</a> project.