Import Hadoop Libraries To IntelliJ Idea

Import Hadoop Libraries To IntelliJ Idea

The post assumes that:

– you have downloaded Hadoop installation binaries and placed them in a target location

– you have an existing Java project in IntelliJ Idea you want to add Hadoop code to, although the steps remain same for any other project (for e.g. a Scala project) as well as a freshly created empty project

I am using the following technologies:

Linux Mint 17.3

Hadoop 2.5.2

IntelliJ Idea 15.02

My Hadoop binaries are located at /usr/share/hadoop-2.5.2/. This is actually your HADOOP_HOME.

1) Open your Java project in IntelliJ Idea.

2) Go to File-→Project Structure

3) Click on Modules and go to Dependencies tab on the right

4) Click on the “+” sign on the right and select the option to add “JARs or directories”

5) Go to the location where you placed your Hadoop binaries (your HADOOP_HOME). In my case it is /usr/share/hadoop-2.5.2/

Inside your HADOOP_HOME go to…..share/hadoop/common. So the full path to navigate to in my case is /usr/share/hadoop-2.5.2/share/hadoop/common

Select the common directory and click OK. This will take you back to the Modules window

6) Click on the “+” sign once more and again select the option to add “JARs or directories”. This time, under your HADOOP_HOME, go to …..share/hadoop/common/lib, which is one level deeper than the previous step. In my case the full path is /usr/share/hadoop-2.5.2/share/hadoop/common/lib.

7) Select the lib directory and click OK. This will take you back to the Modules window. Check the “Export” checkboxes next to both locations you just added, click Apply and then OK.

8) That's it, you're done. If you want to test whether your libraries are actually recognized, create a Java class and try to import Hadoop libraries such as the following:

import org.apache.hadoop.conf.Configuration;

import org.apache.hadoop.fs.FileSystem;

import org.apache.hadoop.fs.FsUrlStreamHandlerFactory;

import org.apache.hadoop.fs.Path;

import org.apache.hadoop.io.IOUtils;

You should not see any errors.