Hadoop / SPARK on Windows

Hadoop on Windows

  • Download the required binaries (e.g., winutils.exe) necessary to run hadoop
  • Download link: https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip
  • Add it to $HADOOP_HOME/bin
  • Set  $HADOOP_HOME, $JAVA_HOME under environment variables
  • Reference: http://stackoverflow.com/questions/19620642/failed-to-locate-the-winutils-binary-in-the-hadoop-binary-path


Spark on Windows

  • While running spark, you can refer to a local path in your computer
  • Spark Master needs to be set to local

String inpath = “C:/New/abc.txt”;
String outpath = “C:/New/New1”;

SparkConf conf = new SparkConf().setAppName(“sparkAction”).setMaster(“local”);

Stay tuned..


About shalishvj : My Experience with BigData

6+ years of experience using Bigdata technologies in Architect, Developer and Administrator roles for various clients. • Experience using Hortonworks, Cloudera, AWS distributions. • Cloudera Certified Developer for Hadoop. • Cloudera Certified Administrator for Hadoop. • Spark Certification from Big Data Spark Foundations. • SCJP, OCWCD. • Experience in setting up Hadoop clusters in PROD, DR, UAT , DEV environments.
This entry was posted in Hadoop, spark, Uncategorized and tagged , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s