Some Curl Commands for BigData

Writing to HDFS

curl -i -X PUT -T $file -L “http://$namenode:50070/webhdfs/v1//$file?op=CREATE&user.name=$user”

Reading from HDFS

curl -i -X GET “http://$namenode:50070/webhdfs/v1//$file?op=OPEN”

In a kerberized environment (Writing to HDFS)

curl –negotiate -ku : -X PUT $file “http://:50070/webhdfs/v1//$file?op=CREATE&user.name=”

OR

curl -iku $userName:$password -L -T $file  -X PUT “http://:50070/webhdfs/v1//$file?op=CREATE&overwrite=true”

From a Windows Machine (with Security)

  • cd to the bin directory where JRE is installed
  • create a kerberos ticket (using kinit)

Make a directory in HDFS

curl –negotiate -iku : -X PUT  “http://:50070/webhdfs/v1//?op=MKDIRS

Copy File

curl –negotiate -iku : -X PUT -T $file -L “http://:50070/webhdfs/v1//$file?op=CREATE

Advertisements

About shalishvj : My Experience with BigData

6+ years of experience using Bigdata technologies in Architect, Developer and Administrator roles for various clients. • Experience using Hortonworks, Cloudera, AWS distributions. • Cloudera Certified Developer for Hadoop. • Cloudera Certified Administrator for Hadoop. • Spark Certification from Big Data Spark Foundations. • SCJP, OCWCD. • Experience in setting up Hadoop clusters in PROD, DR, UAT , DEV environments.
This entry was posted in Rest API, Uncategorized, webhdfs and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s