Category Archives: Uncategorized

Hadoop Cluster : Run Command on ALL Nodes

Its usually tough to run a command on all nodes of a hadoop cluster. Here is a script to do that.. run_command !/bin/bash TPUT=’tput -T xterm-color’ txtund=$(${TPUT} sgr 0 1) # Underline txtbld=$(${TPUT} bold) # Bold txtrst=$(${TPUT} sgr0) # Reset … Continue reading

Posted in Hadoop Cluster Administration, Hadoop Cluster Installation, Uncategorized, Unix | Leave a comment

Some Curl Commands for BigData

Writing to HDFS curl -i -X PUT -T $file -L “http://$namenode:50070/webhdfs/v1//$file?op=CREATE&user.name=$user” Reading from HDFS curl -i -X GET “http://$namenode:50070/webhdfs/v1//$file?op=OPEN” In a kerberized environment (Writing to HDFS) curl –negotiate -ku : -X PUT $file “http://:50070/webhdfs/v1//$file?op=CREATE&user.name=” OR curl -iku $userName:$password -L -T … Continue reading

Posted in Rest API, Uncategorized, webhdfs | Tagged , , , | Leave a comment

Truststore & Keystore

In SSL handshake, TrustStore is to verify credentials stores certificates from third party, Java application communicate or certificates signed by CA(certificate authorities like Verisign) which can be used to identify third party.   KeyStore is to provide credential. stores private … Continue reading

Posted in SSL, Uncategorized | Tagged , , | Leave a comment

Apache Storm

Architecture / Components Nimbus and Supervisor daemons are designed to be fail-fast (process self-destructs whenever any unexpected situation is encountered) stateless (all state is kept in Zookeeper or on disk) Nimbus and Supervisor daemons must be run under supervision using … Continue reading

Posted in Storm, Uncategorized | Tagged , | Leave a comment

Tips: Cluster Installation

For Ambari not to override configs: edit the file corr. to the service          /var/lib/ambari-agent/cache/stacks/HDP/2.0.6/services/YARN/package/templates/yarn-env.sh.j2  

Posted in Hadoop Cluster Installation, Uncategorized | Leave a comment

Hadoop on Windows

Hadoop on Windows Download the required binaries (e.g., winutils.exe) necessary to run hadoop Download link: https://github.com/srccodes/hadoop-common-2.2.0-bin/archive/master.zip Add it to $HADOOP_HOME/bin Set  $HADOOP_HOME, $JAVA_HOME under environment variables Reference: http://stackoverflow.com/questions/19620642/failed-to-locate-the-winutils-binary-in-the-hadoop-binary-path   Spark on Windows While running spark, you can refer to a local path in … Continue reading

Posted in Hadoop, spark, Uncategorized | Tagged , , | Leave a comment

Tips: Maven

Terminologies GroupId: Package name used in the application ArtifactId: Name of the class   Create Uber Jar <project> …. …. <build> <plugins> <!– Maven shade plug-in that creates uber JARs –> <plugin> <groupId>org.apache.maven.plugins</groupId> <artifactId>maven-shade-plugin</artifactId> <version>2.3</version> <executions> <execution> <phase>package</phase> <goals> <goal>shade</goal> … Continue reading

Posted in Maven, Uncategorized | Tagged , , | Leave a comment