Oozie : Trigger coordinator on data availability

Here is a sample to trigger coordinator on data availability..

Coordinator.xml

<coordinator-app name=”oozie-coord-pig-datatrigger” frequency=”3″ start=”2015-06-09T14:25Z” end=”2015-07-05T12:59Z” timezone=”Canada/Eastern” xmlns=”uri:oozie:coordinator:0.1″>
<datasets>
<dataset name=”input1″ frequency=”05″ initial-instance=”2015-06-09T13:30Z” timezone=”Canada/Eastern”>
<!– Below path can be created on HDFS like ${appPath}/feed/2014/03/11/20 –>
<uri-template>/user/oozie/trigger</uri-template>
</dataset>
</datasets>
<input-events>
<data-in name=”coordInput1″ dataset=”input1″>
<start-instance>${coord:current(1)}</start-instance>
<end-instance>${coord:current(20)}</end-instance>
</data-in>
</input-events>
<action>
<workflow>
<app-path>${oozieApplnWFhdfsPath}</app-path>
</workflow>
</action>
</coordinator-app>

Coordinator.job

oozieApplnWFhdfsPath=${nameNode}/user/pigoozietest/oozie-test/job/workflow.xml
oozie.coord.application.path=${nameNode}${oozieCoordXmlHdfsPath}
oozieCoordXmlHdfsPath=/user/oozie/oozie-CoordData/oozie-CoordData
oozie.libpath=<lib path>
nameNode=hdfs://<hdfsmachine>:8020
jobTracker=machine:8021
queueName=default

Running the job

oozie job -oozie http://machine:11000/oozie -config /users/oozieTest/ooziePigTest/oozie-CoordData/coordinator.properties -run

Add a file named _SUCCESS to trigger job

hdfs dfs -put /users/oozieTest/ooziePigTest/oozie-CoordData/_SUCCESS /user/oozie/trigger/_SUCCESS

Advertisements

About shalishvj : My Experience with BigData

6+ years of experience using Bigdata technologies in Architect, Developer and Administrator roles for various clients. • Experience using Hortonworks, Cloudera, AWS distributions. • Cloudera Certified Developer for Hadoop. • Cloudera Certified Administrator for Hadoop. • Spark Certification from Big Data Spark Foundations. • SCJP, OCWCD. • Experience in setting up Hadoop clusters in PROD, DR, UAT , DEV environments.
This entry was posted in oozie and tagged , , , . Bookmark the permalink.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s