1 / 14

Apache Bigtop Working Group

Apache Bigtop Working Group. Cluster stuff. Cloud computing. Bigtop Administration. Make sure you are signed up on the bigtop-dev mailing list. Lots of info which will never get repeated if you miss it Bigtop -user, bigtop-dev Sign up for jira

saber
Download Presentation

Apache Bigtop Working Group

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Apache BigtopWorking Group Cluster stuff

  2. Cloud computing

  3. BigtopAdministration • Make sure you are signed up on the bigtop-dev mailing list. Lots of info which will never get repeated if you miss it • Bigtop-user, bigtop-dev • Sign up for jira • Maybe the component mailing lists also. Not Hadoop, whirr, etc…

  4. Newbie Slide • Structure: • Registration, Join Biocurious. Pays for space nobody takes a cut of this • Registration = AWS Credits. Cancelling IntelliJ. • Do labs, labs too easy • Lab 1 Modified to take 1-2 weeks. Update the wiki with your findings • Lab 2 Build Bigtop 0.3.0; too easy now (0.2.0 was harder, better, compile-native JAVA_HOME bug was good, current dl bug is good) • Lab 2a Create a patch, apply it, do a build to verify it works • Lab 3 setup development environment, setup a maven project using gmaven and surefire • Lab 4 map reduce program • Lab 5 Run the unit tests under the component downloads • Lab 6 Run the integration tests • Lab 7 Puppet, deploy and run • Lab 8 Port a module • Labs are changing; not a class. Cant show up and expect to learn by listening to what others did. Requires time commitment • Demo, doesn’t need to be working; for your benefit not ours

  5. Lab 1 • Install bigtop. Web search for apache bigtop, go to wiki link http://incubator.apache.org/bigtop/ • https://cwiki.apache.org/confluence/display/BIGTOP/Index • https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop

  6. Lab 1 • Install bigtop, run all the components, Hive/Hbase/Pig/Hadoop/Mahout/Oozie • There are bugs, document them • Add the sample programs in quickstart to the wiki. Not all are included yet

  7. Lab 1 • Update the wiki • Sqoop open (User group meeting next week) • Flume/Flume NG (open/nothing) • Zookeeper(open/nothing)

  8. Hadoop Components • Old: Don’t stop at running Pi as test of HDFS • Still missing: Run Terasort in Hadoop, need cluster • https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop • Whirr may need patch depending on where you run it from

  9. Mahout • Don’t run jar like in Hadoop • Scripts handle downloading and clustering, email demo, etc.. Under /examples/bin. • Bigtop puts example/bin under /usr/share/doc/mahout. Is this correct? Not documentation • Add documentation to wiki • Ticket filed

  10. Oozie • Oozie runs, forget the error message, set to highest version

  11. Oozie

  12. Flume/Flume NG • New patch checkinfor Flume NG • Testing

  13. Whirr • sudo apt-get install whirr • Run as: whirr launch-cluster --config /udt/lib/whirr/recipes/mahout-ec2.properties • If successful will see directory under ~/.whirr • whirr.log • mvn clean install

  14. Puppet • sudo apt-get install puppet facter fails

More Related