1 / 17

Apache Bigtop Working Group

Apache Bigtop Working Group. Cluster stuff. Cloud computing. Bigtop Administration. Make sure you are signed up on the bigtop-dev mailing list. Lots of info which will never get repeated if you miss it Bigtop -user, bigtop-dev. Bigtop Administration. Sign up for jira.

eryk
Download Presentation

Apache Bigtop Working Group

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Apache BigtopWorking Group Cluster stuff

  2. Cloud computing

  3. BigtopAdministration • Make sure you are signed up on the bigtop-dev mailing list. Lots of info which will never get repeated if you miss it • Bigtop-user, bigtop-dev

  4. Bigtop Administration • Sign up for jira

  5. Bigtop Administration • Registration, Join Biocurious. Pays for space nobody takes a cut of this • Free drinks • Registration = AWS Credits. Cancelling IntelliJ. Expires end of April. • rvs@apache.org

  6. Newbie Slide • Structure: • Do labs • Lab 1 Modified to take 1-2 weeks. Update the wiki with your findings • Lab 2 Build Bigtop 0.3.0; • Can start projects here, do Jira tickets • Lab 3 map reduce program • Lab 4 Run the unit tests under the component downloads • Lab 5 Run the integration tests • Lab 6 Puppet, deploy and run • Lab 7 Port a module • Labs are changing; not a class. Requires time commitment • Demo, doesn’t need to be working; for your benefit not ours

  7. Lab 1 • Install bigtop. Web search for apache bigtop, go to wiki link http://incubator.apache.org/bigtop/ • https://cwiki.apache.org/confluence/display/BIGTOP/Index • https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop

  8. Lab 1 • Install bigtop, run all the components, Hive/Hbase/Pig/Hadoop/Mahout/Oozie • There are bugs, document them • Add the sample programs in quickstart to the wiki. Not all are included yet

  9. Lab 1 • Update the wiki • Sqoop open (User group meeting next week) • Flume/Flume NG (open/nothing) • Zookeeper(open/nothing)

  10. Hadoop Components • Old: Don’t stop at running Pi as test of HDFS • Still missing: Run Terasort in Hadoop, need cluster • https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop • Whirr may need patch depending on where you run it from

  11. Mahout • Don’t run jar like in Hadoop • Scripts handle downloading and clustering, email demo, etc.. Under /examples/bin. • Bigtop puts example/bin under /usr/share/doc/mahout. Is this correct? Not documentation • Add documentation to wiki • Ticket filed

  12. Oozie • Oozie runs, forget the error message, set to highest version

  13. Oozie

  14. Flume/Flume NG • New patch checkinfor Flume NG • Testing

  15. Whirr • sudo apt-get install whirr • Run as: whirr launch-cluster --config /udt/lib/whirr/recipes/mahout-ec2.properties • If successful will see directory under ~/.whirr • whirr.log • mvn clean install

  16. Puppet • sudo apt-get install puppet facter fails

  17. Ticket Questions/Demo • Bigtop install should include stable for ubuntu? Diff between stable and bigtop-0.3.0-incubating. There used to be a diff. • Monitoring, metrics.properties ->metrics2 • Ganglia or JMX? All components w/daemon; • Bruno has Ganglia recipes to monitor status of cluster. Hadoop monitoring: performance and functionality. Hooked up to kerberos/ commercial version is Cloudera manager. Networking, i/o, block sizes, swap space, disk space. • Stable vs. incubating? • Anwar: LogMining (M/R, clickstream and FE log data, exception on day to day basis);

More Related