1 / 19

Nova System Architecture

Nova System Architecture. Metadata Database. Modules are stateless Keep state externally Shared among multiple instances No sync instead round robin policy. Watchdog. Via Zookeeper Leader election Detects unresponsive server instances & kill them Start new replacements

omar
Download Presentation

Nova System Architecture

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Nova System Architecture

  2. Metadata Database • Modules are stateless • Keep state externally • Shared among multiple instances • No sync instead round robin policy

  3. Watchdog • Via Zookeeper • Leader election • Detects unresponsive server instances & kill them • Start new replacements • Reconfigure load balancer’s routing table

  4. Clients Types • Human clients via command line interface • Web interface via SOAP • NOVA is part of larger environment • Data onboarding and data serving interact with NOVA

  5. User Interface • Provides API for registering channels and workflowettes • Binds workflowettes to channels • (de)registers triggers and insertion of new blocks to channels • Monitors workflowettes execution status -Workflowetts are written in a variant XPDL to describe a directed graph of processing steps

  6. Process manager • Keeps track of registered & bound workflowettes • Responds to trigger events by creating executable instance of workflowettes • Pass executable instance of workflowettesto process executor & has their execution status

  7. Data manager • Maintains list of blocks associated with each channel & task input cursor. • When process manager prepare to execute workflowette, data manager does: • Create pig latin expressions • Reserve output block positions

  8. Process optimizer • Currently implemented: • Merging pairs of workflowette executions that read the same input data and run in the same time • Future plan: • Pipelining workflowettes that form a “chain”

  9. Process executor • Forward execution requests to Oozie(run pig jobs) • Track execution status • Report the execution status back to process manager

  10. Trigger manager • Runs its own tread & fires triggers • Uses metadata DB to avoid conflicts with other concurrent Nova server instances • Triggers cause a workflowette to run via requests to process manager • Triggers garbage collection events that are performed by data manager

  11. Data & metadata replicator • Cross-data-center replication is asynchronous & one way ( stand-by instance) for: • Fail-over • Migration • In Nova complicated, cause data is stored in two places with enormous links • Avoids creating dangling references by: • keep track of references & delaying the replay of transactions until all referenced HDFS files are copied

  12. Scan sharing • Workflowettes scheduled independently, but often multiple workflowette executions that read same data are triggered almost in the same time. • “Load” operation is used once across different workflowettes. • Merging is governed in XML workflowette by : • “ Mergeable ” annotation • “ Maximam queue time “ • Only workflowettes that use ALL consumption mode are supported.

  13. Experiments • Incremental Processing: • Merge overhead • Different block sizes • Incremental Vs. non-incremental Join • Scan Sharing

  14. Merge overhead • Merge function that co-groups records by a key, then applies UDF to examine each record • Pig offers 2 alg. For physical co-group: • Map-side • Reduce-side

  15. Merge overhead “we have not been able to pin down the source of this per-block overhead, i.e. whether it lies mostly in opening an HDFS file or in Pig's le handling code.” • Question: Given a data set with fixed size, how much difference does it make to divide it into large number of small blocks or small number of large blocks?? • Depends on the size of the data size • if large data set the difference is small , • If small data set the difference is large.

  16. Incremental Vs. non-incremental Join • (old A JOIN new B) U (new A JOIN old B) U (new A JOIN new B), which emits a delta block of the join result VS. (All A JOIN All B) • this experiment sorts the data by the join key and bulk-loads it into Zebra (a sorted and sparsely-indexed Hadoopfile format) files and then runs a map-side join.

  17. Scan Sharing

  18. Thank you!!Questions??

More Related