1 / 9

MPC Data Ingest Notes

MPC Data Ingest Notes. Orbit calculation and creation of MOPS objects. Objects include Detections, Orbits (original MPC orbits), Orbits (JPL-calculated) Stored to disk ~19 h running time with one CPU. Additional object creation and DB insertion via MOPS routines.

donagh
Download Presentation

MPC Data Ingest Notes

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MPC Data Ingest Notes

  2. Orbit calculation and creation of MOPS objects • Objects include Detections, Orbits (original MPC orbits), Orbits (JPL-calculated) • Stored to disk • ~19 h running time with one CPU

  3. Additional object creation and DB insertion via MOPS routines • Object creation includes Tracklets, DerivedObjects • 159366 Tracklets • 159357 DerivedObjects • DB insertion includes Detections, Tracklets, DerivedObjects • 31778048 Detections • 32096771 or ~32.09*10^6 objects

  4. DB transaction rates • Additional object creation and DB insertion via MOPS routines • 32.09*10^6 objects + 159357 records in table derivedobject_attrib = ~32.25*10^6 records • 36.5 h running time or ~245 transactions/sec

  5. DB transaction rates • Batch DB insert using optimized SQL directly on the database server • 32.25*10^6 records • 8.73 h running time • ~1025 transactions/sec • Batch DB insert using optimized SQL over the network • 9.75 h running time • ~918 transactions/sec

  6. MOPS overhead • Retrieving MOPS objects from disk and calling the insert method for each object • Includes various data integrity checks • Prepares SQL statements • Executes SQL statements • Processes database return values • 26.75 h additional processing time versus straight SQL insert

  7. Improving the situation • Reducing number of detections • Date-limiting observations • Optimizing the detections table definition • Removing indexes

  8. DB transaction rates • Batch DB insert using optimized SQL with the removal of six indexes in detections table • 32.25*10^6 records • 55 min running time • ~9772 transactions/sec • Undetermined how indexes affect performance using MOPS routines

  9. Conclusions • 245 transactions/sec is a realistic value for object insertion (mostly Detections) via MOPS • Scale of MPC data and/or my implementation may be problematic • Inserted Spacewatch fields and detections along with NEOS model SSM to MPC data instance • Normal runtime for MOPS simulation using Spacewatch data and NEOS model is reported to be half a night • Started DTCTL on Friday, still running… • Received two interruptions with error “Too many links” in attributeOrbits

More Related