1 / 19

Downlink Segment Test Status (S12.0)

Downlink Segment Test Status (S12.0). Russ Laher and Mehrdad Moshir March 8, 2005. Summary. D/L-group software changes implemented for S12.0 105 pipeline-related CRs 67 database-related CRs/AIs 31 ARs D/L segment-testing to check specific CR/AR/AI changes

vaughan-morton
Download Presentation

Downlink Segment Test Status (S12.0)

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Downlink Segment Test Status (S12.0) Russ Laher and Mehrdad Moshir March 8, 2005

  2. Summary • D/L-group software changes implemented for S12.0 • 105 pipeline-related CRs • 67 database-related CRs/AIs • 31 ARs • D/L segment-testing to check specific CR/AR/AI changes • Small cluster of 12 pipeline-drones (mini “ops-like” environment) • SODB (fresh S11 database via EdJackson) • Archiver run on pipeline results for end-2-end verification • ISTs doing science verification of pipeline products and testing of SPICE • QA team verification of QA sub-system upgrades • Test plan documents written by CogEs to help I&T

  3. What’s New for S12.0 • Functionality/infrastructure to process MIPS-24 SUR-mode TPM data • Processing of MIPS-24 flatfield AORS through both flatfield and science pipelines • New MIPS-24 automated “spot” and scan-mirror-dependent flatfield picking • New MIPS-24 scan-leg pointing-refinement pipeline • MIPS:Ge calibration products are now being archived! • New IRAC and MIPS-24 sky median product • Post-BCD IRAC position-refinement pipeline now generates warning, not error, when number of input images < 5 Pipeline-picker upgrade to handle IRS PeakupImage data • Improved and more flexible ensemble creation • New database functionality • new vBest’ing machinery for telemetryFiles records • getCalProducts now returns files with specified releaseLevel range • 12 new QA diagnostics are now being populated in Qa_ptg-xfer table • Ingest now populates SCLK, Frametime, and exposuretime for IRAC in DCEs DB table, to be used for simplifying IRAC first-frame-correction DB query • Updated the following software modules: irs_tune, droopop, profile, basecal, ridge, extract, qaloader, darkest, slremov, snestimator • Various updates to wrappers, cal files, CDFs • New off-line superboresight pipeline: refines original boresight pointing-history file by mapping the IRAC pointing-refinement results back to the original pointing-history data; aim is to use for re-processing; more testing is needed before it can be used in operations • SPICE upgrades for picking version-dependent calibration files, selecting input files via HTTP, and using a remote server for processing • QA sub-system upgrades to SDAT, database functionality, and other web-based tools

  4. Tests Under S12.0 • IRS • All Sci (BCD&BQD), DarkCal, DarkEns pipelines; used fallback cal; all data run through archiver • Details in IRS S12 Test Plan Document • MIPS-Ge • Sci (scanmap, photometry, sed, tpm), darks, flats; used fallback cal; all data run through archiver • Details in MIPS-Ge S12 Test Plan Document • MIPS‑24 • SUR-mode sci scanmap (slow, med, fast cases); SUR- and RAW-mode photometry; SUR- and RAW-mode TPM; flat-field cal w/ pointing transfer, non-linearity cal w/ pointing transfer; SUR & RAW dark cal w/ pointing transfer ; used fallback cal; all data run through archiver • Details in MIPS-24 S12 Test Plan Document • IRAC • sub-, full-array science and cal pipelines for all exposure modes, frame times, and combinations of internal and external repeats; product records and file system contents checked for data run thru each pipeline; used metadata and fallback cal; all data run through archiver • Details in IRAC S12 Test Plan Document • Post-BCD • FIFcreation; pointing refinement;single_frame_pnt_src_ext;multi-frame pnt-src_ext; mosaic pnt-src_ext; overlap correction; image_interp, outlier; image_reinterp; mosaic_coadd for IRAC, MIPS24, 70 &160; all data run through archiver • Details in PostBCD S12 Test Plan Document • QA, SDAT • Loader run for all tested pipelines, SDAT tool run, qa pages and statistics pages brought up • Details in QA/Infrastructure S12 Test Plan Document

  5. S12.0 Functions/Test Data Flows

  6. S12.0 Capabilities

  7. S12.0 Success Criteria • Run, re-run requests that failed on OPS  • Interact with the SODB via perl & c-APIs  • Deposit raw data into the archive file system  • Create pipeline manifests & ensembles  • Run manifests through APES  • Run multiple jobbers on 12 pipeline drones  • Use CDF to get configuration files  • Use CalTransfer to get calibration data (NIT and/or fallbacks)  • Execute post-BCD pipelines including pointing refinemnet  • Put QA statistics into QA tables  • Put processing results in sandbox and set file pointers in DB  • Use SDAT to access and view results on segment test  • Get “better” BCDs & post-BCD prods  • IST sci verification of seg test data & buy off (incremental from IOC)  • Pass the running of archiver  

  8. Calibration SOSDL-SIS-CL-3000 : darkcal output FITS file, dark image SOSDL-SIS-CL-3001 : lincal output FITS file, linearity model cube SOSDL-SIS-CL-3002 : lincal output table file, pixel cal data SOSDL-SIS-CL-3003 : lincal output cmask image SOSDL-SIS-CL-3004 : darkcal output cmask image SOSDL-SIS-CL-3005 : pmask definitions SOSDL-SIS-CL-3006 : Electronic Nonlinearity Calibration File SOSDL-SIS-CL-3007 : MIPS-Ge stim_latent SOSDL-SIS-CL-3008 : function loadcalkeywords SOSDL-SIS-CL-3009 : IRAC-loadcalkeywords version-1 SODB Table SOSDL-SIS-CL-3010 : darkest cmask SOSDL-SIS-CL-3011 : CalTrans Rule List SOSDL-SIS-CL-3012 : IRS Umask SOSDL-SIS-CL-3013 : IRS Lmask SOSDL-SIS-CL-3014 : IRAC flatcal data cube SOSDL-SIS-CL-3015 : IRAC flatcal cmask SOSDL-SIS-CL-3016 : IRS flatcal cmask SOSDL-SIS-CL-3017 : IRS Omask SOSDL-SIS-CL-3018 : IRS Flux Conversion Table SOSDL-SIS-CL-3019 : IRS FOV PSF Table SOSDL-SIS-CL-3020 : IRS COADD2D AMASK SOSDL-SIS-CL-3021 : IRS COADD2D FMASK SOSDL-SIS-CL-3022 : IRS WAVSAMP TABLE SOSDL-SIS-CL-3023 : MIPS-Ge cmask Data Tools SOSDL-SIS-DT-3004 : profile.tbl (IRS BQD) SOSDL-SIS-DT-3005 : ridge.tbl (IRS BQD) SOSDL-SIS-DT-3006 : extract.tbl (IRS BQD) SOSDL-SIS-DT-3007 : irs_tune.tbl (IRS BQD) Executive SOSDL-SIS-EX-3000 : plinexyz.cdf SOSDL-SIS-EX-3001 : run time environment variables Quality Assessment SOSDL-SIS-QA-3000 : QA file (*.qa) format Products SOSDL-SIS-PD-3000 : cvti2r4 output FITS file, 32-bit real data SOSDL-SIS-PD-3001 : cvti2r4 output FITS file, dmask SOSDL-SIS-PD-3002 : muxbleedcorr output DCE FITS file SOSDL-SIS-PD-3003 : imfliprot output FITS file SOSDL-SIS-PD-3004 : MIPS Ge bmask SOSDL-SIS-PD-3005 : imagest output bmask SOSDL-SIS-PD-3006 : tranhead keyword-translate-only list SOSDL-SIS-PD-3007 : Tile/BCD association files SOSDL-SIS-PD-3008 : Mosaic-geometry interface SOSDL-SIS-PD-3009 : BCD delta file SOSDL-SIS-PD-3010 : PREDICTSAT s-mask SOSDL-SIS-PD-3011 : dldeltafile library SOSDL-SIS-PD-3012 : PRF/PSF map SOSDL-SIS-PD-3013 : Point source extraction table SOSDL-SIS-PD-3014 : Detection/Tile/BCD Association File SOSDL-SIS-PD-3015 : imagest output slope image SOSDL-SIS-PD-3016 : latimflag l-mask SOSDL-SIS-PD-3017 : fiducial frame definition parameters SOSDL-SIS-PD-3018 : mosaic_interpolation table SOSDL-SIS-PD-3019 : matched-point-source refined position table SOSDL-SIS-PD-3020 : fiducial frame input parameters SOSDL-SIS-PD-3021 : bandmerged point source output file SOSDL-SIS-PD-3022 : bandmerger pointer dump file SOSDL-SIS-PD-3023 : band-pair registration uncertainties SOSDL-SIS-PD-3024 : bandmerge statistics SOSDL-SIS-PD-3025 : MIPS24 bmask SOSDL-SIS-PD-3026 : FPGen Command Line Input SOSDL-SIS-PD-3027 : FPGen SODB Entries file (input) SOSDL-SIS-PD-3028 : FPGen Template Input file SOSDL-SIS-PD-3029 : Bandmerge (QA) spectral combination counts Pointing Transfer SOSDL-SIS-PT-3000 : pixel-scale table file SOSDL-SIS-PT-3001 : Ptg Refinement Absolute Source List Input Fmt SOSDL-SIS-PT-3002 : MIPS Scan Mirror Parameters Table SOSDL-SIS-PT-3003 : Instrument FOV Table SOSDL-SIS-PT-3004 : Super Boresight Pointing History File SOSDL-SIS-PT-3005 : Refined DCE Position Table File Applicable SISs

  9. Pipeline-Related CRs (105)

  10. Pipeline-Related CRs (105)…

  11. Database-Related CRs/AIs (67) AI #1469 - Change in product types table needed AI #1449 - Ensemble-creation rule changes for MIPS:Si AI #1421 - Change release level from 99 to 100 for a few archive Pids AI #1402 - Data content for new ensRules DB table (set make='f' for ruleId=350) AI #1388 - Add "bcd_median.fits" and "bcd_median_uncert.fits" to ProductTypes AI #1364 - Backfill externalName fields in S12.0 database AI #1363 - Data content changes to productTypes database table (S12.0 DB only) AI #1338 - Remove ruleId=226 from ensRules and ensplScripts DB tables, etc. AI #1212 - Backfill vBest in telemetryFiles database table AI #1219 - New data contents for calExposureTypes database table AI #1215 - New operations-TFS fileType "calexposuretypes" AI #1212 - Backfill vBest in telemetryFiles database table AI #1201 - Update DB productTypes table with comments AI #1191 - Data content for new ensRules and ensPlScripts DB tables AI #1172 - update the product types table per CR 6073 CR #6419 - getCalPRoducts SP needs two release levels min and max CR #6374 - Update instrumentId in runPipelinePicker DB stored-functions CR #6370 - dnlProc_CR6370.sql, addQA_ptg_xfer_CR6370.pm CR #6368 - dnlProc_CR6368.sql (loading mips1fallback) CR #6367 - QA_ptg_xfer_CR6367.sql CR #6363 - Modify pipeline picker to handle new IrsPeakupImage mode CR #6360 - New overloaded updateDce DB stored procedure for Ingest CR #6328 - New S12.0 database capabilities for QA subsystem CR #6327 - New database role qaSuperUserRole CR #6325 - New database table QA_superUsers CR #6324 - New perl-API function getQaAnalystsAndSuperUsers CR #6323 - New DB stored-function getQaAnalystsAndSuperUsers() CR #6318 - New overloaded setDceTimeParameters DB stored procedure - REJECTED CR #6307 - Schema update for mips1fallback in ops and sodb_it databases CR #6284 - New Ingest C-API for populating Dces database table for IRAC CR #6283 - New Dces.atimeend field in database (integer) - REJECTED CR #6282 - Update PlScripts.plscriptnum for plscriptid=1042 CR #6281 - Make plScripts.comment more precise for plScriptId=33 CR #6270 - Remove ruleId=226 and update ensemble-creation document. CR #6250 - vBest'ing machinery for ensembleSets CR #6242 - Modify createEnsembles.pl to have -u option CR #6223 - New database stored function getQaStatusForRequest CR #6224 - New database stored procedure setQaStatusForRequest CR #6225 - New perl-API function getQaStatusForRequest CR #6226 - New perl-API function setQaStatusForRequest CR #6212 - PlScripts-table data-content changes to undo AI #1183 CR #6210 - New DB stored-function getBestTelemetryFilesForCampaign CR #6188 - New perl-API function getRefinedPointingFilesForCampaign CR #6187 - New DB stored-function getRefinedPointingFilesForCampaign CR #6180 - Upgrade perl-API function lockEnsembleProductsVBest CR #6179 - Upgrade DB stored-function lockEnsembleProductsVBest CR #6173 - New perl-API function unlockEnsembleProductsVBest CR #6148 - Add vBest'ing machinery to DB SP registerTelemetryFile CR #6136 - Upgrade database stored-function getDceIdsForRequest CR #6123 - add plscriptId=1550 and 1650 with exptype=ffl CR #6120 - Upgraded perl-API function setQaStatusForCampaignAndPid CR #6106 - New perl-API function getQaStdComments CR #6105 - New database stored-function getQaStdComments CR #6104 - Upgrade stored-procedure setQaStatusForCampaignAndPid CR #6093 - New perl-API function getQaStatusForCampaignAndPid CR #6092 - New DB stored-function getQaStatusForCampaignAndPid CR #6077 - New perl API getLatestEnsembleProduct CR #6076 - New DB stored-function getLatestEnsembleProduct CR #5913 - Mod. to ensemble-creation ruleId=110 CR #5827 - Overloaded addEnsemble and addEnsembleNTC DB stored functions CR #5818 - New perl-API functions for ensemble creation CR #5816 - New DB stored procedures for ensemble creation CR #5815 - New database tables for ensemble creation CR #5738 - New "ruleId" field in Ensembles tables CR #5614 - Stored functions for unlocking vBest CR #5092 - Schema update for QA_irsflatcalprep CR #1209 - Define and implement database procedures for ensemble processing

  12. ARs (31) 5847 Archive Laher_Russ Archive system does not find fallback calibration files for MIPS-Ge

  13. Issues to be tackled in S13 • SDAT fully functioning under Apache proxy server • Improvement of MOPEX documentation for March 22 Observers’ Support Workshop and more extensive improvements over the next few months • Port post-BCD tools to Max OS 10 (goal is by August Observers’ Support Workshop • Database Query of IRSPeakUpImage database table for Final Product Generation – is this needed? • Simplified time-delay calculations for IRAC first-frame correction • MOPEX GUI development • Bandmerge upgrade

  14. Lines of code

  15. Lines of Code

  16. S12.0 LOC breakdown

  17. Sample of new tool for setting QA status

  18. Sample of new ptg diff Q/A statistics

  19. Sample of new ptg diff Q/A statistics

More Related