1 / 39

On-Board Data Compression Concept

On-Board Data Compression Concept. A. N. Belbachir Vienna University of Technology. Signal Description. Photometry mode (Bolometers): 16 bits detector signal 40 Hz readout rate SNR 15000 - 25000 data rate of 1600 Kbits/s SWL detectors with 16x16x8 data values (1280 Kbits/s)

roman
Download Presentation

On-Board Data Compression Concept

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. On-Board Data Compression Concept A. N. Belbachir Vienna University of Technology On-Board Data Compression

  2. Signal Description • Photometry mode (Bolometers): • 16 bits detector signal • 40 Hz readout rate • SNR 15000 - 25000 • data rate of 1600 Kbits/s • SWL detectors with 16x16x8 data values (1280 Kbits/s) • LWL detectors with 16x16x2 data values (320 Kbits/s) On-Board Data Compression

  3. Signal Description • Spectroscopy mode (Photoconductors): • 16 bits detector signal • 256 Hz maximum readout rate • Reset interval from1/32s to 16s • SNR 450 - 650 • data rate of 3600 Kbits/s • SWL detectors with 18x25 data values (1800 Kbits/s) • LWL detectors with 18x25 data values (1800 Kbits/s) • 18x26 data is the science input to the SPU. • Data rate is 3744 Kbits/s On-Board Data Compression

  4. Maximum Data Rate The Maximum data rate = (science data + DEC/MEC header)/s • Photometry: [(161610) channels  16bits + (260 8) header]  40 Hz= 1637.5 Kbits/s • Spectroscopy: [(18262) channels  16bits + (2728) header]  256 Hz= 4032 Kbits/s On-Board Data Compression

  5. Downlink Rate and Compression Ratio • The downlink rate is 100 Kbits/s • Compression ratio for raw data is at least • 16.38 in photometry • 40.32 in spectroscopy • The DEC/MEC header contains • The observation configuration • Timing parameters • Readouts specifications • Observation identification... • The compression parameters On-Board Data Compression

  6. Typical Signals Simulation Photometry Voltage Time On-Board Data Compression

  7. Typical Signals Simulation Photometry Voltage Time On-Board Data Compression

  8. Typical Signals Simulation Spectroscopy Voltage Time On-Board Data Compression

  9. Typical Signals Simulation Spectroscopy Voltage Time On-Board Data Compression

  10. DEC/ MEC DPU SPU SWL compressed data Data command/response / HK SPU LWL compressed data Data command/response / HK SPU PSU S/C Power via DPU 28V Primary Voltage SPU Data Flow On-Board Data Compression

  11. SPU Transmission Modes • Compressed data mode: • Transmission of compressed data + header and raw data of few channels. • Transmission of SPU HK • Raw data mode: • Transmission of lossless compressed data from selected detectors • Transmission of compressed header and SPU HK On-Board Data Compression

  12. SPU Transmission Modes • Compressed data plus raw data mode • Switch between compressed data and raw data transmission mode for the processing of the same data • Transmission of compressed header and SPU HK • Transparent mode • Transmission of data from selected detectors without compression • Transmission of compressed header and SPU HK On-Board Data Compression

  13. Compression Parameters • Default compression ratio • Double the compression ratio • Triple the compression ratio • Quadruple the compression ratio • Do not compress On-Board Data Compression

  14. DPU Commands • Stop compression • Start compression • Perform SPU test in photometry • Perform SPU test in spectroscopy • Perform warm Reset • Start the program for the bolometer test mode • Load to SPU RAM (5 specific load commands) • Dump from SPU RAM • Start Peak-up Software On-Board Data Compression

  15. SPU HLSW Switching Modes • 28 software modes in total • 18 compression modes according to DEC/MEC raw data • 9 software modes according to DPU commands • SPU standby On-Board Data Compression

  16. TOP Communication I/F No Yes Watch Process Yes No Application SW Command Indentification Compression Mode Identification Dump Bol. Test Load Send HK Data Compression(+ OBS HK) Stop OBS Start OBS Warm Reset Test Spec. Peak-up I/F Test Phot. Data Output from SPU to DPU Communication I/F Data Input from DEC/MEC and/or DPU DPU Command SPU HLSW Flowchart On-Board Data Compression

  17. 320 Kbit/s Photometry Spatial Redun. Reduc. Temporal Redun. Reduc. Robust Avera-ging Glitch Detec-tion Sample Rejec-tion Integ-ration 1280 Kbit/s Lossl. Coding Raw Data Selection 100Kbit/s 1800 Kbit/s Spectroscopy Spat. Red. Reduc. Pre-pro-ces-sing Integ-ration Glitch Detec-tion Ramp Rejec- Temp. Red. Reduc. Ramp Fit-ting 1800 Kbit/s Lossl. Coding Raw Data Selection Reduction / Compression Scheme On-Board Data Compression

  18. Preprocessing • This module is still TBD • Ramp linearization necessary or not On-Board Data Compression

  19. Robust Averaging and Ramp Fitting • Robust averaging in photometry • Preprocessing (TBD) • Median fitting • Mean value calculation • Ramp fitting in spectroscopy • Fitting with RANSAC • Least square fitting On-Board Data Compression

  20. Glitch Detection • To ensure not to integrate over invalid sensor readings (i.e. glitches) • Multilevel glitch detection • Intrinsic deglitching at individual sample level • Extrinsic deglitching at ramp/averages level and considering subsequent slopes/averages On-Board Data Compression

  21. Integration in Spectroscopy • On-board integration of sensor readings to achieve the desired compression ratio • Special emphasis: • Guarantee integration over the right readings (i.e. synchronized with positions of chopper) • Not to integrate over ramps affected by glitches • 2 step process: 1. Discard all data of CRE integration blocks with glitches 2. Add slope data of successive ramps within the same chopper position On-Board Data Compression

  22. Redundancy Reduction • Temporal redundancy reduction • Calculation of references frames and differences • Spatial redundancy reduction • Calculation of references pixels and differences On-Board Data Compression

  23. Lossless Coding • Redundancy reduction reduces magnitude of pixel’s values • Implement the run-length encoding algorithm to achieve additional compression On-Board Data Compression

  24. Compression Ratio from Redundancy Reduction (Worst Case) • Photometry: • SNR: 15000 for 16 bits signal • 4 bits for the noise and 12 bits for the offset signal • for 12 averages: the compression ratio is 3.2 • Spectroscopy: • SNR: 450 for 16 bits signal • 10 bits for the noise and 6 bits for the offset signal • for 2 slopes: the compression ratio is 1.23 On-Board Data Compression

  25. Compression Ratio from Lossy Compression (Worst Case) • Photometry: • 5 samples to average in order to fulfill the telemetry requirements of 100 Kbits/s • Total compression ratio: 3.2  5 = 16 • Spectroscopy in the most challenging case with a reset interval of 1/32s: • Compression ratio of 8 from the ramp fitting • Integration of 4 ramps • Total compression ratio: 8  4  1.23 = 39.38 • The remaining ratio is achieved by the spatial redundancy reduction and the lossless coding On-Board Data Compression

  26. Compressed data rate • Photometry: • Compressed science data + compressed header + no data from additional channels • 100.084 Kbits/s without the spatial redundancy reduction and lossless coding algorithms • Spectroscopy: • Compressed science data + compressed header + raw data of 2 channels per board • 99.983 Kbits/s for the most challenging case On-Board Data Compression

  27. column 1 16 17 32 33 48 49 64 row 1 16 17 array 1 array 6 array 8 array 4 array 7 array 5 array 2 array 3 32 array 1 array 2 array 3 header 1 1 1 256 256 256 array 5 array 6 array 7 header 1 1 1 256 256 256 array 4 array 8 header 1 1 256 256 Data Stream from DEC/MEC in Photometry (SWL SPU) On-Board Data Compression

  28. Data Stream from DEC/MEC in Photometry Data Stream: Block1 Block3 Block2 ~1.56K ~1.06K ~1.56K Chopper Move (max. 10 Hz) Frame 1 Next Frames ~4.2K Multiple of 4.2K for the SWL SPU Multiple of 1.06K for the LWL SPU On-Board Data Compression

  29. Data Stream from DEC/MEC in Spectroscopy Data Stream: Frame4 Frame1 Frame3 Frame2 ~1K ~1K ~1K ~1K Next Frames Chopper Move (max. 10 Hz) Block1 Block2 ~2K ~2K Multiple of 1K per each SPU board On-Board Data Compression

  30. Memory Constraints Program Memory (1.5 MB) From DEC/MEC  4032Kbits/s Buffer (0.5MB) Buffer switch of Data Input To/From SPU Board Link Buffer (32KB) Processing Memory (1.44MB) Buffer (0.5MB) Buffer (64KB) To/From DPU 100 Kbits/s SPU RAM Organization On-Board Data Compression

  31. Memory Organization Requirements • Start Processing with 0.5 MB buffer • 2s of observation in Spectroscopy (36KB for the header) • 3s of observation with blue detectors in photometry (21KB for the header) • 12s of observation with red detectors in photometry (28KB for the header) • 32 bits address in the data memory • 16 bits signal per detector data • 2 readouts data per 32 bit address On-Board Data Compression

  32. Memory Requirements • 1.5 Mbytes of RAM for the program memory • 2.5 Mbytes of RAM for the data memory • 2 buffers of 0.5 Mbytes to store DEC/MEC raw data • 1 buffer of 64 Kbytes to store the compressed data + HK • About 1.4 Mbytes for data processing On-Board Data Compression

  33. Processing Requirements in Spectroscopy (Worst Case) • n = 8 samples at 256Hz (Nb. of samples per ramp) • R = 64 ramps for the most challenging case 1/32s reset • Processing Power: • Ramp fitting computationally most demanding 20 FLOPs for 1 Fit 10 FLOPs for 1 Residual 20 FLOPs per sample for glitch detection 20 FLOPs per ramp for glitch detection 20 FLOPs for the integration of 4 ramps 20 FLOPs for the redundancy reduction On-Board Data Compression

  34. Processing Requirements in Spectroscopy • For 450 detectors we have  4500 . R(n2 + 3n + 3) • For 2 s observation  26MFLOPs/2s=13MFLOPs/s On-Board Data Compression

  35. Processing Requirements in Photometry (Worst Case) • n = 5 samples at 40Hz (Nb. of samples to average) • A = 20 averages in a 3 s buffer for the SWL SPU • Processing Power: • Robust averaging computationally most demanding 40 FLOPs for 1 robust averaging 10 FLOPs for 1 Residual 20 FLOPs per sample for Glitch detection 20 FLOPs per average for glitch detection 20 FLOPs for the redundancy reduction On-Board Data Compression

  36. Processing Requirements in Photometry • For 2048 detectors we have  2048  30  A(n + 4) • For 3 s observation  11MFLOPs/3s = 4MFLOPs/s On-Board Data Compression

  37. Blocks (1Byte) Type (1Byte) PIXB/PIXR (2 Bytes) CDMHS (1 Byte) CDMH (CDMHS Bytes) DECID (TBD Bytes) CSD (TBD Bytes) Compressed Entity • Blocks:Nb. of blocks the compressed entity packet has been split into (the block unit is 1004B) • Type: Type of the observing Mode (spectroscopy or photometry) • PIXB/PIXR: relationship between the HK and the science packets ‘the same index’ • CDMHS: Compressed DEC/MEC Header Size • CDMH: Compressed DEC/MEC Header field • DECID: Decompression Code ID • CSD: Compressed Science Data Compressed Entity Packet Structure On-Board Data Compression

  38. Compressed Entity/HK • The compressed entity size is: • ~ 31KB per SPU board in photometry • ~ 13 KB per SPU board in spectroscopy • The SPU HK: compression results/SPU status • SPU HK rate depends on the reset interval and compression efficiency • SPU HK size is 38 Bytes On-Board Data Compression

  39. Summary • The concept of the on-board software finalized • Compression of the science data according to the header • DPU commands to load or dump from the SPU memory or to perform specific exercises • HK to inform about the SPU status • Two different set of compression modules according to: • Photometry to achieve at least a compression of 16 • Spectroscopy to achieve at least a compression of 40 On-Board Data Compression

More Related