eta data processing n.
Skip this Video
Loading SlideShow in 5 Seconds..
ETA Data Processing PowerPoint Presentation
Download Presentation
ETA Data Processing

Loading in 2 Seconds...

play fullscreen
1 / 12

ETA Data Processing - PowerPoint PPT Presentation

  • Uploaded on

ETA Data Processing. Steve Ellingson Low Frequency Software Workshop – Chicago – Aug 10, 2008. RFI Environment: Bad But Manageable. ~ 100 s of noise-limited sensitivity using > 95% of contiguous 5 MHz band around 38 MHz. KEY POINT:

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'ETA Data Processing' - manasa

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
eta data processing

ETA Data Processing

Steve EllingsonLow Frequency Software Workshop – Chicago – Aug 10, 2008

rfi environment bad but manageable
RFI Environment: Bad But Manageable

~ 100 s of noise-limited sensitivity using > 95% of contiguous 5 MHz band around 38 MHz


Can observe here – but need good linearity and narrow channelization

Often, but not always


Search Range

(29-47 MHz)

Primary threat to linearity – receiver design challenge

Ch 6

Ch 5

Ch 3


Ch 4

Ch 2

in band rfi challenges
In-Band RFI Challenges

Galactic background clearly visible underneath sparse RFI

Self-RFI is a relatively

minor problem

6-m Amateur Radio

Self-Generated (PC)

Wideband junk

NC State Police

Impulsive noise

starts to become a problem at resolutions ~100 ms

Wideband junk

Citizen’s Band, other HF

Ionospheric enhancement

Ionospheric enhancement

Wideband junk

offline processing
Offline Processing

Up to 200 x 1GB (17s) Files

7+7 bit complex @ 7.5 MSPS

Data transfer errors (rare but significant)

Sample value histograms / clipping (checking for intermittent RFI swamping)

Data integrity check

1K FFT (yields freq-time resolution 7.324 kHz x 136.5 ms)

Integrate to 8.738 ms (for Crab GP search; also, suppresses impulsive RFI)

Create raw spectragrams

Updated every ~7.5 minutes (timed to track Galactic background variation)

using spectragrams hand-picked for low RFI

Create baseline spectragrams

Remove frequency response;

Linear interpolation between baseline spectragrams to track Galactic background

Calibrate spectragrams

  • Three passes of “plinking” (replacing extreme values with median values):
  • Time-frequency pixels one at a time [th1]
  • All freq pixels for a given time, triggered on total power thresholding [th2]
  • All time pixels for a given freq, triggered on integrated spectrum thresholding [th3]

RFI mitigation

Operates on 7.324 kHz x 8.738 ms spectragrams

w/o interpolation

Incoherent dedispersion

In effect, smoothing to expected resolution of scattered-broadened pulse

(We use 498 ms for Crab)

Integrate time series

Difficult to automate due to RFI and time-domain baseline fluxuations

Manual inspection for pulses

Possible Incoherent combining of polarizations / dipole signals

example of rfi mitigation
Example of RFI Mitigation


Dn = 7.324 kHz

Dt = 498 ms





Plotting power;

Extreme values in this plot are typically within a few % of mean

3600 s


th1 = 0.40 (time-freq)

th2 = 0.03 (time)

th3 = 0.02 (freq)

< 1% pixels plinked

Dn = 7.324 kHz

Dt = 498 ms






Example Simple Pulse Detection (old toolchain – sorry!)

No RFI Mitigation, No Dedispersion


RFI Mitigation, No Dedispersion


RFI Mitigation, DM = 56.791 pc/cm3

DM sweep

Duration ~ 2 s

Peak DM = 56.791 pc/cm3

Est. flux ~ 876 Jy


Example of Relatively Good RFI Conditions

No RFI Mit, No Dedispersion

RFI Mit, No Dedispersion

RFI Mit, DM = 56.791 pc/cm3

off line processing summary
Off-Line Processing Summary
  • Data processing
    • Operates on coherently-sampled voltage data (dipoles or beams)
    • 1 hour of observation is typically about 1 TB raw (data constipation!)
    • 100% new C-language source code / tool chains
    • Nothing special for computing (tend to use existing PC cluster to minimize amount of data transfer)
  • Lessons Learned(from the perspective of a dispersed pulse hunter)
    • Value of extensive diagnostic “pre-analysis” to identify problematic data: Smallest fraction of FLOPS, but greatest fraction of person-hours
      • Weak RFI (histograms over many domains & resolutions)
      • Spurious ionospheric conditions
      • Consistency with sky model (“Error” in time-varying continuum small?)
      • Repeatability (is today within a few tenths of percent of yesterday?)
    • Seems to be more productive to reobserve than to try to salvage “subtly problematic” data, even if only portions look bad.
      • By our standards, we end up throwing out about ½ of data that initially looks good
    • Extent of site multipath (self-inflicted), impact
    • Antenna & cable dispersion, impact
    • Value in keeping coherent dipole voltage data, despite logistics, to maximally facilitate reprocessing
eta a d rx board
ETA A/D-RX Board

120 MHz

System Clock

Altera Stratix EP1S25

25,560 LEs

80 9-bit DSP blocks

1,944,576 memory bits

LVDS direct-connects via Mictor connector


(4b + CLK)



7.5 MSPS

I7+Q7, plus in-band data

(240 Mb/s)




Analog Signal

From ARX


120 MSPS




Reconfigurable Computing Cluster (RCC)

  • 16-node “Virtual FPGA”
    • Each node is a development board with Xilinx XC2VP30 FPGA
    • Edge nodes (“E”) catch streaming LVDS from digital receivers
    • 3.125 Gb/s Infiniband-like interconnects
    • Center nodes (“C”) route between RCC nodes & push results to PC cluster
    • PPCs internal to FPGAs run Linux, perform GPP-type functions

Xilinx ML310


RCC “All Dipoles” Mode

240 MB/s aggregate

(60 MB/s per PC)

Coherent time series, 3.75 MHz BW



John Simonetti Phys

Cameron Patterson CpE

Zack Boor Phys

Sean Cutchins Phys

Kshitija Deshpande EE

Mahmud Harun EE

Mike Kavic Phys

Anthony Lee EE

Brian Martin CpE

Wyatt Taylor EE

Vivek Venugopal CpE

Pisgah Astronomical Research Institute

Supported by: