1 / 53

New Folder Technology Applications and Success Stories

New Folder Technology Applications and Success Stories. New Folder Consulting. Outline. Image and video processing, scene understanding Change detection in vehicle-mounted video (Video aligning, change detection) Automated driving (Video processing, local features, tracking, inference)

rowdy
Download Presentation

New Folder Technology Applications and Success Stories

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. New Folder Technology Applications and Success Stories New Folder Consulting

  2. Outline • Image and video processing, scene understanding • Change detection in vehicle-mounted video (Video aligning, change detection) • Automated driving (Video processing, local features, tracking, inference) • Infrared Tripwire detection (Image processing) • Pedestrian identification and tracking (Adaptive Gaussian mixture models) • Time and frequency domain pattern recognition, machine learning • Gun shot identification (New models for time-series data classification) • Chemical weapon identification (Context-dependent classification)

  3. Change Detection in Video (with NVESD) What’s interesting about this image?

  4. Change Detection In Video: Goal • To leverage pre-collected, vehicle mounted data to detect changes between multiple excursions over a particular road or route Day 1 Day 2 Now, what’s interesting about 2nd image?

  5. Solution • Apply: • Local image features • Statistical pattern recognition • Robust statistics • Dynamic time warping • Learn optimal warping between two paths, combine and isolate differences based on local feature distributions

  6. Video Example

  7. Automated Vehicle Processing and Interaction (with CISCO) • Goal: • Investigate techniques for video understanding(un-solved problem) • Application: • “Play a (racing) video game competitively against a 16 year old” • Real goal is to advance video understanding, video search, and video processing for many different goals

  8. Webcam Setup • Lab setup includes PS3, webcam, connected to PC • Preliminary real-time processing framework in place • OpenCV • Short-term goals: • Track player’s vehicle • Isolate mini-map • Extract information from odometer/speedometer

  9. Video Game Demonstration

  10. Tripwire Identification in IR Data • Tripwires provide a constant source of danger to overseas troops • Some tripwires can be found in IR imagery, but requires careful visualization • Can we provide soldiers with real-time automatic identification of tripwires in IR data? • Significant computational restrictions • Requires efficient algorithm development

  11. ApproximateGround Mask Fused Black Hot 40 Channel 3 ApplyGround Mask Channel 1 Local Demeaning Gradient Based Transform Radon Transform Merge proximate lines and limit minimum length

  12. Pedestrian Detection and Tracking • Identification and tracking of pedestrians has many applications in • Traffic • Security (at many levels) • Military operations • Adaptive Gaussian mixture modeling and Kalman tracking of low-likelihood regions can provide significant tracking capabilities with low computational complexity

  13. Gunshot Detection (with ARO) • Goal: • Accurately identify make and model of different guns based on acoustic muzzle blast • Applications: • Military and security applications for identifying snipers • Secondary research involves accurate localization in complicated environments

  14. Research • Would like to build HMM with Auto-regressive state densities to model changing time-frequency characteristics of gunshots • But what complexity model? (How many states? Weights?) • Require arbitrary complexity model for time series, but can’t search infinite space • Bayesian processing provides automatic tradeoff between model complexity and accuracy • New time-series models: • Uncertain order AR, Stick-Breaking HMM • Tractable inference • Powerful modeling tool

  15. UOAR SBHMM - Acoustic Signal Modeling • Muzzle blast from a Glock Model 17 • Model inferred from 10 examples • Maximum number of states S=25 • 6 states utilized • Most AR components have a low order • (2 or 4) • Broad spectral range • UOAR SBHMM characterizes time-frequency information similar to the STFT

  16. Context-Dependent Chemical Weapon Identification (With ARO & AFRL) • Goal: • Detect trace amounts of chemical weapons in vapor or solid form in varying gasses or on varying substrate • Application: • Early warning system for military and humanitarian applications • Uses LIBS based sensing • Elemental analysis tool, requiring no sample preparation • Provides spectral lines indicating elemental composition of sample under consideration • Extremely rapid analysis • Issues: • LIBS is sensitive to background matrix (context) • Context-dependent classification

  17. Context Dependent Learning Incoming Data Points - x • Instead of trying to mitigate context; exploit context • Same features, information might not be ideal under different sensing scenarios • Infer context, within context, learn decision boundaries • Testing time, aggregate across all contexts weighted by likelihood given data P(C1|x) P(C2|x) P(C3|x) P(Hj|x,C1) P(Hj|x,C2) P(Hj|x,C3) Requires estimates of p(H1|x,Ci) and p(Ci|x) New Folder Consulting

  18. Context Dependent Classification if Chemical Weapon Simulants in Varying Backgrounds • Context-dependent classification improves percent correct classification from 73 to 84 percent correct with relatively straightforward modifications • Improved performance, lower computational complexity, less risk of over training New Folder Consulting

  19. Conclusions • New Folder has many years of experience performing statistical analysis and algorithm development in a wide range of applications • Video, GPR, Cochlear implants, Brain computer interface, LIBS, Acoustics, electromagnetic induction, seismic, etc. • Our focus is on making things work • Not equations on a piece of paper, but real solutions for real problems • How can we help you?

  20. Backup Slides

  21. Change Detection

  22. Video Data & System • Video camera mounted on Nissan Pathfinder • Data collected in heavily wooded area near Duke campus • Objects emplaced along a 100 meter long route • Landmine simulants, UXO objects, crates, boxes, roadside explosive threat simulants etc. • Varying number of objects in each data collection

  23. Local Image Features (Video) • Local image features provide concise set of information describing an image • Fast to calculate, sparse representation • Triangles and squares on right indicate locations of Harris [Harris, 1988] features from two passes • Features consistent, except where new object is encountered (ellipse)

  24. Distance Metric • Need to define distance between two frames of GPR or video data • Given two sets of feature locations, {xi,yi}, from two frames (H1 and H2) in the two video streams, can define • H1 = {xi1,yi1} • H2 = {xj2,yj2} • But D(H1,H2) as defined above is sensitive to outlying feature locations ( ), which will result in large distances

  25. Robust Distance Metric • Robust distance metric should not be adversely affected by certain % of outliers • Remaining work uses a one-sided trimmed mean • Reject top 30% of large distance outliers

  26. Aligning Video Streams • Once feature sets extracted, need a distance between feature sets from two different frames • Many possibilities available • Need invariance under spurious features (noise features) • May require invariance under spatial shifts • Sensor height • Sensor cross-track position? • Once feature distance defined, can use DTW or similar (real-time) approaches to warping video

  27. Detecting Changes (1) • Given aligned video stream, still need to identify changes • 1st match feature locations from original image to new image • Can we find regions with a high density of unmatched feature locations? ?

  28. Detecting Changes (2) • Can estimate density of unmatched local features using kernel smoothing • Resulting disk-based estimate of density shown on bottom • Can use thresholded density as change-detector

  29. Recall Goal • To leverage pre-collected, vehicle mounted data to detect changes between multiple excursions over a particular road or route (Video, and GPR) Day 1 Day 2

  30. Change Detection in GPR • Given: 2 GPR Images at approximately the same spatial location, with alarms • Any significant change in data between passes? • Identify new targets; reject false alarms

  31. Change Detection Examples: GPR • Images on right illustrate capacity of algorithm to identify changes in GPR data • GPR data streams were aligned using distance metric outlined previously • Changes identified by concentration of un-matched feature locations

  32. Direct Detection Rejections • Direct detection algorithms are a large part of GPR processing • Can we “correct” direct detections that occur over objects which have not changed?

  33. Change Detection in Video: 1 Frame

  34. Driving

  35. Video Processing: Vehicle Tracking • Vehicle tracking as a subset of object tracking • Has direct relevance to video search: • Car making right turn with clouds in distance • Car passing police car on left • Current naïve tracking uses localized matched filters with adaptive templates

  36. Extracting Game Meta Data: Map Segmentation • Transparent map on left provides information about road, other vehicles • May be easier to utilize than full video data • Less information in total, but useful meta-data • E.g., can see other vehicles on map prior to on road • Our play-testing found this map very useful

  37. Extracting Game Meta Data: Speedometer • Inferring speed can be difficult at high velocities • Vehicle odometer has significant information, need to perform classification of digits to infer velocity • Can also infer RPM, gear, turbo, and vehicle health

  38. Supervised Learning & Digit Identification • Extracted labeled examples of digits from NFS2 video streams • Project into PCA space • Learn classifier using Gaussian probability density functions • Output MAP classification decision

  39. Gunshot Detection, Weapon ID & Acoustic Signal Classification Gunshot detection and weapon ID require • Discrimination between weapons and other potential false alarms • Weapon Identification Two primary goals • Algorithm adaptability • Recursive Bayesian inference specifically with Variational Bayes • Independent of specific signals • Nonparametric Bayesian methods

  40. Acoustic Signal Classification - Typical Approach • Similar to acoustic surveillance approach • Frame data (20ms – 1s) • Extract Features • Energy and spectral features • Perceptual features (MFCC) • Statistical model • Pattern classification technique • Hidden Markov model • Performance is strongly tied to selected features and statistical model • e.g., Khan et al., 2009; Cavel et al., 2005; Valenzise, 2007; Dufaux, 2000;

  41. Acoustic Signal Classification - Typical Approach Other algorithm specific parameters Frame length Classifier parameters Different feature sets 67%-87% Ntalampiras et al., 2009, EURASIP Journal on Advances in Signal Processing

  42. A Statistical Model for Time-Series • Acoustic signals can be distinguished by their time-frequency structure • Model an acoustic signal as a collection of spectral and energy components • Collection of AR models • How many? • What AR order? (Spectral complexity) • Resulting models must ultimately provide a means of adapting with new information • Nonparametric Bayesian methods provide a principled way to infer model complexity without explicitly enumerating each model • Uncertain-order AR models • Dirichlet process • Variational Bayesian inference provides a parameterized posterior density to enable computationally tractable updating Muzzle Blast Glass Breaking Door Slam

  43. Uncertain-Order AR Models (Morton, 2011) • AR models • AR order (number of previous samples) controls the spectral complexity • Poles in the complex plane • Over estimation results in over fitting • Especially in data limit scenarios • Bayesian Inference • Conjugate prior (computationally simple) • Consider AR order as a random parameter • Infer correct order from data

  44. Nonparametric Bayesian Inference Nonparametric ≠ no parameters Nonparametric = number of parameters determined by the data • Dirichlet Process (Ferguson, 1973) • Probability density function for probability density functions • A draw from a DP is an infinite discrete density • Draw from G - Chinese restaurant process 44

  45. Stick Breaking HMM (Paisley, 2009) • Utilize standard HMM structure but • Model each row of the transition matrix with a stick-breaking construction • Technically truncated to a maximum value “T” (set as high as computationally possible) • Must use Bayesian inference • Model stick proportions with beta densities - [0, 1] • The use of states is penalized to promote automated selection AR Model Parameters HMM

  46. UOAR SBHMM • The density within each state in the SBHMM is AR and uses an UOAR model prior • Parameter Inference • Determines the number of unique spectral and energy components • Determines the spectral complexity of each component • Models occurrence of the components using a Markov assumption • VB approximation enables tractable recursive Bayesian inference • Similar algorithm to standard EM forward backwards • Assume S maximum states (set arbitrarily high) Initial statestick-breaking Transition stick-breaking UOAR models for each state 46

  47. UOAR SBHMM - Example • Synthetic data • 2 States • AR length 4 • AR length 6 • Number of states and AR orders are correctly determined • Synthetic PSD closely matches the STFT 47

  48. UOAR SBHMM - Acoustic Signal Modeling • Muzzle blast from a Glock Model 17 • Model inferred from 10 examples • Maximum number of states S=25 • 6 states utilized • Most AR components have a low order • (2 or 4) • Broad spectral range • UOAR SBHMM characterizes time-frequency information similar to the STFT 48

  49. UOAR SBHMMGeneration of Synthetic Acoustic Signals • UOAR SBHMM is a generative model for time domain data and can be used to generate signals with similar time-frequency characteristics • Spectral information and complexity closely match original signal • Markov assumption does not perfectly model occurrence of the components Bird Chirp Muzzle Blast 49

  50. UOAR SBHMM Gunshot DetectionFalse Alarm Rejection • UOAR SBHMM used to model each class • Glass breaking, Door slam, Wood smashing, Muzzle blasts • 93.75% Correct • Feature based classification • 12 Characterizing features • Relevance Vector Machine • 85% Correct • Tuning may improve application specific performance 50

More Related