Dan Nichols Head of Recording Services Internet2 Multimedia Specialist

1. Your TV IS TOO SLOW Dan Nichols Head of Recording Services Internet2 Multimedia Specialist Northern Illinois University

2. Your TV IS TOO SLOW Your Everything is too slow….. - Cameras - Displays - Microphones - Networks

3. Your etc. IS TOO SLOW The Human Condition & Digital Systems

4. Your etc. IS TOO SLOW Audio Comparisons Nyquist & Pulse Code Modulation (1982) - Data Driven Compromise - Not Fault Tolerant - Fixed Clock - Good Frequency Correlation - Poor Transient Correlation

5. Your etc. IS TOO SLOW Visual Comparisons

6. Your etc. IS TOO SLOW Observable Latency Effects 5ms Unison Impulse Window 221 Samples(44.1 KHz); less than 1/64 Note at 120 BPM Video Frame Sync (-125 ms to 45 ms) ITU Recommendation; Film is +/- 22 ms

7. Your etc. IS TOO SLOW Latency Spirals Positive Latency Spiral: Player/Location A Plays Slightly Ahead Audio Arrives Delayed at Location B (slightly) Player/Location B Plays Slightly Ahead Audio Arrives Delayed at Location A (slightly) If players anticipation is reasonably synchronized with latency, both will feel ahead but the tempo and feel will stabilize If either player is anticipates more than the latency, the tempo will continue to increase as the players attempt to maintain relative anticipation

8. Your etc. IS TOO SLOW Latency Spirals Follow the Leader: - Player/Location A Plays Normally, Leading the Ensemble - Audio Arrives Delayed at Location B (slightly) - Player/Location B Plays in Unison with Player/Location A - Audio Arrives Delayed at Location A (more so) - Location B Receives Synchronous Musical Experience - Location A Experiences Consistent (perhaps tolerable) Lag

9. Your etc. IS TOO SLOW Latency Spirals Negative Latency Spiral: Player/Location A Plays Normally, In Accordance with Musical Instincts Audio Arrives at Location B Delayed Player/Location B Plays Synchronously with Location A Audio Arrives at Location A Delayed Player/Location A Plays Synchronously with Location B Tempo Slows Until Collapse or Stabilization (Dirge)

10. Your etc. IS TOO SLOW In the Real World… Video Delay is Negligible/Nonexistent Audio Delay is Predictable: - 1 ms of Acoustic Delay Originates from 1 ft of Distance - 2.9 ms of Acoustic Delay Originates from 1 m of Distance - Delays from Reflective Surface Integrate into Direct Sound (Timbre-dependent) Under 40 ms - Haas Effect/Law of the First Wavefront - Everything Else is Considered Reverberation (Agreeable Delay) - Delays from Multiple Surfaces Produce Comb Filtering

11. Your etc. IS TOO SLOW In Our Networked World… Sources of Latency Include: - Ingestion Latency - Transmission Latency - Reproduction Latency

12. Your etc. IS TOO SLOW Where Does It Originate? On Camera: Exposure Latency Double the Frame Rate/Halve the Latency Buffering/Processing Latency Collecting/Processing Frames into Compressed Output Audio Interfaces: Audio Buffer Size in Samples Double the Sample Rate/Halve the Latency

13. Your etc. IS TOO SLOW Where Does It Originate? Ingestion/Reproduction: Motherboard Architecture Software Latency Compression Latency Reproduction Display Type Video Repeaters/Switchers/Converters

14. Your etc. IS TOO SLOW Practical, State-of-the-Moment Advice Microphones Selection & Placement: Place Microphones as Close to Sources as Possible Consider Analog Dynamic Compression Supplement Close Mics with Nearfield/Midfield Mics Baffle Reflections to Reduce Echo/Reverb Speaker Selection and Placement: Place Speakers where the Instruments would Naturally be Staged in an Ensemble or Live Performance Use Diffusive Speaker Placement for Diffusive Instruments & Directional Speaker Placement for Directional Instruments Consider Adding Reverb upon Reception of Remote Signal

15. Your etc. IS TOO SLOW Practical, State-of-the-Moment Advice General Audio Considerations: Employ Analog Mixers/Analog Audio Inputs Use Multichannel Audio for Different Sources Replicate Source and Destination Input Levels Employ a dB Meter or Spectrograph Use the Acoustic Feedback Test to Verify Latency is within Interactive Thresholds Close one or both ends of the audio loop and listen for Echo-Delay Feedback or conventional Feedback

16. Your etc. IS TOO SLOW Practical, State-of-the-Moment Advice Video Reproduction Considerations: Consider Video Reproduction Latency CRTs – Less than 8 ms LCDs/LEDs/OLEDs – Greater than 9 ms Projectors – 16 ms or Greater Disable Image Processing http://www.displaylag.com/display-database/ http://www.projectorreviews.com/the-art-of-gaming/ Position Low Latency Display in Natural Location Face Audience and Performers Capture Both Local and Remote Display for Local Projection/Display Repeat Video Signal after Local Reproduction Use Multiple Redundant Outputs on Local Machine for High Latency, Large or Switched Video

17. Your etc. IS TOO SLOW Practical, State-of-the-Moment Advice Video Capture Considerations: Use Higher Frame Rates and Software Compression Higher Frame Rate can Reduce More Latency than Compression Adds Higher Frame Rates/Shutter Speeds will Require More Light

18. Your etc. IS TOO SLOW Development Vectors Faster, More Robust Networks Faster Computers Higher Sample Rate, Fault-Tolerant/Redundant Audio DSD (22.6 Mbps) 192 KHz/384 KHz More Nimble Camera Designers Custom VTC Machine Vision Cameras https://www.ximea.com/en/products/oem-custom-cameras PTZ Control http://www.bhphotovideo.com/c/product/1133719-REG/syrp_0032_0001_genie_mini.html http://www.bhphotovideo.com/c/product/1170815-REG/cinetics_cinetics_axis360_pan_and.html/prm/alsVwDtl http://www.bhphotovideo.com/c/product/1187283-REG/cobracrane_ptu_proline_pan_tilt_head.html/prm/alsVwDtl Lower Latency Displays http://intelextrememasters.com/