media foundation supporting hardware codecs and cameras l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Media Foundation: Supporting Hardware Codecs and Cameras PowerPoint Presentation
Download Presentation
Media Foundation: Supporting Hardware Codecs and Cameras

Loading in 2 Seconds...

play fullscreen
1 / 39

Media Foundation: Supporting Hardware Codecs and Cameras - PowerPoint PPT Presentation


  • 1342 Views
  • Uploaded on

CL34. Media Foundation: Supporting Hardware Codecs and Cameras. Stan Pennington Program Manager Microsoft Corporation. Agenda. Media Foundation (MF) Hardware Codecs, and Asynchronous Media Foundation Transforms (MFTs) in general MF video capture support, with focus on UVC 1.1 HD webcams

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Media Foundation: Supporting Hardware Codecs and Cameras' - benjamin


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
media foundation supporting hardware codecs and cameras
CL34

Media Foundation: Supporting Hardware Codecs and Cameras

Stan Pennington

Program Manager

Microsoft Corporation

agenda
Agenda
  • Media Foundation (MF) Hardware Codecs, and Asynchronous Media Foundation Transforms (MFTs) in general
  • MF video capture support, with focus on UVC 1.1 HD webcams
  • But first, let’s all get up to speed on MF…

MFT

MF

mf background media infrastructures in windows perspective
MF Background: Media Infrastructures in WindowsPerspective
  • Team focus on unmanaged APIs
    • Performance concerns for managed data flow components
      • Non-deterministic finalization of GC
    • Can expose configuration, control, and status through managed. Legacy and roadmap issues with wrappers
  • Four infrastructures
    • Video for Windows (VfW)
    • DirectShow (DShow)
    • Windows Media Format SDK (WM FSDK)
    • MF
mf background media infrastructures in windows perspective cont d
MF Background: Media Infrastructures in WindowsPerspective (cont’d)
  • One or both of two development models for ISVs
    • Applications built on APIs
    • New data flow components using extensibility points – often driven by media format or DSP needs
mf background media infrastructures in windows evolution
MF Background: Media Infrastructures in WindowsEvolution

1990 1995 2000 2005 2010

Current Usage

ACM Codecs

VfW

DVD

TV

DVD, TV (Filter, DMO, ACM, VCM Codecs)

DShow

WM True Streaming (DMO, ACM Codecs)

WM FSDK

Windows 7

Key Mainstream Scenarios & New Features (MFT Codecs)

MF

mf background media infrastructures in windows typical microsoft usage in windows 7
MF Background: Media Infrastructures in WindowsTypical Microsoft usage in Windows 7
  • MF
    • Playback of non-streaming ASF, MP3, MPEG-4, and selected AVI permutations for digital still cameras
    • New metadata handling in Shell Property Handlers
    • Transcoding and new media sharing
  • DShow
    • DVD, TV, MPEG-2, MPEG-1, custom I/O, video editing
  • FSDK
    • Best compatibility for ASF true streaming
    • Historic integration of ASF read, write
mf background media infrastructures in windows use by windows media experiences
MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences
  • Information provided to set expectations, subject to change in servicing or major releases
    • Windows Media Player (WMP) and WMP OCX
      • Opportunistically use MF, FSDK, or DShow as necessary for best end-user playback experience
      • MF (+ DShow source hybrid) for Windows 7 transcoding
    • Windows Media Center (WMC)
      • WMP OCX with custom presenter - for playback other than TV, DVD
    • Shell drag-and-drop transcoding for devices (Windows 7 only)
      • MF (+ DShow source hybrid)
mf background media infrastructures in windows use by windows media experiences cont d
MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences (cont’d)
  • Change: Protected Media Path (PMP) in WMP and OCX
    • Windows Vista: PMP used for MF even with unprotected content, providing seamless cross-fade in mixed DRM/non-DRM playlists
    • Windows 7: PMP only used for DRM-protected content in MF
mf background media infrastructures in windows use by windows media experiences summary
MF Background: Media Infrastructures in WindowsUse by Windows Media Experiences: Summary
  • Cannot assume WMP or WMC extensibility from media infrastructure documentation
    • Only WMP SDK plug-ins are documented as extensibility points, per http://msdn.microsoft.com/en-us/library/dd564682(VS.85).aspx
    • WMC extensibility is per http://msdn.microsoft.com/en-us/library/bb895967.aspx
  • Ask Microsoft for technical clarifications or documentation enhancements
mf background why does microsoft use mf
MF Background: Why does Microsoft use MF?
  • Stream objects (IStream) for media input, output
  • Most connectivity and data flow divorced from streaming components, moved to higher layer
    • No push-pull component issues
    • Better potential for dynamic changes
  • Better control of threading
  • More constrained component roles and graphs topologies
mf background mf features
MF Background: MF Features
  • All COM, C++
  • Windows Vista
    • High-level API: Media Session
    • Formats: ASF w/ WMV and WMA, MP3
  • Windows 7
    • Backward compatibility
    • New high-level APIs: Source Reader, Sink Writer, Transcode API (part of “Fast Transcode” feature)
    • Formats: + AVI w/ DV and MJPEG, + MPEG-4 container w/ MPEG-4 ASP, H.264/AVC, AAC LC, HE v1, HE v2, key ACM wrapping
    • Asynchronous MFT and Hardware Codecs
    • Capture support: Device Proxy
async mft model
Async MFTModel
  • Sync MFT refresher
    • DMO enhanced for MF media types and samples
    • Simplest use is to call ProcessInput, ProcessOutput, repeat until output ready status
  • Need for adding async model
    • Inefficient for high level to poll for hardware completion
  • Async changes
    • Locked as sync for backward compatibility
    • Limited interface use until unlocked
    • Events fired on input needed, output ready, drain complete, marker processed
async mft hardware acceleration
Async MFTHardware Acceleration
  • Candidates
    • Video and audio codecs, processors
  • Why codecs as a new model?
    • No DXVA support for encode
    • No efficient hybrid of DXVA decode and non-GPU encode for transcoding
async mft hardware acceleration usage
Async MFTHardware Acceleration Usage
  • Generically, any ISV can write, any ISV can use
    • Subject to any licensing rules, of course
    • Otherwise, use local registration (MFTRegisterLocal)
async mft hardware acceleration usage cont d
Async MFTHardware Acceleration Usage (cont’d)
  • Picked up by high-level MF APIs, no need to code directly to Async MFTs, if:
    • Overall enable/disable registry keys enabled
    • MF API opt-in when not using WMP or OCX. Set MF_TOPOLOGY_HARDWARE_MODE attribute on topology
    • Codec has Trusted Merit if overlapping in-box functionality, and preferred media subtypes match, or fallback subtypes not satisfied by in-box component
  • Can specify directly in own MF topology (MFTEnumEx)
async mft hardware acceleration usage cont d17
Async MFTHardware Acceleration Usage (cont’d)
  • For Windows Media Experiences and automatic Media Session selection, must constrain preference
    • Codec Trusted Merit cryptographic challenge/response
    • Keys and certificates are managed through Microsoft Hardware Codecs license program
hardware codecs license program
Hardware CodecsLicense Program

Under

the Hood

  • Only for video codecs, video processors
  • Preferred over in-box MFT codecs for Windows Media Experiences if properly licensed through wmla@microsoft.com
  • Only for developer of actual codec core
  • Two hardware models supported
    • GPU – vendor writes complete MFT
    • AVStream – vendor uses Microsoft-provided proxy MFT, or writes complete MFT
  • Test suite protects preference
    • Shares test content with Media Performance Test Kit
hardware codecs license program cont d
Hardware CodecsLicense Program (cont’d)

Under

the Hood

  • Two codec types
    • Transcoding only
    • General – for transcoding and playback
  • Proper MF attributes must be set (see additional resources links)
    • Directly on vendor MFTs
    • Through KS property sets for AVStream
  • AVStream MFTs can use hardware data flow via Hardware Mediums
hardware codecs windows usage
Hardware CodecsWindows Usage

Under

the Hood

  • Windows Media Experiences
    • WMP and Shell leverage the Transcode API for transcoding to device
      • Note: No sink extensibility
    • Several events documented for Hardware Codecs in general are optional in WMP usage. 3rd-party applications may be able to make hardware state machines more efficient by using optional events
hardware codecs merit rules
Hardware CodecsMerit Rules

Under

the Hood

  • Video media subtypes in scope for Trusted Merit *

* Test suite requirements are subject to change per the licensing program

** Existing technology may already provide hardware acceleration

hardware codecs fast transcoding
Hardware CodecsFast Transcoding

Under

the Hood

  • Focused on conversion for portable devices
  • Based on MF Media Session, implemented in WMP and Shell layers
  • Non-public DShow wrapper Media Source used for broad source format support, can output compressed or uncompressed
hardware codecs media session based t ranscoding
Hardware CodecsMedia Session-Based Transcoding

Under

the Hood

Source Resolver

Topology & Loader

Encoding Profile Management

Provided by:

Media Session

Microsoft

Microsoft

or ISV

Scheme Handler

IHV

Video Scaling/Conversion, Other Processing

Byte Stream Handler

File,

Network

Pipeline Topology

Proc

File,

Network

Media Source

MFT Decoder

MFT Encoder

Media Sink

Byte Stream

Byte Stream

or

or

Custom Input

Custom Output

or

GPU-Based Hardware Codec Model Shown

Hardware Acceleration

Webcam

mf capture overview
MF CaptureOverview
  • Device Proxy (DevProxy) – internal object for lighter-weight use of Kernel Streaming (KS) than typical DShow
  • AVStream driver model supported
    • USB Video Class (UVC) 1.1 Webcams
    • Audio capture through WASAPI
    • Other, but not full tuner capabilities
mf capture overview cont d
MF CaptureOverview (cont’d)
  • Picked up by MF high-level APIs through MF Media Source, no need to code directly to DevProxy
      • Create Media Source and pass in to Source Reader using MFCreateSourceReaderFromMediaSource
  • Hardware codecs supported by setting MF_READWRITE_ENABLE_HARDWARE_TRANSFORMS attribute
  • Focus here on UVC capture
    • 1.1 includes value-add plug-in model for camera vendors
    • Diagram: http://msdn.microsoft.com/en-us/library/ee349147.aspx
mf capture use with source reader
MF CaptureUse with Source Reader
  • Source Reader high-level API benefits
    • Easy way to obtain uncompressed or compressed samples from media or device
    • Has constrained extensibility
    • Async and sync APIs. Async is best
    • Good model for feeding a composition engine or adding to existing media app design
mf capture dev proxy source reader
MF CaptureDev Proxy + Source Reader

Provided by:

Specify URL, Byte Stream, or Media Source

Async or Sync Control and Status

Microsoft

IHV

Source Reader

Direct access to consume and modify media samples

File,

Network

Media Source

Custom Input

Webcam

Specific Conversions*

MFT Decoder *

AVStream Hardware Codec Model Shown

* Optional

Hardware Acceleration

mf development tools
MF DevelopmentTools
  • Tracing
    • Challenges
      • Async programming
      • Worker queues for Media Sources
      • Constrained ETW events
      • Some Windows Vista WPP events removed for performance optimizations. Windows TMFs not provided
    • Solution
      • New MFTrace tool, planned for release in 2010
        • Detours hooking of public MF interfaces, adding events
        • Consolidation of existing events
    • Usage
      • Live console output
      • Redirect to file, process with tool such as TextAnalysisNET
      • Topology and timestamp graphing cmds
mf development tools cont d
MF DevelopmentTools (cont’d)
  • General
    • TopoEdit – now aware of hardware-accelerated MFTs
summary
Summary
  • MF high-level APIs in Windows 7 provide integration for:
    • Hardware-accelerated transforms, including Hardware Codecs
    • Resultant fast transcoding
    • Simple AVStream-based audio/video capture, including HD webcams compliant with UVC 1.1
  • Development and tracing tools are being improved
call to action
Call to Action
  • Evaluate MF for new designs
  • Consider MF in opportunities for new functionality or improvements in existing designs
additional resources
Additional Resources
  • MSDN MF documentation in Windows SDK
    • Top level: http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspx
    • Async MFTs: http://msdn.microsoft.com/en-us/library/dd317909(VS.85).aspx
    • Writing a custom MFT: http://msdn.microsoft.com/en-us/library/dd940441(VS.85).aspx
    • Source Reader: http://msdn.microsoft.com/en-us/library/dd940436(VS.85).aspx
    • Video capture: http://msdn.microsoft.com/en-us/library/dd940439(VS.85).aspx
    • TopoEdit help: http://msdn.microsoft.com/en-us/library/ms694197(VS.85).aspx
additional resources cont d
Additional Resources (cont’d)
  • MSDN AVStream documentation in Windows DDK
    • Hardware Codec support in AVStream: http://msdn.microsoft.com/en-us/library/dd568169.aspx
  • MSDN MF Forum: http://social.msdn.microsoft.com/forums/en-US/mediafoundationdevelopment
  • Blog: http://blogs.msdn.com/mf
slide36

YOUR FEEDBACK IS IMPORTANT TO US!

Please fill out session evaluation forms online at

MicrosoftPDC.com

learn more on channel 9
Learn More On Channel 9
  • Expand your PDC experience through Channel 9
  • Explore videos, hands-on labs, sample code and demos through the new Channel 9 training courses

channel9.msdn.com/learn

Built by Developers for Developers….