OCAP RI Training CableLabs Winter Conference Philadelphia March 15-16, 2012
Schedule • Thu Mar 15 – 8:30am to 4:30 pm • Project Overview and History - DaveH • Build system - Chris • Logging and debugging - ScottD • Software architecture overview - MarcinK • Profiling applications with the RI’s JVM – SteveA/ScottD [lunch] • System Information – Prasanna/Craig • Service Selection and DVR subsystem - Craig • PC Platform – Marcin/SteveM/Neha • HN subsystem – Doug/Craig/Marcin/Lori • Fri Mar 16- 8:30am to 2:30 • Graphics - SteveA • Testing – ScottA/Nicolas • Issue reporting and handling - ScottA [lunch] • Stabilization/Robustness project – Marcin/Chris • Miscellaneous topics - TBD
Project Overview and History March 15-16, 2012
Project started June 2008 • Core OCAP I16+ compliant • Several ECNs short of 1.0.0 compliance • DVR I03 extension • Missing ECNs 897, 994, 1017 • Front Panel I02 extension • Approx 200 bugs carried over from legacy code base • Vision Workbench/Clientsim SDK • Closed code base
Current Status • Feb 2012 (1.2.2 Bundle Release) • Core OCAP 1.2.2 • DVR Extension I08 • Front Panel Extension I05 • Device Settings Extension I05 • Home Networking Extension I08 • Home Networking Protocol I07 • JVM • Based on Sun open source PhoneME advanced MR2 • PC Platform • Windows XP initially; now Windows & Linux (several versions) • SDK • Rel-6 released Nov 2011 – no further versions planned • This is an open-source project • http://java.net/projects/ocap-ri/
2012 Schedule • 1.2.1 Bundle released on 01/12/12 • 1.2.2 Bundle (1.2.2 Rel-A RI) released on 02/24/12 • Included VPOP as primary feature • Also MIBObject.getValue • Completed implementation of ServiceResolutionHandler • Completed implementation of DTCP/IP on PC platform • Maintenance against 1.2.1 RI • 1.2.2 Rel-B planned for 04/05/12 • No feature development • Maintenance against 1.2.2 RI • Fixes will be taken off the current trunk • 1.2.2 will not be maintained as a separate branch • 1.2.3 Bundle planned for 05/17/12 • HN features to align with CVP-2 compliance • Maintenance against 1.2.2 RI • Backlog items • Add IPv6 and https support to the PC platform • Stack housekeeping tasks
Other • DTCP platform port for Windows and Linux • MPEOS API changes were included with 1.2.1 RI release • RI Implementation now available for internal testing at CableLabs • Phasing out support from Windows XP to Windows7 • Some performance issues with GStreamer plugin on Win7/newer Linux platforms are being investigated • SDK Rel-6 released on 11/17/11 • No current plans for a future release • Upgrade of our DirectFB library we use (low priority) • Port RI to the MAC (very low priority)
Project site walk through • Main Project Page • Bug Databases • Forums • Wikis • Contribution Process • Bug fix cutoffs • Release notes • Coding standards
Why an RI? OCAP Specifications Stubs & DTDs Clarify Test Release Bundle OCAP Tests (CTP) OCAP Reference Implementation Test
OCAP Release Bundle • Components of a bundle are • Specs • Stubs • DTDs • RI implementation, including integration tests • CTP conformance tests
OCAP RI Requirements • RI runs on a PC • Windows initially – now Linux • RI and PC IDEmust be available on open-source terms • RI and PC IDE must only include components with licenses compatible with the ODL dual-license plans • Components available only under GPL are not OK • Licenses for all third-party RI components must be reviewed by both CableLabs and the ODL legal teams • RI works with existing CableLabs ATE/CTP tests • RI adheres to current and future OCAP core specs • RI adheres to current and future OCAP extensions specs • To ensure backwards compatibility of the spec,MSO guides must run on the RI • To ensure backwards compatibility of stack ports of the RI, any changes to the MPEOS porting layer must be approved by the RI steering committee
Licensing Models • GPL License on java.net • CableLabs OpenCable Project • OCAP Stack, PC Platform, Head-end Emulator • Sun PhoneME Project - JVM • Commercial License • CableLabs Commercial License • Also free • Stack, platform and emulator • RAND IPR commitment • Bug fixes in stack contributed back • Sun or other JVM vendor • Commercial CDC/PBP 1.1 JVMPhoneME JVM
OCAP RI Branching Strategy • Three principal branches • Mainline/Development Branch • Code implemented by internal RI Dev Team • Code from open source contributors that are vetted by RI Tech Leads • Other working branches get merged back to Mainline periodically • Branded Branch (eg, “1.1.4”) • Fixes and enhancements that are tied to the spec and which have been verified by the CTP • Branded branch is maintained separately from mainline • Changes from branded branch eventually migrate back to mainline development • One branded branch per spec release • Experimental Branch • Open source contributors have write access to this directory • No other restrictions • Merging to Mainline on a case-by-case basis
Bug Tracking • Two Bug Tracking Databases • Internal (private) JIRA db (OCORI) at CableLabs, tied to CableLabs CTP bug db • External (public) JIRA db on java.net (IT); hides details of CTP-related issues
RI Build System March 15-16, 2012
Building the RI – The Easy Way See https://community.cablelabs.com/wiki/display/OCORI/Quick+Start for detailed instructions. • Setup development environment • Cygwin + JDK + Ant for Windows • A little more required for Linux (see Wiki) • Get checkout_dev_env.sh (from svn) • Use checkout_dev_env.sh to get source, create setEnv file • Execute ant in appropriate directory. • Builds Platform and Stack. • See Wiki for detailed instructions.
Build System – Environment Variables • Easy to work in several different RI code bases at the same time. • OCAPROOT • The absolute path to the OCAP-1.0 directory. • Required for compilation/execution • Example: E:/Cablelabs/svn/OCAPRI/trunk/ri/RI_Stack • OCAPHOST • Defines the host build environment • Build system reads host environment configuration files from ($OCAPROOT/hostconfig/$OCAPHOST) • Required for compilation only • Example: Win32-Cygwin
Build System – Environment Variables • OCAPTC • The Target Configuration for the build. Basically the port you are working on. • Defines a subdirectory hierarchy where: • build configuration files are found ($OCAPROOT/target/$OCAPTC) • binary intermediate products are built $(OCAPROOT/gen/$OCAPTC) • final binary products are installed and runtime configuration files are kept ($OCAPROOT/bin/$OCAPTC) • Suggested format is: • <org>/<platform>/<os>/[debug|release] • Example: CableLabs/simulator/Win32/debug • Required for compilation/execution
Build tools • Make • Compiles JNI, MPE, MPEOS, and thirdparty native libraries • Ant • Coordinates the entire build system • Wiki contains a list of top-level build targets • JDK (1.4 or higher) • Used to compile stack and test application sources
Win32 Port • Host environment is Cygwin • See Wiki for a full list of Cygwin libraries required to build the RI Stack and Platform • Cross-compile to MinGW (no Cygwin DLL) • Lots of work (including JVM patches) to deal with POSIX vs. Win32-style paths • POSIX for gcc • Win32 for javac, javah, etc.. • VERY SLOW (compared to Linux) • JVM binaries pre-built and checked-in to save compilation time since most won’t be modifying the JVM • WindowsXP, Vista
Linux Port • Known working distros/versions: • Fedora 10/12/13/15 • Ubuntu 10.04/10.10/11.04 • Ubuntu 11.04 • Much faster than Win32 on the same hardware. • See Wiki for detailed instructions.
Logging and Debugging March 15-16, 2012
Stack logging • log4j APIs included in the spec for use by applications • Additional Logger methods avoid String concatenation overhead in most cases • Monitor applications configure logging through DOMConfigurator or PropertyConfigurator • Groups • Multiple loggers can share a common group name, which can be used during configuration
Stack logging continued • New appenders • MIB Appender • AsyncAppender uses an additional thread and a queue to offload writing to the target appender off of the caller thread • New configuration capabilities • Configure at the ‘group’ level or the logger level • Filter support, including ExpressionFilter (ability to use regular expressions for fine-grained control over logging verbosity)
Stack logging continued • Additional information available from the Wiki • https://community.cablelabs.com/wiki/display/OCORI/Configuring+Java+stack+logging
Platform logging • Platform code uses log4c to manage logging • Configuration found in $PLATFORMROOT/log4crc • Additional information available from the Wiki • https://community.cablelabs.com/wiki/display/OCORI/RI+PC+Platform+Logging
Logging and IssueTracker When attaching a log to IssueTracker • Ensure the log contains timestamps • Helpful if Java debug logging is enabled
Java Stack debugging • Possible to step through breakpoints in Java stack code, generate stack traces and thread dumps • Stack trace, thread dumps available via jdb (included with the Sun JDK) • To enable Java debugging, un-comment VMOPT19 & 20 in mpeenv.ini and start debugger or jdb • Re-comment VMOPT 19 & 20 when done..
Platform debugging • gdb can be used to generate a trace if the Platform terminates unexpectedly • ./runRI.sh -gdb
RI Software Architecture March 15-16, 2012
Platform Summary • Full software emulation of STB media decoding and presentation hardware. • Majority of the code is 3rd party support libraries. • Leverages existing frameworks: • GLib – utility library. • Gstreamer / HDHomerun / VLC– tuner control. • GStreamer – media decoding and presentation. • wxWidgets – user interface. • Net-SNMP – Master Agent. • No OS abstraction APIs.
OCAP Summary • Ported to Platform APIs for STB/video-related functionality. • OS functionality MPEOS port maintained together with the stack code. • MPE contains platform-independent / portable C code. • Integrates Advanced phoneME JVM. • OCAP Functionality implemented via pluggable Java Manager implementations.
Profiling applications with the RI’s JVM March 15-16, 2012
phoneME JVM • RI uses the open source (GPL2) phoneME Advanced JVM from Sun/Oracle • Closely based on J2SE 1.4 • Adding JSSE (secure sockets) support in an upcoming release • OCAP requires a JVM compliant with the latest maintenance releases of: • Personal Basis Profile 1.1 • Foundation Profile 1.1 • Connected Device Configuration 1.1 • Last update: October 24, 2009 • Patch common source to fix bugs and build problems • Win32/Cygwin/Make filesystem issues • JDWP (VM debugging) thread sync issues • PNG bug • GIF bug?
JVM Build • All JVM-related files located in $OCAPROOT/jvm • Build disabled by default for RI Win32 – pre-built binaries checked in to SVN • Enable/Disable building the VM with “build.jvm.exclude” entry in $OCAPROOT/target/$OCAPTC/buildrules.properties
ocap_vmlib • Interfaces and classes to assist in integrating a VM with the RI Stack • Includes a full AWT implementation ported to MPE graphics APIs (DirectFB) • Documentation • $OCAPROOT/docs/JVMPortingGuide.doc
Profiling and analysis tools • There are a number of tools available for investigating issues in the Java stack code • NetBeans Profiler • CVM Inspector • HPROF • jdb
Profiling with NetBeans 6.8 • JVMTI-based • Supports profiling of CPU, Memory and high level JVM stats (GC, thread activity) • Used to identify CPU and memory hot spots • Does not support creation or comparison of heap dumps • https://community.cablelabs.com/wiki/display/OCORI/Profiling+the+RI%27s+JVM+using+NetBeans+6.8
CVM Inspector • Set of utility functions and a shell script which can assist in inspecting the state of the JVM • Available by setting a flag in the JVM build • Used to generate and compare heap dumps • Runs in either client-server mode (standard JVM client connects via a socket to the RI) or standalone mode (GDB) • https://community.cablelabs.com/wiki/display/OCORI/Generating+heap+dumps+on+the+RI%27s+JVM+using+CVM+Inspector
HPROF • A command-line profiling tool • Used to generate detailed monitor statistics • https://community.cablelabs.com/wiki/display/OCORI/Generating+thread+monitor+stats+on+the+RI%27s+JVM+using+hprof