realizing the benefit of re using open source software n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Realizing the Benefit of (Re-)using Open Source Software PowerPoint Presentation
Download Presentation
Realizing the Benefit of (Re-)using Open Source Software

Loading in 2 Seconds...

play fullscreen
1 / 52

Realizing the Benefit of (Re-)using Open Source Software - PowerPoint PPT Presentation


  • 61 Views
  • Uploaded on

Realizing the Benefit of (Re-)using Open Source Software. Chris A. Mattmann Senior Computer Scientist, NASA Jet Propulsion Laboratory Adjunct Assistant Professor, Univ. of Southern California Member, Apache Software Foundation. Roadmap. Areas for Open Source within the NASA context

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Realizing the Benefit of (Re-)using Open Source Software' - diella


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
realizing the benefit of re using open source software

Realizing the Benefit of (Re-)using Open Source Software

Chris A. Mattmann

Senior Computer Scientist, NASA Jet Propulsion Laboratory

Adjunct Assistant Professor, Univ. of Southern California

Member, Apache Software Foundation

roadmap
Roadmap
  • Areas for Open Source within the NASA context
  • Strategies and Decision Points
  • Discuss outcome from the NASA OSS
  • Discussion and Recommendations
    • All of the above will take 30-45 mins, then we’ll have Robert Downs present for ~20 mins, and then the last 20-25 mins for discussion
  • This is a shorter version of a longer, half day breakout on NASA Open Source that I led at the 9th NASA Earth Science Data System Working Group Meetings in October 2010
  • http://s.apache.org/anM
  • Also check out NASA OSS slides that inspired this: http://s.apache.org/pw
  • ESIP Summer 2011 Session: http://s.apache.org/Wd0

ESIP-WINTER12

and you are
And you are?
  • Apache Member involved in
    • OODT (VP, PMC), Tika (VP,PMC), Nutch (PMC), Incubator (PMC), SIS (Mentor), Lucy (Mentor) and Gora (Champion), MRUnit (Mentor), Airavata(Mentor)
  • Senior Computer Scientist at NASA JPL in Pasadena, CA USA
  • Software Architecture/Engineering Prof at Univ. of Southern California

ESIP-WINTER12

the nasa esds context
The NASA ESDS Context

Where is open source most useful?

Which area should produce open source software?

ESIP-WINTER12

concerns in the open source world
Concerns in the Open Source World
  • Licensing
    • GPL(v2, v3?), LGPL(v?), BSD, MIT, ASLv2
    • Your own custom license approved by OSS
      • NASA OSS license?
      • Caltech license?
    • Copy-left versus Copy-right
  • Redistribution
    • Can you take open source product X and use it in your commercially interested software Y?
      • If so, do you have to pay for it?
    • Should others pay for your open source product if they use it in their commercial application?
  • Open Source “Help Desk” Syndrome versus Community
    • Are you trying to simply make your open source software (releases) available for distribution (aka help desk)?
    • Are you trying to get others to “buy in” to your open source software?

ESIP-WINTER12

concerns in the open source world1
Concerns in the Open Source World
  • Intellectual Property
    • Who owns it?
    • How does the Open Source Software affect your IP?
  • Open Source Ecosystems
    • Where can you find the “killer app” you need?
    • Which communities are conducive for longevity?
    • How relevant are “generic” open source software communities to NASA Earth Science Data Systems?
  • Contributing
    • Are you even allowed to contribute to a OSS community?
    • Can you do it on “company” time?
    • What’s required?
    • What’s the governance?
  • Responsiveness
    • How response is the OSS community to your projects’ needs?

ESIP-WINTER12

concerns in the open source world2
Concerns in the Open Source World
  • Help/Guidance
    • Is the OSS community/project alive?
    • How can you tell whether the project is alive?
    • How can you “follow the rainbow” to the OSS pot of gold?
  • Interaction
    • What are the best practices for interacting with OSS communities?
  • Implementation strategies
    • Insulation: how do you insulate your project from OSS change?
    • Configuration Management: what are the important CM issues when using Open Source Software in your organization?
  • Architectural strategies
    • How to design your system to take advantage of OSS?
  • Legal strategies
    • How to avoid getting sued by some huge tech company?

ESIP-WINTER12

the nasa esds context1
The NASA ESDS Context

The aforementioned OSS concerns are cross cutting against the whole ESDS enterprise!

ESIP-WINTER12

licensing
Licensing
  • Relates to: redistribution, intellectual property, contributing, legal strategies
  • There are tons of OSS approved licenses
    • What’s the difference between them?
  • The difference mostly has to do with
    • Commercialization
    • Redistribution
    • Attribution

ESIP-WINTER12

redistribution
Redistribution
  • So you’ve built some awesome piece of NASA ESDS Software
  • And you’re wondering
    • What are my options for distributing it to
      • Other DAACs?
      • Other SIPS?
      • Other funded proposal efforts we’re collaborating with (ACCESS, MEASURES)
      • Any other collaborators?
  • Question: what license are you going to choose?
    • Hopefully one that supports redistribution under your own terms!
  • How to redistribute the software?
    • Requires infrastructure
    • Who’s going to set it up?

ESIP-WINTER12

oss ecosystems
OSS Ecosystems
  • Where should you go to for your open source project?
  • Should your org have its own?
  • Should your project (SIPS, DAAC, proposal) have its own?

ESIP-WINTER12

apache maturity model
Apache Maturity Model
  • Start outwith Incubation
  • Grow community
  • Make releases
  • Gain interest
  • Diversify
  • When the project is ready, graduate into
    • Top-Level Project (TLP)
    • Sub-project of TLP
  • Increasingly, Sub-projects are discouraged compared to TLPs

ESIP-WINTER12

apache organization
Apache Organization
  • Apache is a meritocracy
    • You earn your keep and your credentials
  • Start out as Contributor
    • Patches, mailing list comments, etc.
    • No commit access
  • Move onto Committer
    • Commit access, evolve the code
  • PMC Members
    • Have binding VOTEs on releases/personnel
  • Officer (VP, Project)
    • PMC Chair
  • ASF Member
    • Have binding VOTE in the state of the foundation
    • Elect Board of Directors
  • Director
    • Oversight of projects, foundation activities

ESIP-WINTER12

sourceforge a different model
SourceForge (a different model)
  • Project Proposal
    • Accepted? Get going!
  • No foundation-wide oversight
  • Tons of dormant projects with no communities of interest
  • Goal is to host infrastructure andhost technologies
  • Goal is not to build communities
  • No foundation-wide rules or guidelines for committership or for project management
    • Dealt with locally by the progenitor of the project
    • Can lead to BDFL (benevolent dictator for life) syndrome
  • No foundation-wide license requirements
    • BSD, GPL(v2, v3), MIT, LGPL, etc all allowed

ESIP-WINTER12

oss ecosystems1
OSS Ecosystems
  • FTR, there are tons of concerns here
    • Should the ecosystem impose license restrictions?
      • Only support license X or Y?
    • What are the redistribution policies?
    • What’s the community?
    • What’s the IP?
    • What’s the infrastructure support?
    • Is it a foundation with rules, or just free for all?
  • Elephant in the room: What is the exposure? How many people are going to community C and are going to see your software and perhaps want to use it, improve it, file bugs against it, file patches, etc.?
  • Where/how does this matter to ESIP?
    • Standing up our own OSS ecosystem may make sense for large, coarse-grained federations
    • A LOT more difficult to justify OTOH, for fine grained components, and module reuse

ESIP-WINTER12

community versus help desk
Community versus “Help Desk”
  • How do you want to have your open source software project run in the open?
    • Do you expect folks to come to you and only file bugs?
      • Then you and your team are the only ones who can fix them?
      • Then you and your team are the only ones who can release updates?
      • ”Help Desk” open source project
      • Examples: Sourceforge.net, Google Code, etc.
    • Do you expect to grow a community of interest where volunteers actively engage in software development?
      • Are volunteers empowered to pick up a shovel and help dig the hole?
      • Can volunteers (including your own paid employees) file issues?
      • Do you want to give the community a stake/vote in the overall process?
      •  “Community Building” open source project
      • Examples: Apache Software Foundation, Eclipse Foundation, etc.

ESIP-WINTER12

communities
Communities
  • Working on common software
    • Measured not in terms of what center contributor works for, but in terms of
      • Number of patches contributed (high quality)
      • Mailing list questions answered
      • # of releases made, or helped with
      • Tests written
      • Documentation added
    • Take the politics out of it and just work on “core” common code of mutual interest
  • Deciding on the right redistribution mechanism and license
    • Apache Software Foundation and ASLv2 provide openness and ability for center-local redistribution, commerciality and other decisions (for internal distributions and beyond)

ESIP-WINTER12

intellectual property
Intellectual Property
  • Centers can similarly (if they so choose) have their own customizations/distributions of OSS
    • NASA Langley DAAC’s FooBar powered by Apache Hadoop
    • NASA AIRS SIPS’s Baz built on Apache Pivot powered by Linux
  • NASA protects its marks through the New Technology Report process
    • Uncover marks
    • Uncover potential patents
    • License/etc., as appropriate
    • We see this less on the ground side than we do with the flight
    • Mostly NASA wide, but some center specifics (e.g., Caltech)
  • Takeaway: the answer to “who owns it” is still “the Center” or “the project” who initiated the software development, with or without OSS. OSS may have its own restrictions/rules, so need to be wary of that (recall copyleft syndrome)

ESIP-WINTER12

contributing to oss
Contributing to OSS
  • OSS communities are built around “contributions”
    • But what does it mean to contribute?
  • Mailing list help/discussions
    • Yes those are contributions
  • Reporting/filing bugs and new feature improvements
    • Yes those are contributions
  • Writing documentation and user guides
    • Yes those are contributions
  • Volunteering to make releases
    • Yes those are contributions
  • Positively contributing towards the evolution of the project with your thoughts and ideas
    • Yes those are contributions
  • NOTE: What did I leave out?

ESIP-WINTER12

contributing to oss code
Contributing to OSS: code
  • Yes yes, code is important (but not the only contribution)
  • How should you go about your code contributing process to open source?
  • Relates to: “Help Desk”versus Community
  • Do you want the members of your project to be theonly folks who can actual touch the source code of your OSS project?
    • Only bug reports?
    • How “open” are you?
    • How “open” do you want to be?
  • Do you want to accept code contributions from the community?
    • Patches?
    • Allow community members to become “committers” (assist in the development of the code?)
    • RTC versus CTR

ESIP-WINTER12

responsiveness
Responsiveness
  • Measure of a “healthy” community
  • Does your patch sit?
  • Do your mailing list questions go unanswered?
  • Is the project electing new “committers” (if it’s in community mode)?
  • How should your group decide whether or not an OSS project you want to use is responsive?
  • How should your group decide how responsive it will be?
  • Heavily influenced by:
    • Community mode: “Help desk” versus “Community”

ESIP-WINTER12

getting help
Getting help?
  • Using Open Source Software is different than producing software intended for distribution in open source
    • Either way: what’s your strategy for getting help?
    • It used to be: you can’t rely on OSS for any “real time” support
    • Not anymore: companies stood up “around” OSS technology, dedicated to providing support
      • Apache Hadoop: Cloudera
      • Apache Solr/Lucene: Lucid Imagination
      • Apache Cassandra: Riptano
      • Apache Maven: Sonatype
      • …others?
  • How good is the documentation for the OSS Project?
    • Healthy wiki?
    • Healthy “cook books”?
    • Is the documentation baked in or separate?
    • Javadocs (or equivalent) on all public methods?
    • Use cases?

ESIP-WINTER12

getting help diy
Getting help: DIY
  • Build expertise and intellectual capital in existing open source technology
    • If you are pulling it in and using it
    • Involves: “Community” model, but also works in “Help Desk”
    • Involves: active participation
    • Involves: friendly license (especially if you want to redistribute)
  • If you are building your own technology that you want to put into open source
    • Others may “help” you
    • Solve challenges at ZERO cost to you (well not ZERO but minimal)
    • Free cycles of interest

ESIP-WINTER12

interactions with open source
Interactions with Open Source
  • Case Study: Mailing lists communications*

* Taken from: http://wiki.apache.org/nutch/Becoming_A_Nutch_Developer

ESIP-WINTER12

interactions with open source1
Interactions with Open Source
  • Mailing list interaction very important
    • Most immediate feedback you get when contributing to open source, and also when creating your own open source project
    • Is the biggest thing that turns people away to start out with
    • “who are these nerds and why don’t they have any manners?”
    • It’s all about “learning” their “language”
      • Sure, the cost may be high in terms of ego and attitude, but the reward (free work!) is well worth it
  • Following and understanding the practices of the OSS project are hugely important
    • Know the license for an OSS product
    • Know whether they are “community” versus “help desk”
    • Know its history
    • Read its documentation
    • Be informed!
  • Danger: thinking the OSS community is your personal helpdesk (or vice versa, you are its)

ESIP-WINTER12

architectural and implementation issues
Architectural and Implementation issues
  • How to design for open source?
  • Extension points
    • Places in your code for “plug ins”
    • Plugins should be well documented
      • Public facing “javadoc” (or similar)
      • “Cookbooks” for integrating
      • Wiki pages
      • Baked in documentation
  • Use of shared component marketplace
    • Many times PL specific
    • PyPI – Python
    • Maven Central/Ibiblio – Java
    • CPAN – Perl
    • STL, GNU, Autoconf, Make, etc. – C/C++
    • YMMV, some are
      • Webified, have package metadata, searchable, etc.

ESIP-WINTER12

leverage active technologies
Leverage active technologies
  • Don’t expect to use Haskel and have a huge OSS community there to assist you
  • Some modern active OSS languages
    • Python
    • Perl
    • Ruby
    • Java
  • Don’t just be a consumer, be a producer
    • Pick up a shovel and help dig the hole
    • Instead of overlooking the OSScommunity and telling them how the hole should be dug to meetyour needs
    • Will help you get needed OSSfeedback for your technology

ESIP-WINTER12

legal issues
Legal Issues
  • Understand patent law
    • Or pay people to understand it for you
  • Understand OSS licenses and what you sign up for when you use relevant technologies and their licenses
  • Look for open APIs and open specs
    • Understand what “open” means
  • Vendor lock in
    • Try to avoid it through the use of all the techniques of OSS which we’ve discussed
  • Engage the community and understand what’s “really” free an what isn’t
  • Get legal’s buy-in
    • Don’t insulate them

ESIP-WINTER12

legal issues1
Legal Issues
  • Understand “marks”
    • Trademarks
    • Their implication to patents
  • What will your agency support in terms of licensing?
    • Center specific
    • Depends on IP, export control, ITAR, compliance, commercialization
    • If you make it through all of the above and the software is cleared for release, you really are in the driver’s seat
      • No agency wide guidelines
      • Most or all communities allowed
      • Most or all OSS approved licenses allowed
  • Is there a Contributor License Agreement (CLA) or Corporate Contributor License Agreement (CCLA) to sign?
    • What are its implications?

ESIP-WINTER12

free as in beer
Free as in beer
  • “Something like the Apache foundation is the best place for released government software. A previous attempt at release and public distribution via a private company was a truly dismal failure. OpenPBS (portable batch system) is supposed to be available to anyone that asks. However when you do ask a sales rep strings you along for more than a month trying to sell you something that they can't actually assure you will fit your requirements (and is no longer under development) even when the free one is documented as doing so. It was a truly stupid waste of the salesperson's time and mine that would have exceeded the price of providing the file for download or sending by email by several orders of magnitude and generated a lot of ill will. I'll go as far as saying it was blatant false advertising using a government funded open source product to do a bait and switch to try to sell me an unmaintained product they picked up in a corporate take over. My experience appears to have been identical to that of many that attempted to obtain this government funded open source software that NASA had declared was available for anyone. Eventually due to this open source project becoming closed the project just had to fork and the compatible Torque batch system was developed by people that had actually get hold of the original OpenPBS.”– Slashdot user comment

ESIP-WINTER12

nasa open source summit
NASA Open Source Summit

http://www.nasa.gov/open/source/

ESIP-WINTER12

discussion
Discussion
  • Let’s discuss the set of recommendations that came out of the NASA OSS
  • They are very much applicable for the ESIP community as a whole
    • Each participating agency should be able to answer how they are responding to these questions in their agency
    • Forms the agency’s policy and strategy for open source software

ESIP-WINTER12

recommendation 1
Recommendation #1
  • Communication  and  Publicizing  NASA’s  Open  Source  Efforts
    • In order  for  an  Open  Source  policy  to  be  successful,  NASA must  make  an  effort  to  encourage  both   internal  and external  parties  to  participate  in  open  source  development.
    • What  does  the  agency  need  to   do  in  order  to  make  NASA’s open  source  efforts  well  known?

ESIP-WINTER12

recommendation 2
Recommendation #2
  • Licensing
    • The NASA  Open  Source  Agreement  license  (NOSA)  was  originally developed  in  2003  to  enable  NASA  to   provide  software  in source  code  form  to  the  public,  but  software  must  already be  considered  complete.
    • Which licenses should be allowed?

ESIP-WINTER12

recommendation 3
Recommendation #3
  • Barriers  to  Involvement  from  the  Open  Source  Community
    • As a  government  agency  bound  by  significant  regulation  and bureaucracy,  what  are  the  limits  of   community  contribution For  example,  could  a  NASA originated  codebase  ever  be handed  over  to  a   non NASA  community  member  for  long term support  and  maintenance?  Is  this  legal?  If it  were legal, would it be practical?

ESIP-WINTER12

recommendation 4
Recommendation #4
  • Barriers to Development Models and Ongoing Support
    • How do we ensure that “openness”does not conflict with rigor? How narrow should the definition of “development team” be?

ESIP-WINTER12

recommendation 5
Recommendation #5
  • Government Restrictions
    • How to we mesh open source software with decidedly un-open policies such as ITAR?

ESIP-WINTER12

recommendation 6
Recommendation #6
  • Limitations  on  Contributing  to  External  Open  Source Projects  
    • What  are  the  differences  between  contributing minimal, incremental  improvements  or  bug  fixes  and   new  features (as  per  NPR  2210.01)?  And,  who  makes  this  decision?
    • What  lessons  learned  /  best   practices  can  be  drawn  from current  NASA  open  source  development  “pathfinder”  projects (and  can   these  be  applied  “generally”)?  What  should  be the  policy  of  NASA  personnel  contributing  in  their  off-hours?  How  do  you  handle  /  treat  situations  where  people work  off‐hours  on  things  that  derive  directly,   or  are inspired,  by  what  they  worked  on  during  duty  hours?

ESIP-WINTER12

recommendation 7
Recommendation #7
  • How  does  Open  Source  governance  look  within  NASA?  
    • There currently  is  a  lack  of  information  and  awareness  towards licensing,  legal  issues,  and  activity  in  the   open  source software  community  at  NASA.  How  does  one  receive  guidance on  open  source contributions?  What  does  the  process  look like?

ESIP-WINTER12

recommendation 8
Recommendation #8
  • How  should  Open  Source  Efforts  Be  Supported?
    • NASA needs  to  develop  cooperative  support  into  project  structure
      • Project Level  - permitting  code  deemed  outside  the  purview  of ITAR/EAR  to  be  open  source  
      • Budget  Level - allowing for  hiring  of  floating  talent  as  temporary  staff  augmentation
      • Organization  Level -  designing  organization  to  support habits  and  practices  of  open  source   development

ESIP-WINTER12

recommendation 9
Recommendation #9
  • How does NASA open source *everything?*
    • After all, it’s taxpayer money, right?

ESIP-WINTER12

recommendation 10
Recommendation #10
  • How  to  close  the  feedback  loop  between  policy  makers, developers   and  end  users
    • How  do  we  ensure  that  draft policies  have  enough  eyes  on  them,  particularly  from  the people  who  may   be  most  affected  or  who  have  the  most detailed  knowledge  of  those  areas,  and  who  can  thus  best understand  the  implications?  

ESIP-WINTER12

recommendation 101
Recommendation #10
  • How  to  close  the  feedback  loop  between  policy  makers, developers   and  end  users
    • Policies  should  not  put anyone  in  the  position  of  performing  the  mission   by violating  the  policy,  or  adhering  to  the  policy  and thereby  reducing  the  effectiveness  or  inducing  the   failure of  the  mission.  Many  times,  our  policies  derive  (or  are simply  copied)  from  Federal  law,   regulation,  guidance  or, more  commonly,  from  the  policies  of  other  agencies.  

ESIP-WINTER12

recommendation 102
Recommendation #10
  • How  to  close  the  feedback  loop  between  policy  makers, developers   and  end  users
    • Effective  feedback   provides  opportunities  to  modify  or adjust  policy  based  on  practical,  realistic  feedback.  Are we  taking   advantage  of  these  flexibilities?

ESIP-WINTER12

recommendation 11
Recommendation #11
  • How  to  encourage  cultural  change  in  hiring  practices?
  • How can  NASA  attract  more  open  source savvy  people  in  a world  where  companies  like  Red  Hat  &   Google  offer  careers that  encourage  such  participation? NASA  is  competing against  these  companies  for   the  same  skills.

ESIP-WINTER12

recommendation 12
Recommendation #12
  • How  to  Package  Open  Source  Software  to  be  More Accessible? 
    • Collaboration on  open  source  software  is  dependent  on  others  who  find the  software  useful.  The  barrier   for  adoption  of  OSS must  be  kept  low.  This  also  prevents  the  project  dying on  the  vine.  Is  there  a  way  we   can  package  OSS  to make  it  easy  for  others  to  try  and  adapt  to  their  needs?
    • How does NASA ESDSWG SRWG Packaging Rec fit here?

ESIP-WINTER12

recommendation 13
Recommendation #13
  • Combining  open  source  software  development  standards with   Office  of  the  Chief  Engineer  Policies
    • Also interested here in coordinating with the NASA Earth Science Data System WGs, and Software Reuse WG, and ESIP Federation

ESIP-WINTER12

wrapup
Wrapup
  • Open source is critical to the strategy of the organization
  • A great method of sustainability and longevity of software and community beyond organizational boundaries
  • Different licenses, communities, development practices
    • Know what the differences mean

ESIP-WINTER12

alright i ll shut up now
Alright, I’ll shut up now
  • Any questions?
  • THANK YOU!
    • chris.a.mattmann@nasa.gov
    • @chrismattmann on Twitter

ESIP-WINTER12

implementation strategies
Implementation strategies
  • Concerns
    • Insulation: how do I insulate my project from OSS change but still use OSS?
    • How do I make my project’s OSS software available for redistribution?
    • What technologies are the “best” for my project?
    • What communities are the best?
  • Answers:
    • See next slide
    • It depends
    • It depends
    • It depends

ESIP-WINTER12

insulation one strategy
Insulation: one strategy
  • Missions maintaintheir own local CMs
  • Local mission CMs contain forks of existing OSS software
    • Forks can be patchbased or CM based
  • Changes found particularly effectiveare discussed within the comm.And eventually brought before a CCB that reviews their generality, etc.

Fig Credit:Dana Freeborn

ESIP-WINTER12