Integration of version control and code migration processes chapter 3 evolution
Download
1 / 43

Integration of Version Control and Code Migration Processes Chapter 3 Evolution - PowerPoint PPT Presentation


  • 80 Views
  • Uploaded on

Integration of Version Control and Code Migration Processes Chapter 3 – Evolution. JA-SIG - December 4, 2006 - Atlanta, GA Jim Thompson Division of Information Technology University of Wisconsin – Madison https://mywebspace.wisc.edu/jfthomps/vc. University of Wisconsin - Madison.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Integration of Version Control and Code Migration Processes Chapter 3 Evolution' - briar


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Integration of version control and code migration processes chapter 3 evolution

Integration of Version Control and Code Migration ProcessesChapter 3 – Evolution

JA-SIG - December 4, 2006 - Atlanta, GA

Jim Thompson

Division of Information Technology

University of Wisconsin – Madison

https://mywebspace.wisc.edu/jfthomps/vc


University of wisconsin madison
University of Wisconsin - Madison

  • public land-grant university (est. 1848)

  • home of the Badgers

  • Big Ten conference (11 with Penn State)

  • over 40,000 students

  • among top 5 in USA for research funds

  • #1 large school for Peace Corps volunteers

  • Madison is the capital of Wisconsin

JA-SIG 12/4/2006


My work group
My work group

  • part of Division of Information Technology

  • we design, build, acquire, install and support Internet infrastructure applications (a.k.a. upper middleware)

    • E-mail

    • Calendar

    • My WebSpace (see http://www.educause.edu/apps/eq/eqm05/eqm0548.asp)

    • Portal

JA-SIG 12/4/2006


Today s presentation
Today’s presentation

  • Background info on our portals

  • Epicentric (simple design = simple migrations)

    • Regular vs. environment-specific tags in CVS

    • Ported A-B-C design for a Cobol developer (2003)

    • Documented how to build an automated process

  • uPortal (design got more interesting)

    • Maps to locate config files

    • Improved automation for processing maps

    • Modernized process for a Cobol developer (2006)

  • Maybe some of this can work for you

JA-SIG 12/4/2006


My uw madison our portal
My UW-Madison (our portal)

https://my.wisc.edu provides access to

  • E-mail

  • Calendar

  • My WebSpace

  • Application status and Enrollment

  • Financial Aid

  • Classes on the web ([email protected])

  • Course/Advisor/Instructor/Teaching Resources

  • Grades (entry and review)

  • Demo Accounts (Student & Instructor/Advisor)

JA-SIG 12/4/2006


Some my uw madison statistics
Some My UW-Madison statistics

  • September 2003 – over 1.3 million logins

  • Over a million logins/month thru May 2004

  • September 2004 (new usage records)

    • Over 1.85 million logins by 49,900+ users

  • September 2, 2004 (first day of instruction)

    • 85,739 logins by 29,676 users

    • About 6,000 users during lunch hour

  • September 2005 (new UW home page)

    • Over 1.5 million logins by 51,000+ users

JA-SIG 12/4/2006


My uw madison some lingo
My UW–Madison - some lingo

  • MUM = My UW–Madison

  • MUMMI = module inventory

  • MUMMR = migration request

  • MUMsters = developers

  • mumdev = development/test box (Solaris)

  • 4 qa + 8 prod servers named after mummies

JA-SIG 12/4/2006


Mum service goals
MUM Service Goals

  • 24x7 availability (but not all connected systems are always up)

  • Handle peak traffic without slowing down

  • Good response time from called servers

  • Minimize outages (frequency & duration)

  • Weekly updates of production

  • How are we doing?

JA-SIG 12/4/2006


Production updates
Production updates

  • All portal code is on our servers

  • Prefer weekly releases

  • Prepare a staging machine (Monday PM)

  • Deploy early Tuesday (between 5 & 7 AM)

  • Prefer “rolling restarts” to avoid outages

JA-SIG 12/4/2006



Paths to production demo
Paths to production & demo

  • App files migrate thru 6 environments

    • Development

    • Test

    • QA

    • Production

    • Testdemo

    • Demo

  • 1,000+ files in each environment

JA-SIG 12/4/2006


Overview of environments
Overview of environments

  • Development

    • the Wild West

    • CVS available, but not required

  • Test and Testdemo

    • Files must go thru CVS to get to test/testdemo

    • Developers apply test & testdemo tags in CVS

    • Update Mon-Fri 4 & 4:20 AM and “right now”

    • All test migrations stop/restart test

JA-SIG 12/4/2006


Overview of environments1
Overview of environments

  • QA

    • Files flow thru CVS

    • Developers request migration via MUMMR

    • Run load tests here (for about a week)

  • Production

    • Files flow thru CVS

  • Demo

    • Files flow thru CVS

    • Nearly as important as production

JA-SIG 12/4/2006


Files flow thru cvs
Files flow thru CVS

  • Question: How do we control the flow?

    • How to mark a file for promotion?

    • How to know what files got promoted…

      • Which version?

      • Where?

      • When?

  • Answer: CVS tags

    • Drive the process

    • Indicate what’s where, since when

JA-SIG 12/4/2006


Some old cvs tags
Some old CVS tags

JA-SIG 12/4/2006


Cvs tag maintenance
CVS tag maintenance

  • Who maintains which tags?

    • Developers apply and move test/testdemo tags

    • MUM admins apply qa, prod, demo tags

    • Scripts apply and move in and outa tags

    • Scripts create update history tags

  • What’s the purpose of the outa tags?

    • Track deletes (like adds and changes)

    • Encourage developers to clean up

    • Avoid permanent loss of old code (in Attic)

JA-SIG 12/4/2006



Why a mummr
Why a MUMMR?

  • Reproducible instructions

  • QA is where we test the MUMMR

    • Complete list of files?

    • Clear instructions?

    • If we find problems, fix & resubmit MUMMR

    • Increase odds of successful production move

  • Documentation

    • Tell Help Desk and others what’s coming

JA-SIG 12/4/2006


Files flow
Files flow

JA-SIG 12/4/2006


Cvs tags who cares
CVS tags - Who cares?

  • Developers

    • Where’s my code?

  • Project Managers

    • Which version was installed where and when?

  • Migration Controllers

    • What’s scheduled to move?

  • Everyone

    • What’s the difference between X and Y?

JA-SIG 12/4/2006


How comparing tags can help
How comparing tags can help

  • Preparing to update 100+ production files

    • Apply prod tags according to MUMMR

    • Refresh export of files tagged prod

    • Compare inqa and prod

      • If all files are moving, there should be no diffs

      • All differences should be examined

      • Partial migrations are opportunities for problems

  • Developers can use same technique

    • Compare intest and inqa to get MUMMR right

JA-SIG 12/4/2006


That s the main story but
That’s the main story, but…

  • Environmental files

    • DB config settings, log verbosity, etc.

    • permanently branched in CVS (never merged)

    • many branches for Epicentric’s properties.txt

    • convert: Tomcat to WebSphere to WebLogic

    • often had 5 to 8 active Epicentric environments

    • now support 8 uPortal environments

  • Automated updates of environmental files

JA-SIG 12/4/2006


Some environmental branches
Some environmental branches

  • my-predev-tc-under (framework development)

  • my-dev-tc-mumdev

  • my-dev-tc-jh9

  • my-test-tc-mumdev

  • my-testdemo-tc-mumdev

  • my-qa-tc-qa

  • my-prod-tc-prod

  • my-demo-tc-glow

  • 19 Epicentric/Vignette branches (RIP)

JA-SIG 12/4/2006


Controls in cvs
Controls in CVS

  • Regular application files

    • tag changes limited via CVS filter (taginfo)

  • Environmental files

    • anyone can grab a branch

    • commits trigger e-mail notice to MUM admins

      • commits on my-dev or my-test branches are OK

      • commits on other branches, if not by MUM admins, may cause questions

JA-SIG 12/4/2006


Updates were simple as a b c
Updates were simple as A-B-C

  • A = app; B = bin; C = config

  • Epicentric had config files in one directory

  • Update script was simple

    • cd $APPDIR

    • cvs update –r $REGTAG (e.g., test, qa, prod)

    • cd $CONFIGDIR

    • cvs update –r $ENVTAG (e.g., my-…)

  • Simple design yields a simple solution

JA-SIG 12/4/2006


Cloned this solution in 2003
Cloned this solution in 2003

  • We built a similar system for a COBOL app

    • CVS repository and app on AIX

    • 6 files in production app directory

  • Forced Cobol app into A-B-C design

    • Symlink (app to config) for Cobol compiler

  • Contents of document aimed at 3 audiences

    • Managers (General Overview ~ 17 pages)

    • Designers (Design Decisions ~ 5 pages)

    • Implementers (Details & Example Scripts ~ 30 pages)

JA-SIG 12/4/2006


Samples from old document
Samples from “old” document

  • Our CVS taginfo filter calls tagcheck.pl

  • tagcheck.pl lets admins change any tag

  • How do we know tagcheck.pl is working?

  • see S.220 for test plan

  • brief primer on basics of CVS - G.040

  • blank MUMMR – X.110

    https://mywebspace.wisc.edu/jfthomps/vc/intvc

JA-SIG 12/4/2006


Moved to uportal
Moved to uPortal

  • What changed and what didn’t?

    • Good bye, Makefiles… Hello, ant

    • Basic CVS tags and processes continue

    • Simpler shell scripts; more complex build.xml

    • cvs-update.xml (build.xml) calls cvs-logntag.pl

    • Some new tags for uPortal frameworks

    • New (optional) milestone tags

    • Reports to find and compare tags

JA-SIG 12/4/2006


Uportal frameworks in cvs
uPortal frameworks in CVS

  • Import vendor files on vendor branch

  • Separate vendor branch for each vendor

  • Keep our local modifications on trunk

  • CVS tags to identify and control

    • Vendor releases (Vup2-5-1)

    • Our local customizations (Rdev2-5-1)

    • Releases for our environments (Rdev, Rtest…)

    • https://mywebspace.wisc.edu/jfthomps/vc/CVS_up.html

JA-SIG 12/4/2006



Tags for our frameworks
Tags for our frameworks

  • Rdev2-5-1  my-uppredev… (fwk dev)

  • Rdev  my-updev… (shared dev)

  • Rtest  my-uptest…

  • Rtestdemo  my-testdemo…

  • Rqa  my-upqa…

  • Rprod  my.wisc.edu

  • Rdemo  my-demo…

JA-SIG 12/4/2006


More cvs tags
More CVS tags

JA-SIG 12/4/2006


Major families of uportal tags
Major families of uPortal tags

  • Regular (app) tags

    • test, testdemo, qa, prod, demo

    • status (in, outa), update history and milestone tags

  • Framework tags

    • Vup2-5-1, Rdev2-5-1, Rdev, Rtest, Rqa, Rprod, etc.

    • Rxx status (in, outa), update history and milestone tags

  • Environment branches (everywhere)

    • my-dev-tc-mumdev, my-test-tc-mumdev, etc.

    • also have status (in, outa) and milestone tags

JA-SIG 12/4/2006


Which tags are used where
Which tags are used where?

  • Milestone tags are optional (0, 1 or 2)

  • Status tags are optional

  • Update history tags are optional

  • Custom blends are feasible

    • Private (e.g., desktop/laptop) development

    • Update with prod (or qa) code for 90% of files

    • Update to HEAD where you’re working

    • Controlled chaos

JA-SIG 12/4/2006


Finding uportal environment files
Finding uPortal environment files

  • Beyond A-B-C of Epicentric days

  • All over the place

    • in uPortal framework

    • in portlets

    • in batch directories

    • in admin directories

  • Listed in map-cvs-branched-env-files.xml(maps found in multiple locations)

JA-SIG 12/4/2006


Updating environment files
Updating environment files

  • Initially - targets within cvs-update.xml

    • Find and parse each map-cvs-branched-env-files.xml

    • Using 2 XSL style sheets, build and run cvs updates for each mapped file and directory (using supplied ENV branch) – Thanks, Bruce!

  • Now – perl script cvs-use-maps.pl does the job

    • More robust with more functions (updates and tags)

    • Detects problems and recovers when possible

    • Run from ant (cvs-update.xml) or shell scripts

JA-SIG 12/4/2006


Our busy ant build xml
Our busy Ant build.xml

  • Major targets of cvs-update.xml

    • help

    • status (verbose - also run by each update)

    • updateApp

    • updateFwk

    • updateAll

  • Main parameters of cvs-update.xml

    • several to specify, skip or set CVS tags

JA-SIG 12/4/2006


Some uportal updates
Some uPortal updates

  • Predev (for framework development)

    • updateAll–Dfwktag=Rdev2-5-1 –Dregtag=qa …

  • Dev (release new framework)

    • updateFwk–Dfwktag=Rdev –DmodFwkTags=y …

  • Test (usual test code update)

    • updateApp–Dregtag=test –DmodRegTags=y …

JA-SIG 12/4/2006


Typical test update
Typical test update

  • Developer (or cron) runs update script

  • Script tees output and e-mails scanner

  • Scanner (via procmail) parses output

    • Puts copy of raw output (~160k) on the web

    • Builds HTML copy (anchor tags each problem)

    • Builds HTML summary (2k map) on the web

    • Mails summary (map) of results to developers

    • Thank you, Curt!

  • Update creates verbose CVS status report

JA-SIG 12/4/2006


How to track herds of cvs tags
How to track herds of CVS tags

  • Parse the verbose CVS status reports

    • Also tracked Epicentric/Vignette files

  • cvs-show-tags.pl finds and counts tags

    • Defaults to not showing Update History tags

    • Can include branch names (or all tags)

  • cvs-diff-tags.pl compares 2 tags/branches

    • Option to show CVS revs of updated files

    • Summary of all changed files

JA-SIG 12/4/2006


How might you try this
How might you try this?

  • Old Make-based methods are documented

    • for an A-B-C design (separate Apps & Config)

    • shell scripts and some perl

  • Newer Ant/Make-based methods work

    • earlier documents are still relevant

    • maps locate environment (config) files

    • shell scripts, cvs-update.xml and some perl

  • Available via links (here + mywebspace…)

JA-SIG 12/4/2006


What about subversion
What about subversion?

  • Some folks use and like it a lot

  • Repeatedly moving tags in svn is tedious

  • Our 2-phase multi-tag scheme is a challenge

  • Starting to explore options…

  • Others with svn experience?

  • Maybe a Version Control BoF?

  • Maven on the horizon - in uPortal3…

JA-SIG 12/4/2006


Questions
Questions

???

https://mywebspace.wisc.edu/jfthomps/vc

“Prediction is difficult, especially about the future.”

– Yogi Berra

JA-SIG 12/4/2006


ad