Integration of version control and code migration processes chapter 3 evolution
This presentation is the property of its rightful owner.
Sponsored Links
1 / 43

Integration of Version Control and Code Migration Processes Chapter 3 – Evolution PowerPoint PPT Presentation


  • 55 Views
  • Uploaded on
  • Presentation posted in: General

Integration of Version Control and Code Migration Processes Chapter 3 – Evolution. JA-SIG - December 4, 2006 - Atlanta, GA Jim Thompson Division of Information Technology University of Wisconsin – Madison https://mywebspace.wisc.edu/jfthomps/vc. University of Wisconsin - Madison.

Download Presentation

Integration of Version Control and Code Migration Processes Chapter 3 – Evolution

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Integration of version control and code migration processes chapter 3 evolution

Integration of Version Control and Code Migration ProcessesChapter 3 – Evolution

JA-SIG - December 4, 2006 - Atlanta, GA

Jim Thompson

Division of Information Technology

University of Wisconsin – Madison

https://mywebspace.wisc.edu/jfthomps/vc


University of wisconsin madison

University of Wisconsin - Madison

  • public land-grant university (est. 1848)

  • home of the Badgers

  • Big Ten conference (11 with Penn State)

  • over 40,000 students

  • among top 5 in USA for research funds

  • #1 large school for Peace Corps volunteers

  • Madison is the capital of Wisconsin

JA-SIG 12/4/2006


My work group

My work group

  • part of Division of Information Technology

  • we design, build, acquire, install and support Internet infrastructure applications (a.k.a. upper middleware)

    • E-mail

    • Calendar

    • My WebSpace (see http://www.educause.edu/apps/eq/eqm05/eqm0548.asp)

    • Portal

JA-SIG 12/4/2006


Today s presentation

Today’s presentation

  • Background info on our portals

  • Epicentric (simple design = simple migrations)

    • Regular vs. environment-specific tags in CVS

    • Ported A-B-C design for a Cobol developer (2003)

    • Documented how to build an automated process

  • uPortal (design got more interesting)

    • Maps to locate config files

    • Improved automation for processing maps

    • Modernized process for a Cobol developer (2006)

  • Maybe some of this can work for you

JA-SIG 12/4/2006


My uw madison our portal

My UW-Madison (our portal)

https://my.wisc.edu provides access to

  • E-mail

  • Calendar

  • My WebSpace

  • Application status and Enrollment

  • Financial Aid

  • Classes on the web ([email protected])

  • Course/Advisor/Instructor/Teaching Resources

  • Grades (entry and review)

  • Demo Accounts (Student & Instructor/Advisor)

JA-SIG 12/4/2006


Some my uw madison statistics

Some My UW-Madison statistics

  • September 2003 – over 1.3 million logins

  • Over a million logins/month thru May 2004

  • September 2004 (new usage records)

    • Over 1.85 million logins by 49,900+ users

  • September 2, 2004 (first day of instruction)

    • 85,739 logins by 29,676 users

    • About 6,000 users during lunch hour

  • September 2005 (new UW home page)

    • Over 1.5 million logins by 51,000+ users

JA-SIG 12/4/2006


My uw madison some lingo

My UW–Madison - some lingo

  • MUM = My UW–Madison

  • MUMMI = module inventory

  • MUMMR = migration request

  • MUMsters = developers

  • mumdev = development/test box (Solaris)

  • 4 qa + 8 prod servers named after mummies

JA-SIG 12/4/2006


Mum service goals

MUM Service Goals

  • 24x7 availability (but not all connected systems are always up)

  • Handle peak traffic without slowing down

  • Good response time from called servers

  • Minimize outages (frequency & duration)

  • Weekly updates of production

  • How are we doing?

JA-SIG 12/4/2006


Production updates

Production updates

  • All portal code is on our servers

  • Prefer weekly releases

  • Prepare a staging machine (Monday PM)

  • Deploy early Tuesday (between 5 & 7 AM)

  • Prefer “rolling restarts” to avoid outages

JA-SIG 12/4/2006


Integration of version control and code migration processes chapter 3 evolution

JA-SIG 12/4/2006


Paths to production demo

Paths to production & demo

  • App files migrate thru 6 environments

    • Development

    • Test

    • QA

    • Production

    • Testdemo

    • Demo

  • 1,000+ files in each environment

JA-SIG 12/4/2006


Overview of environments

Overview of environments

  • Development

    • the Wild West

    • CVS available, but not required

  • Test and Testdemo

    • Files must go thru CVS to get to test/testdemo

    • Developers apply test & testdemo tags in CVS

    • Update Mon-Fri 4 & 4:20 AM and “right now”

    • All test migrations stop/restart test

JA-SIG 12/4/2006


Overview of environments1

Overview of environments

  • QA

    • Files flow thru CVS

    • Developers request migration via MUMMR

    • Run load tests here (for about a week)

  • Production

    • Files flow thru CVS

  • Demo

    • Files flow thru CVS

    • Nearly as important as production

JA-SIG 12/4/2006


Files flow thru cvs

Files flow thru CVS

  • Question: How do we control the flow?

    • How to mark a file for promotion?

    • How to know what files got promoted…

      • Which version?

      • Where?

      • When?

  • Answer: CVS tags

    • Drive the process

    • Indicate what’s where, since when

JA-SIG 12/4/2006


Some old cvs tags

Some old CVS tags

JA-SIG 12/4/2006


Cvs tag maintenance

CVS tag maintenance

  • Who maintains which tags?

    • Developers apply and move test/testdemo tags

    • MUM admins apply qa, prod, demo tags

    • Scripts apply and move in and outa tags

    • Scripts create update history tags

  • What’s the purpose of the outa tags?

    • Track deletes (like adds and changes)

    • Encourage developers to clean up

    • Avoid permanent loss of old code (in Attic)

JA-SIG 12/4/2006


Integration of version control and code migration processes chapter 3 evolution

JA-SIG 12/4/2006


Why a mummr

Why a MUMMR?

  • Reproducible instructions

  • QA is where we test the MUMMR

    • Complete list of files?

    • Clear instructions?

    • If we find problems, fix & resubmit MUMMR

    • Increase odds of successful production move

  • Documentation

    • Tell Help Desk and others what’s coming

JA-SIG 12/4/2006


Files flow

Files flow

JA-SIG 12/4/2006


Cvs tags who cares

CVS tags - Who cares?

  • Developers

    • Where’s my code?

  • Project Managers

    • Which version was installed where and when?

  • Migration Controllers

    • What’s scheduled to move?

  • Everyone

    • What’s the difference between X and Y?

JA-SIG 12/4/2006


How comparing tags can help

How comparing tags can help

  • Preparing to update 100+ production files

    • Apply prod tags according to MUMMR

    • Refresh export of files tagged prod

    • Compare inqa and prod

      • If all files are moving, there should be no diffs

      • All differences should be examined

      • Partial migrations are opportunities for problems

  • Developers can use same technique

    • Compare intest and inqa to get MUMMR right

JA-SIG 12/4/2006


That s the main story but

That’s the main story, but…

  • Environmental files

    • DB config settings, log verbosity, etc.

    • permanently branched in CVS (never merged)

    • many branches for Epicentric’s properties.txt

    • convert: Tomcat to WebSphere to WebLogic

    • often had 5 to 8 active Epicentric environments

    • now support 8 uPortal environments

  • Automated updates of environmental files

JA-SIG 12/4/2006


Some environmental branches

Some environmental branches

  • my-predev-tc-under (framework development)

  • my-dev-tc-mumdev

  • my-dev-tc-jh9

  • my-test-tc-mumdev

  • my-testdemo-tc-mumdev

  • my-qa-tc-qa

  • my-prod-tc-prod

  • my-demo-tc-glow

  • 19 Epicentric/Vignette branches (RIP)

JA-SIG 12/4/2006


Controls in cvs

Controls in CVS

  • Regular application files

    • tag changes limited via CVS filter (taginfo)

  • Environmental files

    • anyone can grab a branch

    • commits trigger e-mail notice to MUM admins

      • commits on my-dev or my-test branches are OK

      • commits on other branches, if not by MUM admins, may cause questions

JA-SIG 12/4/2006


Updates were simple as a b c

Updates were simple as A-B-C

  • A = app; B = bin; C = config

  • Epicentric had config files in one directory

  • Update script was simple

    • cd $APPDIR

    • cvs update –r $REGTAG (e.g., test, qa, prod)

    • cd $CONFIGDIR

    • cvs update –r $ENVTAG (e.g., my-…)

  • Simple design yields a simple solution

JA-SIG 12/4/2006


Cloned this solution in 2003

Cloned this solution in 2003

  • We built a similar system for a COBOL app

    • CVS repository and app on AIX

    • 6 files in production app directory

  • Forced Cobol app into A-B-C design

    • Symlink (app to config) for Cobol compiler

  • Contents of document aimed at 3 audiences

    • Managers (General Overview ~ 17 pages)

    • Designers (Design Decisions ~ 5 pages)

    • Implementers (Details & Example Scripts ~ 30 pages)

JA-SIG 12/4/2006


Samples from old document

Samples from “old” document

  • Our CVS taginfo filter calls tagcheck.pl

  • tagcheck.pl lets admins change any tag

  • How do we know tagcheck.pl is working?

  • see S.220 for test plan

  • brief primer on basics of CVS - G.040

  • blank MUMMR – X.110

    https://mywebspace.wisc.edu/jfthomps/vc/intvc

JA-SIG 12/4/2006


Moved to uportal

Moved to uPortal

  • What changed and what didn’t?

    • Good bye, Makefiles… Hello, ant

    • Basic CVS tags and processes continue

    • Simpler shell scripts; more complex build.xml

    • cvs-update.xml (build.xml) calls cvs-logntag.pl

    • Some new tags for uPortal frameworks

    • New (optional) milestone tags

    • Reports to find and compare tags

JA-SIG 12/4/2006


Uportal frameworks in cvs

uPortal frameworks in CVS

  • Import vendor files on vendor branch

  • Separate vendor branch for each vendor

  • Keep our local modifications on trunk

  • CVS tags to identify and control

    • Vendor releases (Vup2-5-1)

    • Our local customizations (Rdev2-5-1)

    • Releases for our environments (Rdev, Rtest…)

    • https://mywebspace.wisc.edu/jfthomps/vc/CVS_up.html

JA-SIG 12/4/2006


Integration of version control and code migration processes chapter 3 evolution

JA-SIG 12/4/2006


Tags for our frameworks

Tags for our frameworks

  • Rdev2-5-1  my-uppredev… (fwk dev)

  • Rdev  my-updev… (shared dev)

  • Rtest  my-uptest…

  • Rtestdemo  my-testdemo…

  • Rqa  my-upqa…

  • Rprod  my.wisc.edu

  • Rdemo  my-demo…

JA-SIG 12/4/2006


More cvs tags

More CVS tags

JA-SIG 12/4/2006


Major families of uportal tags

Major families of uPortal tags

  • Regular (app) tags

    • test, testdemo, qa, prod, demo

    • status (in, outa), update history and milestone tags

  • Framework tags

    • Vup2-5-1, Rdev2-5-1, Rdev, Rtest, Rqa, Rprod, etc.

    • Rxx status (in, outa), update history and milestone tags

  • Environment branches (everywhere)

    • my-dev-tc-mumdev, my-test-tc-mumdev, etc.

    • also have status (in, outa) and milestone tags

JA-SIG 12/4/2006


Which tags are used where

Which tags are used where?

  • Milestone tags are optional (0, 1 or 2)

  • Status tags are optional

  • Update history tags are optional

  • Custom blends are feasible

    • Private (e.g., desktop/laptop) development

    • Update with prod (or qa) code for 90% of files

    • Update to HEAD where you’re working

    • Controlled chaos

JA-SIG 12/4/2006


Finding uportal environment files

Finding uPortal environment files

  • Beyond A-B-C of Epicentric days

  • All over the place

    • in uPortal framework

    • in portlets

    • in batch directories

    • in admin directories

  • Listed in map-cvs-branched-env-files.xml(maps found in multiple locations)

JA-SIG 12/4/2006


Updating environment files

Updating environment files

  • Initially - targets within cvs-update.xml

    • Find and parse each map-cvs-branched-env-files.xml

    • Using 2 XSL style sheets, build and run cvs updates for each mapped file and directory (using supplied ENV branch) – Thanks, Bruce!

  • Now – perl script cvs-use-maps.pl does the job

    • More robust with more functions (updates and tags)

    • Detects problems and recovers when possible

    • Run from ant (cvs-update.xml) or shell scripts

JA-SIG 12/4/2006


Our busy ant build xml

Our busy Ant build.xml

  • Major targets of cvs-update.xml

    • help

    • status (verbose - also run by each update)

    • updateApp

    • updateFwk

    • updateAll

  • Main parameters of cvs-update.xml

    • several to specify, skip or set CVS tags

JA-SIG 12/4/2006


Some uportal updates

Some uPortal updates

  • Predev (for framework development)

    • updateAll–Dfwktag=Rdev2-5-1 –Dregtag=qa …

  • Dev (release new framework)

    • updateFwk–Dfwktag=Rdev –DmodFwkTags=y …

  • Test (usual test code update)

    • updateApp–Dregtag=test –DmodRegTags=y …

JA-SIG 12/4/2006


Typical test update

Typical test update

  • Developer (or cron) runs update script

  • Script tees output and e-mails scanner

  • Scanner (via procmail) parses output

    • Puts copy of raw output (~160k) on the web

    • Builds HTML copy (anchor tags each problem)

    • Builds HTML summary (2k map) on the web

    • Mails summary (map) of results to developers

    • Thank you, Curt!

  • Update creates verbose CVS status report

JA-SIG 12/4/2006


How to track herds of cvs tags

How to track herds of CVS tags

  • Parse the verbose CVS status reports

    • Also tracked Epicentric/Vignette files

  • cvs-show-tags.pl finds and counts tags

    • Defaults to not showing Update History tags

    • Can include branch names (or all tags)

  • cvs-diff-tags.pl compares 2 tags/branches

    • Option to show CVS revs of updated files

    • Summary of all changed files

JA-SIG 12/4/2006


How might you try this

How might you try this?

  • Old Make-based methods are documented

    • for an A-B-C design (separate Apps & Config)

    • shell scripts and some perl

  • Newer Ant/Make-based methods work

    • earlier documents are still relevant

    • maps locate environment (config) files

    • shell scripts, cvs-update.xml and some perl

  • Available via links (here + mywebspace…)

JA-SIG 12/4/2006


What about subversion

What about subversion?

  • Some folks use and like it a lot

  • Repeatedly moving tags in svn is tedious

  • Our 2-phase multi-tag scheme is a challenge

  • Starting to explore options…

  • Others with svn experience?

  • Maybe a Version Control BoF?

  • Maven on the horizon - in uPortal3…

JA-SIG 12/4/2006


Questions

Questions

???

https://mywebspace.wisc.edu/jfthomps/vc

“Prediction is difficult, especially about the future.”

– Yogi Berra

JA-SIG 12/4/2006


  • Login