1 / 33

Continuous Delivery on an Enterprise Java Stack

Continuous Delivery on an Enterprise Java Stack. Marc Fasel, Senior Consultant, Shine Technologies @marcfasel http://blog.shinetech.com. Shine Technologies. Specialises in enterprise software development Blue-chip clients Technology focus Enterprise Java Ruby AWS Mobile development.

Download Presentation

Continuous Delivery on an Enterprise Java Stack

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Continuous Delivery on an Enterprise Java Stack • Marc Fasel, Senior Consultant, Shine Technologies • @marcfasel • http://blog.shinetech.com

  2. Shine Technologies • Specialises in enterprise software development • Blue-chip clients • Technology focus • Enterprise Java • Ruby • AWS • Mobile development

  3. What is Continuous Delivery? • Rapid, repeatable, reliable, low risk deployments • Automation of the software delivery process • Extension of Continuous Integration • Continuous check-in by developers • Automated build • Automated unit testing => Detect integration problems early

  4. Business Motivation • Put business in control of software release cycle • Reduce risk • deliver small batches to production • Increase Agility • Allow for short iterations • Trigger deployments at any time

  5. Continuous Delivery Project • Blue-chip Australian retailer • Website relaunch with nation-wide marketing campaign • High-performance web application • Expected load of 1,000,000 requests per hour • 24*7 availability • Greenfield development • Rapid release of new features

  6. Deployment Status Quo • Monthly deployments of software • Highly manual process • Expert knowledge required • Tedious and unpleasant • Failures due to manual process • Not rapid, repeatable, reliable, and low risk

  7. Implementation of Continuous Delivery • Automated deploy to application server • Automated acceptance testing • Push-button promotion to user acceptance testing (UAT) environment • Manual user acceptance testing • Push-button release to production

  8. Continuous Delivery Pipeline Source: Wikipedia

  9. Starting Point • Continuous Integration was already in place • Agile practices were emerging • Automated provisioning of virtual machines using Puppet was in place - machines were reliably the same • Development and deployment was both managed by Shine => DevOps • Access to production machines

  10. Environments Continuous Integration Server Local Deploy Build UAT Production Promote Promote

  11. Production Environment • Cluster of 20 nodes • Load Balancer • Use of Puppet to create nodes • Tomcat and Apache connected to a single Oracle database

  12. 1. Automated Deploy to Build Server • Different parts work together: Build artifact, configuration, data • Build artifact is copied into application server • Configuration files (Tomcat, Apache Httpd) are copied • Manual non-destructive database migration beforehand

  13. Database Migration • Ideal: Automated database migration • Database schema changes are in version control • Automated execution of database scripts during deployment and promotion • Not possible at client • DBA team get change requests with attached scripts • Review of database changes • Execution independent of software release cycle

  14. 2. Automated Acceptance Testing • Selenium • Run through major use cases • Write code don’t record scripts • Use page pattern to aid reusability • Run with every deploy for quick feedback • Tests only ran against Firefox

  15. Problems with Automated Acceptance Testing • Data must be set up/ reset in the environment • No collision with test data used by developers

  16. 3. Push-button Promotion to UAT • Artifact, configuration, and data • Promotion: Copy existing artifact, don’t rebuild • All environment-specific configuration external • Database changes had been done beforehand

  17. 4. Manual User Acceptance Testing • UAT for small features is no problem • Bottleneck were manual regression tests; even with a good test plan this took two hours of tedious work • Batching of features was necessary to reduce the number of times the manual regression tests had to be performed • Good balance was 2 week iterations with a few days of UAT and 1/2 day of manual regression testing

  18. Automated Multi-browser Acceptance Testing • Run Selenium test against supported browsers and operating systems • Maintenance of such an environment time-consuming • Testers were not technical enough to take charge of scripted acceptance testing • => Reliance on manual acceptance tests and - painfully - manual regression tests

  19. Manual Regression Test • Manual regression test after UAT is done, otherwise: If UAT fails manual regression test has to be redone=> lots of work • Time pressure was high because of the 1/2 day needed after UAT was done

  20. 5. Push-button Release to Production • Same script as for UAT promotion: copy artifact, copy production configuration from svn, database setup

  21. Zero Outage Deployment • Deploy at any time • Production deployment needs to be transparent to the user • Round-Robin deployment in cluster • Session Sharing • Programmatically remove node form load balancer cluster

  22. Session Sharing • Different approaches possible • Applications server allow for automatic session replication • Sessions can be stored outside of application server, i.e. • Keep sessions on client • We chose to implement client-side session sharing

  23. Load Balancer Visibility • Programmatically take cluster nodes out of Load Balancer • Remove from cluster • Deploy new software version • Add server back to cluster • Creative solution: health check file in Apache HTTPD

  24. Advantages of Zero-Outage Deployment • Deploy any time vs. deployment window 7:00am Friday morning • Deployment becomes Business-As-Usual • If deployment fails we can redo it an hour later • Support for agile development • Controlled environment: no more manual steps • Deliver hot fixes within hours not days or weeks

  25. Problem: Different Software Versions in Production • Round-robin deployment means multiple software versions are in the same cluster • User lands on server with new front-end feature • User submits page • User lands on server with previous version • Error • Only happens during deployment • We chose to ignore this

  26. Reporting • Monitoring of # of exceptions in logs • Monitoring of down time of servers

  27. Disaster Recovery • Roll Back • Deploy latest working version • No analysis of problem necessary • Almost immediately back in business • Roll Forward • Fix the problem and release again • Analysis may take time • Not feasible for a 24*7 application

  28. Disaster Recovery: Example • Database table was missing index • UAT database had large tables truncated • Long running queries blocked each other • Whole cluster stopped working • Roll-back restored operations • Root-cause analysis

  29. Continuous Deployment • Takes Continuous Delivery one step further • Promote code to Production as soon as it is ready • Reduce batch size to one feature per release • Beyond two week sprints: deliver asap

  30. Disadvantages of Continuous Deployment • Business may request more deployments • Manual testing still the most important test gate for regression; the more you deploy the less thorough this step will be • Continuous Delivery is not for free: Manual testing still takes time • Regression testing time is often more than testing new features

  31. Not Covered • Automated Performance Testing • Spot-check by developers using JMeter • was done after major rewrites of the software by external party • Production database passwords in version control • Systems that are not easily automated

  32. Achievements • Rapid, reliable, repeatable, low risk deployments • Extension of existing agile process • Two week releases • Hotfixes any time • Happy developers • Happy business users

  33. Questions

More Related