slide1 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Hadoop Summit 2013- June 26th, 2013 PowerPoint Presentation
Download Presentation
Hadoop Summit 2013- June 26th, 2013

Loading in 2 Seconds...

play fullscreen
1 / 27

Hadoop Summit 2013- June 26th, 2013 - PowerPoint PPT Presentation


  • 200 Views
  • Uploaded on

Move to Hadoop, Go Faster and Save Millions - Mainframe Legacy Modernization Sunilkumar Kakade – Director IT Aashish Chandra – DVP, Legacy Modernization. Hadoop Summit 2013- June 26th, 2013. Legacy Rides The Elephant. Hadoop is disrupting the enterprise IT processing.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Hadoop Summit 2013- June 26th, 2013' - summer


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
slide1

Move to Hadoop, Go Faster and Save Millions - Mainframe Legacy Modernization

Sunilkumar Kakade – Director IT

Aashish Chandra – DVP, Legacy Modernization

Hadoop Summit 2013- June 26th, 2013

legacy rides the elephant
Legacy Rides The Elephant

Hadoop is disrupting the enterprise IT processing.

recognition contributors
Recognition - Contributors
  • Our Leaders
      • Ted Rudman
      • Aashish Chandra
  • Team
      • Simon Thomas
      • Sunil Kakade
      • Susan Hsu
      • Bob Pult
      • Kim Havens
      • Murali Nandula
      • Willa Tao
      • Arlene Pynadath
      • Nagamani Banda
      • Tushar Tanna
      • Kesavan Srinivasan
mainframe migration overview
Mainframe Migration - Overview
  • In spite of recent advances in computing, many core business processes are batch-oriented running on mainframes.
  • Annual Mainframe costs are counted in 6+ figure Dollars per year, potentially growing with capacity needs. In order to tackle the cost challenge, many organization have considered or attempted multi-year mainframe migration/re-hosting strategies.
batch processing characteristics
Batch Processing Characteristics
  • Characteristics*
  • Large amounts of input data are processed and stored (perhaps terabytes or more).
  • Large numbers of records are accessed, and a large volume of output is produced
  • Immediate response time is usually not a requirement, however, must complete within a “batch window”
  • Batch jobs are often designed to run concurrently with online transactions with minimal resource contention.

*Ref:. IBM Redbook

batch processing characteristics1
Batch Processing Characteristics
  • Key infrastructure requirements:
  • Sufficient data storage
  • Available processor capacity, or cycles
  • job scheduling
  • Programming utilities to process basic operations (Sort/Filter/Split/Copy/Unload etc.)
why hadoop and why now
Why Hadoop and Why Now?

THE ADVANTAGES:

  • Cost reduction
  • Alleviate performance bottlenecks 
  • ETL too expensive and complex
  • Mainframe and Data Warehouse processing  Hadoop

THE CHALLENGE:

  • Traditional enterprises lack of awareness

THE SOLUTION:

  • Leverage the growing support system for Hadoop
  • Make Hadoop the data hub in the Enterprise
  • Use Hadoop for processing batch and analytic jobs
the architecture
The Architecture
  • Enterprise solutions using Hadoop must be an eco-system
  • Large companies have a complex environment:
    • Transactional system
    • Services
    • EDW and Data marts
    • Reporting tools and needs
  • We needed to build an entire solution
batch processing migration with hadoop
Batch Processing Migration With Hadoop

Seamless migration of high MIPS processing jobs with no application alteration

mainframe to hadoop pig conversion example
Mainframe to Hadoop-PIG conversion example

Mainframe JCL

//PZHDC110 EXEC PGM=SORT

//SORTIN DD DSN=PZ.THDC100.PLMP.PRC,

// DISP=(OLD,DELETE,KEEP)

//SORTOUT DD DSN=PZ.THDC110.PLMP.PRC.SRT,LABEL=EXPDT=99000,

// DISP=(,CATLG,DELETE),

// UNIT=CART,

// VOL=(,RETAIN),

// RECFM=FB,LRECL=40

//SYSIN DD DSN=KMC.PZ.PARMLIB(PZHDC11A),

// DISP=SHR

//SYSOUT DD SYSOUT=V

//SYSUDUMP DD SYSOUT=D

//*__________________________________________________

//* SORT FIELDS=(1,9,CH,A)

- 500 Million Records sort took 45 minutes of clock time on A168 mainframe

PIG

a = LOAD 'data' AS f1:char;

b = ORDER a BY f1;

- 500 Million Records sort took less than 2 minutes

More benchmarking studies in progress

mainframe to hadoop pig conversion example1
Mainframe to Hadoop-PIG conversion example

Mainframe JCL

//PZHDC110 EXEC PGM=SORT

//SORTIN DD DSN=PZ.THDC100.PLMP.PRC,

// DISP=(OLD,DELETE,KEEP)

//SORTOUT DD DSN=PZ.THDC110.PLMP.PRC.SRT,LABEL=EXPDT=99000,

// DISP=(,CATLG,DELETE),

// UNIT=CART,

// VOL=(,RETAIN),

// RECFM=FB,LRECL=40

//SYSIN DD DSN=KMC.PZ.PARMLIB(PZHDC11A),

// DISP=SHR

//SYSOUT DD SYSOUT=V

//SYSUDUMP DD SYSOUT=D

//*__________________________________________________

//* SORT FIELDS=(1,9,CH,A)

- 500 Million Records sort took 45 minutes of clock time on A168 mainframe

PIG

a = LOAD 'data' AS f1:char;

b = ORDER a BY f1;

- 500 Million Records sort took less than 2 minutes

More benchmarking studies in progress

mainframe migration value proposition
Mainframe Migration – Value Proposition

Cost Savings

Open Source Platform

Simpler & Easier Code

Business Agility

Business & IT Transformation

Modernized Systems

IT Efficiencies

Optimize

Companies can SAVE 60% ~ 80% of their Mainframe Costs with Modernization

High TCO

Mainframe Optimization: -5% ~ 10% MIPS Reduction

-Quick Wins with Low hanging fruits

Mainframe

Migration

Mainframe ONLINE

-Tool based Conversion

-Convert COBOL & JCL to Java

Inert Business Practices

Convert

Typically 60% ~ 65% of MIPS are used in Mainframes by BATCH processing

PiG / Hadoop Rewrites

Resource

Crunch

Mainframe BATCH

-ETL Modernization

-Move Batch Processing to Hadoop

Estimated 45% of FUNCTIONALITY in mainframes is never used

mainframe migration traditional approach
Mainframe Migration – Traditional Approach
  • Traditional approaches to mainframe elimination call for large initial investments and carry significant risks – It is hard to match Mainframe performance and reliability.
  • Many organizations still utilize mainframe for batch processing applications. Several solutions presented to move expensive mainframe computing to other distributed proprietary platform, most of them rely on end-to-end migration of applications.
mainframe batch processing metascale architecture
Mainframe Batch Processing MetaScale Architecture
  • Using Hadoop, Sears/MetaScale developed an innovative alternative that enables batch processing migration to Hadoop Ecosystem, without the risks, time and costs of other methods.
  • The solution has been adopted in multiple businesses with excellent results and associated cost savings, as Mainframes are physically eliminated or downsized: Millions of dollars in savings based on MIP reductions have been seen.
metascale mainframe migration methodology
MetaScale Mainframe Migration Methodology
  • Key to our Approach:
  • allowing users to continue to use familiar consumption interfaces
  • providing inherent HA
  • enabling businesses to unlock previously unusable data

1

2

3

4

5

6

mainframe migration benefits
Mainframe Migration - Benefits

“MetaScale

is the market leader in moving mainframe batch processing to Hadoop”

summary
Summary
  • Hadoop can revolutionize Enterprise workload and make business agile
  • Can reduce strain on legacy platforms
  • Can reduce cost
  • Can bring new business opportunities
  • Must be an eco-system
  • Must be part of an data overall strategy
  • Not to be underestimated
the learning
The Learning

Over two years of Hadoop experience using Hadoop for Enterprise legacy workload.

  • We can dramatically reduce batch processing times for mainframe and EDW
  • We can retain and analyze data at a much more granular level, with longer history
  • Hadoop must be part of an overall solution and eco-system
  • We developed tools and skills – The learning curve is not to be underestimated
  • We developed experience in moving workload from expensive, proprietary mainframe and EDW platforms to Hadoop with spectacular results

HADOOP

  • We can reliably meet our production deliverable time-windows by using Hadoop
  • We can largely eliminate the use of traditional ETL tools
  • New Tools allow improved user experience on very large data sets

IMPLEMENTATION

UNIQUE

VALUE

the horizon what do we need next
The Horizon – What do we need next?
  • Automation tools and techniques that ease the Enterprise integration of Hadoop
  • Educate traditional Enterprise IT organizations about the possibilities and reasons to deploy Hadoop
  • Continue development of a reusable framework for legacy workload migration
legacy modernization service offerings
Legacy Modernization Service Offerings
  • Leveraging our patent pending and award-winning niche` products, we reduce Mainframe MIPS, Modernize ETL processing and transform business and IT organizations to open source, cloud based, Big Data and agile platform
  • MetaScale Legacy Modernization offers following services –
  • Legacy Modernization Assessment Services
  • Mainframe Migration Services
    • MIPS Reduction Services
    • Mainframe Application Migration
  • Legacy Distributed Modernization
    • ETL Modernization Services
    • Modernize Proprietary Systems and Databases
  • Managed Applications Support
  • Support Transition Services
legacy modernization made easy
Legacy Modernization Made Easy!

www.metascale.com

Follow us on Twitter @LegacyModernizationMadeEasy

Join us on LinkedIn: www.linkedin.com/company/metascale-llc

For more information, visit: