using lua for stored procedures in a high availability database l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Using Lua for Stored Procedures in a High Availability Database PowerPoint Presentation
Download Presentation
Using Lua for Stored Procedures in a High Availability Database

Loading in 2 Seconds...

play fullscreen
1 / 20

Using Lua for Stored Procedures in a High Availability Database - PowerPoint PPT Presentation


  • 129 Views
  • Uploaded on

Using Lua for Stored Procedures in a High Availability Database. VeriSign I2S Team Name of Presenter: John Rodriguez Date: July 14, 2008. Agenda. 1 VeriSign High Availability Database Background (VHAD) – Simplicity 2 Why Choose Lua for Stored Procedures?

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

Using Lua for Stored Procedures in a High Availability Database


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
using lua for stored procedures in a high availability database

Using Lua for Stored Procedures in a High Availability Database

VeriSign I2S Team

Name of Presenter: John Rodriguez

Date: July 14, 2008

agenda
Agenda

1 VeriSign High Availability Database Background (VHAD) – Simplicity

2 Why Choose Lua for Stored Procedures?

3 VHAD Integrated Development Environment – IDE

introduction
Introduction
  • VeriSign’s mission is making the Internet and telecommunications networks more reliable, more intelligent, and more secure.
  • VeriSign runs the registry (database) for the .com, .net, .cc, and .tv TLDs.
  • The motivation for VHAD was driven by historic and projected transactional growth and system availability requirements
  • VHAD is a very high availability (99.9995%), very fast, replicated, memory-based database, optimized for Online Transaction Processing (OLTP).
  • Are you crazy? Why write your own database?
    • Big, replicated commercial database systems just have too many moving parts.
    • Cost rises exponentially when we push well beyond normal availability levels.
  • I still think you’re crazy. What makes you think you can actually do this?
    • Carefully, aggressively simplify the design and architecture.
    • Target only the critical subset of system functionality.
mission
Mission
  • Continuous Availability
    • Availability is achieved by minimal administration, narrowly focused scope, simple and clear design, in-memory architecture
    • Approach is NOT added redundancy over a lower-availability database.
  • High Performance
    • Designed for 1+ billion update transactions per day (~ 50K/second).
    • Designed for an unlimited volume of read-only transactions.
  • Online Transaction Processing
    • Assumes Insertion/Update of approximately 10 rows per transaction.
    • Not designed for general purpose set processing – no huge transactions.
  • Storage Engine is fully ACID compliant
    • Atomicity – Transactional commit/abort – all or nothing.
    • Consistency – Foreign key / uniqueness relationships are enforced.
    • Isolation – A transaction never sees someone else’s partial change.
    • Durability – Acknowledgement of a commit guarantees data persistence.
running continuously
Running Continuously
  • What kinds of things don’t stop but just keep going, going, going…?
    • The moon keeps orbiting.
    • Simple tools just work.
    • What makes them differentfrom things that break, ormust be stopped occasionally?
  • Continuous Availability:
    • No “maintenance windows”, no fine print.
    • No excuses, it doesn’t matter why.
    • Not an affirmation that we will fix it,but a guarantee that we won’t need to.
  • How targeting really high availability effects design choices.
continuous availability
Continuous Availability
  • Commercial Availability - routine outages
    • 90% 2 hours down/day - nightly batch jobs
    • 99% 1 hour down/week - weekly backups
    • 99.9% 1 hour down/month - monthly reorganization
  • High Availability - occasional outages
    • 99.95% 1 hour down/quarter - quarterly upgrade
    • 99.99% 1 hour down/year - reboot after failure,
  • Continuous Availability - no scheduled outages
    • 99.999% 5 minutes down/year - automated switch over99.9995% 2 minutes down/year - Project Goal99.9999% 30 seconds down/year - Wow
  • Perfect Availability - no visible failures
    • 99.99999% failure is never visible to work in progress
topology

Prime

Guard

Replica

Replica

Mirror

Mirror

Mirror

Replica

Replica

Topology
  • Prime performs all changes.
    • If the Prime fails, a Mirror is automatically selected to be promoted to Prime .
  • Mirrors confirm persistence.
    • Provide persistence in memory.
    • Must include a separate site.
  • Guard only writes log to disk.
    • Protects against poison data.
    • Not subject to errors in the database logic.
  • Replicas support massiveread-only access.
    • Replicas don’t vote, which simplifies recovery analysis.
    • Replicas can be promoted to Mirror when needed.

Prime processes changesand manages data consistency

Guard defends against poison data

Mirrorsconfirmpersistence

Replicassupport unlimitedread access

As many levels as desired

agenda8
Agenda

1 VeriSign High Availability Database Background (VHAD) – Simplicity

2 Why Choose Lua for Stored Procedures?

3 VHAD Integrated Development Environment – IDE

rationale for choosing lua for stored procedures
Rationale for Choosing Lua for Stored Procedures
  • Lua shares major VHAD design principle – keep it small and simple.
  • Lua is mature and very stable – critical core code is not expanding.
  • Most databases realistically require a service outage when a stored procedure is added or modified, and that change isn’t transactional.
  • VHAD requires more dynamic behavior there, because anything that compromises even a single transaction is considered an outage.
  • Lua gives us the dynamic behavior we need – different interpreters can have different code in use simultaneously for a procedure name.
  • Highly parallel operation is needed: thousands of threads at once – Lua interpreter is fully thread safe and reentrant, unlike most others.
  • Lua Garbage Collection via interpreter heap – it’s a huge advantage not to have to explicitly deallocate or use reference counts.
  • Efficient direct access to C++ objects gives a big speed advantage.
cursor example the schema in sql
Cursor Example – The Schema in SQL
  • Assume there is a database table like the following:
    • CREATE TABLE DOMAIN_T (DOMAINNAME VARCHAR(256), REGISTRARVARCHAR(256), DATE TIMESTAMP);
cursor example userdata and metatable
Cursor Example – Userdata and Metatable
  • One of the basic types that Lua supports is userdata.
  • Lua provides APIs to create userdata types and associate a metatable of metamethods to a userdata type:
    • Static const struct luaL_reg cursor_metatable[] = {

{“__newindex”, setColumn},

{“__index”, getColumn},

{“__gc”, cleanup},

{NULL, NULL}

}

    • luaL_newmetatable(lstate, “Session.Cursor”)
    • luaL_register(lstate, NULL, cursor_metatable)
  • Associating metamethods to userdata:
    • luaL_getmetatable(lstate, “Session.Cursor”)
    • Lua_setmetatable(lstate, -2)
cursor example lua stored procedure
Cursor Example - Lua Stored Procedure
  • A sample metamethod in C:

int setColumn (lua_state *lstate) {

luaL_checkudata(lstate, 1, “Session.Cursor”)

colName = lua_tostring(lstate, 2)

colData = lua_tonumber(lstate, 3)

}

  • A sample stored procedure that uses the Cursor type:

function INSERT_DOMAIN (args)

domain = Cursor.new (“REGY”, “DOMAIN”)

domain.DOMAINNAME = args.DOMAINNAME

domain.RESISTRAR = args.REGISTRAR

domain.DATE = args.DATE

Cursor.insert(domain)

commit()

end

agenda13
Agenda

VeriSign High Availability Database (VHAD) Background – Simplicity

Why Choose Lua for Stored Procedures?

VHAD Integrated Development Environment – IDE

vhad integrated development environment ide
VHAD Integrated Development Environment - IDE
  • Need an environment that lets a user:
    • Write a stored procedure
    • Load a stored procedure dynamically
    • Debug a stored procedure
    • List schemas, tables, column values, etc.
    • Rapid prototyping of stored procedures
  • Developing a database tool like sqlplus – but better
    • Display schemas, stored procedures, etc.
    • Display columnar data
    • Execute a stored procedure
    • Sqlplus does not have a stored procedure debugger!
slide15
ldb
  • Developed a command line Lua debugger - ldb
  • The ldb language
    • list firstline [lastline]
    • break line
    • clear line
    • display [varname]
    • step
    • continue
    • print stack|trace|breakpoints
    • quit
    • help
ldb example
ldb Example

ldb> break 11

setting breakpoint at ./sort.lua line 11

11 local i=l+1

ldb> continue

original

Jan,Feb,Mar,Apr,May,Jun,Jul,Aug,Sep,Oct,Nov,Dec

stopping at breakpoint in ./sort.lua line 11

11 local i=l+1

ldb> display m

Find local name m:

name m, type number, value 1.000000

Find global m:

Did not find global m

ldb example continued
ldb Example - continued

ldb> print stack

7: 2

6:`Jan'

5: 1

4: function

3: 12

2: 1

1: table

ldb> print trace

level# name,type,language,source,currentline,linedefinedat

0# qsort,global,Lua,./sort.lua,12,5

1# testsorts,global,Lua,./sort.lua,55,51

2# ,,main,./sort.lua,66,0

conclusion difficulties faced
Conclusion – Difficulties Faced
  • Execution of stored procedures from memory.
  • Thread pools of Lua_states are reused.
  • Decimal adaptation and other numeric types.
  • Debugging stored procedures.
thank you

Thank You

"Make everything as simple as possible, but not simpler." Albert Einstein