1 / 38

Data Representation, Data Integration and API Delivery of PDB Data

Data Representation, Data Integration and API Delivery of PDB Data. John Westbrook RCSB/PDB Rutgers University. Introduction. What are the underlying data requirements for: Data validation ? Data acquisition and exchange ? Robust interoperable APIs and databases ?.

melchior-magda
Download Presentation

Data Representation, Data Integration and API Delivery of PDB Data

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Data Representation, Data Integration andAPI Delivery of PDB Data John Westbrook RCSB/PDB Rutgers University

  2. Introduction What are the underlying data requirements for: • Data validation ? • Data acquisition and exchange ? • Robust interoperable APIs and databases ?

  3. It all starts with a data specification … • Characteristics of a successful data specification: • semantically precise and comprehensive • adequately models the domain • fully electronically accessible • well supported by software • easily integrated with related metadata specs

  4. What’s Driving Data Specification for Structure Data ? • IUCr sponsored community effort (1989 -> ) • Automated data acquisition • Data management and data exchange for PDB • New technologies (e.g. cryo-electron microscopy) • High-throughput structure determination and structural genomics (via International Task Forces) • Data deposited will be at the level of journal “materials and methods” section • Additional description of X-ray, NMR experiments and new items describing protein production (~3x increase in content scope)

  5. Current Data Dictionarieshttp://deposit.pdb.org/mmcif/ • PDB Exchange Dictionary Including extensions for structural genomics and automated data extraction • mmCIF • Ligand data • NMR • 3D-EM • Target Registration • Protein Production • Modeling • Crystallization • Symmetry • Image data • BIOSYNC > 3000 public definitions

  6. Broadening the Audience • Common archival XML representation for all PDB collaborators • Build on informatics structure of PDB Exchange Dictionary • Retain simple logical data organization of Exchange Dictionary • Preserve straightforward mapping to relational data model • Automated translation of PDB Exchange Dictionary in the form of XML schema and RDFS/OWL

  7. Mapped Dictionary Metadata(XML schema) • Data Attributes • Definition • Examples • Data type(primitive type/regular expression patterns) • Range or allowed values • Classes • Categories • Subcategories • Category groups • Associations • Parent-child relationships • Interdependencies/exclusivity • Methods Red - mapped Green - partially mapped Blue - not mapped

  8. Mapping Dictionary Semantics to XML • Data blocks mapped to an unordered sequence of category elements • Category elements include unique attributes with xpath expressions used in conjunction with key/keyref attributes to describe parent-child relationships • Internal category structure mapped to sequence of data item elements within a complexType • Data item features (e.g. data type, range, enumerations) mapped to restrictions within simpleTypes or unions or SimpleTypes • Definitions and examples mapped to annotation/documentation attributes

  9. Mapping Dictionary Semantics to OWL Web Ontology Language • Combines - • DAML - DARPA Agent Markup Language • OIL - Ontology Integration Language • RDFS - Serialized using Resource Description Framework Schemas • Concept oriented representation - GO, BioPAX • Designed to integrate domain metadata, acts as a global schema language • Supports decision logic and reasoning applications • Centralized vs. distributed integration http://deposit.rcsb.org/mmcif/

  10. Supporting Software Tools • Validating Parsers for Files and Dictionaries (CIFPARSE) • Dictionary access and presentation tools (CIFOBJ) • File format translation tools (MAXIT, CIFTr) • PDB Validation Suite • Data acquisition and editor tool (ADIT) • Database Builder and Loader (mmCIFLOADER) • XML translation tool for data files and dictionaries (mmCIF2XML) • Data extraction and merging tools (PDB_EXTRACT) • Others: BioPerl, BioPython, mmLib (py), CCP4

  11. Availabilityhttp://deposit.pdb.org/software • WWW and CDROM Distribution • Source and Binary Distributions • Open Source License • Supported on Linux, IRIX, ALPHA, SUNOS, and Mac OSX

  12. We have a Specification …What about the data? • Dictionary compliant files are provided in mmCIF and XML formats for all PDB entries. • These files reflect all of the RCSB data uniformity efforts • Legacy PDB-format files remain unchanged • Software tools are provided to produce PDB-format from mmCIF data files

  13. Data Uniformity • Sequence • Resolve anomalies relative to Swiss-Prot/Uniprot, GenBank • Resolve anomalies between sequence and atom • Atom nomenclature • Atom naming problems in 40% of structures • Redundant atom labels • Errors in chirality • Biologically active molecule described • Ligands • Names standardized • Bond types and connectivity verified • http://deposit.pdb.org/public-components-erf.cif • Ligand Depot - http://ligand-depot.rutgers.edu/ • Functional assembly ftp://beta.rcsb.org/pub/pdb/uniformity/data/mmCIF/ The Protein Data Bank: Unifying the Archive. Nucleic Acids Research 2002, 30:245-248

  14. It’s not your Grandmother’s PDB Archive… FAT not FLAT files • mmCIF and XML files in combination with dictionaries and external reference files contain all of the information required to build relational database… • PDB-format data files are also provided containing the coordinates of functional assemblies…

  15. Data Sharing Nightmare

  16. Data AcquisitionContent Coverage

  17. Data AcquisitionContent Coverage Summary • Average number of populated data items in all entries released in 2003 is 334 • Average number of populated data items in structural genomics entries is 337

  18. Data Extraction/Acquisition Flow DATA COLLECTION/REDUCTION pdb_extract_sf Process logs and output files Deposit STRUCTURE SOLUTION pdb_extract Process logs and output files STRUCTURE REFINEMENT Sequence and Source Details mmCIF Validation Interfaces: Command line Macro Language Web Interface CCP4 V5 Deposit

  19. Depositor Validation MAXIT Data ADIT AutoDep Input Tool Database Loader Reports Final Files Metadata Dictionaries Data Views RCSB System for Data Acquisition and Archiving

  20. Browser or Program Client Server File Parsers/Ftp/HTTP/CGI WSD/IDL Client Server HTTP/SOAP Database System XML/OWL Schema Database Server Client SQL, JDBC, OODB, Xpath. Racer. … API Types Web or Corba Service

  21. Goals for Corba API Delivery • Provide application and database access to macromolecular structure data • Follow standards-based approach (OMG MMS finalized 2001) • Build on informatics structure of PDB data ontology • Provides high performance access • Direct access to compact binary data structures (e.g. coordinates) • Provide broad granularity of access (individual atoms to biological assemblies)

  22. Program Level Access to the Details of Molecular Structure Ligand – Which ligands are contained within the entry? Chain/Entity – Extract the sequence and coordinates for each molecular entity. Secondary Structure – Extract helices and sheets for the entry. Residues/Atoms - What is the environment of this residue? Extract the coordinates for a selection of atoms or residues.

  23. API Architecture Features • API organization based on PDB Exchange Data Dictionary - access methods are provided at the level of data categories/classes • PDB Exchange Dictionary provides the content to automatically generate: • OMG Interface Definition Language (IDL) and access classes • SQL queries required to support Corba server • Software to load PDB data files in memory or into a supporting relational database engine

  24. Automatic Production of Macromolecular Structure API Components PDB Exchange Dictionary + API Specific Data Dictionaries Metamodel Framework CORBA IDL, SQL Schema, XML DTD/Schemas, Data Loaders Database Access Classes

  25. Macromolecular Structure API Data Flow mmCIF Parsers Applications XML Files mmCIF Data Files (Data Reference Standard) Relational Database API Servers

  26. Metadata Framework • PDB Exchange Dictionary • Defines content model • Grouping Dictionary • Maps dictionary content to API organization • Assigns attributes to API aggregate data types and indices • Schema Mapping Dictionary • Maps content to physical storage layer

  27. Current Server Availability • OpenMSS toolkit provides Java interface to Oracle/MySQL using JDBC • C++ server using native interface to DB2 implemented on 4-node Linux cluster at the Nucleic Acid Database (NDB) • Installation of DB2 at SDSC underway to support high-performance access (DataStar)

  28. Client Program Examples A primary requirement of the design was that it present an interface that was clearly defined and easy to use from the point of view of developing new applications. The code examples in this section illustrate how client programs can use the API to quickly access macromolecular structure data. As a simple example the following Python code fragment will print out the atom identifier and the Cartesian (x, y, z) position for atoms in the macromolecule 4hhb. Example 1. Retrieving the AtomSite list for hemoglobin (4HHB) and printing the atomic coordinates. try: sid = ”4HHB" e = ef.get_entry_from_id(sid); except: print "cannot get entry %s, exiting!" % sid sys.exit(1) print "got entry!" # Get the atom site list atoms = e.get_atom_site_list() print "got %d atoms total" % (len(atoms)) print "A few atoms:" for a in atoms[:10]: print "%s\t%.3f %.3f %.3f" % (a.id, a.cartn.x, a.cartn.y, a.cartn.z)

  29. Example 2. Listing symmetry information and the residues ranges for the helices of the hemoglobin (4HHB). # Get the symmetry information s = e.get_sym_info() print "space group: %s" % s.space_group print "cell constants: " c = s.acell.unit_cell print "a=%.3f, b=%.3f, c=%.3f" % \ (c.length_a, c.length_b, c.length_c) print "alpha=%.3f, beta=%.3f, gamma=%.3f" % \ (c.angle_alpha, c.angle_beta, c.angle_gamma) # Get the secondary structures sconfs = e.get_struct_conf_list() print "Secondary structures:" for a in sconfs: print a.id, '\t', \ a.beg_auth.asym.id, a.beg_auth.comp.id, a.beg_auth.seq.id, \ '\t-->', \ a.end_auth.asym.id, a.end_auth.comp.id, a.end_auth.seq.id

  30. Client Availability • Example clients provide category-level access in Java OpenMMS and C++ native servers • Clients available in Java, C++ and Python • C++ API extended to support efficient detailed molecular selections (e.g. coordinates of secondary structure elements, symmetry related molecular elements, biological assemblies)

  31. Web ServicesWeb Service Description Language (WSDL) • Message definitions - taken from XML schema • Operations - abstract definitions of messages that can be sent and received • Binding - concrete format of messages and transmission protocol (typically HTTP/SOAP) • Service - actual address of the web service • Supported by a registration and discovery protocol - Universal Description, Discovery and Integration (UDDI)

  32. Web Services- really easy - A Python Example from SOAPpy import SOAPProxy server = SOAPProxy("http://pdbbeta.rcsb.org/ \ jboss-net/services/rcsbWebService") print server.getSequenceForStructureAndChain \ ("1KIP", "A")

  33. Web Services - applications - • Widely used in e-commerce • Provide underlying infrastructure for the Grid • Infrastructure for Distributed Annotation System (DAS) used by e-Family, BioSapiens, OmniGene, TIGR, KEGG, and many others … • BioMoby, EBI, PDB… • Standardization and integration is informal http://pdbbeta.rcsb.org/jboss-net/services/rcsbWebService?wsdl http://www.ebi.ac.uk/msd-srv/docs/api/ http://www.ebi.ac.uk/Tools/webservices/ http://ncicb.nci.nih.gov/core/caBIO http://biomoby.org http://www.wsindex.org http://biodas.org

  34. C C C C C R R R R Insilico Workflow Description Computation

  35. Workflow Representation BPEL4WS - Business Process Execution Language for Web Services • Combines XLANG from Microsoft and WSFL from IBM • Defines complex workflows of web services • Supports contingencies, scheduling and error handling • Supported by tools to execute the define workflow http://www-106.ibm.com/developerworks/library/ws-bpel

  36. Summary • PDB Exchange Dictionary content provides the infrastructure for validation and exchange of structure and experimental data. • The data dictionary also provides the foundation for database local construction. Integration is provided by incorporation of external data. • OWL may provide a mechanism for global schema integration • Tools exist to readily move existing FAT files into database systems. • Web and Corba services provide integrative APIs but further work is required to achieve a level standardization. • Portable representation of complex workflows is still a research problem.

  37. Access to RCSB Resources • RCSB Protein Data Bank Site • http://www.pdb.org/ • OpenMMS site (Java implementation) • http://openmms.sdsc.edu • RCSB/PDB Software Download Site (C++ and Python implementation, NDB server) • http://deposit.pdb.org /mmcif/FILM/ • RCSB/PDB Dictionary Resource Site • http://deposit.pdb.org /mmcif/ • RCSB/PDB Beta Data Site • ftp://beta.rcsb.org/pub/pdb/uniformity/data/

  38. http://www.pdb.org/ Operated by three members of the RCSB: Rutgers, The State University of New Jersey; San Diego Supercomputer Center at the University of California, San Diego; Center for Advanced Research in Biotechnology/UMBI/NIST. The RCSB PDB is supported by funds from the National Science Foundation (NSF), the National Institute of General Medical Sciences (NIGMS), the Office of Science, Department of Energy (DOE), the National Library of Medicine (NLM), the National Cancer Institute (NCI), the National Center for Research Resources (NCRR), the National Institute of Biomedical Imaging and Bioengineering (NIBIB), and the National Institute of Neurological Disorders and Stroke (NINDS).

More Related