1 / 56

MOLGENIS For rapid prototyping of web applications biologists want to have

MOLGENIS .org For rapid prototyping of web applications biologists want to have. Morris Swertz Joeri van der Velde Joris Lops NBIC/BioAssist programmers meeting Utrecht, 18 December 2009. EBI. Biobanking platform. Outline. Introduction Motivation Conceptual strategy

iliana-neal
Download Presentation

MOLGENIS For rapid prototyping of web applications biologists want to have

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. MOLGENIS.orgFor rapid prototyping of web applications biologists want to have Morris Swertz Joeri van der Velde Joris Lops NBIC/BioAssist programmers meeting Utrecht, 18 December 2009 EBI Biobanking platform

  2. Outline • Introduction • Motivation • Conceptual strategy • MOLGENIS implementation • Expected results • Underlying architecture • Practical • Install a MOLGENIS virtual machine • Work through examples in manual • See what it can do for you

  3. Motivation

  4. Biological challenges ! Many materials ! HTP processes ! Complex, distr. workflows ! Large data ! Trace dependencies ! Adapt to new protocols 10 10.000 Main work flow Data dependency Biomaterial/result Lab/analysis process Scale of information Associated data files material 10.000 process strains genome 10,000 markers inbreed 100 1,000,000 10,000 individuals genotype genotypes map QTL profiles correlate 100,000 10,000,00 hybridize expressions preprocess norm exprs. network 100 100,000 microarrays probes

  5. For example Large datasets Dozens of samples Processing Complex relationships

  6. For example Swertz et al (2004) Bioinformatics 20, 2075-83

  7. For example Swertz et al (2004) Bioinformatics 20, 2075-83

  8. For example Swertz et al (2004) Bioinformatics 20, 2075-83

  9. Informatic challenges processing infrastructure user interaction infrastructure data infrastructure researchers bioinformatician Communication + integration infrastructure Swertz & Jansen (2007) Nature Reviews Genetics

  10. … try hitting a moving target € 10 100.000 strains genome SNP arrays inbreed 100 10,000,000 10,000 individuals genotype genotypes map QTL profiles correlate 1000 1000 LC/MS mass peaks preprocess aligned peaks network bioinformatician softw engineers biologist biologist suitable infrastructure biological challenges

  11. … repeatedly Biologist challenges Software solutions Work NextGenSeq Metabolomics support platform MassSpec/Metab. Experiments http://www.molgenis.org Swertz & Jansen (2007) Nature Revies Genetics 8, 235-243

  12. Strategy

  13. Standardization + Variation http://www.molgenis.org http://www.molgenis.org Swertz & Jansen (2007) Nature Reviews Genetics 8, 235-243

  14. Model variation, Reuse software Small model <!-- entity organization --> <entityname="Experiment"label="Experiment"> <fieldname="ExperimentID"key="1“ readonly="true" label="ExperimentID(autonum)"/> <fieldname="Medium" type="xref" xref_field="Medium.name"/>/> <fieldname="Protocol" label="Experiment Protocol"/> <fieldname="Temperature"type="int" Model of a variant <!-- entity organization --> <entityname="Experiment"label="Experiment"> <fieldname="ExperimentID"key="1“ readonly="true" label="ExperimentID(autonum)"/> <fieldname="Medium" type="xref" xref_field="Medium.name"/>/> <fieldname="Protocol" label="Experiment Protocol"/> <fieldname="Temperature"type="int" Code generation Step 2. Automate common Patterns (informatics) Step 1: Model variation Points (biology) + 10.000 10.000 strains genome markers inbreed 100 1,000,000 10,000 individuals genotype genotypes map QTL profiles correlate 100,000 10,000,00 hybridize expressions preprocess norm exprs. network 100 100,000 Step 3. Reuse in family of projects microarrays probes

  15. Add new biology to model Model new protocols Model of a variant <!-- entity organization --> <entityname="Experiment"label="Experiment"> <fieldname="ExperimentID"key="1“ readonly="true" label="ExperimentID(autonum)"/> <fieldname="Medium" type="xref" xref_field="Medium.name"/>/> <fieldname="Protocol" label="Experiment Protocol"/> <fieldname="Temperature"type="int" Your model <!-- entity organization --> <entityname="Experiment"label="Experiment"> <fieldname="ExperimentID"key="1“ readonly="true" label="ExperimentID(autonum)"/> <fieldname="Medium" type="xref" xref_field="Medium.name"/>/> <fieldname="Protocol" label="Experiment Protocol"/> <fieldname="Temperature"type="int" Reusable framework and generators + 10.000 Use new protocols 10.000 strains genome 15 markers inbreed 100 1,000,000 10,000 individuals genotype genotypes map QTL profiles correlate 100,000 10,000,00 hybridize expressions preprocess norm exprs. network 100 100,000 microarrays probes

  16. Add new features to software Add features once Model of a variant <!-- entity organization --> <entityname="Experiment"label="Experiment"> <fieldname="ExperimentID"key="1“ readonly="true" label="ExperimentID(autonum)"/> <fieldname="Medium" type="xref" xref_field="Medium.name"/>/> <fieldname="Protocol" label="Experiment Protocol"/> <fieldname="Temperature"type="int" Your model <!-- entity organization --> <entityname="Experiment"label="Experiment"> <fieldname="ExperimentID"key="1“ readonly="true" label="ExperimentID(autonum)"/> <fieldname="Medium" type="xref" xref_field="Medium.name"/>/> <fieldname="Protocol" label="Experiment Protocol"/> <fieldname="Temperature"type="int" Reusable framework and generators + 10.000 Added automatically 10.000 strains genome markers inbreed 100 1,000,000 10,000 16 individuals genotype genotypes map QTL profiles correlate 100,000 10,000,00 hybridize expressions preprocess norm exprs. network 100 100,000 microarrays probes

  17. … repeatedly Biologist challenges Software solutions Work NextGenSeq Metabolomics support platform MassSpec/Metab. Experiments http://www.molgenis.org Swertz & Jansen (2007) Nature Revies Genetics 8, 235-243

  18. MOLGENIS in practice

  19. Step 0: Brainstorm Observable feature * Observation target Observed value * time Panel Individual Observed Relation Inferred Value * *

  20. Step 1a: define the data model 1: molgenis_db.xml 506 lines of XML code 16 entities, 67 fields

  21. Step 1b: add ui model 1: molgenis_db.xml 50 lines of XML code (forms, menu’s and plugins)

  22. Language elements <?xml version="1.0" encoding="UTF-8"?> <!ELEMENT molgenis (description? , (module | entity |form | menu | plugin)*)> <!ATTLIST molgenis name CDATA #REQUIRED> <!ATTLIST molgenis label CDATA #IMPLIED> <!ATTLIST molgenis version CDATA #IMPLIED> <!ELEMENT description ANY> <!ELEMENT module (description?, entity+)> <!ATTLIST module name CDATA #REQUIRED> <!ELEMENT entity (description?,field*,unique*)> <!ATTLIST entity name CDATA #REQUIRED> <!ATTLIST entity abstract (true|false) #IMPLIED> <!ATTLIST entity implements CDATA #IMPLIED> <!ATTLIST entity extends CDATA #IMPLIED> <!ATTLIST entity decorator CDATA #IMPLIED> <!ATTLIST entity description CDATA #IMPLIED> <!ELEMENT field EMPTY> <!ATTLIST field name CDATA #REQUIRED> <!ATTLIST field type CDATA #IMPLIED> <!ATTLIST field label CDATA #IMPLIED> <!ATTLIST field length CDATA #IMPLIED> <!ATTLIST field xref_field CDATA #IMPLIED> <!ATTLIST field xref_label CDATA #IMPLIED> <!ATTLIST field enum_options CDATA #IMPLIED> <!ATTLIST field default CDATA #IMPLIED> <!ATTLIST field auto (true|false) #IMPLIED> <!ATTLIST field nillable (true|false) #IMPLIED> <!ATTLIST field unique (true|false) #IMPLIED> <!ATTLIST field readonly (true|false) #IMPLIED> <!ATTLIST field hidden (true|false) #IMPLIED> <!ATTLIST field description CDATA #IMPLIED> <!ELEMENT unique EMPTY> <!ATTLIST unique fields CDATA #REQUIRED> <!ATTLIST unique subclass (true|false) #IMPLIED> <!ATTLIST unique description CDATA #IMPLIED> <!ELEMENT form (form*,menu*,plugin*)> <!ATTLIST form name CDATA #REQUIRED> <!ATTLIST form entity CDATA #REQUIRED> <!ATTLIST form label CDATA #IMPLIED> <!ATTLIST form view (list|record) #IMPLIED> <!ATTLIST form readonly (yes|no) #IMPLIED> <!ELEMENT menu (form*,menu*,plugin*)> <!ATTLIST menu name CDATA #REQUIRED> <!ATTLIST menu label CDATA #IMPLIED> <!ELEMENT plugin (form*,menu*,plugin*)> <!ATTLIST plugin name CDATA #REQUIRED> <!ATTLIST plugin type CDATA #REQUIRED> <!ATTLIST plugin label CDATA #IMPLIED> See practical.

  23. 0 INFO [myFactory] working dir: D:\Development\Molgenis33Workspace\molgenis4phenotype 78 INFO [myFactory] MOLGENIS version 3.3.0-testing 94 INFO [myFactory] Using options: model_database = [pheno_db.xml] #File with data structure specification (in MOLGENIS DSL). model_userinterface = pheno_ui.xml #File with user interface specification (in MOLGENIS DSL). Can be same file as model_database output_src = generated/java #Output-directory for the generated project. output_hand = handwritten/java #Output-directory for the generated project. output_sql = generated/sql #Output-directory for the generated sql files. output_doc = WebContent/doc #Output-directory for the generated documentation. output_type = #Output type of the project, either war (for use in tomcat) or jar (standalone). output_web = WebContent #Output-directory for any generated web resources db_driver = com.mysql.jdbc.Driver #Driver of database. Any JDBC compatible driver should work. db_user = molgenis #Username for database. db_password = xxxxxx #Password for database. db_uri = jdbc:mysql://localhost/pheno #Uri of the database. Default: localhost db_filepath = attachedfiles #Path where the database should store file attachements. Default: null db_jndiname = jdbc/molgenisdb #Used to create a JDBC database resource for the application object_relational_mapping = subclass_per_table #Expert option: Choosing OR strategy. Either 'class_per_table', 'subclass_per_table', 'hierarchy_per_table'. Default: class_per_table mapper_implementation = multiquery #Expert option: Choosing wether multiquery is used instead of prepared statements. Default: false exclude_system = true #Expert option: Whether system tables should be excluded from generation. Default: true force_molgenis_package = false #Expert option. Whether the generated package should be 'molgenis' or the name specified in the model. Default: false auth_loginclass = org.molgenis.framework.security.SimpleSecurity #Expert option. verbose = true #This switch turns the verbose-mode on. compile = false #This switch makes the factory also compile (usefull outside IDE). mail_smtp_protocol = #Sets the email protocol, either smtp, smtps or null. Default: null meaning email disabled mail_smtp_hostname = localhost #SMTP host server. Default: localhost mail_smtp_port = 25 #SMTP host server port. Default: 25 mail_smtp_user = #SMTP user for authenticated emailing. Default: null. mail_smtp_password = #SMTP user for authenticated emailing. Default: null. 110 INFO [MolgenisLanguage] parsing db-schema from [pheno_db.xml] 780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable 780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable 780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable 780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable 780 WARN [Entity] [WARNING]: missing key 0 for entity Nameable 797 WARN [Entity] [WARNING]: missing key 0 for entity Nameable 797 WARN [Entity] [WARNING]: missing key 0 for entity Nameable 844 INFO [MolgenisLanguage] parsing ui-schema 937 INFO [main] generating .... 1717 INFO [TableDocGen] generated WebContent\doc\tabledoc.html 2076 INFO [EntityDocGen] generated WebContent\doc\objectmodel.html 2436 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram.dot 2545 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.system.dot 2748 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.observation.dot 2842 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.target.dot 2998 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.variable.dot 3138 INFO [DotDocGen] generated WebContent\doc\entity-uml-diagram-pheno.protocol.dot 3997 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-minimal-diagram.dot 4184 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.system.dot 4388 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.observation.dot 4606 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.target.dot 4731 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.variable.dot 4887 INFO [DotDocMinimalGen] generated WebContent\doc\entity-uml-diagram-minimal-pheno.protocol.dot 5184 INFO [ClassDocGen] generated WebContent\doc\classmodel.html 5293 INFO [InMemoryDatabaseGen] generated generated\java\ui\data\InMemoryDatabase.java 5609 INFO [MySqlCreateSubclassPerTableGen] generated generated\sql\create_tables.sql 5671 INFO [JDBCDatabaseGen] generated generated\java\ui\JDBCDatabase.java 5921 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Identifiable.java 5921 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Nameable.java 5968 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\OntologySource.java 6014 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\OntologyTerm.java 6030 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Investigation.java 6061 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservableFeature.java 6124 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservedValue.java 6170 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservedRelationship.java 6217 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\InferredValue.java 6233 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ObservationTarget.java 6280 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Individual.java 6311 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Panel.java 6326 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\CodeList.java 6327 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Code.java 6374 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol.java 6390 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ProtocolApplication.java 6405 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ProtocolParameter.java 6437 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\ParameterValue.java 6452 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\InferredValue_derivedFrom.java 6468 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Panel_individuals.java 6483 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol_observableFeatures.java 6499 INFO [DataTypeGen] generated generated\java\pheno\core\data\types\Protocol_protocolComponents.java 6624 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\OntologySourceMapper.java 6655 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\OntologyTermMapper.java 6671 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InvestigationMapper.java 6702 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservableFeatureMapper.java 6733 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservedValueMapper.java 6780 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservedRelationshipMapper.java 6827 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InferredValueMapper.java 6842 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ObservationTargetMapper.java 6873 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\IndividualMapper.java 6889 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\PanelMapper.java 6905 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\CodeListMapper.java 6936 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\CodeMapper.java 6951 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolMapper.java 6983 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolApplicationMapper.java 6998 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ProtocolParameterMapper.java 7029 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\ParameterValueMapper.java 7045 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\InferredValue_derivedFromMapper.java 7061 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Panel_individualsMapper.java 7076 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Protocol_observableFeaturesMapper.java 7092 INFO [MultiqueryMapperGen] generated generated\java\pheno\core\data\mappers\Protocol_protocolComponentsMapper.java 7217 INFO [JDBCMetaDatabaseGen] generated generated\java\ui\JDBCMetaDatabase.java 7263 INFO [CountPerEntityGen] generated generated\sql\count_per_entity.sql 7310 INFO [CountPerTableGen] generated generated\sql\count_per_table.sql 7341 INFO [FillMetadataTablesGen] generated generated\sql\insert_metadata.sql 7405 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\OntologySourceCsvReader.java 7420 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\OntologyTermCsvReader.java 7420 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InvestigationCsvReader.java 7436 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservableFeatureCsvReader.java 7452 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservedValueCsvReader.java 7467 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservedRelationshipCsvReader.java 7483 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InferredValueCsvReader.java 7498 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ObservationTargetCsvReader.java 7514 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\IndividualCsvReader.java 7514 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\PanelCsvReader.java 7530 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\CodeListCsvReader.java 7545 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\CodeCsvReader.java 7545 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolCsvReader.java 7561 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolApplicationCsvReader.java 7561 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ProtocolParameterCsvReader.java 7576 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\ParameterValueCsvReader.java 7576 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\InferredValue_derivedFromCsvReader.java 7592 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Panel_individualsCsvReader.java 7608 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Protocol_observableFeaturesCsvReader.java 7608 INFO [CsvReaderGen] generated generated\java\pheno\core\data\csv\Protocol_protocolComponentsCsvReader.java 7748 INFO [REntityGen] generated generated\java\pheno\core\R\OntologySource.R 7748 INFO [REntityGen] generated generated\java\pheno\core\R\OntologyTerm.R 7764 INFO [REntityGen] generated generated\java\pheno\core\R\Investigation.R 7779 INFO [REntityGen] generated generated\java\pheno\core\R\ObservableFeature.R 7779 INFO [REntityGen] generated generated\java\pheno\core\R\ObservedValue.R 7795 INFO [REntityGen] generated generated\java\pheno\core\R\ObservedRelationship.R 7795 INFO [REntityGen] generated generated\java\pheno\core\R\InferredValue.R 7810 INFO [REntityGen] generated generated\java\pheno\core\R\ObservationTarget.R 7810 INFO [REntityGen] generated generated\java\pheno\core\R\Individual.R 7826 INFO [REntityGen] generated generated\java\pheno\core\R\Panel.R 7826 INFO [REntityGen] generated generated\java\pheno\core\R\CodeList.R 7842 INFO [REntityGen] generated generated\java\pheno\core\R\Code.R 7857 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol.R 7857 INFO [REntityGen] generated generated\java\pheno\core\R\ProtocolApplication.R 7873 INFO [REntityGen] generated generated\java\pheno\core\R\ProtocolParameter.R 7873 INFO [REntityGen] generated generated\java\pheno\core\R\ParameterValue.R 7888 INFO [REntityGen] generated generated\java\pheno\core\R\InferredValue_derivedFrom.R 7888 INFO [REntityGen] generated generated\java\pheno\core\R\Panel_individuals.R 7888 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol_observableFeatures.R 7904 INFO [REntityGen] generated generated\java\pheno\core\R\Protocol_protocolComponents.R 7998 INFO [RApi] generated generated\java\source.R 8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\IdentifiableHtmlForm.java 8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\NameableHtmlForm.java 8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\OntologySourceHtmlForm.java 8044 INFO [HtmlFormGen] generated generated\java\pheno\core\html\OntologyTermHtmlForm.java 8060 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InvestigationHtmlForm.java 8060 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservableFeatureHtmlForm.java 8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservedValueHtmlForm.java 8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservedRelationshipHtmlForm.java 8076 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InferredValueHtmlForm.java 8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ObservationTargetHtmlForm.java 8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\IndividualHtmlForm.java 8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\PanelHtmlForm.java 8091 INFO [HtmlFormGen] generated generated\java\pheno\core\html\CodeListHtmlForm.java 8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\CodeHtmlForm.java 8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolHtmlForm.java 8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolApplicationHtmlForm.java 8107 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ProtocolParameterHtmlForm.java 8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\ParameterValueHtmlForm.java 8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\InferredValue_derivedFromHtmlForm.java 8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Panel_individualsHtmlForm.java 8122 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Protocol_observableFeaturesHtmlForm.java 8138 INFO [HtmlFormGen] generated generated\java\pheno\core\html\Protocol_protocolComponentsHtmlForm.java 8138 INFO [MolgenisServletContextGen] generated WebContent\META-INF\context.xml 8169 INFO [MolgenisContextListenerGen] generated generated\java\servlet\ContextListener.java 8232 INFO [MolgenisServletGen] generated generated\java\MolgenisServlet.java 8403 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\InvestigationsForm.java 8560 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ObservableFeaturesForm.java 8591 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\PanelsForm.java 8654 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\Panels\IndividualsForm.java 8701 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ObservedValuesForm.java 8732 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplicationsForm.java 8825 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\ParameterValuesForm.java 8857 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\ObservedValuesForm.java 8888 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenu\InferredValuesForm.java 9013 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\InferredValuesForm.java 9044 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservableFeaturesForm.java 9137 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargets\IndividualsForm.java 9169 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargets\PanelsForm.java 9200 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\ProtocolsForm.java 9293 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenu\ParametersForm.java 9325 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenu\ProtocolComponentsForm.java 9496 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologyTermsForm.java 9528 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologySourcesForm.java 9606 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\OntologySources\OntologyTermsForm.java 9638 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\CodeListsForm.java 9700 INFO [FormScreenGen] generated generated\java\ui\screen\TopMenu\Main\Ontologies\CodeLists\CodesForm.java 9965 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenuMenu.java 10012 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\MainMenu.java 10059 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenuMenu.java 10152 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Investigations\InvestigationMenu\ProtocolApplications\ProtocolApplicationMenuMenu.java 10230 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\ObservationTargetsMenu.java 10293 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\Protocols\ProtocolMenuMenu.java 10324 INFO [MenuScreenGen] generated generated\java\ui\screen\TopMenu\Main\OntologiesMenu.java 11354 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Main\ReportPlugin.java 11557 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Main\Ontologies\OntologyManagerPlugin.java 11604 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\Model_documentationPlugin.java 11604 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\RprojectApiPlugin.java 11620 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\HttpApiPlugin.java 11635 INFO [PluginScreenGen] generated Molgenis33Workspace\molgenis4phenotype\generated\java\ui\screen\TopMenu\WebServicesApiPlugin.java 11651 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\report\InvestigationOverview.ftl 11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\OntologyBrowser\OntologyBrowserPlugin.ftl 11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\DocumentationScreen.ftl 11807 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\RprojectApiScreen.ftl 11823 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\HttpAPiScreen.ftl 11823 WARN [PluginScreenFTLTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\SoapApiScreen.ftl 11854 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\report\InvestigationOverview.java 12057 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\OntologyBrowser\OntologyBrowserPlugin.java 12072 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\DocumentationScreen.java 12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\RprojectApiScreen.java 12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\HttpAPiScreen.java 12088 WARN [PluginScreenJavaTemplateGen] Skipped because exists: handwritten\java\plugin\topmenu\SoapApiScreen.java 12103 INFO [MolgenisServletContextGen] generated WebContent\META-INF\context.xml 12259 INFO [SoapApiGen] generated generated\java\ui\SoapApi.java 12353 INFO [CsvExportGen] generated generated\java\tools\CsvExport.java 12431 INFO [CsvImportByNameGen] generated generated\java\tools\CsvImportByName.java 12636 INFO [CopyMemoryToDatabaseGen] generated generated\java\ui\tools\CopyMemoryToDatabase.java Step 2: Generate (MOLGENIS + Eclipse) Generates 150 files, 30k lines of Java, SQL and R code + docs (tomcat/mysql; hsqldb, psql, jpa/hibernate, jetty in alpha )

  24. Step 2: Generate Model file XML  customize...  user interaction infrastructure MyScript Plugins Generate  FormGen TreeGen MenuGen PluginGen APIs in Java, R, Web services and HTTP MatrixGen JDBCMapGen Communication infrastructure JTypeGen JReadCsvGen JListGen RListGen JDatabaseGen RMatrixGen HSQLGen WSGen data infrastructure MySQLGen

  25. Step 3: Use and evaluate 1: molgenis_db.xml 2: generate 3: use Demo: http://wwwdev.ebi.ac.uk/microarray-srv/pheno/ Source: https://svn.gene.le.ac.uk/gen2phen/pheno-model

  26. Step 4: optionally create plugins

  27. Alternative: existing database Generate Connnects to MolgenisExtractModel (automatic) Existing db Xml model

  28. Expected results

  29. Result: Web GUI 1: molgenis_db.xml 2: brainstorm doc* 3: exchange format* 4: back and frontend* *autogenerated Demo: http://wwwdev.ebi.ac.uk/microarray-srv/pheno/ Source: https://svn.gene.le.ac.uk/gen2phen/pheno-model

  30. Result: Documentation (generated-docs/objectmodel.html) 1: molgenis_db.xml 2: documentation* *autogenerated

  31. Result: Exchange format my GaP my Data my Data Investigation, Protocol Material, Data annotations Raw and processed data researcher researcher Between labs Central access

  32. Result: Exchange format 1: molgenis_db.xml 2: documentation* 3: exchange format* *autogenerated • Tools • Db • Import • Export

  33. Result: Exchange format 1: molgenis_db.xml 2: documentation* 3: exchange format* *autogenerated

  34. REST interface (/api/find and /api/add) • http://localhost:8080/molgenis_distro/api/find/example.Data?name=mydata

  35. Result: R interfaces(/api/R) source(“http://localhost:8080/xgap/api/R”) MOLGENIS is connected #download data traits <- get.metabolitedata(name=“mytraits”) 25 metabolite downloaded in 30ms genotypes <- get.markerdata(name=“mygenotypes") 744 marker downloaded in 30ms #calculate ... #upload results for others to use add.data(qtls, name=“myqtls”) 18.600 data items added in 2sec Swertz et al, CASIMIR consortium, GEN2PHEN consortium (submitted) / 36

  36. Result: SOAP interface(/api/soap?wsdl) Smedley, Swertz, Wolstencroft et al (2008) Brief. in Bioinf.

  37. Edit & trace your data UML documentation of your model Connect to R statistics Workflow ready web-services find.investigation() 102 downloaded obs<-find.observedvalue( 43,920 downloaded #some calculation add.inferredvalue(res) 36 added Import/export to Excel plugin your own scripts (OntBrowse) Tech keywords: object oriented data models, multi-platform java, tomcat/glassfish web server, mysql/postgresql database, Eclipse/Netbeans IDE, Java API, WSDL/SOAP API, R-project API, MVC, freemarker templates and css for custom layout, open source.

  38. Under the hood

  39. Architecture Model file XML  customize...  user interaction infrastructure MyScript Plugins Generate  FormGen TreeGen MenuGen PluginGen APIs in Java, R, Web services and HTTP MatrixGen JDBCMapGen Communication infrastructure JTypeGen JReadCsvGen JListGen RListGen JDatabaseGen RMatrixGen HSQLGen WSGen data infrastructure MySQLGen

  40. Generators? <#listentitiesasentity> CREATE TABLE ${SqlName(entity)} ( <#listdbFields(entity)asf> <#iff_index!=0>, </#if> ${SqlName(f)}${mysql_type(model,f)} <#if!f.nillable> NOT </#if>NULL <#iff.getDefaultValue()?exists>${f.getDefaultValue()}</#if> <#iff.auto&&f.type=="int"> AUTO_INCREMENT</#if> <#iff.type=="xref"> , FOREIGN KEY (${SqlName(f)}) REFERENCES ${SqlName(model.getEntity(f.XRefEntity).getRootAncestor() )} (${SqlName(f.XRefField)}) ON DELETE RESTRICT </#if> </#list> <#listentity.getAllKeys()askey> <#ifkey_index==0> , PRIMARY KEY(${csv(key.fields)}) <#else> , UNIQUE(${csv(key.fields)}) </#if> </#list> <#listentity.getAllIndices()asi> , INDEX ${SqlName(i)} (${csv(i.fields)}) </#list> ) ENGINE=InnoDB;

  41. Generator results CREATETABLE Investigation ( ` id INTEGERNOTNULL AUTO_INCREMENT , name VARCHAR(255) NOTNULL , description TEXT NULL , PRIMARYKEY(id) , UNIQUE(name) , UNIQUE(id) ) ENGINE=InnoDB;

  42. Database API: • db.Database • count(filter) • find(filter) • add(entity) • add(list) • update(entity) • update(list) • remove(entity) • remove(list) • beginTx(),commitTx() • add(CsvReader) • find(CsvWriter) • db.Query • equals(field,value), • greaterThan(f,v) • lessThan(f,v) • limit(f,v) • offset(f,v) • or() • etc. • util.Entity • find(db,filter) • get(db,id) • query(db)

  43. //find • Database db = new app.JDBCDatabase("molgenis.properties"); • for(Contact c: db.find(Contact.class)) System.out.println(c); • //query • Query<Address> q = db.query(Address.class); • q.like(“lastname",“Lops"); • for(Address t: q.find()) System.out.println(t); • //update • Contact one = new Contact(); • one.setFirstName(“Joeri"); • db.add(one); • //update in batch • List<Contact> many = new ArrayList<Contact>(); • for(int i = 0; i < 10; i++) • { • Contact c = new Contact(); • c.setLastName(“friend_"+i); • many.add(c); • } • db.add(many);

  44. MVC user interface framework • Tree • MVC triple main response Menu request(target,action) FormView .ftl (layout) Form FormController.java (logic) menu handleRequest() reload() layout FormModel.java (state) form form Database API

  45. Plugin: your own MV(C) publicclassFileServingPlugin extends PluginModel(AndController) { public String getViewTemplate() { return"plugin/examples/file/FileServingPlugin.ftl"; } public String getViewName() { return"plugin_examples_file_FileServingPlugin"; } publicvoid handleRequest(Database db, Tuple request) { //if( request.getAction("do_add") ) } publicvoid reload(Database db) { //Query q = db.query(Experiment.class); //q.like("name", "test"); //List<Experiment> recentExperiments = q.find(); } } File with view Name of view (next slide) What do on request? What do on page refresh/after request?

  46. Plugin: view in freemarker <#macroplugin_examples_file_FileServingPluginscreen> <formmethod="post"enctype="multipart/form-data"name="${screen.name}"> <inputtype="hidden"name="__target"value="${screen.name}"" /> <inputtype="hidden"name="__action"/> <!-- this shows a title and border --> <divclass="formscreen"> <divclass="form_header"id="${screen.getName()}"> ${screen.label} </div> <#--optional: mechanism to show messages--> <#listscreen.getMessages()asmessage> <#ifmessage.success> <pclass="successmessage">${message.text}</p> <#else> <pclass="errormessage">${message.text}</p> </#if> </#list> <divclass="screenbody"><divclass="screenpadding"> <#--begin your plugin--> Create your plugin layout code here. <#--end of your plugin--> </div></div> </div> </form> </#macro>

  47. Future • ‘Cooler’ UI framework? • GWT, ServerFaces, YUI, Flex? • Semantic integration? • /api/rdf, semantically enhanced model • Large files • Hadoop, BinaryMatrix, … • Computation/services/workflows • Galaxy integration, GridGain, … • Documentation! • Your wish list here

  48. Practical

  49. Practical • Copy and install VirtualBox (dvd/usb) • Copy and run molgenis-vm.vdi image • Pick up a manual • Start generating 

More Related