Internship report by Ž ygimantas Gatelis Supervisor: Giovanni Franzoni

1. Development of aService to Manage Requests of Data and Monte Carlo Samples ofDevelopment and VALIDATIONat CMS Internship report by ŽygimantasGatelis Supervisor: Giovanni Franzoni Home institution: Vilnius University

2. Project Project name - RelVal Machine • RelVal - request samples dedicated to testing and validating either novel event reconstruction updates or updates to the calibration constantsand alignment parameters • RelVal Machine - web application to manage requests of data and Monte Carlo samples for development and validation inCMSSW.

3. Problematic How users perform tests now? • Now users use runTheMatrix.py tool for tests, validation and submission for production. • runTheMatrix.py - python tool that stores the configuration ofcommon samples production requests in .py files, and simplifies running of particular workflow. • cmsRun - is a command that performs both simulation of CMS detector and reconstruction, afterthatreturns results in .root file format. • cmsDriver.py is tool that generates complex configuration for cmsRun. It works like interface to csmRun • when running runTheMatrix.py it creates cmsDriver.py command that generates configuration for cmsrun

4. Problematic How users perform tests now?

5. Problematic runTheMatrix.py drawbacks • Only limited amount of requests are available • If user want to submit custom request he/she has to insert new request in runTheMatrix.py code by hand • If custom request is not pushed to git then after next code checkout custom request will be lost • Has only command line interface

6. Purpose of RelVal Machine • RelVal Machine stores requests in a database All users can add new requests or edit existing. User will not loose his requests because they are stored in database, he can reuse them later or clone with some changes. • RelVal Machine solves requests bookkeeping problem. Users can bookkeep his requests, clone requests made bysomeone else and customize them • Using RelVal Machine user can perform tests using existing infrastructure It means that RelVal Machine integrates with runTheMatrix.py tool in order to perform tests. Using existing infrastracture is a big plus. We know that it works and generates correct configuration for cmsrun. • Automate submission of relval samples for release validation There are about 100 bi-weekly relval samples that should be submitted in order to perform release validation

7. Technologies RelVal Machine architecture

8. Technologies For database we choose OracleDb technology:

9. Technologies Flask - python web framework was chosen as main server side framework We chose between Flask and CherryPy: • In general both frameworks are similar • Both are lightweight python server side frameworks • Only plus for cherryPy is that it has been used in McM • Flask has bigger community and better documentation • Flask has many plugins that helps accomplish general tasks easier • Flask is faster

10. Technologies Client side (frontend) technologies • AngularJs (v1.2.12) - javascript framework for creating dynamic web application also used in McM • Bootstrap v3.1.0 - CSS framework, to make application look attractive. • And few other javascript frameworks to make life easier

11. What I have done Created relational database schema, its implementation and programmed all required operations with database. Simplified database schema: Full database schema: https://twiki.cern.ch/twiki/pub/CMS/PdmVRelValMachine/relval_04-22.png

12. What I have done

13. What I have done Created web pages fordata representation and search/sort capability

14. What I have done

15. What I have done Created web pages for inserting/editing/cloning existing data

16. What I have done

17. What I have done RelVal Machine deployed into development machine under CERN SSO (Single Sign On). • Only users that have CERN account can access application. • Application accessible beyond CERN network. • Application link: • https://cms-pdmv-dev.cern.ch/relvalmachine/

18. What I have done Integration  to existing CMS infrastructure • During integration development changes in existing runTheMatrix.py tool was made and pushed into newest CMSSW release git repository. • Programmed integration to runTheMatrix.py from RelVal Machine side . • For communication between RelVal Machine and runTheMatrix.py I use JSON file, where request information is stored .

19. What I have done Users requests that takes relative long time is asynchronous. User can have fast response, that task is submitted and then service on different thread runs runTheMatrix.py  that takes from 2 to 20 minutes command using external machine.

20. What I have done Documentation for RelVal Machine • Created deployment descriptor. Detail instructions how to deploy application in plain virtual machine. • Created documentation of RelVal Machine application for future developers that will improve application. • Code stored in github repository under PdmV account and accessible for all.

21. Future improvements Thing that should be improved by future developers • Implement requests submission for production • Search by any field (for now user can search only by title) • Batches submission for localtests and production • Make better client side validation • Improve steps creation page, allow user to paste whole command not only by flag/value pairs. Now application has convenient infrastructure so it should be easy for future developers change and refactor existing code.

22. Questions