quattor core modules status update
Download
Skip this Video
Download Presentation
Quattor: core modules status update

Loading in 2 Seconds...

play fullscreen
1 / 13

Quattor: core modules status update - PowerPoint PPT Presentation


  • 124 Views
  • Uploaded on

Quattor: core modules status update. 3 rd Quattor Workshop Trinity College Dublin 03/2007 Presented by Marco Emilio Poleggi – CERN-IT/INFN-CNAF. Major CERN developments…. ( Not including CERN-specific developments!) Support for Namespaces (done) Default template set

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' Quattor: core modules status update' - daryl-wilcox


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
quattor core modules status update

Quattor: core modules status update

3rd Quattor Workshop

Trinity College Dublin

03/2007

Presented by Marco Emilio Poleggi – CERN-IT/INFN-CNAF

major cern developments
Major CERN developments…

(Not including CERN-specific developments!)

  • Support for Namespaces (done)
    • Default template set
    • NCM components’ templates -> ‘tplconvert’
    • Need to discuss obsoletion of old template set
  • CDB: SOAP-layer improvements (ongoing)
    • Latest SOAP-Lite under testing
      • Still problems with badly escaped sequences (base64 works around)
      • Huge memory consumption
  • CDB backend (ongoing)
    • Current SQL-based back-end is slow, and aysnchronous
    • Rewrote CDB2SQL in multithreaded Python with a fast XML parsing library – but not deployed yet!
  • SWRepSOAP (with BARC India)
    • Several enhancements done

done

WIP

WIP

WIP

done

major cern developments1
…Major CERN developments
  • CCM extensions (started)
    • CCM to accept non-local profiles
    • Essentially, what AII does, but fully CCM-integrated
  • Release build framework (ongoing)
    • 99%-automated procedure (see “Quattor Release Process” slides)
    • ETICS integration: still missing
  • Release 1.3 under preparation
    • Namespaces + cumulated fixes + core modules and components enhancements
  • NOTD
    • First version in CVS, not functional yet
  • CERN manpower still limited 
    • ~1.4 FTE (counting 1 FTE contributed by CNAF, and 0.2 FTE contributed by BARC India)

WIP

WIP

WIP

WIP

No change

developers and users wish list
Developers’ and users’ wish list
  • CDB
    • Restrict template naming per namespace
  • CDB2SQL
    • Direct back-end interfacing, like native XML DB and Oracle
      • Avoid re-parsing XML profile just generated by PAN
  • PAN Compiler
    • Easier programmatic IF integration by adding a new ‘data’ type template
      • Template type where only key-value entries can be defined – easy parsing/writing by applications
  • NCM (cf. separate discussion)
    • Automated web-based documentation for components
    • Generic ‘provides’ for NCM components (#17681)
      • Request for “meta-components”
  • We probably need a new indirection level for device identification
    • Device identifiers (e.g. ‘eth0’, ‘sdc’) depend on the kernel used (2.4 vs. 2.6)

Pending

No progress

still requested

WIP

Pending

Pending

developments at cern ccm
Developments at CERN: CCM
  • UTF8 support (thanks to Cal)
  • getRecHash() function for converting config tree to Perl data structure
  • Correct handling of --force option
  • No more crashes on stale locks
  • Better handling of HTTP 304 “no change”
developments at cern ncm
Developments at CERN: NCM
  • cdispd
    • PID now recorded in a file (thanks to Uwe)
    • Log-rotation (thanks to Uwe)
    • Avoid instantiating twice the same configuration with same PID
    • All instabilities should have been fixed now… any complaints?
  • ncd
    • Log ccm-fetch information into log file when running ncm_wrapper
    • Avoid locking when called with –-list
    • Backup now optional in NCM::Check::lines
  • template
    • Added tags <LFOREACH>...<ENDLFOR>, <PACKAGELIST> and modified <PACKAGEPATH> (thanks to Ronald)
developments at cern ncm components
Developments at CERN:NCM components
  • Namespace support
    • Template conversion through ncmtplconvert (shipped with quattor-sw-util):

$ ncmtplconvert –-line-model “match-this-line”\

> --map-rule=“this=that” –-dont-touch=“foo_must_stay”\

> foo_bar_quux.tpl

    • Build framework integration via make tplconvert
    • New tree structure:

components/

|-- foo

| |-- config.tpl

| |-- functions.tpl

| `-- schema.tpl

`-- quux

|-- config.tpl

`-- schema.tpl

  • When to get rid of old templates
    • Next quattor 1.3 update? Quattor 1.4?
  • Many new components! Thanks to contributors
    • Review needed
developments at cern pan templates
Developments at CERN:pan-templates
  • New namespace structure

|-- clusters

| `-- mycluster

| |-- config.tpl

| `-- software.tpl

|-- components

| `-- aii

| |-- config.tpl

| `-- schema.tpl

|-- hardware

| |-- card

| | `-- nic

| | `-- intel_e100.tpl

| |-- cpu

| | `-- GenuineIntel_Pentium_4_2600.tpl

| |-- disk

| | `-- STD_80.tpl

| |-- machines

| | `-- asus_terminator_p4_533a.tpl

| `-- ram

| `-- 1024.tpl

|-- os

| `-- i386_sl3

| |-- aii.tpl

| `-- rpms

| |-- base.tpl

| |-- defaults.tpl

| |-- quattor.tpl

| `-- user.tpl

|-- pan

| |-- functions

| | |-- filesystem.tpl

| | |-- general.tpl

| | |-- network.tpl

| | `-- validation.tpl

| |-- profile_base.tpl

| |-- structures

| |-- structures.tpl

| |-- types.tpl

| `-- units.tpl

|-- profiles

| `-- mynode.tpl

|-- quattor

| `-- repository_cleanup.tpl

|-- repository

| |-- myrep_i386_sl3.tpl

| `-- myrep_x86_64_sl4.tpl

`-- site

|-- aii.tpl

|-- config.tpl

`-- repositories.tpl

  • When to get rid of old templates
    • Next quattor 1.3 update? Quattor 1.4?
developments at cern spma swrep
Developments at CERN: SPMA, SWRep
  • SPMA/rpmt
    • Support for checking RPM signatures
      • But not working with current rpmt-py yet..
  • SWRepSOAP
    • Can now upload the same package to multiple platforms
    • Added check for empty packages
    • Added authorship info in find and query commands
    • New chtag command to move packages across “areas”
    • Support for source RPMs (SRPMS)
    • Support for last-updated timestamp
    • Support for operation authorship
    • Package integrity checking via MD5
    • Support for authentication with Kerberos 5
developments at cern tools
Developments at CERN: tools
  • wassh
    • Parallel command execution engine on top of ssh
    • Now distributed with quattor, ready for next release
      • Support for multiple clusters and subclusters
    • Useful as a manual notification tool… don’t abuse of it ;-)

wassh –l root --cluster lxplus ‘ncm-ncd –-configure foo’

    • Plug-in mechanism for resolving attributes such as cluster name, architecture, location, etc into lists of hosts
      • Wassh does work without this backend, but requires list of hots
      • Plug-in for CERN instance of CDBSQL available
      • Volonteers for writing a XMLDB back-end?
developments at cern tools1
Developments at CERN: tools
  • pangraph
    • Graph representation of Pan templates (virtually, any kind of inclusion-based hierarchy)

$ /usr/lib/perl/quattor-sw-util/pangraph\

--base-dir <cvs-checkout>/pan-templates/namespaces\

--format png --tpl-file-ext .tpl.cin profiles/mynode.tpl.cin\

> graph.png

    • First step towards a Web-based template navigator?
      • Integration with Lemon’s XML parser, which will be donated to quattor
    • Try it 
developments at cern others
Developments at CERN: others
  • Work in progress on new partitioning scheme
    • See Luis’ presentation
  • CDB update
    • See Marco Emilio’s presentation
  • Plan for developments @ CERN can be found under:
    • http://quattor.org/documentation/management/quattor-plans-Q42006-Q42007.htm
    • Not updated yet since last workshop..
ad