1 / 19

Frontier rpms

Frontier rpms. June 2011 David Front Weizmann Institute. Why rpms?. Once a site (takes the effort and) moves to use rpms upgrading SW is faster, and easier compared to using tarballs It should also be more safe … assuming that proper testing of the new rpm release has been done

ahmed-odom
Download Presentation

Frontier rpms

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Frontier rpms June 2011 David Front Weizmann Institute frontier rpms

  2. frontier rpms

  3. Why rpms? Once a site (takes the effort and) moves to use rpms • upgrading SW is faster, and easier compared to using tarballs • It should also be more safe … assuming that proper testing of the new rpm release has been done Yum package manager is supported by frontier rpms • With yum, installing, upgrading and removing is easier that via rpm: - yum is aware of rpm dependencies - No need to specify rpm version, only package name • Automatic upgrade possibility: - Periodically do ‘yum upgrade package’ (via crontab) • Note, however, that relocation is not supported by yum frontier rpms

  4. Frontier rpms – history and status • Frontier has been written by CMS and distributed as tarballs • FaviaDonno did create initial frontier rpms for ATLAS: • frontier-squid • frontier-tomcat • frontier-servlet • frontier-awstats • A new release of frontier rpms is available, is in use at CERN ATLAS product servers since today, and is planned to be defined as ‘production ready’ soon: • In particular, there is a need for rpms to be tested out of CERN first • Current use of frontier rpms: • ATLAS (and CMS?) has squids installed as rpms • The other frontier rpms are installed only at ATLAS servers at CERN: CMS, and non CERN ATLAS frontier servers are installed via tarballs. frontier rpms

  5. Installing frontier rpms: Prerequisites • The (configurable) user to own rpm files should exist prior to rpm installation: Default: • ‘squid’ for squid • ‘dbfrontier’ for the other rpms • If you are installing frontier rpms, instead of existing non rpm (tarball) package/s: First uninstall the existing non rpm SW: • Remove the existing directory in which the software was installed • Take care to remove/undo changes at other files that have been edited manually or by scripts. In particular, undo crontab (manual) changes. • Verify that squid/tomcat service actually stopped: Be aware that: Stopping squid/tomcat may take a long time, or: ‘service frontier-{squid or tomcat} status’ may say by mistake ‘stopped’, while ‘ps –efw|grep{squid or tomcat}’ is not empty. frontier rpms

  6. Installing frontier rpms: configuration • Manually take care of per rpm and per site and per site/machine files, if any. For details, consult later slides, and: http://frontier.cern.ch/dist/rpms/_README • Configuration changes: As a rule of thumb, configuration changes are applied to files as follows: • The directory is /etc/{squid|awstats(|tomcat)} • Configuration changes of customize.sh are applied when the appropriate server restarts. • Hence, there should be no need to reinstall an rpm in order to apply (such) configuration changes. • In addition, for frontier-squid, squidconf is applied at install time frontier rpms

  7. Installing frontier rpms (cont.) • In order to install frontier rpms, as user root, import the gpg public key once (per machine): rpm --import http://frontier.cern.ch/dist/rpms/cernFrontierGpgPublicKey • Using the command 'rpm': as user root: rpm -Uvh [--prefix <RPM_INSTALL_PREFIX>] http://frontier.cern.ch/dist/rpms/<rpm> • Relocating an rpm: The defaults of RPM_INSTALL_PREFIX are: frontier-squid: / frontier-awstats: / frontier-tomcat: /data/dbfrontier frontier-servlet: /data/dbfrontier/tomcat/webapps/atlr (for servlet atlr) The frontier rpms are relocatable Exception: frontier-tomcat and frontier-servlet rpms can not be relocated to / The plan is that / will be the default for them as well. frontier rpms

  8. The current rpm versions At:http://frontier.cern.ch/dist/rpms/: • frontier-awstats-6.0-1.noarch.rpm • frontier-servlet-3.29-1.noarch.rpm • frontier-tomcat-6.0.32-1.noarch.rpm • frontier-squid-2.7.STABLE9-5.3.sl5.x86_64.rpm frontier rpms

  9. Installing via yum • get cernFrontier repository file once per machine: wget -O /etc/yum.repos.d/cern-frontier.repo http://frontier.cern.ch/dist/rpms/cern-frontier.repo • Now you may use yum as usual. for example: yum install package1 [package2] [...] yum upgrade package1 [package2] [...] yum remove package1 [package2] [...] frontier rpms

  10. Testing the installation: tomcat, squid For a simple test of tomcat and squid servers, using fnget.py: you may adjust and run commands like the following, from bash: wget -O /tmp/fnget.py http://frontier.cern.ch/dist/fnget.py chmod +x /tmp/fnget.py server=atlasfrontier1.cern.ch # Adjust this to your server servlet=atlr # Adjust this to you servlet port=8000 # To test tomcat. Adjust if needed /tmp/fnget.py \--url='http://'${server}':'${port}'/'${servlet}'/Frontier' \--sql="select 'TOMCAT_IS_HAPPY' from dual" \--refresh-cache port=8080 # To test squid. Adjust if needed /tmp/fnget.py \--url='http://'${server}':'${port}'/'${servlet}'/Frontier‘ \--sql="select 'SQUID_IS_HAPPY' from dual“ \--refresh-cache If all went well, you should get records like the following: 'TOMCAT_IS_HAPPY' CHAR 'SQUID_IS_HAPPY' CHAR frontier rpms

  11. Testing the installation: awstats, logrotate To see that awstats works, one may wait a while and see that the numbers or the upgraded squid do grow Logrotates: Logs should not grow too much: /data/tomcat_logs/catalina.out /data/tomcat_logs/modifiedqueries.out /data/squid_logs/access.log To see the allowed sizes: • sudo –u <squid/tomcat user> crontab –l • Find there when logrotate should work and sizes are allowed frontier rpms

  12. Testing the installation: load (After a while,) monitor the upgraded server/s host load, verifying that it did not rise or drop considerably, as a result of the upgrade: • CERN: use lemon https://lemonweb.cern.ch/ • Else where: ganglia or whatever host monitoring tool is in use frontier rpms

  13. Building a frontier rpm • rpms should be built at voatlas137 • The related sources should be checked out from svn, under: /data/$USER/rpms • Build an rpm: cd /data/$USER/rpms/<rpm Name> pushd ./scripts; ../../scripts/rpmbuild_.sh;popd 2) Copy rpm to the web: cd /data/$USER/rpms/<rpm Name> pushd ./scripts; ../../scripts/rpm2web.sh ; popd frontier rpms

  14. Plan • Once the new rpms will be defined as production ready,it will besuggested that all ATLAS frontier sites will use these frontier rpm/s instead of existing rpm or tarball/s • Enhance frontier rpms to support CMS • Support more than one servlet at a site.ATLAS does not require this because it has only one servlet per site • Add a /etc/tomcat/servlets.conf configuration file • Add a /etc/tomcat/servlets.passwd configuration file(will reside at sindes at CERN) 3) Avoid file duplication at underlying (CMS) tarball and at the rpms code (mainly at frontier-squid). This will make the maintenance of frontier code simpler 4) TBD: Add more monitoring tools into the rpms frontier rpms

  15. frontier rpms

  16. Configuration: frontier-squid Optional configuration files: 1) /etc/squid/squidconf: Used at rpm installation Consists of export commands Main usage: to specify user and group for frontier-squid, other than default user and group: squid, squid Example (as of CERN): export FRONTIER_USER=dbfrontier export FRONTIER_GROUP=users 2) /etc/squid/customize.sh Used per restart of the squid server. Consists of specific commands that control the content of squid.conf file. Examples: Appear at /etc/squid/customize.sh.example_* and at https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-squid/trunk/src/frontier-squid-utils/etc/squid -See also: FlaviaDonno's documentation for previous version of frontier-squid rpms: https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2 frontier rpms

  17. Configuration: frontier-servlet Per servlet configuration file should be edited manually. May be done after installation: <install dir>/tomcat/conf/Catalina/localhost/<servlet name>.xml Example of such a file name: /data/dbfrontier/tomcat/conf/Catalina/localhost/atlr.xml frontier rpms

  18. Configuration: frontier-awstats Optional file to control environment variables at install time /etc/awstats/awstats.conf Syntax: export <variable>=<value> Example: export FRONTIER_USER=<something other than dbfrontier> Editing of the followingfiles may be done (before or) after rpm installation: 1) A file for rsync with the password of user $SITE$PROJ (Example: cernatlas) /etc/awstats/password-file 2) An optional per awstats client site/machine config file. For CERN clients, this file is not needed, but for other sites it may be needed. File: /etc/awstats/customize.sh ## WARNING Care should be taken when changing variable in this file, ( and in particular PATH_LOG). Testing of frontier rpms has not been done with changing of these variables The rpm installs an example file: /etc/awstats/customize.sh.example_CERN: https://svnweb.cern.ch/trac/frontier/browser/rpms/frontier-awstats/trunk/src/etc/awstats/customize.sh.example_CERN frontier rpms

  19. urls • Documentation of new versions of frontier rpms: http://frontier.cern.ch/dist/rpms/_README • The main url of frontier: http://frontier.cern.ch • Current frontier rpms: http://frontier.cern.ch/dist/rpms/ • To browse svn files of frontier rpms: https://svnweb.cern.ch/trac/frontier/browser/rpms • To checkout a frontier rpms svn directory (example: frontier-tomcat): svn co svn+ssh://svn.cern.ch/reps/frontier/rpms/frontier-tomcat/trunc frontier-tomcat • Atlas frontier documentation by John Destefano: https://www.racf.bnl.gov/docs/services/frontier/ • Documentation of (previous) frontier server rpms, by FlaviaDonno: https://twiki.cern.ch/twiki/bin/view/LCG/WLCGFrontier • Documentation of (previous) versions of frontier-squid rpm, by FlaviaDonno: https://twiki.cern.ch/twiki/bin/view/PDBService/SquidRPMsTier1andTier2 • Previous versions of frontier rpms, created by FlaviaDonno, are at: http://grid-deployment.web.cern.ch/grid-deployment/flavia/?C=M;O=D • CMS has a lot of frontier documentation frontier rpms

More Related