1 / 12

User traceability and log analysis tools

User traceability and log analysis tools. Eygene Ryabinkin , RRC Kurchatov Institute Giuseppe Misurelli , INFN-CNAF (speaker) Daniel Kouril , CESNET EGEE09 Conference September 22, 2009 Barcelona. Outline. Log analysis How to figure out what’s going on

uriah
Download Presentation

User traceability and log analysis tools

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. User traceability and log analysis tools EygeneRyabinkin, RRC Kurchatov Institute Giuseppe Misurelli, INFN-CNAF (speaker) Daniel Kouril, CESNET EGEE09 Conference September 22, 2009 Barcelona

  2. Outline • Log analysis • How to figure out what’s going on • How to relate the records at various sources • What OSCT has been provided so far • gLite-LB tracer (lbtrace) • lcg-CE tracer (dig-lcgce) • How does OSCT distribute the tools • RPMs available at? • Documentation • Support • Future plans

  3. Log analysis: security officer viewpoint “A security incident affected my site. I need to do forensics among a lot of row logs in various forms” “ “The jobs running in my WN passed through different machines: UI, WMS/LB” “I wish I had utilities able to analyze log files for me?”

  4. Log analysis: OSCT suggested solution The OSCT startingpoint: create at least some usabletools and show themto the public “useour log analysistoolsthatprovideconsistent interface and thatcouldbechainedtogether”

  5. gLiteservicescurrentlytraced gLite-LB (lbtrace) lcg-CE (dig-lcgce) Whywestartedwiththesetwoservices? • they keep the majority of the information about user jobs • Not counting data transfer • CREAM CE? Not yet investigated, but in TODO list • easily relation between log records and relevant attributes to be searched • Job IDs • User DNs • VOMS attributes

  6. gLite-LB tracer: lbtrace Queries LB hostsaboutuserjobs and bookkeeping information • job hopsthrough the differentGridservices Can list job recordsbasing on simplequerysyntax • status eqdone and dsteq <CE> Currently can interrogate only the live part of the LB database • job records are periodicallypurgedinto the offline record library Needscreationofindices on the LB backend • easy but a bit annoying on production LB (servicesrestartrequired)

  7. gLite-LBtracer: usageexample #lbtrace -k host -H octopus.grid.kiae.rulistownereq '/DC=ch/DC=cern/OU=OrganicUnits/OU=Users/CN=samoper/CN=582979/CN=Judit Novak' and status eqdone and destinationeqsnowpatch-hep.westgrid.ca:2119/jobmanager-lcgpbs-ops --- Job 1: JobId: https://octopus.grid.kiae.ru:9000/B1uLHolwYwpTYh1uwudmjQ Owner: /DC=ch/DC=cern/OU=OrganicUnits/OU=Users/CN=samoper/CN=582979/CN=Judit Novak Source: sam111.cern.ch JobState: Done StatusReason: Job terminatedsuccessfully Destination: snowpatch-hep.westgrid.ca:2119/jobmanager-lcgpbs-opsCondorID: 664 GlobusID: [none] PBSOwner: [none] PBSNode: [none] Lookingfor information aboutjobsrecordedby a given LB • Security officerneedsforensics on a specific job

  8. lcg-CE tracer: dig-lcgce Usesjobmaprecordsfrom the gatekeeper A SQL-like interface to the job records • Understandsconditionalexpressionsthatcouldbecombined Can invokeLRMS-specifictoolsto trace selectedjobs down to the batch system logginglayer • Think <<tracejob>> Can beused on centrallogginghost • Needsonlyjobmapfiles and installedPython

  9. lcg-CEtracer: usageexample #dig-lcgce -s 20090901 -e 20090916 userDNeq '/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=giuseppemisurelli' { 'localUser': '18700', 'ceID': 'gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert', 'timestamp': '2009-09-07 14:00:21', 'userFQAN': ['/dteam/Role=NULL/Capability=NULL', '/dteam/italy/Role=NULL/Capability=NULL', '/dteam/italy/INFN-CNAF/Role=NULL /Capability=NULL'], 'userDN': '/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=giuseppemisurelli', 'jobID': 'https://lb009.cnaf.infn.it:9000/ZTHAJucuJpw4mgwKysV_2A', 'lrmsID': '118258.gridit-ce-001.cnaf.infn.it‘ } Diggingfor information relatedto a suspecteduser DN • Site CSIRTsnotifiedaboutmalicious job submittedby a given DN

  10. Distributing the tools Sources live in SA1 subversionrepository • https://www.sysadmin.hep.ac.uk/svn/security Packageswillbeprovidedby the SA1 repository • https://twiki.cern.ch/twiki/bin/view/EGEE/EGEESA1PackageRepository • basicallyyuminstall <<package_name>> Documentation and usageexamplesprovided at the OSCT twiki web site • https://twiki.cern.ch/twiki/bin/view/LCG/LogTracing Packagesinstall standard Unix man pages • man lbtrace • man dig-lcgce

  11. Support and future plans Support and featurerequests are handledthrough the SA1 Savannah section • https://savannah.cern.ch/projects/sa1tools/ Future plans • UsegLite Job Provenancetoaccess data from offline LB recordsstore • Add more SQL-likefeaturesusefulforoverviewof the useractivity • orderby, count • Writetoolsfor CREAM CE • Investigate the possibilityfortracing the data movementsoverstorageelements • Anythingsensiblerequestedby the end-usersof the tools

  12. Thanksforyourattention Questions, comments, featurerequests? You’re welcome!

More Related