user traceability and log analysis tools n.
Skip this Video
Loading SlideShow in 5 Seconds..
User traceability and log analysis tools PowerPoint Presentation
Download Presentation
User traceability and log analysis tools

Loading in 2 Seconds...

  share
play fullscreen
1 / 12
Download Presentation

User traceability and log analysis tools - PowerPoint PPT Presentation

mercedes-flores
182 Views
Download Presentation

User traceability and log analysis tools

- - - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

  1. User traceability and log analysis tools EygeneRyabinkin, RRC Kurchatov Institute Giuseppe Misurelli, INFN-CNAF (speaker) Daniel Kouril, CESNET EGEE09 Conference September 22, 2009 Barcelona

  2. Outline • Log analysis • How to figure out what’s going on • How to relate the records at various sources • What OSCT has been provided so far • gLite-LB tracer (lbtrace) • lcg-CE tracer (dig-lcgce) • How does OSCT distribute the tools • RPMs available at? • Documentation • Support • Future plans

  3. Log analysis: security officer viewpoint “A security incident affected my site. I need to do forensics among a lot of row logs in various forms” “ “The jobs running in my WN passed through different machines: UI, WMS/LB” “I wish I had utilities able to analyze log files for me?”

  4. Log analysis: OSCT suggested solution The OSCT startingpoint: create at least some usabletools and show themto the public “useour log analysistoolsthatprovideconsistent interface and thatcouldbechainedtogether”

  5. gLiteservicescurrentlytraced gLite-LB (lbtrace) lcg-CE (dig-lcgce) Whywestartedwiththesetwoservices? • they keep the majority of the information about user jobs • Not counting data transfer • CREAM CE? Not yet investigated, but in TODO list • easily relation between log records and relevant attributes to be searched • Job IDs • User DNs • VOMS attributes

  6. gLite-LB tracer: lbtrace Queries LB hostsaboutuserjobs and bookkeeping information • job hopsthrough the differentGridservices Can list job recordsbasing on simplequerysyntax • status eqdone and dsteq <CE> Currently can interrogate only the live part of the LB database • job records are periodicallypurgedinto the offline record library Needscreationofindices on the LB backend • easy but a bit annoying on production LB (servicesrestartrequired)

  7. gLite-LBtracer: usageexample #lbtrace -k host -H octopus.grid.kiae.rulistownereq '/DC=ch/DC=cern/OU=OrganicUnits/OU=Users/CN=samoper/CN=582979/CN=Judit Novak' and status eqdone and destinationeqsnowpatch-hep.westgrid.ca:2119/jobmanager-lcgpbs-ops --- Job 1: JobId: https://octopus.grid.kiae.ru:9000/B1uLHolwYwpTYh1uwudmjQ Owner: /DC=ch/DC=cern/OU=OrganicUnits/OU=Users/CN=samoper/CN=582979/CN=Judit Novak Source: sam111.cern.ch JobState: Done StatusReason: Job terminatedsuccessfully Destination: snowpatch-hep.westgrid.ca:2119/jobmanager-lcgpbs-opsCondorID: 664 GlobusID: [none] PBSOwner: [none] PBSNode: [none] Lookingfor information aboutjobsrecordedby a given LB • Security officerneedsforensics on a specific job

  8. lcg-CE tracer: dig-lcgce Usesjobmaprecordsfrom the gatekeeper A SQL-like interface to the job records • Understandsconditionalexpressionsthatcouldbecombined Can invokeLRMS-specifictoolsto trace selectedjobs down to the batch system logginglayer • Think <<tracejob>> Can beused on centrallogginghost • Needsonlyjobmapfiles and installedPython

  9. lcg-CEtracer: usageexample #dig-lcgce -s 20090901 -e 20090916 userDNeq '/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=giuseppemisurelli' { 'localUser': '18700', 'ceID': 'gridit-ce-001.cnaf.infn.it:2119/jobmanager-lcgpbs-cert', 'timestamp': '2009-09-07 14:00:21', 'userFQAN': ['/dteam/Role=NULL/Capability=NULL', '/dteam/italy/Role=NULL/Capability=NULL', '/dteam/italy/INFN-CNAF/Role=NULL /Capability=NULL'], 'userDN': '/C=IT/O=INFN/OU=Personal Certificate/L=CNAF/CN=giuseppemisurelli', 'jobID': 'https://lb009.cnaf.infn.it:9000/ZTHAJucuJpw4mgwKysV_2A', 'lrmsID': '118258.gridit-ce-001.cnaf.infn.it‘ } Diggingfor information relatedto a suspecteduser DN • Site CSIRTsnotifiedaboutmalicious job submittedby a given DN

  10. Distributing the tools Sources live in SA1 subversionrepository • https://www.sysadmin.hep.ac.uk/svn/security Packageswillbeprovidedby the SA1 repository • https://twiki.cern.ch/twiki/bin/view/EGEE/EGEESA1PackageRepository • basicallyyuminstall <<package_name>> Documentation and usageexamplesprovided at the OSCT twiki web site • https://twiki.cern.ch/twiki/bin/view/LCG/LogTracing Packagesinstall standard Unix man pages • man lbtrace • man dig-lcgce

  11. Support and future plans Support and featurerequests are handledthrough the SA1 Savannah section • https://savannah.cern.ch/projects/sa1tools/ Future plans • UsegLite Job Provenancetoaccess data from offline LB recordsstore • Add more SQL-likefeaturesusefulforoverviewof the useractivity • orderby, count • Writetoolsfor CREAM CE • Investigate the possibilityfortracing the data movementsoverstorageelements • Anythingsensiblerequestedby the end-usersof the tools

  12. Thanksforyourattention Questions, comments, featurerequests? You’re welcome!