1 / 21

Overview of TIO-index implementations

Overview of TIO-index implementations. The DAG, GIDS and Desire TIO/LDAP index servers. Henny Bekker. Overview of TIO-index implementations. Agenda. General overview of LDAP/TIO-indexes What are TIO indexes The generic model Some specific implementations

johana
Download Presentation

Overview of TIO-index implementations

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript


  1. Overview of TIO-index implementations The DAG, GIDS and Desire TIO/LDAP index servers Henny Bekker

  2. Overview of TIO-index implementations Agenda • General overview of LDAP/TIO-indexes • What are TIO indexes • The generic model • Some specific implementations • The generic Desire TIO index server • The Ericsson DAG server • The GIDS server • Open Issues • The scope and communication between LDAP/TIO index servers • Exchanging TIO’s • Local access policy • Access restrictions • Security requirements • Senario’s What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  3. General overview of LDAP/TIO-indexes Tagged Index Object’s A TIO consists of: • Meta information such as • A mime header defining the object • An object type identifier that uniquely identifies the subtree and scope • One or more URI's that will form the base of the created referrals • The security options and credentials such as a PGP or S/MIME key • The update type indicating the type of TIO (e.g. full or incremental) • The payload • The tokenization types headers (e.g. Full, Token, RFC822 etc) • Indicating which information is ‘tokenizated’ and which delimiters to use • The TAG list • Containing multiple consecutive tags which might be grouped using a dash. What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  4. General overview of LDAP/TIO-indexes Content-Type: application/index.obj.tagged; dsi="1.3.6.1.4.1.5062.1.99.1.114"; base-uri= "weetmuts.surfnet.nl:389/o=SURFnet, c=NL" Content-Length:6219 version: x-tagged-index-1 updatetype: total thisupdate: 950688539 BEGIN IO-Schema sn: FULL cn: FULL . o: TOKEN END IO-Schema BEGIN Index-Info sn: 22/Arends -6/Bezemer -4/Bos -8/Neggers . -2-3,5-9,11,14-15,18-19/+31 302 305 305 -12/030-2305327 o: 1/SURFnet END Index-Info What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  5. General overview of LDAP/TIO-indexes Tagged Index Object’s (cont.) What is it used for: • Provide pointers to servers which most likely contains the requested information • The number of false hits is depending on the choice of attribute tokenization types • Performing phrase searches is depending on the tokenization of the fields • Features a full or incremental update (which uses potentially less bandwidth) What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  6. General overview of LDAP/TIO-indexes The generic model • A TIO interface • For importing, deleting and in some cases exporting TIO’s from the index • Implementing authentication control • A TIO searchable index • For searching the index on referrals to other information services • Accessible through the TIO query interface • The LDAP query interface • Containing a LDAP gateway to the query interface of the TIO index • Can act as an LDAPv2 chaining server or as an LDAPv3 referral server What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  7. Some specific implementations The generic Desire TIO index server • Sponsored by the European Community and build by SURFnet & DFN in cooperation with Dante. • The server consists of: • The TIO index server • Using the MySQL database engine for storing and searching the TIO’s • Containing a TIO push/pull interface and a database for storing TIO’s. • An HTTP frontend for direct access to the TIO index server by the NPS. • A Native Protocol Server (NPS) for access using the LDAP protocol • For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. • For connecting directory servers with a specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. • An LdapCrawler for gathering and converting LDIF files to TIO’s • Currently no encryption of TIO’s implemented • Currently only support for LDAPv2. (no characterset conversion problem) What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  8. Some specific implementations The generic Desire TIO index server (cont.)

  9. Some specific implementations The Desire LDAP/TIO index server (cont.)

  10. Some specific implementations The Desire LDAP/TIO index server (cont.) • Unfortunately we don’t have yet any performance figures  • The package is on the brink of being completed  • Presumably the GIDS index server will be faster • The generic MySQL engine is slow compared to a dedicated TIO database. • Current implementation • Available on Linux and (hopefully) on Digital Unix • The source code and executable for Linux of • The LdapCrawler with an integrated LDIF2TIO converter • The TIO index (using MySQL v3.23.6) • The LDAP NPS implemented using the Open-LDAP v1.2.10 with an API to the TIO index What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  11. Some specific implementations The Ericsson DAG server • Offspring of the TISDAG project • Aimed to provide a solution for an uniform telephone directory containing numbers without a centralized database • The server consists of: • The DAG (Directory Access Gateway) index server • Implemented using the TimesTen “In-Memory” database engine for storing and searching the TIO’s. • One or more CAP (Client Access Point) modules • For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. • One or more SAP (Server Access Point) modules • For connecting directory servers with a specific communication protocols such as LDAPv2, LDAPv3 or WhoIS++. What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  12. Some specific implementations The Ericsson DAG server (cont.)

  13. Some specific implementations The Ericsson DAG server (cont.) • Unfortunately we aren’t allowed to present exact figures  • The next version is said to be much faster  • Performance figures • Response times • Use LDAPv3 referral requests to measure the response time of the referral server without doing chaining or following referrals. • The mean response time related to the number of parallel search queries. (measured with a large number of queries) • Number of queries/second (or minute??) • The number of parallel requests • Related to the response time • Maximum number of entries in the TIO index • Bounded by the memory size and the algorithm used to search the index What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  14. Some specific implementations The GIDS server • Offspring of the TISDAG project • Second implementation of the TISDAG TIO index server • The server consists of: • An index server • Using a dedicated database engine for storing and searching the TIO’s. • Is using a dedicated communication protocol (analogous with LDAP) to communicate with the CAP and SAP modules. • One or more CAP (Client Access Point) modules • For connecting clients using specific communication protocols such as LDAPv2, LDAPv3 and HTTP • One or more SAP (Server Access Point) modules • For connecting directory servers with a specific communication protocols such as LDAPv2 and LDAPv3 • An LdapCrawler for gathering and converting LDIF files to TIO’s • With support for LDAPv2 and LDAPv3 and character-set conversion What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  15. Some specific implementations The GIDS server (cont.) • Performance figures • Response times • Measured with LDAPv3 requests (an LDAPv3 bind, sending the query, receiving the message, doing an unbind operation) • With one sequence of LDAPv3 requests, a mean time of approximately 23 msec. per LDAPv3 request. • With 10 simultaneously LDAPv3 requests approximately 150 msec per LDAPv3 request. • Maximum number of queries/second • Approximately 65 LDAPv3 queries/second • With LDAPv2 the number will be lower because the server has to do chaining. • Maximum number of entries in the TIO index • Bounded by the memory size and the algorithm used to search the index • Current demo implementation (CH, DE, NL, NO & SE) 120K tokens of 450 different data sets. (which consumes about 35-Mbyte of memory). What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  16. Open issues The scope and communication between TIO index servers • Scope • Centralized versus distributed LDAP/TIO engines • Location of the TIO/LDAP-index server • Located close by (in network terms) the end users to minimize the RTT • Located close by the referred LDAP servers to minimize the RTT related to LDAPv2 chaining • Exchanging TIO’s • Global TIO collection versus distributed collections on country level • Distributed to country level • Knowledge base or ‘where to find what’?? • Encrypted transport via HTTP • Push or pull ?? What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  17. Open issues Local access policy • Security requirements • Personal data is subjected to privacy legislation • For public data other security requirements might be imposed • No unauthorized access to local directory servers • Only accessible by local inhabitants and peer countries • All applications able to access the index should be known • Only a limited number of referrals might be returned • No ‘access denied’ messages • Don’t show entries which are not accessible • Access restrictions • Restrict access to the TIO/LDAP-index server • Restrict access to the LDAP servers containing the information • Chaining versus LDAPv3 referral • HTTP access control versus LDAP access control • Access via HTTP proxies versus LDAP proxies What is a TIO index bla bla bla bla bl abl ablbbabbnsjdsa bla bla

  18. Open issues Senario’s • Create trusted relation between country level TIO servers • Only peers will communicate with each other • Besides the local LDAP clients • A peer will enforce their own local access rules • The TIO index server should only be accessible by known clients • The LDAP query will be chained to the remote peers • The TIO objects of the peer country should deliver referrals which will point to a known access point e.g. an LDAP proxy or the FLDSA • An LDAP search requests from a known LDAP client must be chained to the known access point. • The number of known access points should be limited • The TIO objects cannot be duplicated between the peers

  19. Open issues Senario’s (cont.)

  20. Discussion..

More Related