html5-img
1 / 162

OVERVIEW OF SERVER-SIDE PROCESSING

OVERVIEW OF SERVER-SIDE PROCESSING. Doug Clough SYNERGETICS Engineered Systems. 12 January 2009 www.ptagis.org. Data Basics (15 min) PTAGIS Specification Document Tagging & Interrogation Files Core Database Tables PTTP – Client and Server Implementations (15 min) Request processing

zed
Download Presentation

OVERVIEW OF SERVER-SIDE PROCESSING

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. OVERVIEW OF SERVER-SIDE PROCESSING Doug Clough SYNERGETICS Engineered Systems 12 January 2009 www.ptagis.org

  2. Data Basics (15 min) PTAGIS Specification Document Tagging & Interrogation Files Core Database Tables PTTP – Client and Server Implementations (15 min) Request processing Request transformation FDVL – Field Data Validating Loader (30 min) Validation and Loading User Feedback: PTTP and Email Loader performance report IDL – Interrogation Data Loader (30 min) Validation and Loading Loader performance report Interrogation System performance reports Server-Side Processing

  3. Data Basics (15 min) PTAGIS Specification Document Tagging & Interrogation Files Core Database Tables PTTP – Client and Server Implementations (15 min) Request processing Request transformation FDVL – Field Data Validating Loader (30 min) Validation and Loading User Feedback: PTTP and Email Loader performance report IDL – Interrogation Data Loader (30 min) Validation and Loading Loader performance report Interrogation System performance reports Server-Side Processing

  4. PIT Tag Specification Document Tagging & Interrogation Files Core Database Tables Data Basics

  5. PIT Tag Specification Document Tagging & Interrogation Files Core Database Tables Data Basics

  6. PIT Tag Specification Document • PIT Tag Steering Committee (PTSC) • Tagging & Interrogation Files • Tagging File pg 14 • MiniMon Interrogation File pg 27 • MULTIMON File pg 30 • Core Database Tables • Appendix F • Excludes ‘process-specific tables’ A subcommittee of the Fish Passage Advisory Committee of the Columbia Basin Fish and Wildlife Authority. The Committee is made up of technical representatives of the Basin’s fisheries agencies and tribes. The Committee's function is to provide guidance in the development of tagging methodologies and data standards and to serve as a technical forum for the operation of PTAGIS and the PIT tag data collection and data distribution system. V. Glossary of Terms – pg 71

  7. PIT Tag Specification Document Tagging & Interrogation Files Core Database Tables Data Basics

  8. Tagging Files • Multiple versions are anticipated • All versions maintained on server • FDVL provides insert / update / delete capability • Latest version is in database • FILE TITLE Header Record • Auto-generated by P3 • xxxYYDDD.zzz • xxx Valid Coord ID • YYDDD Day of Year • zzz Arbitrary Extension • External file name must match internal FILE TITLE • When choosing zzz extension, user must be sure not to create a conflict with an existing file

  9. MiniMon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records YES • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.an • sss Site Code • YYDDD Day of Year • a Sequence character • n 1 or 2 (Primary/Backup) • External file name must match internal FILE TITLE

  10. MiniMon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records YES • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.an • sss Site Code • YYDDD Day of Year • a Sequence character • n 1 or 2 (Primary/Backup) • External file name must match internal FILE TITLE Fish Tag Interrogation

  11. MiniMon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records YES • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.an • sss Site Code • YYDDD Day of Year • a Sequence character • n 1 or 2 (Primary/Backup) • External file name must match internal FILE TITLE Timer Tag Interrogation

  12. MiniMon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records YES • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.an • sss Site Code • YYDDD Day of Year • a Sequence character • n 1 or 2 (Primary/Backup) • External file name must match internal FILE TITLE TASS Transceiver Diagnostics

  13. Multimon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records NO • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.nnn • sss Site Code • YYDDD Day of Year • nnn Sequence number100’s primary, 200’s backup • External file name must match internal FILE TITLE

  14. Multimon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records NO • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.nnn • sss Site Code • YYDDD Day of Year • nnn Sequence number100’s primary, 200’s backup • External file name must match internal FILE TITLE DGE ControlState Events

  15. Multimon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records NO • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.nnn • sss Site Code • YYDDD Day of Year • nnn Sequence number100’s primary, 200’s backup • External file name must match internal FILE TITLE DGE Interrogation Events

  16. Multimon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records NO • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.nnn • sss Site Code • YYDDD Day of Year • nnn Sequence number100’s primary, 200’s backup • External file name must match internal FILE TITLE TASS Transceiver Diagnostics

  17. Multimon Interrogation Files • Multiple versions not supported • Files are maintained on server • IDL provides insert capability only • Format defined in SpecDoc? • Interrogation records NO • Transceiver diagnostics NO • Varies by model / firmware • FILE TITLE Record • Auto-generated by MiniMon • sssYYDDD.nnn • sss Site Code • YYDDD Day of Year • nnn Sequence number100’s primary, 200’s backup • External file name must match internal FILE TITLE Timer Tag Interrogation

  18. PIT Tag Specification Document Tagging & Interrogation Files Core Database Tables Data Basics

  19. Core Database Tables • Have existed from ‘day one’ • All contain data of interest to ‘typical user’ except … • TEST_TAGS • TEST_TAG_DATA • Many non-core tables exist to support PTAGIS processes

  20. Core Database Tables These support IDL

  21. Data Basics (15 min) PTAGIS Specification Document Tagging & Interrogation Files Core Database Tables PTTP – Client and Server Implementations (15 min) Request processing Request transformation FDVL – Field Data Validating Loader (30 min) Validation and Loading User Feedback: PTTP and Email Loader performance report IDL – Interrogation Data Loader (30 min) Validation and Loading Loader performance report Interrogation System performance reports Server-Side Processing

  22. Request processing PTTP has processing, reporting, and messaging capabilities What happens to ‘data payload’ depends on request and type of data PTTP – Client and Server Implementations

  23. Request processing PTTP has processing, reporting, and messaging capabilities What happens to ‘data payload’ depends on request and type of data PTTP – Client and Server Implementations Primary job

  24. Request processing PTTP has processing, reporting, and messaging capabilities What happens to ‘data payload’ depends on request and type of data Request transformation PTTP deployment began in Jan 2001 Previously used upload or email for tagging files Previously used upload or email for interrogation files Deployment took time; backward compatibility was required Interrogation files emailed to intdata@psmfc.org Turned into PTTP transactions by PTTP_EmailClient PTTP stages files for processing by IDL Tagging files transmitted by PTTP Original email interface to FDVL was retained PTTP_Dispatcher invokes PTTP_FDVL via PTTP_Handler Sends email to pitdata@psmfc.org to invoke FDVL PTTP – Client and Server Implementations Important secondary job

  25. PTTP Overview PTTP ‘Transaction’ Zip-Files • Name: TX<UUID>.zip • Transmitted by FTP • Unpacked in per-transaction directory • One serialized PTTP_Req (XML) • Zero or more data files • Results available in PTTP_Ack (XML) Remote PTTP Clients • P3 - Tagging files • Minimon, Multimon – Interrogation files Local PTTP Clients • PTTP_EmailClient • Generates PTTP transaction for interrogation data sent by email to intdata@psmfc.org

  26. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type

  27. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type

  28. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type

  29. PTTP Detail PTTP_Req examines data payload. If interrogation files are included, the transaction identifier is posted to input queues of auxiliary analytical processes: • TASS_Monitor • DGE_Monitor Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type

  30. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type For ‘Invocation’ event PTTP_Dispatcher … • Invokes handler • Handler reads possible Input file • Handler performs action and queues‘Completion’ event

  31. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type For ‘Invocation’ event PTTP_Dispatcher … • Invokes handler • Handler reads possible Input file • Handler performs action and queues‘Completion’ event

  32. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type For ‘Invocation’ event PTTP_Dispatcher … • Invokes handler • Handler reads possible Input file • Handler performs action and queues‘Completion’ event For ‘Completion’ event PTTP_Dispatcher • Instantiates PTTP_Ack to log completion • Calls genNextEvent() to queue possible follow-on ‘Invocation’ event

  33. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type For ‘Invocation’ event PTTP_Dispatcher … • Invokes handler • Handler reads possible Input file • Handler performs action and queues‘Completion’ event For ‘Completion’ event PTTP_Dispatcher • Instantiates PTTP_Ack to log completion • Calls genNextEvent() to queue possible follow-on ‘Invocation’ event

  34. PTTP Detail Process flow … • watchFtp launches processPittagFtp • PTTP_Server unpacks zip-file to per-TX directory and instantiates PTTP_Req • PTTP_Server initializes PTTP_Ackand queues ‘Invocation’ event • PTTP_Server wakes PTTP_Dispatcherwhich proceeds according to event type For ‘Invocation’ event PTTP_Dispatcher … • Invokes handler • Handler reads possible Input file • Handler performs action and queues‘Completion’ event For ‘Completion’ event PTTP_Dispatcher • Instantiates PTTP_Ack to log completion • Calls genNextEvent() to queue possible follow-on ‘Invocation’ event

  35. Data Basics (15 min) PTAGIS Specification Document Tagging & Interrogation Files Core Database Tables PTTP – Client and Server Implementations (15 min) Request processing Request transformation FDVL – Field Data Validating Loader (30 min) Validation and Loading User Feedback: PTTP and Email Loader performance report IDL – Interrogation Data Loader (30 min) Validation and Loading Loader performance report Interrogation System performance reports Server-Side Processing

  36. Validation and Loading User Feedback PTTP Email Loader performance report FDVL – Field Data Validating Loader

  37. Validation and Loading User Feedback PTTP Email Loader performance report FDVL – Field Data Validating Loader

  38. FDVL – Recording ‘Field Events’ in Database Tables Original File Types • Tagging • ReleaseInformation • MonitoredRelease • Mortality

  39. FDVL – Recording ‘Field Events’ in Database Tables Since 2001 • Tagging • ReleaseInformation • MonitoredRelease • Mortality Since 2001 Per-fish flag codes are usedto indicate recaptureand mortality eventswithin a Tagging file

  40. FDVL Overview

  41. FDVL Overview PARALLEL

  42. FDVL Overview SERIAL

  43. FDVL Overview Staging eliminated with fd_quickLoad in August 2002

  44. FDVL Software Components

  45. FDVL - Meta-Data • Stored and maintained in MS Access database • spec_doc.mdb • Exported to CSV-files for use • FD_FileMetaData.txt • FD_FileTableMap.txt • CSV-files populate in-memory data structures in fdc_receiveEmail.pl • SpecDoc • FileMap

  46. Specification Document: FDVL Model of PTAGIS File Types

  47. FDVL: Mapping of Record Fields to Columns in Database Tables

  48. FDVL: Modeling Content of a User’s File

  49. User’s Tagging File • Must contain all mandatory records • Values in optional fields must be valid • As Validator parses file it • Assembles FileData tree • Tallies ‘leaves’ on the tree • Assesses their validity • Errors exist if valid count is less than item count • Processing stops with user message if errors exist • Otherwise Loader is told to run • May be running already • May have been halted

  50. Validator Log File For each record group • item_count • Tally of required records • valid_count • Tally of valid records If errors are encountered • Specifics are posted to log • Log entries are incorporated into user message

More Related