tools and services vision fy12 fy14 n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Tools and Services Vision FY12-FY14 PowerPoint Presentation
Download Presentation
Tools and Services Vision FY12-FY14

Loading in 2 Seconds...

play fullscreen
1 / 19

Tools and Services Vision FY12-FY14 - PowerPoint PPT Presentation


  • 122 Views
  • Uploaded on

Tools and Services Vision FY12-FY14. C. Thompson. SOTO Raster Processing Pipeline: FY12. gridding. n ew g ranule collection. tiling. Dataset A. unified data structure. reader/ conditioner A. config file. config file. Dataset C. reader/ conditioner B. KML pyramids. MRF files.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Tools and Services Vision FY12-FY14' - miles


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
soto raster processing pipeline fy12
SOTO Raster Processing Pipeline: FY12

gridding

new

granule

collection

tiling

Dataset A

unified

data

structure

reader/

conditioner A

config

file

config

file

Dataset C

reader/

conditioner B

KML

pyramids

MRF

files

reader/

conditioner C

full

MRF

generation

global

geoTIFFtile

generation

global

PNG

generation

Dataset B

KML

generation

floating tile

repository

soto raster processing pipeline fy13
SOTO Raster Processing Pipeline: FY13

DMAS

(execute)

gridding

new

granule

collection

unified

data

structure

universal

file

format

tiling

Dataset A

reader/

conditioner A

config

file

config

file

universal

reader

Dataset C

reader/

conditioner B

trigger

DMAS

(ingest)

MRF

files

reader/

conditioner C

full

MRF

generation

global

geoTIFFtile

generation

Dataset B

generate

metadata

floating tile

repository

soto raster processing pipeline fy14
SOTO Raster Processing Pipeline: FY14

DMAS

(execute)

gridding

new

granule

collection

tiling

Dataset A

unified

data

structure

reader/

conditioner A

PO.DAAC

Image

Generation

Subsystem

config

file

config

file

universal

file

format

global &

per granule

hdf5-MRF

generation

universal

reader

floating

MRF

generation

Dataset C

reader/

conditioner B

trigger

DMAS

(ingest)

MRF

files

reader/

conditioner C

full

MRF

generation

global

geoTIFFtile

generation

Dataset B

generate

metadata

floating tile

repository

hitide web services fy12
HITIDE Web Services: FY12

reader/tiler A

tile DB

crawlers

XML

ingestor

reader/tiler B

space

time

dataset(s)

Dataset A

reader/tiler C

XML

L2

OpenSearch

WS

L2

extract

WS

L2

WMS

Dataset C

dynamic

extraction

dynamic

image

generation

imaging

plugin C

imaging

plugin A

imaging

plugin B

Dataset B

extract

plugin A

extract

plugin B

extract

plugin C

hitide web services fy13
HITIDE Web Services: FY13

space

time

dataset(s)

data summaries

DMAS

(execute)

reader

A

tiler

A

universal

tiler

reader/tiler A

tile DB

trigger

reader

B

tiler

B

crawlers

XML

ingestor

reader/tiler B

space

time

dataset(s)

Dataset A

reader

C

tiler

C

reader/tiler C

config

file

XML

universal

file

format

L2

OpenSearch

WS

L2

extract

WS

L2

WMS

Dataset C

dynamic

extraction

dynamic

image

generation

universal

writer

universal

file

format

imaging

plugin C

imaging

plugin A

imaging

plugin B

universal

reader

Dataset B

extract

plugin A

extract

plugin B

extract

plugin C

hitide web services fy14
HITIDE Web Services: FY14

DMAS

(execute)

space

time

dataset(s)

data summaries

reader

A

universal

tiler

tile DB

reader

B

crawlers

XML

ingestor

Dataset A

reader

C

config

file

XML

universal

file

format

Tiled WMS

L2

OpenSearch

WS

L2

extract

WS

L2

WMS

Dataset C

dynamic

extraction

dynamic

image

generation

hdf5-MRF

granule files

universal

writer

universal

file

format

imaging

plugin C

imaging

plugin A

imaging

plugin B

universal

reader

Dataset B

extract

plugin A

tiled wms fy12 fy13 gwinn
Tiled WMS: FY12/FY13 (GWiNN)
  • Editable color palette (fixed bin size/data range)
  • WMTS compatible
  • Partial MRF updates
  • Dynamic tile resizing (“overzoom”)
  • Server architecture optimization
  • Re-organization of source code for easier deployment
  • Bolster documentation
  • Per granule MRF generation
  • Multiple projections
  • Sample clients
  • RESTful calls to individual tiles
  • Vector support
  • More flexible key-value pair ordering

client-side solution

completed

high priority

client-side solution

high priority

high priority

high priority

mid priority

mid priority

completed

mid priority

high priority

mid priority

soto raster processing pipeline fy141
SOTO Raster Processing Pipeline: FY14

DMAS

(execute)

gridding

new

granule

collection

tiling

Dataset A

unified

data

structure

reader/

conditioner A

PO.DAAC

Image

Generation

Subsystem

config

file

config

file

universal

file

format

global &

per granule

hdf5-MRF

generation

universal

reader

floating

MRF

generation

Dataset C

reader/

conditioner B

trigger

DMAS

(ingest)

MRF

files

reader/

conditioner C

full

MRF

generation

global

geoTIFFtile

generation

Dataset B

generate

metadata

floating tile

repository

tiled wms fy13 fy14 po daac
Tiled WMS: FY13/FY14 (PO.DAAC)
  • Design and development of hdf5-MRF format
  • Benchmark hdf-MRF against MRF access times
  • Build hdf5-MRF Apache plug-in

colorized

cylindrical

or

reprojected

image

(cached request)

gridding

tiling

colorized

reprojected

image

(new request)

unified

data

structure

universal

file

format

global &

per granule

hdf5-MRF

generation

floating

MRF

generation

colorized

reprojected

image

(new request)

universal

reader

colorized

cylindrical

image

(new request)

colorized

cylindrical

image

(new request)

fy13 soto
FY13: SOTO
  • Enhance user experience
    • Reduce latency
      • Move to MRF
        • Modify raster image generation backend to produce daily MRF’s (Jeff, Charles)
        • Re-imagine vector processing backend (U/V components) utilizing MRF’s (Jeff, Charles, GWiNN)
    • Update GUI controls
      • Add flat Earth view
        • Integrate OpenLayers view (Christian, GWiNN)
      • Perform usability review
        • Identify use cases (Eric, Charles, Michelle)
        • Generate requirements and design document (Eric, Charles, Michelle, Christian)
        • Generate GUI mockup (Christian, GWiNN)
        • Implement new GUI design (Christian, GWiNN)
  • Improve maintainability
    • Streamline process execution
      • Port to python
        • Rewrite floating-tiles-to-image-tiles package (Jeff, Charles)
        • Rewrite dataset-to-floating-tiles package (Jeff, Charles)
  • Unbundle processing and GUI packages
    • Transition processing package into baseline imaging subsystem component
      • Identify data storage structures
        • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed)
        • Design and define output: MRF contents (Charles, Andy, Eric, Mike, Jeff)
      • Revise and update architecture
        • Generate functional requirements and design document (Eric, Charles, Jeff)
  • Integrate with PO.DAAC system
    • Connect to DMAS
      • Initiate processing via DMAS-Execute
        • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)
      • Archive MRF’s via DMAS-Ingest
        • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)
fy13 hitide
FY13: HITIDE
  • Stabilize system
    • Mitigate tile database connectivity issues
      • Re-architect existing tile database
        • Review current schema and tables (Richard, Mike)
        • Generate requirements and design document (Eric, Richard, Mike)
        • Identify revised or new schema and tables (Richard, Mike)
        • Build database (Richard, Mike)
        • Update all services connecting to database (Richard, Mike)
  • Improve maintainability
    • Upgrade installation and deployment procedures
      • Streamline ingest
        • Review existing procedures and documentation and update as necessary (Richard, Mike)
    • Update messaging
      • Add error, information, and warning notifications
        • Insert calls to SigEvent service throughout ingestion module (Richard, Mike)
        • Insert calls to SigEvent service regarding database “health” (Richard, Mike)
  • Enhance user experience
    • Add/update functionality
      • Revise extraction web service
        • Review existing extraction output: organization, format, precision (Richard, Mike)
        • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles)
        • Modify web service as necessary (Richard, Mike)
      • Revise search web service
        • Apply indexing scheme to increase speed (Richard, Mike)
        • Make 100% OpenSearch compatible by returning total count (Richard, Mike)
      • Revise imaging web service
        • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles)
      • Allow search by data summaries
        • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles)
        • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed)
    • Update GUI controls
      • Perform usability review
        • Identify use cases (Eric, Charles, Ed)
        • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis)
        • Generate GUI mockup (Sargis, Richard)
        • Implement new GUI design (Sargis, Richard)
  • Integrate with PO.DAAC system
    • Connect to imaging subsystem
      • Utilize encapsulated WMS
        • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike)
    • Connect to DMAS
      • Initiate tiling via DMAS-Execute
        • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed)
        • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)
fy13 tiled wms
FY13: Tiled WMS
  • Stabilize system
    • Mitigate tile database connectivity issues
      • Re-architect existing tile database
        • Review current schema and tables (Richard, Mike)
        • Generate requirements and design document (Eric, Richard, Mike)
        • Identify revised or new schema and tables (Richard, Mike)
        • Build database (Richard, Mike)
        • Update all services connecting to database (Richard, Mike)
  • Improve maintainability
    • Upgrade installation and deployment procedures
      • Streamline ingest
        • Review existing procedures and documentation and update as necessary (Richard, Mike)
    • Update messaging
      • Add error, information, and warning notifications
        • Insert calls to SigEvent service throughout ingestion module (Richard, Mike)
        • Insert calls to SigEvent service regarding database “health” (Richard, Mike)
  • Expand functionality
    • Add/update functionality
      • Revise extraction web service
        • Review existing extraction output: organization, format, precision (Richard, Mike)
        • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles)
        • Modify web service as necessary (Richard, Mike)
      • Revise search web service
        • Apply indexing scheme to increase speed (Richard, Mike)
        • Make 100% OpenSearch compatible by returning total count (Richard, Mike)
      • Revise imaging web service
        • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles)
      • Allow search by data summaries
        • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles)
        • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed)
    • Update GUI controls
      • Perform usability review
        • Identify use cases (Eric, Charles, Ed)
        • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis)
        • Generate GUI mockup (Sargis, Richard)
        • Implement new GUI design (Sargis, Richard)
  • Integrate with PO.DAAC system
    • Connect to imaging subsystem
      • Utilize encapsulated WMS
        • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike)
    • Connect to DMAS
      • Initiate tiling via DMAS-Execute
        • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed)
        • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)
fy13 event service
FY13: Event Service
  • Stabilize system
    • Mitigate tile database connectivity issues
      • Re-architect existing tile database
        • Review current schema and tables (Richard, Mike)
        • Generate requirements and design document (Eric, Richard, Mike)
        • Identify revised or new schema and tables (Richard, Mike)
        • Build database (Richard, Mike)
        • Update all services connecting to database (Richard, Mike)
  • Improve maintainability
    • Upgrade installation and deployment procedures
      • Streamline ingest
        • Review existing procedures and documentation and update as necessary (Richard, Mike)
    • Update messaging
      • Add error, information, and warning notifications
        • Insert calls to SigEvent service throughout ingestion module (Richard, Mike)
        • Insert calls to SigEvent service regarding database “health” (Richard, Mike)
  • Enhance user experience
    • Add/update functionality
      • Revise extraction web service
        • Review existing extraction output: organization, format, precision (Richard, Mike)
        • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles)
        • Modify web service as necessary (Richard, Mike)
      • Revise search web service
        • Apply indexing scheme to increase speed (Richard, Mike)
        • Make 100% OpenSearch compatible by returning total count (Richard, Mike)
      • Revise imaging web service
        • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles)
      • Allow search by data summaries
        • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles)
        • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed)
    • Update GUI controls
      • Perform usability review
        • Identify use cases (Eric, Charles, Ed)
        • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis)
        • Generate GUI mockup (Sargis, Richard)
        • Implement new GUI design (Sargis, Richard)
  • Integrate with PO.DAAC system
    • Connect to imaging subsystem
      • Utilize encapsulated WMS
        • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike)
    • Connect to DMAS
      • Initiate tiling via DMAS-Execute
        • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed)
        • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)
fy13 consolidated web services
FY13: Consolidated Web Services
  • Stabilize system
    • Mitigate tile database connectivity issues
      • Re-architect existing tile database
        • Review current schema and tables (Richard, Mike)
        • Generate requirements and design document (Eric, Richard, Mike)
        • Identify revised or new schema and tables (Richard, Mike)
        • Build database (Richard, Mike)
        • Update all services connecting to database (Richard, Mike)
  • Improve maintainability
    • Upgrade installation and deployment procedures
      • Streamline ingest
        • Review existing procedures and documentation and update as necessary (Richard, Mike)
    • Update messaging
      • Add error, information, and warning notifications
        • Insert calls to SigEvent service throughout ingestion module (Richard, Mike)
        • Insert calls to SigEvent service regarding database “health” (Richard, Mike)
  • Enhance user experience
    • Add/update functionality
      • Revise extraction web service
        • Review existing extraction output: organization, format, precision (Richard, Mike)
        • Generate use cases, requirements, design (Eric, Richard, Mike, Ed, Andy, Charles)
        • Modify web service as necessary (Richard, Mike)
      • Revise search web service
        • Apply indexing scheme to increase speed (Richard, Mike)
        • Make 100% OpenSearch compatible by returning total count (Richard, Mike)
      • Revise imaging web service
        • Replace existing WMS with Tiled WMS granule MRF’s (Richard, Mike, Jeff, Charles)
      • Allow search by data summaries
        • Identify standard set of data summaries across datasets to add to tile database (Richard, Mike, Ed, Andy, Charles)
        • Review and revise process for adding unique data summaries per dataset to tile database (Richard, Mike, Ed)
    • Update GUI controls
      • Perform usability review
        • Identify use cases (Eric, Charles, Ed)
        • Generate requirements and design document (Eric, Richard, Charles, Ed, Sargis)
        • Generate GUI mockup (Sargis, Richard)
        • Implement new GUI design (Sargis, Richard)
  • Integrate with PO.DAAC system
    • Connect to imaging subsystem
      • Utilize encapsulated WMS
        • Update calls via Tiled WMS to standard WMS with optional keywords as necessary, e.g., scaling, colors, projection (Richard, Mike)
    • Connect to DMAS
      • Initiate tiling via DMAS-Execute
        • Design and define input: universal PO.DAAC data format model (Andy, Eric, Charles, Thomas, Michelle, Ed)
        • Generate requirements and design document (Jeff, Eric, Charles, Thomas, Mike)