henry kucera hydraspace solutions inc www hydraspace com l.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Second Generation Spatial Information Warehouse Architecture PowerPoint Presentation
Download Presentation
Second Generation Spatial Information Warehouse Architecture

Loading in 2 Seconds...

play fullscreen
1 / 49

Second Generation Spatial Information Warehouse Architecture - PowerPoint PPT Presentation


  • 254 Views
  • Uploaded on

Henry Kucera HydraSpace Solutions Inc www.hydraspace.com. Second Generation Spatial Information Warehouse Architecture. HydraSpace Development Partners. System Integrator. Henry Kucera, President, HydraSpace. Development Teams. Edric Keighan, President, CubeWerx. Pierre Lafond

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Second Generation Spatial Information Warehouse Architecture' - benjamin


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
hydraspace development partners
HydraSpace Development Partners
  • System Integrator
  • Henry Kucera,
    • President, HydraSpace
  • Development Teams
  • Edric Keighan,
    • President, CubeWerx
  • Pierre Lafond
    • President, Holonics
  • Gail Kucera
    • President, Mercator
strategic goals
Strategic Goals
  • The strategic goal is to:
    • achieve greater interoperability by establishing a common information infrastructure for use by all stakeholders
    • promote and leverage existing and evolving Technology and expertise (e.g., AGMIID, OGC Web Mapping, CEONet, MID-C, AquaGIS, III/OGDI, and others);
    • gain competitive advantage by consolidating various independent initiatives;
    • provide entry to foreign markets by cooperating with similar international activities;
    • ensure the compliance of products by funding standards-based activities to influence international standards evolution;
    • demonstrate a solution for the global delivery of integrated geospatial information, applications and services.
information components
Information Components

Framework information

Stakeholder Information

Endangered species

Siesmic surveys

Facilities information

Marine habitat

Geological mapping

Climatology

Shipping and aeronautical

Topographic base

Survey and GPS

Hydrographic Charts

Land disposition (leases, tenures, water licences, etc.)

Administrative areas

Satellite imagery

Tide and Current

connectivity barriers

Inconsistent data formats

  • Proprietary software & hardware systems
  • Varying spatial & temporal resolutions
  • Variable accuracy & precision
  • Multiple views of common entities
Connectivity Barriers:
the enterprise information model for supporting regulation and governance

Pesticide Permit

Mine Operations

Environmental

Assessment Act

Forest Tenures

X

Watersheds

Management

Waste Permit

Water Licenses

X

X

Lakes Database

Fisheries Stream class A

Ecosystem

Integrity

X

Fish Stock Releases

Rare & Endangered

Species

Hatchery Mgt

X

X

X

Petroleum

Exploration

(biophysical habitat

WETLAND)

Fisheries

Coastal Zone

Management

Shipping of

products

Modified from BC Ministry of Environment (1997)

The Enterprise Information Model ….For supporting regulation and governance
the enterprise information model for supporting industry same data different view

Production

testing

Export

Applications

Environmental

Assessment

Engineering

Historical

Research

Reservoir

Testing

Refining

Marketing

Data

Synthesis

Providing

Delivery

Infrastructure

Production

Exploration

Drilling

Seismic

Profiling

Land &

Permit

Acquisition

Partnerships

Consumer

The Enterprise Information Model for Supporting Industry… Same data - different view
distributed spatial architecture requirements
Distributed Spatial Architecture Requirements

The data handling requirements are:

  • a data warehouse, conforming to international standards, for the storage and integration of spatial and non-spatial information
  • a distributed architecture for locating and accessing data through a simple single-window search
  • a suite of tools and services for data load, integration and retrieval, Web browsing and report generation
  • The system must be designed to deliver information in a form readily accessible and useable by decision-makers in support of economic,environmental and environmental obligations.
distributed spatial system functions
Distributed Spatial System Functions

The system must support the following:

  • integrated spatial and non-spatial data capture and storage
  • capability to catalog data warehouse contents
  • linkages to multiple warehouses
  • timely location, access and retrieval of operational and warehouse data
  • multi-resolution views of data
  • reporting, display and visualization of raw, processed and abstracted data
enterprise support for multiple tools
Enterprise support for Multiple Tools
  • Data Access Tools
          • formulate requests
          • extract geodata
          • visualize geodata
          • navigate in the data warehouse
  • Analysis Tools
          • spatial analysis
          • capture, compilation, edit
          • any long transaction
  • Data Mining Tools
          • application specific
multi tier spatial architecture components

GIS

& applications

(ESRI, CARIS, INGR)

Integration Tools

Tools for modeling,

cleaning, integrating

and loading data.

Access Tools

Tools for query, analysis

and reporting.

(Web-based preferred)

Tier 3

Application

Environments

Tier 2

Application

Servers

DB servers

(CubeSERV)

File servers

(SAFE FME)

App Servers

(CIDAS)

Data Access

Protocols &

APIs

Network Interface APIs - (OGDI, OGC & CGI)

File manager

Meta-Data Management (Repository)

Tier 1

Data Management

&

Data Server

Environment

Data files

Data Management Tools

Object-Relational DBMS + with Spatial

enhancements

Multi-tier Spatial Architecture Components
mapping multi tier to the spatial data warehouse architecture

Medium & Thick

OGDI Clients

Thin

Client

Medium & Thick

SQL Clients

TIER1:

Applications

Web

Server

TIER 2:

Application Server

Components

OGDI

Server

CubeSERV

Server

CubeSTOR Server

SQL Engine

Text

Cartridge

CubeSTOR

Spatial Cartridge

SQL std.

Cartridge

Other

Cartridges

TIER 3:

Oracle Data Server

Oracle Network Computing

Architecture

File manager

File manager

Data files

File manager

Data files

Data files

  • Spatial data
  • Attribute data
  • Meta data

Database

Files

Mapping Multi-tier to the Spatial Data Warehouse Architecture
ogc service model for levels of distributed data access

Picture Case (Thin Client)

Display

Graphic Element Case (Medium Client - OGDI

& SAFE FME with GIS browsers)

Render

Service

Data Case (Thick Client - SQL Web Applications in

conjunction with either of the above)

DEG*

Service

* DEG =Display Element Generator

† SF = Simple Features

Filter

(SF† Query)

Service

OGC Service Model for Levels of Distributed Data Access
information access the picture case
Information Access:The Picture Case
  • OGC Compliant CubeVIEW using Common Gateway Interface (CGI)

The “Picture” case

Using a CGI-encoded set of instructions, a portrayal can be constructed into a GIF or JPEG. That GIF or JPEG travels to the browser via HTTP.

information access the graphic element case
Information Access:The Graphic Element Case
  • Open Geospatial Datastore Interface (OGDI)

The “Graphic Element” case

A Java applet or ActiveX component connects to a service (in this case using GLTP but in the future it will use XML/HTTP, IIOP, etc.) and requests either Display Elements (raster vector) or Rendered Images (e.g. transparent GIFs). These pieces can be rendered or layered on the client side.

Currently MapObjects, FME ArcView, GeoMedia, PCI and other applications can access data using the OGDI.

information access the data case

GIS

or file

archive

Spatial

Browser

Custom

application

Vanilla

HTML

Stakeholder

Data

FME

Translator

FME

with

OGDI

OGDI

Server

SQL

Html/

CGI

OGDI

Reader

Multi-

dimensional

Data

Format

(MDF)

CubeSTOR

Convert

Framework

Data

CubeSTOR

Load

  • Spatial data
  • Attribute data
  • Meta content
Information Access:The Data Case

The “Data” case

There is interaction with online sources via links and client initiated service connections. The data files might contain several feature sets, tables and some symbolization information.

slide17

Distributed Warehouse: Access to

data at any level from anywhere

DATA ACQUISITION

Picture, Graphic elements or Data

OPERATIONAL &

EXTERNAL

DATABASES

LOCAL DATA

WAREHOUSE

MASTER

DATA

WAREHOUSE

LOCAL DATA

WAREHOUSE

LOCAL DATA

WAREHOUSE

WEB PLUGINS

&

HTML/JAVA

TOOLS &

APPLICATIONS

DATA

VIEWS

Open Distributed Spatial Environment

Http://www.mercator-sys.com

protecting your information warehouse is based on international standards

ISO SQL3 & SQL/MM

Common Language

Common Data Model

Common Spatial Functions

Repository, Dictionary

& ADT Support

APIs (OGDI/CLI/ RDA)

Application

Level:

User-defined

Repository

Design

Physical

View

Level:

Level:

Level:

Database

Subject Area

Information

Model

Model

Schema

ISO TC211

OGIS

Common

Meta

-Model

COM/CORBA Compliance

Spatial Features & Operators

Simple Feature SQL

Modelling Methods

JAVA & OLE/DB

Meta

-Information

Web Mapping Standards

Services Definition

Protecting your information!Warehouse is based on International Standards:
warehouse products
Warehouse products:
  • CubeSTOR Convert
  • CubeSTOR Spatial Cartridge for Oracle 7 and 8
  • CubeSTOR Server
  • CubeSTOR Load
  • CubeSTOR Archive
  • CubeSERV (Web based product)
  • CubeVIEW (Web based product)
  • GenSystem (Network and Generalization tools)
  • SAFE FME/OGDI (& FME Direct)
  • Compusult CIDAS and MetaManager
  • Geomedia/OGDI
  • MapObject/OGDI and ArcView/OGDI (ArcInfo/OGDI in progress)
  • PCI/OGDI
  • …. others
cubestor access
CubeSTOR Access...
  • Existing CubeSTOR/Oracle Database Interface (release 2.09)
    • CubeSTOR Dictionary Views
    • ODI Packages
    • Seamless storage of all types of vector data
    • seamless storage of multi-channel imagery
    • Storage that supports hierarchical cellular data structures
    • Query through OGC compliant web mapping servers
    • Support for storage and return of OGC/SQLMM WKB
    • Support for query through OGDI
    • Integrated CGI interface
    • 3-D points, lines and polygons
    • Open API to multiple tools
cubestor access21
CubeSTOR Access...
  • Additional function in the Next Release CubeSTOR/Oracle Database Interface ( SQL3 Oracle 8 features for release Dec. 1999)
    • Feature partitioning managed by SQL engine
    • Insert, Update, Delete operations using standard SQL
    • Base Functions on Well-Known-Types
    • Metric Functions on Well-Known-Types
    • Geometric Manipulation Functions on Well-Known-Types
    • Support for ADT’s
    • Topological Operators (4 intersection method: Egenhofer and Clementi)
installed cubestor warehouse systems
Installed CubeSTOR Warehouse Systems
  • Canadian Forest Service - National Forest Information System (60 GB aiming at 350GB in the next 4 months and expansion capability to 1TB +. The immediate expansion is to load the BC Forest Inventory spatial data (FC1 + FIP) as well as yet to be determined work with Ontario Ministry of Natural Resources.
  • Department of National Defense - Eight installations to manage and map topographic and intelligence. (100 GB per installation)
  • US Army Topographic Engineering Center - Multiscale information management. (20 GB with growth to 100 GB in 2000)
  • Health Canada - Spatially enabled health statistics. Digital Chart of the world and National Topographic Database plus other datasets. (Estimated size 50 - 60 GB)
  • NRCan Build of an image library (underway). (12 GB - may grow to terabytes)
  • NRCan Host of Digital Chart of the World (20 GB)
  • Canadian Hydrographic Service - Manage soundings and chart data at five locations (Terabytes)
slide23

Input Data

Integration

SLF

ITD

1:50K

PITD

1:250K

  • Imagery
  • LIDAR
  • SAR
  • etc

Tactical

Data

Vector

Data

VPF

VMAP

1 & 2

VITD

1:50K

UVMAP

1:10K

CADRG

1:250K

Business Data

Usage

INFORMATION GENERATION

COMMAND &

CONTROL

REQUIREMENTS

& TRAINING

POLICY &

PLANNING

Data

Assimilation &

Management

INTEGRATION

& CONFLATION

SERVICES

DATA MANAGEMENT

&

PRODUCT DISSEMINATION

DISTRIBUTED

PRODUCTION

Case Study 1: US ARMY TEC- AGMIID

slide24

SLF

VPF

Existing

interface

VITD

1:50K

ITD

1:50K

PITD

1:250K

VMAP1 & 2

CADRG

1:250K

UVMAP

1:10K

Enhanced

interface

Year 2

interface

OGDI

Enhanced

FME

Retrieve

features

w/ IDs

CubeSTOR/Oracle Database System

Features

with

meta-content

Linked

Features

with

Meta-content

Information

“Products”

with

meta-content

OGDI

GEODEX

Visualize

feature data

Feature Linking

Service (FLS)

Conflation/

Generalization

& Best Path

Links in MSAccess

Year 1: US ARMY TEC

slide25

DYNAMIC

CONFLATION

+

+

Imagery

Vector

Data

Elevation

Data

Complex query

New “Product”

VALIDATION &

VERIFICATION

Linked Query w/ Boolean

CubeSTOR/Oracle Database System

with FME/OGDI/OGC Interface

4

Interface 1:

Update data and meta

Clean

Source

and/or

Features

Linked

Features

with

Meta-content

Conflated

Persistent

Information

“Products”

1

2

3

DATA MINING

& PLANNING

Complex Query

Interface 2:

Plus new data

Interface 4

Interface 3

SIMPLE

QUERY

Conflation/

Generalization

Analysis and linking

Year 2 & 3: US ARMY TEC

us tec agmiid target architecture dynamic multi scale visualization and analysis

Multiple datasets

Medium & Thick

OGDI Clients

Thin

Client

Medium & Thick

SQL Clients

TIER1:

Applications

Web

Server

TIER 2:

Application

Server

Components

Spatial Load Facility (FME & OGDI)

Data Cleansing , Topology Validation

& Quality Assurance

OGDI

Server

CubeSERV

Server

CubeSTOR Server

Network Analysis

SQL Engine

Generalization

GenSystem

Text

Cartridge

CubeSTOR

Spatial Cartridge

SQL std.

Cartridge

Other

Cartridges

Engine

TIER 3:

Oracle Data

Server

Oracle Network Computing

Architecture

  • Spatial data
  • Attribute data
  • Meta data

Database

Files

US TEC-AGMIID Target Architecture:Dynamic Multi-scale Visualization and Analysis
slide27

Topology integration

Version integration

Visualization

& reporting

Metacontent maintenance

Schema transform

Load & QA

Best path

Feature extraction

Generalization

Spatial

Warehouse

Data Server

Targetted Services to be delivered to TEC through AGMIID

case 2 nfis cgdi
Case 2: NFIS - CGDI

“The National Forest Information System (NFIS) is a policy and science framework designed to support Canada’s national and international commitments by providing timely and accurate spatial and non-spatial forest resources information”

nfis project deliverables
NFIS: Project Deliverables
  • The first phase of the project completed in March 1999 with the following deliverables:
    • Design and implementation of a working multi-gigabyte spatial data warehouse based on an Oracle/Unix platform
    • Population of the warehouse with selected metadata and data sets including: CANFI, NatGRID 1km & 100m elevation data, classified satellite imagery, protected areas, selected provincial forest inventory information, climate change data
    • Criteria & Indicators Reference to 1km NatGRID
    • Incorporate a Web-based reporting tool for National Forest Inventory
the technology environment for forestry reporting

Spatial Load

Interface

ORACLE 8 Environment

Open

GeoData

Interface

(OGDI)

SQL Language Interface

Output

Server

GIS

CubeWerx Spatial Cartridge

Spatial

- ESRI EOO

- Bentley DGN

- Satellite

- DEM

- SAIF

Feature

Manipulation

Engine

(FME)

Abstract Data Types

(OGIS and SQLMM

Compliant)

OGDI Server Driver

& CGI/BIN API

4GL Apps

Open

Geodata

Interface

(OGDI)

Oracle Database Kernel

Common

Gateway

Interface

(CGI, HTML

JAVA)

Meta-content Schema

Non - Spatial

- Oracle

- DB2

- flat files

Web Apps

Non-spatial Schema

Spatial Schema

Seamless, pluralistic

Standards-Based

Diverse Data Formats

Common

Application Server

Interfaces

Client

Applications

Multi-media Database Server

The technology environment for forestry reporting
slide32

NFIS: Criteria and Indicators Pilot

C&I Test Data

Forest Inventories

Protected Areas

Ecosections

Forest Regions

Biogeoclimatic

Watershed Atlas

Species Status (CDCs)

Allowable Cut Data

NatGRID

C & I

Spatial and non-spatial data

data updates

NFIS Data Warehouse

NFIS Catalog of Available Data

Reporting

Requirements

Other Data Warehouses

Gap Analysis

C&I CCFM Reporting

1.1.1; 1.1.2; 1.1.3;

2.2.2; 2.3.1; 4.4.4;

4.5.1; 5.1.2; 6.5.1

C & I Analysis and Synthesis

NFIS Data Access Methods

slide33

Biogeoclimatic (Provincial)

Forest Cover 1:20000 (Local)

Terrestrial Ecozones (National)

Spatial and Attribute Data Warehouse

NFIS Data Sampling

Every data set will be sampled at a one second interval

slide34

Sampling points HHCODED

1 second grids on the forest cover.

slide35

Data Sampling

Sampling at every second by every second grid.

Every sample point will have a unique HHCODE along with the coordinates, and attributes from the source data.

A 1:20,000 mapsheet is 6 minutes by 12 minutes

6 minutes x 60 seconds = 360 seconds

12 minutes x 60 seconds = 720 seconds

360 x 720 = 259,200 samples per mapsheet.

100 6’by12’ mapsheets per letter block = 25,920,000 points

Study area, 3 letter blocks = 77,760,000 points

slide36

Data Loads

A 1:250,000 is (100 1:20,000 ) 1 degree by 2 degree

3600 x 7200 = 25,920,000

There are approximately 7000 1:20,000 mapsheets in B.C.

259200 x 7000 = 1,814,400,000

1,814,400,000 records for British Columbia

approx. 20,000,000,000 records for Canada

case 3 national health surveillance system nhss

Enable Analysis & Decision Support

  • Handle Large Volume of Spatial Data,
  • Leverage existing technology,
  • Use Basic Tools to keep training and acquisition costs low
Case 3: National Health Surveillance System (NHSS)
  • Link NHSS GIS Prototypes
slide38

Any GIS client

CubeView

Formida

Applications

(next phase)

client

Presentation Layer

Distributed Web interface

Page (OGC)

Nation Health Surveillance System

application server

Application Server Layer

(Currently in implementation

Data Server Layer

OGDI Interface and Drivers

foundation

spatial data

foundation

spatial data

Proprietary GIS

Vital statistics

Air quality data

format & vital

CSD & detailed

attribute

- SO

Aggregate data

2

Detailed data

stats & socio-

- Battery flares

economic

(Alberta Only)

CSD & EA

CSD, EA &

On Client/

Tunney's Pasture

CubeWerx Hull

Socio-Economic

Replicated

WIN/NT

Win/95

CubeWerx Hull

ODBC interface

Tunney's Pasture

Grasslands

MID/MIF and DBF

UNIX

UNIX

WIN/NT

OGDI interface

OGDI interface

OGDI interface

MID/MIF

MID/MIF

MID/MIF

slide39

Case 4: CHS Source DBFrom paper charts to digital data

From

  • Map sheet paradigm
  • Closed, single purpose, file based GIS applications
  • Single purpose internal

“cartographic” systems

  • Standards for data exchange
  • Moderate data volume

To

  • Continuous coverage
  • Open, object-relational DBMS with many end-user tools
  • Multi-purpose “decision support” systems
  • Standards for data access
  • Massive data volume
chs back office architecture flow

RealTime Data

Metacontent

Clean

Data

Data

Products

“Refined”

Source Data

Source

Data

CHS Back Office Architecture & Flow
slide41

Application

Access

Federated Data Warehouse

Interface Layer

  • Open Interface
  • Flexible Architecture
  • Expandable
  • Neutral
  • Rapid Access
  • Easy

GIS Tools

RealTime Data

Metacontent

Data

Products

Visualization Tools

Web Browser

“Refined”

Source Data

CHS Front Office: Open

Access to clients

metacontent management through the business life cycle
Metacontent Management through the Business Life cycle

Prospective

Metacontent

CUSTOMER

Marketing

Distribution

Planning

Production

Product Data Management

External

collection

Survey/

internal collection

Data Acquisition

Static and Dynamic

Metacontent

distributed process architecture

Halifax

St John’s

Montreal

Ottawa

Distributed process architecture
  • Remote collaboration handled through standard distributed database architecture and networking

Survey hydrographer

Validation hydrographer

Cartographer

Production Quality Control

case 5 geoide feature extraction and generalization linking industry and academia
CASE 5: GEOIDE Feature Extraction and GeneralizationLinking industry and academia

“develop a service to extract information from high resolution imagery using collaborative data and place it into an open distributed data architecture that can be used for such data analysis and warehousing capabilities as feature linking, generalization tools, aggregation, and visualization.”

goals for feature extraction
Goals for Feature Extraction

Images

Multi-resolution

Multispectral

Multisensor

User requirements / goals

  • Semi-automated Feature extraction
  • Line features
      • roads (CCRS)
      • rivers (UVIC)
      • coastlines (JOINT)
  • Area Features
      • lakes
      • islands
      • forest stands
      • agricultural areas etc

Operator

simple stream extraction algorithms complete
Simple stream extraction algorithms complete
  • The project is using commercially available imagery such as 4m KFA 100, 1m IKONOS (when available), and 1m digital orthophotos.
  • UVIC grad students have also developed remote sensing tools to capture necessary information.
dynamic multi scale update generalization visualization and analysis

Medium & Thick

OGDI Clients

Thin

Client

Medium & Thick

SQL Clients

Multiple datasets

Web

Server

Spatial Load Facility (FME & OGDI)

Data Cleansing & Topology Validation

& Quality Assurance

OGDI

Server

CubeSERV

Server

GeoMedia

CubeSTOR Server

Network Analysis

SQL Engine

Feature Linking

Service (FLS)

Alpha

Feature Extraction

Service

(UVIC-CCRS)

Generalization

Text

Cartridge

CubeSTOR

Spatial Cartridge

GenSystem

SQL std.

Cartridge

Other

Cartridges

Engine

Oracle Network Computing

Architecture

  • Spatial data
  • Attribute data
  • Meta data

Database

Files

Dynamic Multi-scale update, generalization, visualization and analysis
  • Seamless information
  • generalized products
  • automated data mining
proposed feature extraction deliverables
Proposed Feature Extraction Deliverables
  • The first phase of Feature Extraction to be completed by October 30, 1999 with the following deliverable:
    • Design and implement a working (not integrated) feature extraction demonstration
  • The second phase deliverables in early 2000 would include:
    • interfaces from the Feature Extraction service to the spatial data warehouse based on the CubeSTOR/Oracle implementation on a Unix platform
    • Population of the warehouse with selected metadata and data sets from potential partners including: LIDAR, SAR, VMAPX, VITD, PITD, ITD, classified satellite imagery, selected tactical information.
  • The third phase in late 2000 - early 2001 would:
    • Develop and demonstrate the data fusion enhancements for terrain analysis and extraction of higher-dimensional features
benefits of adding feature extraction to agmiid
Benefits of adding feature extraction to AGMIID

The enhancements would result in:

  • a feature extraction service for multi-resolution imagery
  • images, hi-res vector, and classified output in the data warehouse, accessible to other services (e.g., feature linking and terrain analysis
  • a distributed architecture for acquisition and update of data through a single portal
  • a suite of tools for data load, integration and retrieval, Web browsing, and report generation
  • The system must be designed to deliver information in a form readily accessible and useable by decision-makers.