Next Generation 4D Distributed Modeling and Visualization
1 / 57

Automated 3D Model Construction for Urban Environments - PowerPoint PPT Presentation

  • Uploaded on

Next Generation 4D Distributed Modeling and Visualization. Automated 3D Model Construction for Urban Environments. Christian Frueh John Flynn Avideh Zakhor. University of California, Berkeley. June 13, 2002. Presentation Overview. Introduction Ground based modeling Mesh processing

I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
Download Presentation

PowerPoint Slideshow about 'Automated 3D Model Construction for Urban Environments' - troy-walls

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.

- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
Automated 3d model construction for urban environments

Next Generation 4D Distributed Modeling and Visualization

Automated 3D Model Construction for Urban Environments

Christian Frueh John Flynn Avideh Zakhor

University of California, Berkeley

June 13, 2002

Presentation overview
Presentation Overview

  • Introduction

  • Ground based modeling

    • Mesh processing

  • Airborne modeling

    • Aerial photos

    • Airborne laser scans

  • 3D Model Fusion

  • Rendering

  • Conclusion and Future Work


Goal: Generate 3D model of a city for virtual walk/drive/fly-thrus and simulations

  • Fast

  • Automated

  • Photorealistic


For Fly-Thru:

For Walk/Drive-Thru:

  • 3D Model of terrain and buildings tops & sides

  • coarse resolution

  • 3D model of street scenery & building façades

  • highly detailed


Airborne Modeling

Ground Based Modeling

  • Laser scans/images from plane

  • Laser scans & images from acquisition vehicle


Complete 3D City Model


3D Model of terrain and building tops

3D model of building façades

Airborne modeling
Airborne Modeling

Acquisition of terrain shape and top-view building geometry


  • Available Data:

  • Aerial Photos

  • Airborne laser scans

Texture: from aerial photos

Geometry: 2 approaches:

I) stereo matching of photos

II) airborne laser scans

Airborne modeling1
Airborne Modeling

Approach I : Stereo Matching

(last year)

Stereo photo pairs from city/urban areas, ~ 60% overlap




  • Segmentation

  • Camera parameter computation,

  • Matching,

  • Distortion reduction,

  • Model generation

Stereo matching
Stereo Matching

Stereo pair from downtown Berkeley and the estimated disparity after removing perspective distortions

Stereo matching results
Stereo Matching Results

Downtown Oakland

Airborne modeling2
Airborne Modeling

Approach II: Airborne Laser Scans

Scanning city from plane

  • Resolution 1 scan point/m2

  • Berkeley: 40 Million scan points

point cloud

Airborne laser scans
Airborne Laser Scans

  • Re-sampling point cloud

  • Sorting into grid

  • Filling holes

Map-like height field

usable for:

  • Monte Carlo Localization

  • Mesh Generation

Textured mesh generation
Textured Mesh Generation

1. Connecting grid vertices to mesh

2. Applying Q-slim simplification

3. Texture mapping:

  • Semi-automatic

  • Manual selection of few correspondence points: 10 mins/entire Berkeley

  • Automated camera pose estimation

  • Automated computation of texture for mesh

Airborne model
Airborne Model

East Berkeley campus with campanile

Airborne model1
Airborne Model

Downtown Berkeley

Ground based modeling
Ground Based Modeling


2D laser







Acquisition of highly detailed 3D building façade models


  • Scanning setup

  • vertical 2D laser scanner for geometry capture

  • horizontal scanner for pose estimation

  • Acquisition vehicle

  • Truck with rack:

  • 2 fast 2D laser scanners

  • digital camera

Scan matching initial path computation

(ui, vi, i)

(ui-1, vi-1, i-1)

(u2, v2, 2)

(u1, v1, 1)

Scan Matching & Initial Path Computation

Horizontal laser scans:

  • Continuously captured during vehicle motion

  • Overlap

Relative position estimation by scan-to-scan matching

Translation (u,v)

Rotation 

Adding relative steps (ui, vi, i)

t = t0


t = t1

(u, v)

path (xi,yi,i)

Scan matching

3 DOF pose (x, y, yaw)

6 dof pose estimation from images
6 DOF Pose Estimation From Images

  • Scan matching cannot estimate vertical motion

    • Small bumps and rolls

    • Slopes in hill areas

  • Full 6 DOF pose of the vehicle is important; affects:

    • Future processing of the 3D and intensity data

    • Texture mapping of the resulting 3D models

  • Extend initial 3 DOF pose by deriving missing 3 DOF (z, pitch, roll) from images

6 dof pose estimation from images1
6 DOF Pose Estimation From Images

Central idea: photo-consistency

  • Each 3D scan point can be projected into images using initial 3 DOF pose

  • If pose estimate is correct, point should appear the same in all images

  • Use discrepancies in projected position of 3D points within multiple images to solve for the full pose

6 dof pose estimation algorithm
6 DOF Pose Estimation – Algorithm

  • 3DOF of laser as initial estimate

  • Project scan points into both images

  • If not consistent, use image correlation to find correct projection

  • Ransac used for robustness

6 dof pose estimation results
6 DOF Pose Estimation – Results

with 3 DOF pose

with 6 DOF pose

Monte carlo localization 1
Monte Carlo Localization (1)

Previously: Global 3 DOF pose correction using aerial photography

a) path before MCL correction

b) path after MCL correction

After correction, points fit to edges of aerial image

Monte carlo localization 2
Monte Carlo Localization (2)

Extend MCL to work with airborne laser data and 6 DOF pose


No perspective shifts of building tops, no shadow lines

  • Fewer particles necessary, increased computation speed

  • Significantly higher accuracy near high buildings and tree areas

Use terrain shape to estimate z coordinate of truck

  • Correct additional DOF for vehicle pose (z, pitch, roll)

  • Modeling not restricted to flat areas

Monte carlo localization 3
Monte Carlo Localization (3)

Track global 3D position of vehicle to correct relative 6 DOF motion estimates

Resulting corrected path overlaid with airborne laser height field

Path segmentation

Path Segmentation

24 mins, 6769 meters

vertical scans:107,082

scan points: ~ 15 million

Too large to process as one block!

Path segmentation1
Path Segmentation

Resulting path segments overlaid with edges of airborne laser height map

Simple mesh generation1

Side views look “noisy”

Remove foreground: extract facades

Simple Mesh Generation


Point cloud


  • Problem:

  • Partially captured foreground objects

  • erroneous scan points due to glass reflection

Fa ade extraction and processing 1

split depth

main depth

local minimum

2. Histogram analysis over vertical scans

split depth

main depth

depth value sn,υfor a scan point Pn,υ



scan nr

ground points


Façade Extraction and Processing (1)

1. Transform path segment into depth image

Fa ade extraction and processing 2
Façade Extraction and Processing (2)

3. Separate depth image into 2 layers:

foreground=trees, cars etc.

background=building facades

Fa ade extraction and processing 3
Façade Extraction and Processing (3)

4. Process background layer:

  • Detect and remove invalid scan points

  • Fill areas occluded by foreground objects by extending geometry from boundaries

    • Horizontal, vertical, planar interpolation, RANSAC

  • Apply segmentation

  • Remove isolated segments

  • Fill remaining holes in large segments

  • Final result: “clean” background layer

Fa ade extraction examples 1
Façade Extraction – Examples (1)

with processing

without processing

Fa ade extraction examples 2
Façade Extraction – Examples (2)

without processing

with processing

Fa ade extraction examples 3
Façade Extraction – Examples (3)

without processing

with processing

Mesh generation
Mesh Generation

Downtown Berkeley

Automatic texture mapping 1
Automatic Texture Mapping (1)

Camera calibrated and synchronized with laser scanners

Transformation matrix between camera image and laser scan vertices can be computed

1. Project geometry into images

2. Mark occluding foreground objects in image

3. For each background triangle:

Search pictures in which triangle is not occluded, and texture with corresponding picture area

Automatic texture mapping 2

Typical texture reduction: factor 8..12

Automatic Texture Mapping (2)

Efficient representation: texture atlas

Copy texture of all triangles into “mosaic” image

Automatic texture mapping 3

Texture synthesis: preliminary

  • Mark holes corresponding to non-textured triangles in the atlas

  • Search the image for areas matching the hole boundaries

  • Fill the hole by copying missing pixels from these image

Automatic Texture Mapping (3)

Large foreground objects: Some of the filled-in triangles are not visible in any image!

“texture holes” in the atlas

Automatic texture mapping 4

Texture holes filled

Automatic Texture Mapping (4)

Texture holes marked

Ground based modeling results
Ground Based Modeling - Results

Façade models of downtown Berkeley

Ground based modeling results1
Ground Based Modeling - Results

Façade models of downtown Berkeley

Model fusion
Model Fusion

Fusion of ground based and airborne model to one single model


Façade model

Airborne model

Model Fusion:

  • Registration of models

  • Combining the registered meshes

Registration of models

Which model to use where?

Registration of Models

Models are already registered with each via Monte-Carlo-Localization !

Preparing ground based models
Preparing Ground Based Models

Intersect path segments with each other Remove degenerated, redundant triangles in overlapping areas

original mesh

redundant triangles removed

Preparing airborne model
Preparing Airborne Model

Ground based model has 5-10 times higher resolution

  • Remove facades in airborne model where ground based geometry is available

  • Add ground based façades

  • Fill remaining gaps with a “blend mesh” to hide model transitions

Preparing airborne model1
Preparing Airborne Model

Initial airborne model

Preparing airborne model2
Preparing Airborne Model

Remove facades where ground based geometry is available

Combining models
Combining Models

Add ground based façade models

Combining models1
Combining Models

Fill remaining gaps with a “blend mesh” to hide model transitions


Ground based models:

  • Up to 270,000 triangles, 20 MB texture per path segment

  • 4.28 million triangles, 348 MB texture for 4 downtown blocks

  • Difficult to render interactively!

  • Subdivide model and create multiple level-of-details (LOD)

  • Generate scene graph, decide which LOD to render when

Multiple lods for fa ade meshes
Multiple LODs for façade meshes

Highest LOD

  • Qslim mesh simplification

  • Texture subsampling

Lower LOD

  • Geometry: 10%

  • Texture: 25%

of original mesh

Fa ade model subdivision for rendering
Façade Model Subdivision for Rendering

Subdivide 2 highest LODs of façade meshes along cut planes





Path segment




Global scene





Interactive rendering
Interactive Rendering

Downtown blocks: Interactive rendering with web-based browser!

Future work
Future Work

  • Resolution enhancement and post-processing of LIDAR data

  • Devise new data acquisition system and algorithms to capture

    • both sides of street simultaneously

    • texture for upper parts of tall buildings

  • Include foreground objects in model

  • Add temporal component to dynamically update models

  • Compact representation

  • Interactive rendering