National datawarehouse for traffic information big data supplier
Download
1 / 24

National Datawarehouse for Traffic Information – Big Data supplier - PowerPoint PPT Presentation


  • 75 Views
  • Uploaded on

National Datawarehouse for Traffic Information – Big Data supplier. Els Rijnierse. Contents. Introducing NDW Experiences with our big data Challenges, choices and changes. Posting.

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about ' National Datawarehouse for Traffic Information – Big Data supplier' - conley


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript

Contents
Contents

Introducing NDW

Experiences with our big data

Challenges, choices and changes


Posting
Posting

The last slide will ask you to post your impression, to share what struck you most with all conference attendees


Ndw is a collaborative venture
NDW is a Collaborative venture

24 Road authorities

National

6 out of 12 provinces

Cities, either independent or in an alliance

Covering >6000 km road network(total Dutch road network is 130.000 km)

Introducing NDW


What is our aim
What is our aim?

Develop and maintain a joint database for traffic data. Up-to-date, complete and unambiguous with known quality

Create efficiency by working together and sharing information

Stimulate effective use of this data for:- real time traffic management - real time traffic information - analyses, policy making and research

Introducing NDW


Trafficmanagement
Trafficmanagement

Central source for all road authorities

Introducing NDW


Objectives
Objectives

Less traffic jams

Predictability

Safer roads

Less emission

More collaboration

Data voor doorstroming


Happy road users
Happy road users

Introducing NDW


Supervisory Board

Accountability

Supervision

Selection from

data

Common data

need

Supply

NDW

Demand

Participatinggoverments (IDP)

Roadauthorities

Individual data

supply

Individual

Data need

Commercial parties(EDP)

Service providers

Infrastructure

supply

System provider (external)

Introducing NDW


Data types 1
Data types - 1

Traffic flow per lane per vehicle class on 14818 measuring sites

Travel time (realised or estimated) per lane on 9424 measuring sites

Traffic speed per lane per vehicle class on 13410 measuring sites

(measuring sites may produce more kind of data)

Every minute, traffic data from more than 24,000 measuring sites is collected, processed and within 75 seconds distributed to the users

Introducing NDW


Data collection
Data collection

Introducing NDW


Some figures on figures
Some figures on figures

Over 24,000 measurement sites

Giving aprox. 460,000 figures on speed, flow and travel time each minute

=> >27 Million per hour

=> >600 million per day

=> >240 billion per year

+ meta data on these figures


Real time traffic data february 2012 5 cm snow
Real-time traffic data (February 2012: 5 cm snow)

Introducing NDW


Data types 2
Data types - 2

Road works, planned and actual

Reports of congestion and accidents

Status (open/closed) of bridges

Near future: Status (open/closed) of peak lanes and regular lanes

On occurrence data on availability of the road is collected

Introducing NDW


Cooperation between cbs en ndw
Cooperation between CBS en NDW

NDW collects and distributes raw data, we do not aim to do any statistical analysis.

CBS started with small NDW datasets (1 day) and is now working on a larger set (3 months) to determine new methodology

Conclusion:

Forget everything you learned about statistics

Experiences


When to start calculating experiences with big data 1
When to start calculating (Experiences with big data – 1)

When using big data:

This traditional way of working does not produce statistics quicker.

This requests huge datastores for raw data storage

Strongly advised is starting with statistical analyses the moment data is streaming in and storing only aggregated in between results

Adapt you algorithms to be able to handle correct any unpredictable gaps in the raw data that will occur

Traditional statistical methodology:

gather and store everything and perform the statistical analyses on certain times.

Experiences


Technical issues experiences with big data 2
Technical issues (Experiences with big data - 2)

Traditional relational databases but also statistical tools (SPSS/SAS/R) are not fast enough, run far out of memory and do not have enough performance for quick retrieval of raw data.

When using a data storage technique suitable for fast recovery of raw data then some coding and programming has to be done on the raw data.

Recalculating because of wrong choices or methods takes an increasing amount of time as the amount of raw data grows quickly every day.

Experiences


Challenges choices changes
Challenges, Choices, Changes

Devils Triangle

Contents awareness

Statistical knowledge

IT knowledge

Challenges, Choices, Changes


Challenges choices changes1
Challenges, Choices, Changes

Challenge

Government policy is that public data are open data, which means our raw data are on the WWW (www.ndw.nu/datalevering)

Anybody can download them and produce surveys, statistics, tables, draw conclusions and publish these (long) before statistical office does.

Be aware of publicity this might cause, discussions on ‘the truth’ and the status of a response or statement from the statistical office.

Take on the challenge of producing real time statistics

Challenges, Choices, Changes


Challenges choices changes2
Challenges, Choices, Changes

Choice

Traditional storage of raw data used for statistics is at thestatistical office.

Big data should be left at their origin and withdrawn when needed.

Challenges, Choices, Changes


Challenges choices changes3
Challenges, Choices, Changes

Change

Look for appropriate IT infrastructure and develop a new way of handling data

Challenges, Choices, Changes


www.sendsteps.com

Prepare to react; keep your phone ready!

Internet

Go to sendc.com

1

Log in with Session

2

Type WS3 <space> your answer

3

TXT

Text to +316 4250 0030

1

Type Session <space> WS3 <space> your answer

2

Posting messages is anonymous

No additional charge per message



Challenges changes and choices when using these amounts for statistics
Challenges, Changes and Choices when using these amounts for Statistics

Forget everything you learned on statistics: How to produce 1 representative figure

on traffic intensity from this:


ad