data science with spotfire for opening government data for innovators and entrepreneurs n.
Download
Skip this Video
Loading SlideShow in 5 Seconds..
Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs PowerPoint Presentation
Download Presentation
Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs

Loading in 2 Seconds...

play fullscreen
1 / 46

Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs - PowerPoint PPT Presentation


  • 380 Views
  • Uploaded on

Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs. Dr. Brand Niemann Director and Senior Enterprise Architect – Data Scientist Semantic Community http://semanticommunity.info/ AOL Government Blogger http://gov.aol.com/bloggers/brand-niemann/

loader
I am the owner, or an agent authorized to act on behalf of the owner, of the copyrighted work described.
capcha
Download Presentation

PowerPoint Slideshow about 'Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs' - yitta


An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -
Presentation Transcript
data science with spotfire for opening government data for innovators and entrepreneurs

Data Science With Spotfire for Opening Government Data for Innovators and Entrepreneurs

Dr. Brand Niemann

Director and Senior Enterprise Architect – Data Scientist

Semantic Community

http://semanticommunity.info/

AOL Government Blogger

http://gov.aol.com/bloggers/brand-niemann/

February 18, 2012

tibco spotfire 4 01
TIBCO Spotfire 4.0
  • The Top 5 Reasons Why Spotfire Analytics is Better and Smarter:
  • Clarity of Visualization
  • Freedom of Spreadsheets
  • Relevance of Applications
  • Confidence of Statistics
  • Reach of Reports

http://spotfire.tibco.com/

tibcosilver spotfire features matrix
TIBCOSilverSpotfire Features Matrix

Free

https://silverspotfire.tibco.com/us/get-spotfire/feature-matrix

tibcosilver spotfire tutorials
TIBCOSilver Spotfire Tutorials

https://silverspotfire.tibco.com/us/tutorials

my silver spotfire data science library in the cloud
My Silver SpotfireData Science Library in the Cloud

I will show you examples

of how I built these later.

Federal Budget 2013

in a day!

Data Science Library in the Cloud

federal budget 2013 dashboard1
Federal Budget 2013 Dashboard

Silver Spotfire Web Player

the value proposition of spotfire
The Value Proposition of Spotfire
  • More to Do with Less?
    • Take Control of Your Business Data
    • Visualize Your Data - Drag and Drop Your Spreadsheets
    • Customize Your Dashboards - Instantly Add New Visualization
    • Share Your Insights - Publish Your Dashboard
    • Get Trial of Silver Spotfire
  • Agile Analysis:
    • Fastest to Actionable Insight
    • Insight Into the Unknown
    • Self-Service Discovery
    • Universal Analytics Platform

Source: https://silverspotfire.tibco.com/us/home

Source: http://www.gartner.com/technology/reprints.do?id=1-196U0P5&ct=120207&st=sg

the value proposition of agile analysis invert your bath tub with spotfire analytics
The Value Proposition of Agile Analysis - Invert your "bath tub" with Spotfire Analytics

Spotfire offers dimension-free data exploration, data mashups, predictive and event driven, contextual collaboration and enterprise class technology.

Source: Jim Hawley, Spotfire Federal Government

spotfire is part of tibco
Spotfire is Part of TIBCO

Video Presentation

the value proposition of data science
The Value Proposition of Data Science
  • We are interested in learning about Taxonomy and Enterprise Vocabulary for fundamental architectural elements to enable interoperability and provide consistent understanding of shared architecture information across the enterprise.
    • Source: Walt Okon, Senior Architect Engineer, Enterprise Architecture & Standards, Department of Defense Chief Information Officer, October 4th, Email.
  • Aneesh Chopra: Government’s Big Data Opportunity. “The Federal Government needs Data Science and Data Scientists!”
    • Source: O’Reilly STRATA Conference New York, September 20, 2011.
what is data science
What is Data Science?
  • Data science enables the creation of data products.
  • Data science is a holistic approach.
  • The first step of any data analysis project is “data conditioning,” or getting the data into a state where it is usable.
  • Statistics is the “grammar of data science.”
  • Edward Tufte’s Visual Display of Quantitative Information is a foundational text for anyone practicing data science. He calls himself a data scientist!
  • Data scientists are patient, inherently interdisciplinary, and can think outside the box.
  • Some References:
    • Data Science Graduate Class at RPI, Troy, NY
    • Data Science
    • AOL Government
data science architecture
Data Science Architecture
  • 1. Create an inventory of documents and data sets.
  • 2. Build that inventory in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard.
  • 3. Provide a sample knowledgebase of each of the four types of documents (Word, PDF, PowerPoint, and Excel).
  • 4. Provide the multiple sample knowledgebases in a Spotfire dashboard so they can be seen, compared, merged, harmonized, sorted, searched, downloaded, and shared on mobile devices (e.g. iPad).
  • 5. Scale the previous architectural pattern with more content volume and types if necessary.
knowledgebase
Knowledgebase
  • What is a knowledgebase?
    • Knowledgebase = Model + Instances
    • Model = Vocabulary, Taxonomy, and Ontology/Rules
    • Instances = Linked Data Semantically Linked to the Model
  • How is a knowledgebase built?
    • Model = Vocabulary – Glossary in MindTouch
    • Taxonomy – Contents and Resources in MindTouch
    • Ontology/Rules in Be Informed 4
    • Instances = Linked Data Semantically Linked to Model – MindTouch, Excel and Spotfire
the knowledgebase in mindtouch
The Knowledgebase in MindTouch
  • MindTouch is often referred to as the “Swiss Army Knife” of collaboration tools! See MindTouch Web Site.
  • So I make MindTouch look like a “Knowledge Hub” (e.g., on top of SharePoint Portal like the Army Corps of Engineers Knowledge Hub) and feature key documents and data sets.
  • Relating one or more Spotfire dashboards to the key document and data sets points to the ability to track progress. It’s all about metrics!
mindtouch social knowledge base social help center
MindTouch Social Knowledge Base – Social Help Center

MindTouch provides exceptional, purpose-built social help desks and knowledge bases for some of the world’s largest and most respected technology and media brands. Our solutions layer social and collaborative capabilities over existing systems and deliver strategic value to our customers. Product help is strategic for user assistance teams, product and marketing teams, community managers, and product evangelists as they look to build engaged communities around their brands to increase top and bottom line revenues.

http://www.mindtouch.com/solutions/knowledge_base

army corps of engineers knowledge hub
Army Corps of Engineers Knowledge Hub
  • The Knowledge Hub is a dynamic online destination to feature products developed by the US Army Corps of Engineers as well as to engage end-users and others in innovative and intuitive interaction. Within the Knowledge Hub is a Navigation Community which provides a forum on which navigation personnel can discuss, share, learn, explore and search products, project and programs of concern them. One goal of the Hub is to be a web-based framework for enterprise decision support and tech transfer within the Corps of Engineers.
    • POC: Marty Kittrell, Martin.C.Kittrell@usace.army.mil.

Source: http://chl.erdc.usace.army.mil/Media/1/2/2/0/Nav_eNews_Mar-2011.pdf

mindtouch knowledgebase
MindTouch Knowledgebase

AOL Government Story

Spotfire Dashboard

Research Notes (Metadata)

Complete Budget Document

Attachments (see next slide)

Comments (see next slide)

http://semanticommunity.info/Budget_of_the_United_States_Government_Fiscal_Year_2013

mindtouch knowledgebase1
MindTouch Knowledgebase

http://semanticommunity.info/Budget_of_the_United_States_Government_Fiscal_Year_2013

data science is part of my system of systems architecture
Data Science is Part of My System of Systems Architecture

Dynamic Case Management (e.g. Be Informed)

Data Science Library (e.g. Spotfire)

Data Science Products (e.g. Spotfire)

S

Semantic Index of

Linked Data (e.g. Excel)

agile methods questions on our minds
Agile Methods:Questions on Our Minds
  • What Should We Do with Enterprise Architecture?
    • Be like a building architect that provides a blueprint with building specifications and a scale (able) model.
  • How Should We Do That?
    • With Be Informed, an internationally operating, independent software vendor that has been recognized recently by Gartner and Forrester.
  • What is Be Structured?
    • It is complimentary to various well-known development, compliance and architecture frameworks, including ITIL, Cobit, Prince II, RUP, TOGAF, Zachman, SCRUM, Cogniam, DEMO, and Pronto. Note: See my tutorials.
working within a broader context
Working Within A Broader Context
  • Begin with the End in Mind (see Next Slide):
    • Open Innovator's Toolkit
      • President Obama emphasizes a “bottom-up” philosophy that taps citizen expertise to make government smarter and more responsive to private sector demands. This philosophy of “open innovation” has already delivered tangible results in public and regulated sectors of the economy – areas like health IT, learning technologies, and smart grid – that are poised to deliver productivity growth and grow the jobs of the future. We have surfaced new or improved policy tools deployed by our government to achieve them. We’ve posted the Open Innovator’s Toolkit as a roster of 20 leading practices that an “open innovator” should consider when confronting any policy challenge – at any level of government. Our aspiration is to build upon this list, adding new tools and case studies to form an evidence base that will help to scale “open innovation” across the public sector.
  • Follow 5 Easy Steps:
    • 1. Build an table of contents-like index of complex documents with well-defined web addresses in MindTouch.
    • 2. Build that index in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard.
    • 3. Build a Spotfire knowledgebase with that Excel spreadsheet.
    • 4. Build multiple knowledgebases in a Spotfire dashboard so they can be seen, compared, merged, harmonized, sorted, searched, downloaded, and shared on mobile devices (e.g. iPad).
    • 5. Scale the previous architectural pattern with more content volume and types if necessary.
open government initiative opening data for innovators and entrepreneurs
Open Government Initiative:Opening Data For Innovators and Entrepreneurs

Our aspiration is to build upon this list, adding new tools and case studies to form an evidence base that will help to scale “open innovation” across the public sector.

http://www.whitehouse.gov/open/toolkit

slide25
Step 1. Build an table of contents-like index of complex documents with well-defined web addresses in MindTouch.

http://semanticommunity.info/AOL_Government/Open_Innovator's_Toolkit-Taking_the_Challenge

2 build that index in an excel spreadsheet so it supports faceted search in a spotfire dashboard
2. Build that index in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard.

Note: This MindTouch table

copies directly to Excel in the

next slide.

http://semanticommunity.info/AOL_Government/Open_Innovator's_Toolkit-Taking_the_Challenge#Data_Table_(Excel)

2 build that index in an excel spreadsheet so it supports faceted search in a spotfire dashboard1
2. Build that index in an Excel spreadsheet so it supports faceted search in a Spotfire dashboard.

http://semanticommunity.info/@api/deki/files/17378/=OpenInnovator'sToolkitTable02182012.xlsx

3 build a spotfire knowledgebase with that excel spreadsheet1
3. Build a Spotfire knowledgebase with that Excel spreadsheet.
  • Building Steps:
    • 1 - Drag and Drop Spreadsheet Onto Spotfire (see Scatter Plot automatically).
    • 2 - Add New Table to Display Spreadsheet Data (make any adjustments/corrections and Refresh Data).
    • 3 - Adjust Scatter Plot Axes, Color by, Shape by, Size by to produce desired display.
    • 4 - Add New Test Area, Rename Page as Dashboard and Add MindTouch and Excel with Web links to sources of metadata and data.
    • 5- Insert Action Controls to Reset All Filters and Unmark Marked Rows.
    • 6 - Save Spotfire file to hard drive with desired name and then save to Library.
    • 7 - Test Web Player version and embed in MindTouch.
follow 5 easy steps
Follow 5 Easy Steps
  • Step 4. Build multiple knowledgebases in a Spotfire dashboard so they can be seen, compared, merged, harmonized, sorted, searched, downloaded, and shared on mobile devices (e.g. iPad).
    • Another example: How To Simplify Benefits Website For Veterans (AOL Government, MindTouch, Excel, Spotfire, and PowerPoint Tutorial).
  • Step 5. Scale the previous architectural pattern with more content volume and types if necessary.
    • My Silver Spotfire Library in the Cloud!
new features in spotfire 4 0
New Features in Spotfire 4.0

http://stn.spotfire.com/stn/Site/News40.aspx

new features in spotfire 4 01
New Features in Spotfire 4.0
  • Information At A Glance:
    • Dynamic Values
    • Conditional Icons
    • Sparklines
    • Graphical Summary Table
  • Look and Feel:
    • All New Graphical Profile
    • Pop-Over Filter Panel
    • Pop-Over Legend
    • Individual Control Over Axis Label Visibility
    • More Control Over Legend Contents and Placement
    • Fixed Size Layout
    • Mix Filters and Controls on the Page
    • Nicer Looking Tables
    • Combine Different Slices of Data on the Same Page
    • Toolbars and Information
new features in spotfire 4 0 continued
New Features in Spotfire 4.0(Continued)
  • Navigation and Interaction:
    • Actions
    • Page History Navigation
    • Embed Interactive Controls
  • Building Dashboards:
    • Preserve Information When Switching Visualizations
    • Change All Fonts in One Place
    • Easier Access to Toggling Visualization Features
    • More Predefined Categorical Coloring Schemes
    • Manage Document Color Schemes
    • Better Defaults When Creating Visualizations
    • Toggle Auto Column Additions Off
    • Analysis Previews
    • Control Over Table Header Font
new features in spotfire 4 02
New Features in Spotfire 4.0
  • Collaboration:
    • Share with TIBBR
    • Add TIBBR Discussions to the Analysis
    • Embed Dashboards in Other Web Pages
  • Other Enhancements:
    • Export Footer
    • Stepped Linecharts
    • Automation Services 4.0
new features in spotfire 4 03
New Features in Spotfire 4.0
  • Information At A Glance:
    • Dynamic Values:
      • What – Dynamically display single values in text areas that responds to filtering and parameter changes.
      • Why – Look at most important numbers first before diving into more details.
      • My Note: See Next Slides.
    • Conditional Icons:
      • What – Dynamically calculated conditional icons that respond to filtering and parameter changes.
      • Why – Indicate change, comparisons to target and highlight important events.
      • My Note: See Next Slides.
    • Sparklines:
      • What – Dynamically calculated sparklines that respond to filtering and parameter changes.
      • Why – Show at a glance and when drilling in whether a metric is trending down, up or varies a lot.
      • My Note: See Next Slides.
    • Graphical Summary Table:
      • What – Dynamic values, conditional icons and sparklines in one compact table broken down by some category.
      • Why – Visually show everything you need on a single screen.
      • My Note: See Next Slides.
slide37
New Features in Spotfire 4.0:Dynamic Values, Conditional Icons, Sparklines, and Graphical Summary Table

PC Desktop Spotfire

slide38
New Features in Spotfire 4.0:Dynamic Values, Conditional Icons, Sparklines, and Graphical Summary Table

Filter for

Debt Service

PC Desktop Spotfire

new features in spotfire 4 04
New Features in Spotfire 4.0
  • Collaboration:
    • Share with TIBBR:
      • What – Right Click on any visualization or page and share the view in tibbr with a link back to the analysis
      • Why – Easy sharing of insights and findings.
      • My Note: Tibbr host name has to be set by Administrator.
    • Add TIBBR Discussions to the Analysis:
      • What – Integrated tibbr discussions right in the analysis filtered to a particular subject.
      • Why – Discuss insights and findings with colleagues directly in the analysis. Subscribe and get notified when someone posts a comment on an analysis you are interested in.
      • My Note: See Next Slide.
    • Embed Dashboards in Other Web Pages:
      • What – One click access to HTML fragments that displays a Spotfire page that can be pasted directly into portals and other web pages.
      • Why – Put a link to the analysis in your corporate blog or wiki. Integrate Spotfire analysis displays into SharePoint WebPart and other portals.
      • My Note: I was already doing this!
new features in spotfire 4 05
New Features in Spotfire 4.0
  • Other Enhancements:
    • Export Footer:
      • What – Include a footer when exporting or printing pages.
      • Why – Make it clear where the printout came from or indicate to the reader that the contents is confidential.
      • My Note: See Next Slide.
    • Stepped Linecharts:
      • What – Draw stepped linecharts that only show a change in value at the exact point where the value changed.
      • Why – Better representation of discrete data that avoids misleading the user by interpolating values in between data points.
      • My Note: See Next Slide.
    • Automation Services 4.0:
      • What – New task added to remap Information Services catalogs and schemas during an automated Library import.
      • What – Allow for the automation of migrating a Spotfire Information Model from a test to production environment in instances when the test and production instances of the data source are in different database catalogs or schemas.
      • My Note: See Slides That Follow.
new features in spotfire 4 0 other enhancements automation services 4 0
New Features in Spotfire 4.0Other Enhancements: Automation Services 4.0

TIBCO Spotfire Automation Services

Selecting a "Set data source credentials" task in the job builder will now allow you to go back and select a different certificate if the first one selected is invalid.

http://stn.spotfire.com/stn/Site/News.aspx

new features in spotfire 4 0 other enhancements automation services 4 01
New Features in Spotfire 4.0Other Enhancements: Automation Services 4.0

http://stn.spotfire.com/stn/Platform/InformationServices.aspx

new features in spotfire 4 0 other enhancements automation services 4 02
New Features in Spotfire 4.0Other Enhancements: Automation Services 4.0

My Note: Customize Spotfire Documentation in MindTouch

http://semanticommunity.info/Build_DoD_in_the_Cloud/Enterprise_Information_Web_for_Semantic_Interoperability_at_DoD/Spotfire_Information_Designer