1 / 31

Ensuring Compliance of Patient Data with Big Data and BI

Ensuring Compliance of Patient Data with Big Data and BI. Ayad Shammout & Denny Lee. Please silence cell phones. Agenda. A Quick Big Data Primer Healthcare and Big Data Compliance and Auditing SQL Compliance Project Compliance and Auditing with Big Data and BI

lazaro
Download Presentation

Ensuring Compliance of Patient Data with Big Data and BI

An Image/Link below is provided (as is) to download presentation Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author. Content is provided to you AS IS for your information and personal use only. Download presentation by click this link. While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server. During download, if you can't get a presentation, the file might be deleted by the publisher.

E N D

Presentation Transcript

  1. Ensuring Compliance of Patient Data with Big Data and BI AyadShammout & Denny Lee

  2. Please silence cell phones

  3. Agenda A Quick Big Data Primer Healthcare and Big Data Compliance and Auditing SQL Compliance Project Compliance and Auditing with Big Data and BI Big Data: Unstructured Volumes of Data Analytics: PowerPivot, Power View

  4. What is Big Data? Volume Exceeds physical limits of vertical scalability Velocity Decision window small compared to data change rate Variety Many different formats makes integration expensive Variability Many options or variable interpretations confound analysis

  5. Dataexplosion 10x increase every five years By 2015, organizations that build a modern information management system will outperform their peers financially by 20 percent. • – Gartner, Mark Beyer “Information Management in the 21st Century” 85% from new data types Volume Velocity Variety Cheap, Distributed Storage & Processing Hadoop Easy Accessibility of External Data Cloud

  6. Large Data Volumes New Data Sources New Technologies Non-traditional Data Types New Economics New Questions & New Insights

  7. Big Data Business Value 140,000-190,000 more deep analytical talent positions 15 out of 17 sectors in the US have more data stored per company than the US Library of Congress 1.5 million more data savvy managers in the US alone €250 billion Potential annual value to Europe’s public sector 50-60% increase in the number of Hadoop developers within organizations already using Hadoop within a year $300 billion Potential annual value to US healthcare

  8. Data • becomes the new currency

  9. Hadoop: The most visible face of Big Data MapReduce Layer Task tracker Task tracker HDFS Layer Job tracker Name node Data node Data node

  10. HDInsight: Visit HadoopOnAzure.com

  11. Healthcare and Big Data

  12. Healthcare and IT Often the laggard in technology Yet application of IT to healthcare can radically change what we can do Genomic Sequencing Proteomic sequencing Incidence Prediction

  13. Healthcare Big Data Example Scenarios Clinical Trial Deviations Originally Viagra was developed to lower blood pressure and treat Angina Now its used to help newborn pulmonary hypertension and altitude sickness Incidence Prediction Missed 4 or more visits, twice as likely to have an asthmatic incident Particular Cardiac monitor sine wave points to highly likelihood of heart attack Campaigns Social media and advertising campaigns to understand user behavior and sentiment Patient Satisfaction Social media and advertising campaigns to understand user behavior and sentiment

  14. BIDMC Auditing Scenario Auditing is critical component HIPAA in ensuring patient privacy 1 Billion rows+ of audit data 146 mission critical clinical applications Comprehensive audits yield 300-500k transactions/day HIPAA requires audit system with 20 years of data Auditing Project Available to community as part of Compliance SDK Updating for SQL Server 2012, HDInsight, Power View, and MobileBI* Creating an enterprise tool for consolidated storage, reporting and alerting of all application audit data - that's cool! John Halamka’s Cool Technology of the Week (Wellsphere Top Health Blogger, Health Impact Award)

  15. BIDMC Compliance Project Use Excel 2013 PowerPivot and Power View HDInsight Azure HDInsight Windows SSAS (tabular) SQL Server 2008/2012 ETL Logs to HDFS Audit Logs

  16. Auditing Sensitive Information

  17. Storage Infrastructure Audit Logs Transfer files to ASV via AzCopy, CloudExplorer, etc.

  18. Storage Infrastructure Azure Storage Vault (ASV) Azure Blob Storage Azure Flat Network Storage Hadoop on Azure Compute Nodes (Medium VMs)

  19. Storage Infrastructure Push data Back to Storage Azure Storage Vault (ASV) Azure Blob Storage Stream data To compute Azure Flat Network Storage Hadoop on Azure Compute Nodes (Medium VMs) map sort shuffle reduce http://dennyglee.com/2013/03/18/why-use-blob-storage-with-hdinsight-on-azure/

  20. SSIS to HDInsight

  21. SSIS Processing

  22. SSAS Tabularof HoAAudit Data

  23. Hadoop/ Auditing: File sizes • Currently testing gz vs. raw • E.g. 12MB raw text file vs. 633Kb gz file (~20x compression) • 20x smaller size, ~same query time • Approx same map / reduce task utilization • File Size is 250MB-1GB • SSIS package takes care of the size • Future testing: avro, protobuf

  24. Hadoop / Auditing: Formats • For ease of processing, replace carriage returns within embedded SQL statements, e.g. • select col1, col2 • from tableA • to • select col1, col2 from tableA • This allows you to create a Hive table using CR as row delimiter (i.e. does not have things like SQL quoted identifiers)

  25. BI Connectivity SQOOP, HiveODBC, Templeton, CSV, etc

  26. Big Data … Excel-lerated! 2 Server, 3mo 110 GB binary files SSIS extraction 1.2GB of text 120MB gz Hadoop to PowerPivot 6MB

  27. PowerPivot workbook of HoA Audit data

  28. Power View of HoA Audit Data

  29. Win a Microsoft Surface Pro! Complete an online SESSION EVALUATION to be entered into the draw. Draw closes April 12, 11:59pm CTWinners will be announced on the PASS BA Conference website and on Twitter. Go to passbaconference.com/evalsor follow the QR code link displayed on session signage throughout the conference venue. Your feedback is important and valuable. All feedback will be used to improve and select sessions for future events.

  30. Thank you! Platinum Sponsor Diamond Sponsor

More Related