The Challenges of Repeatable Experiment Archiving – Lessons from DETER
This presentation is the property of its rightful owner.
Sponsored Links
1 / 9

Overview of DETER PowerPoint PPT Presentation


  • 57 Views
  • Uploaded on
  • Presentation posted in: General

The Challenges of Repeatable Experiment Archiving – Lessons from DETER Stephen Schwab SPARTA, Inc. d.b.a. Cobham Analytic Solutions May 25, 2010. DETER Highlights 3 distributed clusters, ~500 nodes Combination of DETER developed software and legacy Emulab DETER Capabilities Federation

Download Presentation

Overview of DETER

An Image/Link below is provided (as is) to download presentation

Download Policy: Content on the Website is provided to you AS IS for your information and personal use and may not be sold / licensed / shared on other websites without getting consent from its author.While downloading, if for some reason you are not able to download a presentation, the publisher may have deleted the file from their server.


- - - - - - - - - - - - - - - - - - - - - - - - - - E N D - - - - - - - - - - - - - - - - - - - - - - - - - -

Presentation Transcript


Overview of deter

The Challenges of Repeatable Experiment Archiving – Lessons from DETERStephen SchwabSPARTA, Inc. d.b.a. Cobham Analytic SolutionsMay 25, 2010


Overview of deter

DETER Highlights

3 distributed clusters, ~500 nodes

Combination of DETER developed software and legacy Emulab

DETER Capabilities

Federation

Security Experimentation

Environment (SEER)

Templates

DETER

Emulab

GMPLS

GMPLS

Emulab

WAIL

Overview of DETER

SEER

Federator

CEDL

DRAGON

Internet

Provisioned

Connectivity

Credentials

USERS

Plug-ins to configure

federants


Overview of deter

DDoS Experiment on DETER (circa 2005)

Background Traffic:

REPLAY | NTCG | HARPOON

HIGH FIDELITY TRAFFIC

Topology:

BUILDING-BLOCKS |

JUNIPER ROUTER CORE

REALISTIC CONNECTIVITY AND SCALE-DOWN

Attack Traffic:

DETER-INTEGRATED ATTACK SCRIPTING

AUTOMATION OF VARIETY OF SCENARIOS UNDER STUDY

Instrumentation:

PACKET AND HOST STATISTICS CAPTURE | SPECTRAL ANALYSIS | METRICS CALCULATION | INTEGRATED VISUALIZATION

SEER: TOOLBOX FOR RIGOROUS INVESTIGATION OF RESULTS

CORE

AS-11357

ATTACK TRAFFIC

BACKGROUND

TRAFFIC


Overview of deter

Security Experiment Methodology & Tools (circa 2005)

DETER -- integrated workbench & tools for experimenters…

Experimenter’sselect from apalette of predefined elements: Topology, Background and Attack Traffic, and Data Capture and Instrumentation

Our Methodology frames standard, systematic questions that guide an experimenter in selecting and combining the right elements

Experiment Automation increases repeatability and efficiency by integrating the process within the DETER testbed environment

TOPOLOGY TRAFFIC ATTACK DATA-CAPTURE

PALETTESs

METHODOLOGY

& GUIDANCE

?

EXPERIMENT

AUTOMATION

… but this level of abstraction leads to major drawbacks


Worm botnet experiment 2009

Worm/Botnet Experiment (2009)

831 Virtual Nodes on 63 Physical PCs


Experiment specification

Experiment Specification

  • Large and Complex Experiments are more suitably constructed

  • by combining abstract elements

  • modeling different aspects (topology, traffic, networking devices, etc.)

  • with constraints on behavior

    Example of such an experiment on previous slide

  • hand-crafted (e.g. hand-compiled) experiment from abstract elements


Initial approach archiving it all

Initial Approach: Archiving it All

  • Intuition drawn from analogy with physical (discovery) sciences…

  • Record all aspects of experiment to ensure (ideal) reproducibility

  • Software

    • Artifacts being investigated (often the researcher’s new system!)

    • Operating Environment (OS, standard software on clients, servers in experiment scenario, network routers, firewalls, etc.)

    • Experiment & Test Infrastructure (initialization, control, data collection, data reduction, data analysis, data visualization, …)

  • Hardware

    • All end-systems and routers/switches

    • All firmware

    • All chips/chipset variants (Tulip 21140As are not Tulip 21140Es!)

  • Procedures

    • All scripts and manual interactions required to run the experiment

  • … networked systems require large and growing (unbounded) detail

    to describe precisely… which ideal reproducibility would seem to demand


Challenges to ideal archiving

Challenges to Ideal Archiving

  • Separating Invariants from Contingencies

  • An experiment requires certain properties; these are the essence of the experiment

  • But every configuration detail must be specified; these are contingencies – merely choices (perhaps important to record)

  • Repeatability should be primarily defined with respect to explicit invariants

    Experiment Internals

  • Publications do not capture full details because increasing complexity of software, hardware and networking technologies result in (exponential?) growth in description of these aspects

  • Peer-review process does not provide incentives to capture full details (noted in other position papers)

  • Funding agencies do not provide sufficient funding to do so (how much detail can, will and should be demanded? Where is the limit on returns for dollars invested?)

    Granularity of Reuse

  • Individual researchers are interested in examining, studying and re-running different elements of any given experiment

  • Experiments that archive everything do not clearly delineate the various pieces


Future deter capability vision

Future DETER Capability & Vision

  • DETER is developing the capability to

  • Specify Experiments Declaratively

  • Reason about the software (or hardware) alternatives that may be available to realize each element in a testbed

  • Select implementations that are sufficient to perform the experiment correctly

  • … and ensure detection of fidelity-loss through the use of monitored invariants

  • Resolve global conflicts among local element to implementation mappings

    DETER vision is to foster

  • Reuse through sharing of tools, technology, results & ideas among researchers

  • … and to promote this vision by providing abstractions, models and elements that are supported by our experiment life-cycle framework and tools

  • Facilitate individuals and researchers focused on specific topics to create their own abstractions, models and elements


  • Login