A Software-Defined Storage for Workflow Applications Samer Al-Kiswany

A Software-Defined Storage for Workflow Applications Samer Al-Kiswany

A Software-Defined Storage for Workflow Applications Samer Al-Kiswany The University of Waterloo Matei Ripeanu The University of British Columbia 1 Opportunity: Application-Optimized Storage

Application Storage System file access pattern Storage System Application file location 2 Current Storage System Architecture Limitation Limits building application-optimized storage: Limits the flow of information across layers Prohibits optimizing the storage operations Our Solution: Software-Defined Storage Architecture

FlexStore: Flexible and Extensible Storage System Provides application control of storage operations Extensible Evaluation highlight: Evaluation with synthetic and real science applications, up to: 6x higher performance 10x lower network load Outline Target Application Domain Software-Defined Storage Architecture Architecture FlexStore Evaluation

Summary Workload Characteristics - Workflow Applications File dependency Computation Reduce Local access Local access Reduce Montage workflow 6

System Architecture Application hints (e.g., access patterns) Workflow Runtime Engine Storage hints (e.g., location information) POSIX API Compute

Nodes task Local storage task task Local storage Local

storage Intermediate Storage (shared) Stage In/Out Backend Filesystem (e.g., GPFS, NFS) 7 Application-Optimized Storage Challenges Passing hints while maintaining the current API Design an extensible storage system 8

Outline Target Application Domain Software-Defined Storage Architecture Architecture FlexStore Evaluation Summary Solution: Hints through Custom Attributes Application-optimized storage: Application Storage System file access pattern

File System API Custom Metadata Storage System Application file location 10 Solution: Hints through Custom Attributes Advantages: Application-agnostic Maintains layered architecture benefits

Maintains standard API Provides an incremental adoption path Old Application File System API Custom Metadata Cross Layer Storage System File Custom

System APIMetadata Cross Layer Application File System API Custom Metadata Old Storage System 11

FlexStore: Flexible and Extensible Storage System Scheduler Client Metadata Manager Storage Nodes 12 FlexStore Architecture

Flexibility through external control Software storage architecture Scheduler control defined Control Plane Operations Plane Versions of the same operation

. Op. 1 Op. 2 Primitives Plane . Pr. 1 Pr. 2 13 FlexStore Architecture Flexibility through external control Software defined storage architecture

Extensibility common primitives, isolate operations. Scheduler control Control Plane Operations Plane Versions of the same operation . Op. 1 Op. 2

Primitives Plane . Pr. 1 Pr. 2 14 FlexStore Design Scheduler control Dispatch based design Message tagging

15 Evaluation Montage workflow Platform: 102 nodes cluster Workload: 10 stages ~4,000 tasks 6,770 files generated (size of 27GB) Ceph FlexStore-D FlexStore

Up to 25% performance gain 2x gain with synthetic benchmarks. 16 Summary Contributions: Cross layer communication trough tags Design first software-defined extensible storage system Evaluation: Up to 6x higher performance with synthetic benchmarks Up to 10x lower network load Up to 70% higher performance with real applications

17 Thank you 18 Layered System Architecture Problem: Limits flow of information [Patil HotCloud 09, Grider CMU Report 06, Seltzer HotOS 09, HECE Working Group] Proposed solutions: New API [UrsaMinor, BitDew, HDFS, GreenStore]

Modify or extend existing API. [Mesnier SIGOPS 11, Patterson SOSP 95] 19 Cross Layer Optimization Challenges Passing hints while maintaining the current API Design an extensible storage system 20 Opportunity Application-optimized storage:

Application Storage System file access pattern File System API Custom Metadata Storage System Application file location 21

Recently Viewed Presentations

  • 1 Chapter One Integrated Marketing Communications 1-1 1

    1 Chapter One Integrated Marketing Communications 1-1 1

    Integrated Marketing Communications. 1. Integrated marketing communications : is the coordination and integration of all marketing communication tools, avenues, and sources within a company into a seamless program that maximize the impact on customers and stakeholders at a minimal cost.
  • PLAT-751T: 3D Graphics in Metro Style Apps and Games

    PLAT-751T: 3D Graphics in Metro Style Apps and Games

    In-line code to favor readability over factoring of functionality. Simple samples. ... [PLAT-756T] Building Xbox LIVE games for Windows 8. www.buildwindows.com. Windows 8 is the ideal platform for 3D apps. www.buildwindows.com. Use 3D in Metro style apps.
  • Exposure Java 2013 APCS Edition Chapter 3 Slides

    Exposure Java 2013 APCS Edition Chapter 3 Slides

    Real numbers do not have remainder division in any practical sense. There also is the issue that Java is based on C++, which does not allow remainder division with real number data types. Even though the following examples do not...
  • Electron Spin Resonance Spectroscopy

    Electron Spin Resonance Spectroscopy

    Electron Spin Resonance Spectroscopy or It's fun to flip electrons! Electron Paramagnetic Resonance spectroscopy Electron Spin Resonance spectroscopy Principles of EMR spectroscopy B 0 D E h n Classical theory: Electron spin moment interacts with applied electromagnetic radiation Quantum theory:...
  • Task Group G Report - IEEE Standards Association

    Task Group G Report - IEEE Standards Association

    Matthew B. Shoemake Task Group G Chairperson November 15, 2002 Letter Ballot 49 Resolution Resolved all comments from Letter Ballot 49 Comment resolutions in document 11-02-711r8 Of technical comments: 60 Accepted 29 Countered 58 Rejected Balloting History State of the...
  • Forming a Government - St. Joseph Catholic Church

    Forming a Government - St. Joseph Catholic Church

    Economic Problems. Congress no control over interstate commerce..trade between the states. Different trade laws…different interests..DIFFERENT. War debts needed to be paid..states had no money…issued paper money…no value not backed by gold or silver..just paper
  • Day 2_PPT02

    Day 2_PPT02

    ICAO Language Proficiency Requirements Implementation - The Italian situation - Professions Based on the risk analysis ENAC decided to postpone the time limit for private pilots (with or without IR).
  • Biology - riverviewsc.weebly.com

    Biology - riverviewsc.weebly.com

    Sponges are asymmetrical; they have no front or back ends, no left or right sides. Sponges carry out basic functions, such as feeding and circulation, by moving water through their bodies. ... As water moves through the sponge, food particles...