Big Data Open Source Software and Projects ABDS

Big Data Open Source Software and Projects ABDS

Big Data Open Source Software and Projects ABDS in Summary XXV: Layer 17 Part 1: Pre-Cloud Data Science Curriculum March 1 2015 Geoffrey Fox [email protected] http://www.infomall.org School of Informatics and Computing Digital Science Center Indiana University Bloomington Functionality of 21 HPC-ABDS Layers 1) Message Protocols: 2) Distributed Coordination: 3) Security & Privacy: 4) Monitoring:

5) IaaS Management from HPC to hypervisors: 6) DevOps: Here are 21 functionalities. 7) Interoperability: (including 11, 14, 15 subparts) 8) File systems: 9) Cluster Resource Management: 4 Cross cutting at top 10) Data Transport: 17 in order of layered diagram 11) A) File management starting at bottom B) NoSQL C) SQL 12) In-memory databases&caches / Object-relational mapping / Extraction Tools 13) Inter process communication Collectives, point-to-point, publish-subscribe, MPI: 14) A) Basic Programming model and runtime, SPMD, MapReduce: B) Streaming:

15) A) High level Programming: B) Application Hosting Frameworks 16) Application and Analytics: 17) Workflow-Orchestration: Part 1) Pre-Cloud 2) Cloud Apache Oozie https://oozie.apache.org/ http://hortonworks.com/hadoop/oozie/ Oozie is a scalable, reliable and extensible server-based workflow scheduler system to manage Apache Hadoop jobs coming from Yahoo. Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) which specify a sequence of actions to execute. Oozie also provides Coordinator jobs, which are Workflow jobs which can be scheduled to recur based on time or data availability. Oozie can schedule various types of Hadoop jobs (mapreduce, Pig, Hive and others) as well as other types of jobs such as Java programs and shell scripts.

Apache ODE The Orchestration Director Engine ODE http://ode.apache.org/ executes business processes written following the WS-BPEL standard. This is Web Service approach to orchestration/workflow and stresses control flow whereas usually dataflow more natural Side-by-side support for both the WS-BPEL 2.0 OASIS standard http:// docs.oasis-open.org/wsbpel/2.0/OS/wsbpel-v2.0-OS.html and the legacy BPEL4WS 1.1 vendor specification. (BPEL = Business Process Execution Language) Supports 2 communication layers: one based on Axis2 (Web Services http transport) and another one based on the JBI http://en.wikipedia.org/wiki/Java_Business_Integration standard (http://jcp.org/en/jsr/ detail?id=208). Support for the HTTP WSDL binding (initial SOAP-based Web Service protocol), allowing invocation of REST-style (this won) web services. Possibility to map process variables externally to a database table of your choice. High level API to the engine that allows you to integrate the core with virtually any communication layer. Hot-deployment of your processes. Compiled approach to BPEL that provides detailed analysis and validation at the command line or at deployment.

Management interface for processes, instances and messages. There were other open source BPEL projects but maybe this is only remaining one? http://en.wikipedia.org/wiki/List_of_BPEL_engines ActiveBPEL This was top open source BPEL implementation from Active Endpoints which was acquired by Informatica The last release was 5.0.2 in 2008 http://activebpel502.sourceforge.net/ See http:// www.activevos.com/cp/329/active-endpoints-releases-milestone1-of-activebpel-community-edition-with-bpel4people Replaced by ActiveVOS (Visual Orchestration System) http ://www.activevos.com/; proprietary coding of open BPEL standard and supports people in the loop (BPEL4People) Implements BPMN visual interface (Business Process Model and Notation)

Apache Airavata https://airavata.apache.org/ Apache Airavata is a workflow framework to enable the scheduling and execution of jobs and workflows on distributed computing resources, including local resources, national computing resources, and computing clouds. Four main components: workflow suite application wrapper service registry service message broking service Airavata evolved out of work at Indiana Universitys Extreme Computing Lab, originally supporting the Linked Environments for Atmospheric Discovery (LEAD) project. The Open Gateway Computing Environments (OGCE) project abstracted and generalized the workflow developed for LEAD, making it applicable to other science gateways. Airavata became an incubator project in 2011 and a top level project in 2012.

Pegasus https://pegasus.isi.edu/ Grid workflow engine capable of executing tasks connected by data flow graphs as shown The Pegasus project encompasses a set of technologies that help workflow-based applications execute in a number of different environments including desktops, campus clusters, grids, and clouds. Pegasus bridges the scientific domain and the execution environment by automatically mapping high-level workflow descriptions onto distributed resources. It automatically locates the necessary input data and computational resources necessary for workflow execution. Pegasus enables scientists to construct workflows in abstract terms without worrying about the details of the underlying execution environment or the particulars of the low-level specifications required by the middleware

(Condor, Globus, or Amazon EC2). Pegasus also bridges the current cyberinfrastructure by effectively coordinating multiple distributed resources. Kepler, BioKepler The Kepler Project (UC Davis, UC Santa Barbara, and UC San Diego) https://kepler-project.org/ is dedicated to furthering and supporting the capabilities, use, and awareness of the free and open source, scientific workflow application, Kepler. Kepler is designed to help scientists, analysts, and computer programmers create, execute, and share models and analyses across a broad range of scientific and engineering disciplines. Kepler can operate on data stored in a variety of formats, locally and over the internet, and is an effective environment for integrating disparate software components, such as merging "R" scripts with compiled "C" code, or facilitating remote, distributed execution of models. Using Kepler's graphical user interface, users simply select and then connect

pertinent analytical components and data sources to create a "scientific workflow"an executable representation of the steps required to generate results. The Kepler software helps users share and reuse data, workflows, and components developed by the scientific community to address common needs. Kepler is a java-based application that is maintained for the Windows, OSX, and Linux operating systems. http://www.biokepler.org/ bioKepler is a Kepler module of scientific workflow components to execute a set of bioinformatics tools Kepler Example Workflows https://kepler-project.org/users/sample-workflows Galaxy

http://galaxyproject.org/Galaxy http://en.wikipedia.org/wiki/Galaxy_(computational_biology) is an open source scientific workflow system written in Python. Workflow systems provide a means to build multi-step computational analyses akin to a recipe. They typically provide a graphical user interface for specifying what data to operate on, what steps to take, and what order to do them in. Galaxy is also a data integration platform for biological data. It supports data uploads from the user's computer, by URL, and from many online resources (such as the UCSC Genome Browser, BioMart and InterMine). Galaxy supports a range of widely used biological data formats, and translation between those formats. Galaxy provides a web interface to many text manipulation utilities, enabling researchers to do their own custom reformatting and manipulation without having to do any programming.

Galaxy Objects include Histories are computational analyses (recipes) run with specified input datasets, computational steps and parameters. Histories include all intermediate and output datasets as well. Datasets, workflows and pages are other objects Swift Apache license http://swift-lang.org/main/ is a scripting language approach to workflow as in foreach protein in proteinList { runBLAST(protein); } This is called Many task computing Swift is easy: Short, simple scripts can do large-scale work. The same script runs on multicore computers, clusters, grids, clouds, and supercomputers. Swift is parallel: it runs multiple programs concurrently as soon as their inputs are available, reducing the need for complex parallel programming.

Taverna Taverna http://www.taverna.org.uk/ http:// en.wikipedia.org/wiki/Taverna_workbench is an LGPL open source and domainindependent Workflow Management System a suite of tools used to design and execute scientific workflows and aid in silico experimentation. Well developed and robust system linking to tools like Bioconductor Various service types available: WSDL-style and RESTful Web services, BioMart, BioMoby, SoapLab, R, Beanshell, Excel and csv spreadsheets Taverna has been created by the myGrid team and is currently funded though FP7 projects BioVeL, SCAPE and Wf4Ever.

The Taverna tools include the Workbench (desktop client application), the Command Line Tool (for a quick execution of workflows from a terminal), the Server (for remote execution of workflows) and the Player (Web interface plugin for submitting workflows for remote execution). Taverna Online lets you create Taverna workflows from a Web browser. Also featured in http://www.myexperiment.org/ workflow sharing environment Taverna Triana Triana Apache license http://www.trianacode.org/ from Cardiff termed a problem solving environment an old fashioned name It was one of two testbed applications for GridLab, a large EU funded project. The aim of GridLab was to develop a simple and robust grid application toolkit (GAT) enabling applications to exploit the power of the Grid. TrianaCloud uses RabbitMQ to link modules together

Triana is a Java system with one implementation based on the Sun's JXTA protocols and distributed across many mobile devices Generally built around distributed modules Triana has a traditional GUI interface for linking components together It can link to Pegasus Trident Apache license https://www.microsoft.com/mscorp/tc/trident.mspx http://tridentworkflow.codeplex.com / Scientific Workflow Workbench is no longer active

Trident is a set of tools based on the Windows Workflow Foundation that addresses scientists need for a flexible, powerful way to analyze large, diverse datasets. It includes graphical tools for creating, running, managing, and sharing workflows. This central control present in many workflow systems but not obviously good. Trident Workflow Composer provides graphical tools for creating workflows. For large data sets, Trident can run multiple workflows in parallel on a Windows HPC Server 2008 cluster. It had classic mistake of a centralized controller Trident Workflow Application - available in Windows Presentation Foundation and Silverlight versions provides a simple way to run Trident workflows. The Microsoft Silverlight version of the Trident workflow application enables users to run workflows remotely using a Silverlight-enabled browser via Web services. Trident is integrated with myExperiment - a workflow collaboration portal - so scientists can easily share their Trident workflows with colleagues. Trident security model supports users and roles that allows scientists to control access rights to their workflows. The Trident Registry maintains libraries of workflows and activities, workflow inputs and outputs, workflow versions, and associated data products; each user can have a personal workflow library and can share workflows with other users. Trident Management Studio manages the Trident Registry and workflow execution, schedules workflows, and monitors workflow executions locally or remotely.

IPython Scripting as in Swift is an important approach to workflow/orchestration Several projects have used Python as the scripting language as have related (in nature of problem) System specification languages like Ansible https://wiki.openstack.org/wiki/NovaOrchestration/Workflo wEngines IPython notebook preserves provenance of activity Taverna can be executed from inside Ipython http://www.pro-ibiosphere.eu/news/4654_the%20running% 20of%20taverna%20workflows%20within%20an%20ipython %20notebook

Recently Viewed Presentations

  • Began in Italy Spread north through Europe Arrived

    Began in Italy Spread north through Europe Arrived

    Began in Italy Spread north through Europe Arrived late in England
  • Viral Hepatitis - Yola

    Viral Hepatitis - Yola

    Hepatic And Post-hepatic Jaundice Sonal Pruthi Roll Number - 82 Defective Hepatic Uptake Abnormal Conjugation Hepatocellular Damage Gilbert's Syndrome Drugs Rifampicin Novobiocin Probenecid Viral Hepatitis Alcoholic Hepatitis Drug Induced Hepatitis Autoimmune hepatitis Toxic Hepatitis Criggler Najjar Syndrome Viral Hepatitis Drugs...
  • NJViD Presentation  Internet2 FMM - November 2, 2010

    NJViD Presentation Internet2 FMM - November 2, 2010

    William Paterson (VALE), Rutgers Libraries, and NJEDge were awarded a $1M IMLS Grant to provide funding for a rather unique initiative to support academic video on-demand services. ... Sakai or Moodle. Online students can access the same content as classroom...
  • Copyright management and open access Presenter: Affiliation: Hasmik

    Copyright management and open access Presenter: Affiliation: Hasmik

    Copyright management and open access Presenter: Hasmik Galstyan eIFL-IP expert [email protected] Affiliation: American University of Armenia
  • Part two  Ataxic gait  Treatment options Ataxia gait

    Part two Ataxic gait Treatment options Ataxia gait

    Dysdiadochokinesia. Dysmetria. Trunk ataxia tests . Tandem walking. Control of voluntary movements, ability to co-ordinate movements. Focus on ability to perform simple than more complex tasks. Function such as eye-hand co-ordination, right left co-ordination. Lack of co-ordination / lack of...
  • Data Visualization Research Lab

    Data Visualization Research Lab

    When should be use words? Dual coding theory What is a visual language? - does it make sense? How to integrate images and words The Narrative thread Gillnet Set Surface Buoy Lobster Trap Trawl Surface Buoy Landry 2002 Theory: Graphics...
  • English Learner Series: Linguistically Appropriate Goals

    English Learner Series: Linguistically Appropriate Goals

    The broken down speaking/listening standards in SIRAS take into account the disability but aren't necessarily linguistically appropriate. The speech and language specialists may have broken down the same speaking/listening standard with speech criteria in mind, but may or may not...
  • Welcome! College of the Environment, Forestry and Natural

    Welcome! College of the Environment, Forestry and Natural

    College of the Environment, Forestry and Natural Sciences (CEFNS) Orientation for new faculty. August 22, 2019. Jamie Baxter, Administrative Director