PROVE as a Web Service

PROVE as a Web Service

Application development on EGEE with P-GRADE Portal Gergely Sipos MTA SZTAKI [email protected] EGEE Training and Induction EGEE Application Porting Support 1 Running jobs in EGEE from command line User Interface Write JDL, Submit workload

Workload Management System EGEE VO Retrieve status & (small) output files Workload: Single job One executable & one or more files Sequential or MPI Workflow DAG of jobs

Parameter study Independent, similar jobs 2 Defining a single job in JDL $ vi OR nano OR my.jdl JobType = "Normal"; Executable = gridTest; StdError = stderr.log; StdOutput = stdout.log; InputSandbox = {/home/sipos/gridTest}; OutputSandbox = {result.txt, stderr.log, stdout.log}; $ glite-wms-job-submit my.jdl JobID $ glite-wms-job-status JobID $ glite-wms-job-output JobID

3 Managing data in EGEE from command line User Interface Run file management commands Upload, transfer, register files EGEE VO Download files Delete files File catalog & Storage Elements $ lfc-mkdir myCatalogEntry

$ lfc-rmdir myCatalogEntry $ lcg-cr vo voName d SEName l myCatalogEntry file . . . 6 Short History of P-GRADE portal Parallel Grid Application Development Environment Initial development started in the Hungarian SuperComputing Grid project in 2003 Motivation: Simplify grid usage for application developers and end users It has been continuously developed since 2003 Around 30 manyear development + training + user support

Detailed information: Open Source community since January 2008: Current version: 2.8 7 Portal installations P-GRADE Portal services: Several VOs of EGEE: Biomed, Astronomy, Central European, NA4,... GILDA: Training VO of EGEE Southern Eastern European Grid Several national Grids (UK National Grid Service, HunGrid, Turkish Grid, etc.)

US Open Science Grid, TeraGrid OGF Grid Interoperability Now (GIN) VO Under development: KnowledgeGrid of Malaysia MIMOS & SZTAKI Portal services and account request: Account request form on portal login page 9 Design principles of P-GRADE portal P-GRADE Portal is not only a user interface, it is a General purpose Application Development and Execution Environment

P-GRADE Portal includes a high-level middleware layer for orchestrating jobs on grid resources inside a grid among several different grids and several VOs P-GRADE Portal is grid-neutral: It is not tailored to any particular middleware grid type Can be connected to various grids based on different grid middleware LCG-2, gLite, GT2, GT4, ARC, Unicore, etc. Implementation for GT2, GT4, gLite, ARC, LSF GUI is the same no matter which type of grid is connected to it 10 What is a P-GRADE Portal workflow?

A directed acyclic graph where Nodes represent jobs (batch programs to be executed on a computing element) Ports represent input/output files the jobs expect/produce Arcs represent file transfer operations semantics of the workflow: A job can be executed if all of its input files are available 11

Three levels of parallelism Multiple instances of the same workflow process different data files Job level: Parallel execution inside a workflow node (MPI job as workflow component) Workflow level: Parallel execution among workflow nodes (WF branch parallelism) PS workflow level: Parameter study execution of the workflow

Multiple jobs run parallel Each job can be a parallel program 12 Example: Computational Chemistry ~100 independent jobs to run Department of Chemistry, University of

Perugia SOLUTION OF SCHRODINGER EQUATION FOR TRIATOMIC SYSTEMS USING TIMEDEPENDENT (RWAVEPR) OR TIME INDEPENDENT (ABC) METHOD A single execution can be between 5 hours and 10 hours Many simulations at the same time SEQUENTIAL FORTRAN 90 13 Typical user scenario Job compilation phase UPLOAD JOB SOURCE(S)

Portal server Client COMPILE EDIT Grid services DOWNLOAD BINARI(ES) 14 Typical user scenario Workflow development phase


BINARIES DSpace WF repository 15 Typical user scenarios Workflow execution phase MyProxy Certificate servers TRANSFER FILES, SUBMIT JOBS DOWNLOAD PROXY CERTIFICATES


(SMALL) RESULTS DOWNLOAD (SMALL) RESULTS 16 P-GRADE Portal structural overview Source: Jia Yu and Rajkumar Buyya: A Taxonomy of Workflow Management Systems for Grid Computing, Journal of Grid Computing, Volume 3, Numbers 3-4 / September, 2005 Web browser DSpace workflow repository

Java Webstart workflow editor Workflow language: Extended DAGMan Globus and gLite information systems (Globus GIIS, gLite BDII) Workflow scheduler: Extended DAGMan Grid interface: command line clients + scripts EGEE, Globus (and ARC) Grid services

(gLite WMS, LFC,; Globus GRAM, ) 18 Web interface - Portlets 19 Email notifications NOTIFY 20 Workflow portlet WORKFLOW EDITOR

21 Graphical workflow editing To define a graph: 1. Drag & drop components: jobs and ports 2. Define their properties 3. Connect ports by channels (no cycles, no loops) System generates JDL for each job automatically 24

Workflow Editor Properties of a job Properties of a job: Executable file Type of executable (Sequential / Parallel) Command line parameters Which resource to use? Which VO? Broker or Computing element? 25 Workflow Editor

Defining input-output files File properties Type: input: the executable reads output: the executable generates File type: local: comes from my desktop remote: comes from an SE File: location of the file Internal file name: Executable uses this e.g. fopen(, ) File storage type (output files only): Permanent: final result Volatile: temp. data channel

26 Scaling up a workflow to a parameter study Complete workflow Grid files (files in the same EGEE catalog entry) Results stored in grid files 27 Advanced parameter studies Generator

component(s) Initial input data Generate or cut input into smaller pieces Complete workflow Collector component(s) Aggregate

result 28 Concept of parameter study workflows GEN Generator part generates the input parameter space SEQ SEQ SEQ SEQ

Parameter study part COLL Collector part evaluates and integrates the results 29 Example: Image processing /grid/gilda/sipos/InputImages Image.0 Image.1 Image.2 /grid/gilda/sipos/XCoordinates

XCoordinate.0 XCoordinate.1 /grid/gilda/sipos/YCoordinates YCoordinate.0 YCoordinate.1 3 x 2 x 2 = 12 execution of the whole workflow CROSS PRODUCT of data items /grid/gilda/sipos/Output ImagePart.0 ImagePart.1 ...

30 Upload a workflow from client side or from FTP server UPLOAD STORED on FTP server 32 Import a workflow from DSpace repository 34 Proxy creation:

The first step of grid usage Workload Management Write JDL, Submit job (executable + small inputs)System (WMS) User Interface (UI) or Portal Information System (BDII) query create proxy Retrieve status & (small) output files

Retrieve output Job status publish state Submit job Loggin g Site X of YOUR VO Computing Element (CE)

Job status VO Management Service (VOMS) Logging and bookkeeping (LB) process Storage Element (SE) Read/write files

36 Certificate and proxy management Portlet 37 OGF GIN interoperability portal by P-GRADE Acccessing Globus, gLite and ARC based grids/VOs simultaneously Proxy 1 P-GRADE Proxy 6 P-GRADE GEMLCA portal

Portal Proxy 2 Proxy 5 GEMLCA Repository Proxy 3 Proxy 4 38 Application execution 39

Detailed view of a PS workflow Generator job(s) Overall statistics of workflow instances Workflow instances Collector job(s) 40 Fault-tolerant execution Utilizing Condor DAGMans rescue mechanism Resubmit only failed jobs, not completed jobs! If the EGEE broker leaves a job stuck in a CEs

queue, the portal automatically kills the job on this site and resubmits the job to the broker by prohibiting this site. As a result the portal guarantees the correct submission of a job as long as there exists at least one matching resource job submission is reliable even in an unreliable grid 41 Information system visualization 42 LFC-SE file browser portlet

43 Compilation support 44 Learn once, use everywhere Develop once, execute anywhere Thank you! [email protected] 45

Recently Viewed Presentations

  • Five Steps to a Successful Endowment Kathryn W. Miree ...

    Five Steps to a Successful Endowment Kathryn W. Miree ...

    Then, we'll go through the process of building an endowment in five steps. Building endowment requires a proactive process. It's easy to get lost without a plan - so the last step is to make a plan and set goals.
  • SOIL - Fundamental Concepts

    SOIL - Fundamental Concepts

    Soil Forming Processes Translocations Transformations Additions Losses Major Components Mineral matter Organic matter Air Water Physical Properties of Soil Soil texture Soil structure Soil color Bulk density Three Fractions of Mineral Matter Sand Silt Clay Aspects of Soil Structure The...
  • Sindromul de suprapopulare bacteriana a intestinului subtire

    Sindromul de suprapopulare bacteriana a intestinului subtire

    O lanternă: Probabil că o folosea pentru a găsi cheile pierdute mai devreme Cel mai straniu obiect : din această serie l-am păstrat la sfârşit. În timpul unei partide de sex un cuplu de homosexuali a decis să încerce ceva...
  • HaH Template Calibri

    HaH Template Calibri

    [email protected] Service Model Overview Care closer to home: Medically stable patients complete their acute care pathway in their own home Core service hours 0700-2200, 7 days per week Support to patients 24/7: Patient, carer & clinician access to a nurse...
  • Personal Finance Sixth Edition - Anvari.Net

    Personal Finance Sixth Edition - Anvari.Net

    Future value interest factor (FVIF): a factor multiplied by today's savings to determine how the savings will accumulate over time. Can be calculated using the future value table or a financial calculator. Future value table shows various interest rates (i)...
  • Pacifica Synod Workshop for Congregational Treasurers ...

    Pacifica Synod Workshop for Congregational Treasurers ...

    Church Income/Expenses. Recordkeeping/Records Retention. Employer/Employee Relationships & Reporting. Salaries/Benefits. ... Check is signed by persons authorized under the bank account agreement. Dual signatures are recommended. The pastor should not be an authorized signer.
  • The Consequences of Unintended Dispersal of Industrial Chemicals

    The Consequences of Unintended Dispersal of Industrial Chemicals

    The first meeting of the Conference of the Parties took place from 2-6 May 2005 in Punta del Este, Uruguay PoPs in Swedish Breast Milk Toxicity of PoPs Related to structure Related to 'most toxic substance' 2,3,7,8 tetrachloro dibenzo dioxin...
  • Biological Responses - NCEA Level 3 Biology

    Biological Responses - NCEA Level 3 Biology

    How do we know some rhythms are Endogenous? The rhythm may persist when the . organism is moved . from one part of the world to another. e.g. periwinkles move up and down the rock face in response to the...