The Italian Grid Infrastructure (IGI) HPC 2008 Cetraro

The Italian Grid Infrastructure (IGI) HPC 2008 Cetraro

From the EGEE project to the new EGI organization and IGI PON Final Workshop 10/2/2009 Catania Mirco Mazzucato EGI-DS MB Italian Grid Infrastructure Coordinator INFN CNAF Director EGEE-III EGEE: Flagship Grid infrastructure project co-funded by the European Com. Main Objectives Expand/optimise existing EU grid infrastructure, include more resources and user communities Prepare migration from a projectbased model to a sustainable federated infrastructure based on National Grid Initiatives Duration: 2 years Consortium: ~140 organisations across 33 countries EC co-funding: 32Million 2

EGEE Achievements EGEE Production Grid Infrastructure Steady growth over the lifetime of the project Improved reliability The EGEE project - Bob Jones - EGEE'08 - 22 September 2008 3 First EGEE site in Senegal Add to other EGEE International sites EU-India EU-China EU-Asia EU-Med EE-Lat. Am. ..and other National Grids: Open Science Grid (US) Naregy (Japan)

10 September 2008 IDG4 EGEE Applications >270 VOs from several scientific domains Astronomy & Astrophysics

Civil Protection Computational Chemistry Comp. Fluid Dynamics Computer Science/Tools Condensed Matter Physics Earth Sciences Fusion High Energy Physics Life Sciences EU users: LHC , Other VOs ESFRI communities Applications have moved from testing to routine and daily usage ~80-95% efficiency The EGEE project - Bob Jones - EGEE'08 - 22 September 2008 5 EGEE added value Successful coordinated European process for Open Source Grid middleware development and distribution (gLite) Used by ~ 10.000 Researchers in production in ~30 Countries and ~250

Compute Centres Unified QA certification, test and release process Done once for all. Avoid duplication of same activities in all countries and sites Coordinated structured operation of the grid infrastructure services at national and European level Regional Operation Centers control operation in one country and perform all internal tasks Avoid N*(N-1) interactions between sites Coordinated user communities support and grow of collaborative research activity at pan-European and international scale EGEE has shown that it is possible to provide a common access infrastructure to national or international projects or disciplines, enabling the sharing of computing resources and data archives with minimal overhead by distriburted research teams .. and that it is possible to extend it to support innovation in other sectors : Cyclops, eNMR 6 e-Infrastructure sustainability Users need now to rely on a permanent, common general European Grid

infrastructure Sustainable independently of short project funding cycles ->The European Grid Initiative (EGI) -> new organizational model Operate the European level of the production Grid infrastructure for a wide range of scientific disciplines to link national e-Infrastructures National Grid Initiatives (NGI) EGI.Org Central Organization Coordinate the integration and interaction between National Grid Infrast. (NGIs) 7 The EGI model Research Teams Research Institutes NGI2 NGI1 European Grid Initiative EGI NGIn

Resource Centres National Grid Initiatives (NGIs) 8 Grids in Europe Each NGI should be a recognized national body with a single point-of-contact should mobilize national funding and resources should ensure the operation of a national e-Infrastructure should support user communities should contribute and adhere to intl. standards and policies 9 The EGI Users Research Teams (RTs) establish one or more Virtual Organizations (VOs) for each User Community and use the Grid services to share their IT resources

VOB2 VOA Discipline User Community A VOB1 Discipline User Community B IT Resources Researchers / RTs VOZ VOB3 Discipline User Community Z

IT Resources Research Institutes 1 The grid added value Support the increase of scale, interdisciplinary and capability to handle the data deluge necessary for a Competitive collaborative research and innovation within countries and in Europe Teams can easily share their local computing resources and their data in a managed way that ensures the owner control The grid services (software, operational, user oriented, policies) have overcome the technical and political barriers limiting a global and shared access to distributed resources different administrative domains large variety of legacy solutions adopted by the European Compute centres Very different usage policies

Grids do not deal with new ICT hardware resource ownership Resources are available from the normal governments' lines of budget to support reseach and innovation in all countries Allocated to the Research teams by their departments, universities, research laboratories and national facilities. In the future, if required, EGI may also consider to become an "organiser (or enabler) of commodity low cost ICT resources provision Interesting for innovative communities like Civil Protections 1 EGI added value New organizational model to provide in a sustained way the common grid access infrastructure to national or international projects or disciplines as a new legal Entity providing a pan-European overall coordination framework and central services to NGIs National Grid Initiatives (NGI) established in each member state managing the national grid infrastructure Sustainability obtained via progressive introduction of service charges

The NGI represents the users and the resource providers of all interested research domains in one country and act as unique interface to and other NGIs EGI support international research collaborations in all domains while also maintaining national control and independence of computing resources conditionally shared according to the owner policies 1 International Implications Coordination necessary at national and European level Help to solve the technology challenges in a coordinated way Help to solve the worse than n * (n-1) problem EGEE and related projects have shown that this is possible and successful Here comes the EGI Blueprint Description of necessary EGI functions and services

At NGI and levels coordination body size and role 1 The Road to the Current Blueprint Requ i from D5.1: Draft Definition of EGI Org D5.2: Draft Convention re m NGIs ents , EGE E WP3 consolidation WS 29-31-Jan-08

MW: Coordination no development Fe ed ba ck Geneva workshop 30-June-08 Feedba ck Feedback from: NGIs, EGEE, other Grids PRACE, WLCG, etc. D4.4: Early Draft of Blueprint

dba F ee k c Provision of a large-scale production Grid infrastructure Rome workshop 13-14-March-08 Second Draft of Blueprint Istanbul workshop 22-Sep-08 Final Blueprint Prague Policy Board 20-Jan-09 Jan 31, 2020 1 Blueprint Draft V1 V2 V3

103 pages (with Appendix) 46 pages 33 now Function details go to D3.1 and D3.2 Middleware development not part of EGI financing More explanation of business case More emphasis on user support Clarification: is not the top of a hierarchy Funding: NGI resources: largely available - scalable Operations cost decreasing Innovation will require sustained funding More specific on EC funding for and NGIs Jan 31, 2020

J. Knobloch - EGI_DS Policy Board 20-Jan-2009 15 1 The Business Model for EGI International and interdisciplinary research projects and communities need to share data and compute resources available in individual countries The current pan-European grid infrastructure established by EGEE and related projects is heavily used the (growing) user communities rely on continuity and sustainability EGI can ensure a common infrastructure avoiding divergence between countries by establishing a National Grid Initiative in each member state and a relatively small central offering general services Jan 31, 2020 1

EGI Management/Governance Members NGI1, NGI2, NGI3, NGIn User Forum Steering Committee Associate Members Non-voting Representatives e.g. EIROforum member, extra-EU NGIs, Chair of UFSC, EGI Council (UFSC) User Forum (UF)

EGI Director Advisory Committees e.g. Middleware Coordination Board (MCB) UCO User Coordination CTO Middleware Maintenance CAO Admin & PR COO Operations User Community

Services Middleware Unit Administration & PR Unit Operations Unit EGI Jan 31, 2020 1 and NGI Tasks NGI NGI NGI NGI Jan 31, 2020 tasks NGI international tasks NGI local tasks 1 Resources and Funding EGEE and similar European Grid infrastructure projects rest on three pillars for financial support: 1. 2. 3. Co-funding by the EC Matched national funding National/institutional funding for computing infrastructures and their operation Primary EGI funding is at the national level EC funding for

Jan 31, 2020 Coordinating national efforts Activities spanning several countries Initial funding during startup Supporting innovation 1 Resources for Jan 31, 2020 2 Resources in NGIs Based on current experience These resources exist in countries with operational grid infrastructure

Jan 31, 2020 2 EGI Financial Reference Model NGI International tasks Middleware Operations User services maintenance Total Costs EGI Effort (FTEs) EGI costs (M/year) 225 110 70 51

20.25 9.90 6.30 4.59 41 1,0 1 Income NGI membership fees (M/year) EC Contribution (M/year) 10.125 4.95 3.15 1.80

20 NGI, Middleware consortia and EIRO/ESFRI co-funding Contribution (M/year) 10.125 4.95 3.15 1.80 20 Jan 31, 2020 2 Funding view of NGI 600 k 600 k

EC Funding National Co-Funding NGI Average medium sized NGI Membership Fees 30 k Jan 31, 2020 2 Funding Evolution Jan 31, 2020 2 Transition to EGI

From here: Limited duration To there: TopDown EU co-funded projects started in 2001 with the European DataGrid, followed by a series of EGEE and similar projects. Sustainable NGIs forming a coherent and sustainable pan-European eInfrastructure Jan 31, 2020 2 Transition Period Transition from EGEE and similar European projects Already moving towards the EGI model:

Joint Research Units (JRU) are already established in 21 countries a good starting point for NGIs De-centralisation is an essential part of the EGEEIII strategy Tight collaboration between EGEE-III and EGI_DS has been established and a common workplan is in place EGEE-III Deliverable DNA 1.4: Plan for a Transition to a Sustainable Infrastructure to be complemented by EGI_DS document D5.5: Transition Scenario from Current Grid Infrastructures to EGI Jan 31, 2020 2 Statements approved The EGI Policy Board acknowledges the significant progress made in the production of the next blueprint version, taking into account the extensive feedback on previous drafts provided by the EGI Policy Board and other experts. The current document provides a firm basis for the construction of the institution and the submission of the EGI -and

other related- proposals (including one on the middleware activities) to the EC The EGI Policy Board authorizes the EGI_DS project to create a task force for the identification of the best options for EGI funding and for the preparation of the EGI proposal The EGI Policy Board authorizes EGI_DS to prepare the position of director and open it as soon as the location is decided Approved by the EGI Policy Board with 20 votes, 1 contrary and 1 abstention on January 20 in Prague Jan 31, 2020 2 Blueprint: Functions and Services

Baseline Hypothesis National level Primary functions are necessary to be run at the national level They guarantee the national Grid does exists and is operational More reflection of the best practices already deployed, not direct consequence of EGI EGEE used as a model, but same set of functions and services found in other grids, too, they are not implied by gLite/EGEE model International (EU) level Functions and services needed for Coordinating national Grids Activities being done once and make available to all The EGEE experience is visible, because it is the major European/World Grid today and a model for many others However, the resulting pattern is not specific to EGEE, it is a genuine general model 2 EGI Tasks

Authentication of users Allocation of users to virtual organizations Allocation of computing resources Authorization of VOs to run jobs, Distribution and scheduling of jobs, Monitoring of submitted jobs, Accounting of users and VOs Manage and transfer Data Reporting usage allocation to NGIs Other centrally coordinated functions does not directly own any grid infrastructure provides central coordination functions for Management Operation of the grid infrastructure Middleware common distribution, interfaces and certification

User support and application development National National Grid Grid Initiative Initiative 2 EGI Operations = critical services + NGI international tasks critical central common services NGI international tasks NGI local tasks EGI NGI NGI

NGI NGI NGI 3 Operat. Services (1/2) Operation of tools and services Grid configuration repositories Grid accounting repositories Grid repositories for SLA compliance and performance monitoring Grid operations portal NGI Grid oversight User support Central ticket handling system Gathering requirements for user support tools 3 Operat. Services (2/2) Security

Security policy development and maintenance Coordination of security and incident response Expert team for security vulnerabilities in MW Other international tasks Middleware deployment/roll-out and support Resource allocation and brokering support Interoperations between NGIs and with other grids Network support Definition of best practises, procedures, requirements Catch-all production grid core services 3 The middleware issue in Europe EGEE Grid: focus on production quality gLite based, strong hierarchical model. Include Globus/ Condor Guarantee stable and high quality services

NorduGrid ARC based, many commonalities with EGEE, more flexible model DEISA as the supercomputing Grid UNICORE based, less overlap with ARC/gLite Smaller Grid islands Mostly Globus and Condor based Default choice for general grid R&D publications and prototypes Some serving specific application communities Includes also activities of some European countries Commercial products Usually single components, basic functions, limited use in academic world Cloud: Amazon S2 and S3 services still not used by Science 3 The EGI choice

Ian-Carl definition in the Anatomy of the Grid: Grid : set of services which enable dynamic ICT resource sharing (CPU, Storage, Data, Archives, Instruments) in multi-administrative domains for the benefit of multi-institutional virtual organizations Did we practically achieve this after 8 years? Not yet!!!! ->A set of different implementations not inter-operating each other Globus, Condor, gLite, Unicore, Arc, Naregi. A shared large scale production infrastructure like EGI could be theoretically achieved Trying to interconnect free individual Grid islands however no proofs that scalability and quality problem can be solved can not remove barriers (lack of widely accepted standards) for building easily global collaboration Accepting a unified standard compliant middleware certified solution no barriers, quality and scalability at the price of less flexibility, some danger of middleware o(g)ssification

->Problems pushed at the borders but EGEE is there to show this strategy can be successful EGI has certainly valued the high quality and no barrier the best choice in line with EGEE Put effort on making the grid access simpler 3 EGI Middleware should ensure continuity should coordinate procurement and distribution should steer the process of providing the necessary requirements and feedback to the middleware developers should certify selected components should promote interaction with standardization bodies 3

Universal Middleware Distribution (UMD) Proposed by the Middleware Consortia. gLite ARC UNICORE To foster convergence of the current solutions through adoption of common standard profiles Repository contains high quality middleware components satisfying EGI criteria Selected by a Middleware Coordination Board (MCB) constituted by Users, Operations and Middleware Consortia 3 Role of in Middleware Central technical coordination by Middleware Unit (led by CTO) Objectives: EGI middleware distribution. Coordinates

Procurement Maintenance Support Final Certification Establishment of common requirements for interoperability and new developments No development of middleware within 3 EGI User Communities Services Goal: To provide significant added value for existing and new application user communities Provides overall coordination point for application support and community building

Communities organized in Special Support Centers Event organization Services for new communities Coordination of SSC interfacing activities Grid planning Training tasks 3 EGI Funding Model Membership fees General Central Coordination + Common services

Service Charges Project Funds Service Provisioning Projects Operations EGI International Activities Not initially applied INCOME COST CENTERS COSTS COVERED 3

The Italian Grid Infrastructure (IGI) Mirco Mazzucato Italian Grid Infrastructure Coordinator INFN CNAF Director IGI today The talian Grid Infrastructure (IGI) is an EU Joint Research Unit Based on a MoU signed in December 2007 Recognized and supported by the Italian Ministry of the University and Research Recognized by the EU Commission Providing a unique International interface for what concern the Italian grid infrastructure Providing a common coordination of the Italian Grid infrastructure for e-Research by public Institutions Open to new partners 4 IGI partners

Istituto Nazionale di Fisica Nucleare (INFN), Ente per le Nuove tecnologie, lEnergia e lAmbiente (ENEA), Consiglio Nazionale delle Ricerche (CNR) Istituto Nazionale di Astrofisica (INAF), Istituto Nazionale di Geofisica e Vulcanologia (INGV), Universit degli Studi di Napoli Federico II, Universit degli Studi della Calabria, Sincrotrone Trieste S.C.p.A. (ELETTRA), Consorzio COMETA, Consorzio COSMOLAB,

Consorzio SPACI,, Consortium GARR LUniversita di Perugia LUniversita del Piemonte Orientale ..ongoing discussions with Compunting Centres 4 IGI main Objectives Svolgere attivit di ricerca e sviluppo nel campo dei servizi e delle infrastrutture di Grid Partecipare, attraverso lintegrazione delle proprie risorse computazionali e di dati, al consolidamento ed allespansione dellinfrastruttura europea di Grid, al servizio dellArea della Ricerca Europea (ERA) e, per il tramite di questa, stabilire i collegamenti con altre Grid a livello internazionale Consolidare, rafforzare ed espandere, partecipando ai bandi del VII P.Q., linfrastruttura di Grid europea gi realizzata attraverso i progetti del VI P.Q. E gia un ottimo punto di partenza come NGI 4 IGI: Managerial structure EGI Council units Coordinator Unit R&D Pianning Coordination Committee Unit Software Release Unit Operation Management

Responsible Amministration Unit Tarining & Outreach IGI Management Structure a Proposal 45 4 Strategy Keep initially the structure as simple as possble to make it sustainable Allow representation of all active Institutions both at technical and managerial level Leverage from what already exists (from EGEE) and adapt to a broader scope 4 Units with technical responsibilities: 1 Coordination Committee Coordinator Responsible of the administration 4

IGI Units: taks and services (1/2) Unit Operation Management (Infrastructure and Grid Services) Management of Grid central services and catch-all for production infrastructure and test Planning of infrastructure and interoperability Help desk for users site manager Monitoring, accounting Management of other tools and services supporting opertions Grid infrastructure management Grid security (security and incident response coordination in the region) Contribution to EGI operations, technical collaboration with (Unit Operations) Unit software Release (Release mw and certification) Adapt gLie/UMD (Unified Middleware Distribution) to national needs , add required missing components Integration and certification of missing components componenti aggiuntive Technical Collaboration with (Middleware Unit) 4

IGI Units (2/2) Unit Planning (Planning, R&D): Scope: verify user requirements and plan grid infrastructure enhacement Partecipants: 1 rappresentative per IGI Institution Promote partecipations to R&D national and international Calls Plan common R&D activitites (mw and management tools) Unit Training and Application support (Applications and Training) Support to porting of applications Collect new requirements and user feedback Training for users and site managers Dissemination and contacts with industrial partners All IGI Units collaborate and coordinate their activities with

to get common things done with a common unique effort 4 IGI tasks and issues Provide common services for e-Research Common middleware releases General Guidelines Common operation of the Italian Grid Infrastructure Common interface to EGI Define initial set of supported services:

National CA Provision and support of the Grid portal Management of Grid Services (also for applications) Monitoring tool Accounting tool Management and control infrastructure Users support 4 Guidelines for IGI: Operational Model Autonomy: Master common services for local and international work for greater efficiency Avoid multiplication of gateways Sustainability: Pass from project activities to the offer of useful services in long term paid by Research Institutions and Resource Providers Subsidiarity: do things at as local a level as possible ( pulls things together) Reliability: Increase it through pushing responsibility down to sites Scalability Preserve current one in presence of more

middleware stacks to be supported, more non-EGEE Grids integrated, ... 5 EGI Guidelines for NGIs NGIs are expected to: operate in autonomy secure Grid infrastructures in the countries coordinate Grid operations in the countries collaborate to the definition of common operational procedures, policies, standards/specifications adhere to standards/specifications to ensure interoperability support users and operational problems Very much in line with IGI strategy 5 Users and sites support

Devlop the Italian Regional Operation Centre ticketing system based on XOOPS/xHelp Integrated with EGEE/EGI GGUS? (Global Grid UserSupport) ticketing system Web services are used to: Transfer tickets from the global to national system Transfer tickets from the national to the global system The IGI user support group handles the tickets according to addressing At national level when appropriate Either send them to GGUS 5 Access to the IGI GRID Via User Interface (UI). Could be: A dedicated server, installed in a similar way as the others grid

elements UI Plug-and-Play (UI PnP), a software one can install on any pc without root privilegies A web portal: 5 The General Purpose Services Resource Brokers Logging & Bookeeping Top Level BDII

voms servers + replicas gridice server LCG File Catalog server Server MyProxy FTS server SRM Data storage service 5 The future evolution At National level IGI as the other NGIs should make the steps to: Become a legal recognized national organization (NGI) IGI JRU sufficient for the initial phase Take over the operation of the national e-Infrastructure Represent and support all national Research user communities and resource providers (application independent ) Develop MOU for resource sharing

In coordination at European Level with: The European Grid Initiative (EGI) EGI.Org Central Organization Other National Grid Initiatives (NGI) Contribute to prepare a permanent, common production EU Grid infrastructure. Effective Transition from EGEE,DEISA, SEE grid. Promote the integration between National Grid Infrastructures (NGIs) Operate the production Grid infrastructure on a European level for a wide range of scientific disciplines 5 Summary and Conclusions EGEE and related projects have proven that grid infrastructures provide a considerable added value to the research and innovation in Europe

Increasing of scale, Multi-disciplinary, handling of data deluge EGI is now taking over and setting up a new organizational model Resources from current lines of national budget Consolidated grid services (by or NGIs) progressively covered by service charges Technology Innovation and service enhancement EC co-funded Very tight Timetable January 20: NGI Policy Board Blueprint approved March 2009: Select location site (Evaluation ongoing) Set up and prepare EGI EC project End of June: Completed the creation of the EGI Council ( formal acceptance by NGIs of Blueprint + fees) 5

Recently Viewed Presentations