Project

General

Profile

D64 OpenAIRE Maintenance Report » History » Version 52

Marek Horst, 16/12/2016 01:38 PM
removing comments, describing Zenodo Content Status.

1 1 Marek Horst
h1. D6.4 OpenAIRE Maintenance Report (v1, 9th of December 2016)
2
3
{{toc}}
4
5
h2. Overview
6
7 37 Marek Horst
This document contains information about the deployment, status of OpenAIRE2020 services and content, and the history of major modifications of the system hosted at ICM, Poland. Zenodo repository is hosted at CERN, Switzerland.
8 1 Marek Horst
9
The official maintenance of the OpenAIRE2020 services began on January 1st, 2015 when the project started.
10
11 37 Marek Horst
*TODO: elaborate the following*
12 18 Alessia Bardi
The deliverable will consist in a high-level report on the status of 
13
* OpenAIRE workflows (CNR),
14
* services (ICM), 
15 30 Marek Horst
* and Information Space (UNIBI).
16 1 Marek Horst
17 18 Alessia Bardi
h2. Information Space
18 1 Marek Horst
19 44 Jochen Schirrwagen
The OpenAIRE [[Core Data Model]] comprises the following interlinked entites: **results** ( in form of publications, datasets and patents), **persons**, **organisations**, **funders**, **funding streams**, **projects**, and **data sources** (in form of institutional, thematic, and data repositories, Current Research Information Systems (CRIS), thematic and national aggregators, publication catalogues and entity registries)).
20 43 Jochen Schirrwagen
21
22 46 Jochen Schirrwagen
h3. Data Sources
23
24 50 Jochen Schirrwagen
Data sources in OpenAIRE are divided in 
25
* data providers (~805) OpenAIRE continuously collects information from (_downloadFrom_) and need to comply with the "OpenAIRE Guidelines":https://guidelines.openaire.eu
26
* data sources (~6562) where metadata are collected from compatible data providers (usually aggregators)
27
28 46 Jochen Schirrwagen
|*data source type*|*compatibility*|*count*|
29 47 Jochen Schirrwagen
|institutional repository |openaire-3.0 |122 |
30 48 Jochen Schirrwagen
| |openaire-2.0+ |103 |
31
| |openaire-2.0 |10 |
32
| |openaire-basic |252 |
33 47 Jochen Schirrwagen
|thematic repository |openaire-3.0 |5 |
34 48 Jochen Schirrwagen
| |openaire-2.0+ |9 |
35
| |openaire-basic |22 |
36
|other publication repository |openaire-3.0 |8 |
37
| |openaire-2.0+ |1 |
38
| |openaire-2.0 |2 |
39
| |openaire-basic |34 |
40 47 Jochen Schirrwagen
|data repository |openaire-data-2.0 |11 |
41
|publication repository aggregator |openaire-3.0 |5 |
42 48 Jochen Schirrwagen
| |openaire-2.0 |1 |
43
| |openaire-basic |7 |
44 47 Jochen Schirrwagen
|data repository aggregator |openaire-data |1 |
45 48 Jochen Schirrwagen
|journals |openaire-3.0 |71 |
46
| |openaire-2.0 |2 |
47
| |openaire-basic |77 |
48 47 Jochen Schirrwagen
|journal aggregator / publisher |openaire-3.0 |4 |
49 48 Jochen Schirrwagen
| |openaire-basic |26 |
50 1 Marek Horst
| |proprietary |1 |
51
|publication catalogue |proprietary |12 |
52 50 Jochen Schirrwagen
|repository registries|proprietary|2 |
53 1 Marek Horst
|funder databases |proprietary |8 |
54 51 Jochen Schirrwagen
55 52 Marek Horst
The number of data sources is based on the OpenAIRE-index from December 15, 2016.
56 49 Jochen Schirrwagen
57 20 Jochen Schirrwagen
h3. Content Status
58 18 Alessia Bardi
59 45 Jochen Schirrwagen
|*data type*|*count*|
60
|publication metadata|17460368|
61 1 Marek Horst
|dataset metadata|3226586|
62
|projects|653268|
63 18 Alessia Bardi
|organizations|64591|
64 20 Jochen Schirrwagen
|authors|16188328|
65
|EuropePMC XML fulltext|1574358|
66 1 Marek Horst
|PDF fulltext|2227458|
67 29 Marek Horst
68
|*inference type*|*count*|
69
|datasets matched|88610|
70
|projects matched|351302|
71
|software urls references|21481|
72
|protein db references|196462|
73
|research initiatives references|7294|
74
|documents classified|2405869|
75
|similar documents found|164602477|
76
|citations matched by reference text|11390293|
77 1 Marek Horst
|citations matched by id|3929053|
78
79 30 Marek Horst
Tables based on IIS report generated on November 20, 2016 for OpenAIRE production infrastructure.
80
81
h3. Zenodo Content Status
82
83
|*data type*|*count or size*|
84
|records|96771|
85 1 Marek Horst
|managed files|181778|
86 30 Marek Horst
|files total size|8TB|
87
88 52 Marek Horst
Presented numbers are based on Zenodo repository status on December 14, 2016.
89 18 Alessia Bardi
90
h2. [[OpenAIRE workflows]]
91
92
The OpenAIRE aggregation system is based on the "D-NET software toolkit":http://www.d-net.research-infrastructures.eu/. D-NET is a service-oriented framework specifically designed to support developers at constructing custom aggregative infrastructures in a cost-effective way. D-NET offers data management services capable of providing access to different kinds of external data sources, storing and processing information objects of any data models, converting them into common formats, and exposing information objects to third-party applications through a number of standard access API. Most importantly, D-NET offers infrastructure enabling services that facilitate the construction of domain-specific aggregative infrastructures by selecting and configuring the needed services and easily combining them to form autonomic data processing workflows. 
93
94
The Enabling Layer contains the Services supporting the application framework. These provide functionalities such as Service registration, discovery, subscription and notification and data transfer mechanisms through ResultSet Services. Most importantly, these Services are configured to orchestrate Services of other layers to fulfil the OpenAIRE specific requirements and implement the *[[OpenAIRE workflows]]*.
95
96
h2. Services
97
98
How the system is maintained
99
100
h3. Software life-cycle
101
102
h4. D-NET services
103 1 Marek Horst
104
The D-NET services are shipped as web applications and deployed on the tomcat application server (v7.0.52) on three distinct systems: dev, beta, production. To support the deployment process all the software artifacts are automatically built on a continuous integration system ("Jenkins":https://ci.research-infrastructures.eu) and hosted on a dedicated maven repository ("nexus":http://maven.research-infrastructures.eu/nexus), while webapp builds are made available via "http server":http://ppa.research-infrastructures.eu/ci_upload. The mentioned tools supporting the software lifecycle are maintained by CNR.
105
106
The D-NET services deployment is performed in subsequent stages:
107
* The development infrastructure plays the role of test bench for the integration of the software developed by different institutions. It is maintained by CNR and runs mostly un-released code and contains mock or subsets of the data available on the production system.
108
* The beta infrastructure runs only released code. It is maintained by ICM and consists of the final integration stage where all the system workflows are tested on the real data (not necessarily the same data as the production system) before making them available to the production system. Although the software running on the beta system is not yet production ready, its portal is publicly accessible in order to showcase new features and data.
109
* The production infrastructure is maintained by ICM and runs only code that was tested on the beta system.
110 18 Alessia Bardi
111 1 Marek Horst
D-NET backend services are packed in four different web applications, each of them running on a dedicated tomcat instance.
112 18 Alessia Bardi
113 1 Marek Horst
h4. Information Inference Service
114 26 Marek Horst
115
"Information Inference Service":https://github.com/openaire/iis versioning and deployment is described on "IIS versioning and deployment":https://issue.openaire.research-infrastructures.eu/projects/openaire/wiki/IIS_versioning_and_deployment wiki page. 
116
117 1 Marek Horst
Formerly IIS was being deployed on CDH4 cluster. Since October 1st, 2015 dedicated "cdh5 branch":https://github.com/openaire/iis/commits/cdh5 was created where new SPARK modules were introduced and existing modules were optimized. On November 20, 2016 for the first time all inferences in production infrastructure were generated by IIS deployed on new CDH5 OCEAN cluster. Both stability and major performance increase were noticed, inference generation time decreased from over 2 days to 12 hours.
118 18 Alessia Bardi
119 1 Marek Horst
h4. Portal
120 23 Marek Horst
121
OpenAIRE portal is hosted at ICM. It uses Joomla! 3.6.2, a free dynamic portal engine and content management system (CMS).
122
123
The Joomla! depends on other upstream applications:
124
125
* Apache 2.4.7
126
* PHP 5.5.9
127
* MySQL 5.5.53
128 1 Marek Horst
* OpenLDAP 2.4.31
129 18 Alessia Bardi
130 1 Marek Horst
h4. Zenodo
131
132
Zenodo repository employs an instance of the Invenio software, developed by CERN.
133 31 Marek Horst
134
Repository is deployed in a production system (https://zenodo.org) and a QA system (https://sandbox.zenodo.org). In total the two systems are running on 30 VMs hosted in CERNs OpenStack infrastructure. All machines are configured using Puppet and are running on top of CERN CentOS 7.2.
135
136
Zenodo/Invenio depends on the following applications:
137
- HAProxy for load balancing
138
- Nginx for serving static content and proxying request to application server
139
- UWSGI as application server for Zenodo/Invenio application
140
- Redis for memory cache
141
- RabbitMQ as message broker
142
- Celery as distributed task queue
143
- PostgreSQL as database
144
- Elasticsearch as search engine
145
146 1 Marek Horst
Deployment process is described at http://zenodo.readthedocs.io/projectlifecycle.html#release-process
147 33 Marek Horst
148
See https://github.com/zenodo/zenodo/commits/production for changes to Zenodo production system (does not include changes to Invenio modules).
149 35 Marek Horst
150
Zenodo repository was relaunched on Invenio v3 alpha on September 12, 2016.
151 18 Alessia Bardi
152 1 Marek Horst
h3. Infrastructure services
153 18 Alessia Bardi
154 1 Marek Horst
Because OpenAIRE2020 services are a continuation and incremental extension of the services already present that resulted from OpenAIRE+ project, so they are still hosted on that same machines. More details are available at "OpenAIRE+ WP5 Maintenance Report":http://wiki.openaire.eu/xwiki/bin/view/OpenAIREplus%20Specific/WP5%20Maintenance%20Report.
155 18 Alessia Bardi
156 1 Marek Horst
h4. Hadoop clusters
157 18 Alessia Bardi
158 1 Marek Horst
h5. DM hadoop cluster
159
160
CDH version: @cdh4.3.1@
161
162
h5. IIS hadoop cluster
163 24 Marek Horst
164
There were two IIS clusters deployed:
165 39 Marek Horst
* old CDH4 IIS cluster, version @cdh4.3.1@, in operation until December 9, 2016 
166 1 Marek Horst
* new CDH5 IIS cluster deployed on March 22, 2016 in OCEAN infrastructure, supports MRv2 on YARN and SPARK
167 24 Marek Horst
168
CDH5 cluster version history:
169 27 Marek Horst
** @5.5.2@ deployment on March 22, 2016
170
** @5.5.2 -> 5.7.5@ upgrade on November 30, 2016
171 24 Marek Horst
** @5.7.5 -> 5.9.0@ upgrade on December 8, 2016
172 1 Marek Horst
173
h4. Databases
174 28 Marek Horst
175
|_database type_|_usage_|_version_|
176 24 Marek Horst
| postgress | statistics | @9.1.23@ |
177
| postgress | DNet services | @9.3.14@ |
178
| mongodb | DNet services | @3.2.6@ |
179 1 Marek Horst
| virtuoso | LOD | @7.2@ |
180 41 Marek Horst
181 18 Alessia Bardi
h4. Piwik analytics platform
182 24 Marek Horst
183
Currently deployed Piwik version: @2.17.1@, since December 6, 2016.
184 40 Marek Horst
185 24 Marek Horst
h4. ownCloud filesync platform
186 40 Marek Horst
187 18 Alessia Bardi
Deployed at https://box.openaire.eu. Current version: @8.2.7@, since October 19, 2016.
188 1 Marek Horst
189 22 Marek Horst
h3. Architectural changes
190 1 Marek Horst
191 18 Alessia Bardi
[[D64_Servers_Administration_Operations_Changelog|Change Log for servers administration operations]]
192
193 1 Marek Horst
h4. Introducing CDH5 IIS cluster hosted in OCEAN infrastructure
194
195
Slave node specification:
196
* Huawei RH1288 V3
197
* CPU: 2x Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz (24 cores, 48 threads)
198
* RAM: 128GB
199
* HDD: 4x SATA 6TB 7.2K RPM (HDFS)
200
201
Cluster summary (16 slaves):
202
* CPU: 384 cores, 768 threads
203
* RAM: 2048GB
204
* HDD: 384TB (HDFS)
205
206
YARN available resources:
207
* vcores: 640
208
* memory: 1.44TB
209 18 Alessia Bardi
* HDFS: 344TB
210 1 Marek Horst
211
h4. Incorporating resources from old CDH4 IIS cluster into existing DM CDH4 cluster
212 25 Marek Horst
213 1 Marek Horst
This task was possible after shutting down old IIS CDH4 cluster what happened on December 9, 2016.
214 25 Marek Horst
215
h4. Deploying DNet postgress and mongodb databases on separate machines
216
217
Separating @openaire-services@ database instances into dedicated ones (since June 27, 2016):
218
* @openaire-services-postgresql@
219
* @openaire-services-mongodb@
220 34 Marek Horst
221 1 Marek Horst
h4. Updating Zenodo repository infrastructure at CERN
222 34 Marek Horst
223
Several architectural changes were introduced in CERN's infrastructure:
224
* changed storage backend from OpenAFS to CERN EOS (18 PB disk cluster) for better scalability
225
* changed from self-managed MySQL database to CERN Database Team managed PostgreSQL database
226
* deployment of Elasticsearch clusters (6 VMs)
227
* SLC to CentOS 7 on all 30 VMs
228 1 Marek Horst
229
h3. System downtimes
230 36 Marek Horst
231
h4. @ICM
232 1 Marek Horst
233 36 Marek Horst
* [planned] November 14, 2016, 2 hours. #2423 dealing with Linux Dirty COW vulnerability: kernel upgrade, OpenAIRE services restart.
234
235
h4. @CERN
236
237
* [unplanned] September 16, 2016, 3 hours. Preparation of a new load balancer caused an automatic update of CERN outerperimeter firewall that automatically closed access to the operational load balancers.
238
* [planned] September 12, 2016, 8 hours. Complete migration from old infrastructure to new infrastructure
239 1 Marek Horst
* minor incidents until September 12, 2016 due to overload of the legacy system