Change Log for servers administration operations¶

2nd Period (14/12/2016 - 30/06/2018)¶

2nd period list of tasks was based on tickets defined in OpenAIRE's redmine issue tracker and mantis - ICM's internal task tracker.

June 2018:

#3756 Add new user on beta.services.openaire.eu
#3746 Update beta.openaire.eu joomla
#3688 monitor.openaire.eu - launch publicly available
#3651 Problem with LDAP login in beta
#3479 LDAP service migration at CNR
#3409 [PROD] New Solr server cluster deployment
#3704 Access issues LDAP - administrator portal
#2097 Update arXiv pdfs on the ICM ftp server (repeated every month)
#3712 Older instance of postgres backup
#3391 Rename beta.repomanager.openaire.eu
#3703 Open ssh access of Beta from new CNR machine
#3710 Configuration for profiles API

May 2018:

#3667 Cannot update beta index
#2913 Occasional 502 Proxy Error while communicating with Oozie server deployed on IIS cluster
#3646 Lost ssh access to openaire-beta vm
#3657 Let Alessia enter on mongodb.services.openaire.eu
#3648 Delete piwik site
#3575 Location of front end portal files
#2834 Access to piwik site generation script from beta.openaire
#3469 Monitoring mongodb on mongodb.services.openaire.eu
#3630 access to box.openaire.eu
mantis#9574 Uploading two updated configuration files required by integration tests suite on ci.ceon.pl (performed several times throughout the project)
mantis#9768 Prometheus and Grafana monitoring services deployment

April 2018:

#3622 Add Robert Paciorek to CNR's LDAP and ICM Sysadmins group
#3619 Internal oozie server error when accessing web console via HTTP
#3442 Request for VM (Application Server)
#3610 Cloudera Manager (CDH) 5.9 issue with HUE and LDAP authentication
#3209 Machine for Shibboleth IDP
#3425 Network FS between virtuoso VM and DM cluster
#3579 Configuration of uoa-admin-tools
#3592 Handshake failure ("unrecognized_name")
#3204 Machine to host the OpenAIRE SSO service
mantis#9736 Making ResourceManager being responsive again on IIS CI Test cluster

March 2018:

#2785 DLI services and ElasticSearch cluster deployment
#3502 Data source harvesting - IP problem(?)
#3557 Transactional mail service
#3414 MongoDB in beta.services.openaire.eu
#3189 OpenAIRE API rate limits
#2449 Portal SEO
#3114 sms: visualizing tool LOD
#3449 Make statistics databases deployed in beta and production accessible from PTM machine
#3486 Error in Matomo (piwik)
#3208 Update BETA user management service with reset password and remind username fuctionalities
#3215 BETA 8380 in some cases fails to restart
#3003 repository hostname not resolved

February 2018:

#3393 howto access to new Hadoop cluster cdh5
#3387 Support contact email for OpenAIRE AAI
#3112 Give sudo access to panagiotis.kanakakis to pilot.services.openaire.eu
#3184 Export project details for Zenodo
#3389 Statistics on LOD visits

January 2018:

#3193 openaire.eu mail senders/SPF settings for openaire.eu
#3330 LD-R is not available
#3438 Files are missing from box.openaire.eu
#2872 Access to Arthur Lackner
#3164 url encoding problem
#2869 LOD Link not working
#3379 Access on ICM to Arthur Lackner
#3195 Machine for the OpenAIRE anonymization tool
#3364 Increase disk space for /vol/openaire_objectstore_pdfs on services.openaire.eu
#3290 Add eduPerson schema to beta ldap
#3352 Openaire DM Hadoop: gateway:/mnt/tmp - out of inodes

December 2017:

#2284 www.openaire.eu returns sometimes ERR_CONNECTION_REFUSED
#3362 Clean up tmp diretory of services.openaire.eu
#3334 user account
#2855 Prepare dedicated machine for PTM
#3348 Unable to connect with sites via HTTPS requiring AES 256 ciphers
#3354 Remove authentication from demo.openaire.eu
#3301 monitor.openaire.eu/administrator returns internal server error
#3212 OpenAIRE newsletter statistics
#3333 guidelines.openaire.eu does not rebuild from github repo sources
#3312 Connection problem to hadoop

November 2017:

#3329 In demo.openaire.eu: add alias beta.explore.openaire.eu
#3325 api.openaire.eu: add hostname develop.openaire.eu
#2809 Bots and deep paging
#3306 is there a postgres available for beta.repomanager.openaire.eu
#3261 New node js app for demo portal
#3294 Freestyle support plugin, need of a table
#3241 provide access to the hadoop cluster
#3017 A new vm machine for monitoring - Joomla
#3287 check monitor.openaire.eu after rrecovery due to crash
#3284 www.openaire.eu returns 403 forbidden
#3159 503 Service Unavailable when opening 10 tabs with publication details
#3277 install curl on gateway.hadoop.dm.openaire.eu
#2984 A new vm machine for Joomla
#3279 please help with ssh login
#2597 Bad response from api.openaire.eu/oai_pmh

October 2017:

#3223 Need to run dist-upgrade on beta.services.openaire.eu and services.openaire.eu
#2566 Stats tab of big aggregators like La Referencia does not load in reasonable time
#3217 Domain for datathon
#3179 New api documentation pages
#3160 URL encoding: different settings on beta and production?
#3143 Production upgrade to Java8

September 2017:

#3161 Update proxy mappings for api.openaire.eu
#2950 Upgrade beta aggregator to java8
#3121 Disable connections from beta.services.openaire.eu to postgresql.services.openaire.eu
#3083 Federeted Queries over SPARQL endpoint give different results

August 2017:

#3071 CNR LDAP - request for account

July 2017:

#3016 Problem accessing project API: 503 Service Unavailable
#2840 setup ldap authentication for OCEAN hadoop services
#2999 New spring boot web application to deploy
#2549 Machine for a new portal
#2246 investigate usage of api.openaire.research-infrastructures.eu
#2876 LOD Ontology (vocabs)
#3006 enable CORS at analytics.openaire.eu/piwik.php
#2541 Enable CORS for Vocabularies API
#2313 dismiss dnet-hadoop-parent
#2763 struts2 vulnerability in the validator
#2754 Hardening the services machines
#2998 analytics.openaire.eu is not responding
#2979 Spark History Server on IIS cluster is down
#2974 enable access from claims service to user-management service
mantis#9693 Making spark history server available again on CDH5 cluster
mantis#9675 Fixing spark history server webapp returning 503 error
mantis#9654 Configuring apache httpd on hadoop clusters
mantis#9525 Adjusting puppeta and monit configuration on hadoop-ci-test
mantis#9738 Dropping openaire libs out of local maven repo on jenkins machine
mantis#9737 Providing jenkins worker SSH access to iis-ci-test.ocean.icm.edu.pl

June 2017:

#2968 information system backup frequency
#2957 openaire-dm-hadoop-s4 problems
#2945 Pilot proxy changes
#2463 trigger the generation of $idsite in Piwik for dataproviders
#2705 new datasource: glasstree
#2947 IIS CDH5 Ocean cluster is down
#1828 Monitor the access to the OpenAIRE API
#2865 Access to Google webmaster tools
#2914 jobtracker(s) temporarily unavailable
#2323 Limit access to the direct index API
#2798 catalina.out of 8280: record is not compliant, transformation is interrupted.
mantis#9735 Fixing IIS build failures

May 2017:

#2813 Preparing demo.openaire.eu
#2837 User Management log files
#2804 Enable https in demo.openaire.eu
#2639 More RAM for openaire-solr-beta.vls
#2580 Environment Preparation
#2761 Different containers logs design for a production & beta services
#2800 Cannot login on the validator
#2831 Metrics API
#2825 disk space on beta solr server
#2812 "dnet8480" in 8480 has no "w" rights on /var/log/dnet/search/
#2817 UBONN access to ocean cluster

April 2017:

#2477 Use cached values for information coming from IS
#2799 More RAM for openaire-services.vls
#2790 Different size of PostgreSQL "stats" databases on openaire-stats-beta.vls and openaire-stats.vls
#2771 create dnet8280 user on DM cluster
#2769 Shutdown of furkot2 (aka old.openaire.eu)
mantis#9725 Hooking up OpenAIRE IIS with sonar.ceon.pl
mantis#9678 Optimizing memory configuration IIS CDH5 cluster

March 2017:

#2760 Copy directories from netapp1-nfs:/vol/openaire_objectstore_pdfs to netapp2-nfs:/vol/openaire_beta_objectstore_pdfs
#2758 Switching api.openaire.eu from furkot2.vls to openaire-www.vls
#2753 Need access to the IIS HDFS
#2249 jobtracker returns error 500
#2581 Virtuoso for LOD production
#2175 MongoDB backend for the production services
#2584 ftp account for exeley
#2431 Piwik Authentication Token
#2280 google fails to fetch robots.txt
#2520 Java and SSL: Could not generate DH keypair
#2051 Jobtracker@DM error 500
mantis#9723 Fixing lost node on hadoop-ci-test.vls.icm.edu.pl
mantis#9724 Providing access to yarn web interfaces on hadoop-ci-test
mantis#9719 Fixing Resource Manager failures ON IIS CI Test cluster
mantis#9709 Renaming OCEAN cluster hosts

February 2017:

#2513 Gathering portal piwik logs for website usage analysis
#2637 DIGITAL.CSIC Piwik Tracking
#2426 Comparison of usage logs
#2603 [solr.openaire.eu] memory size
#2594 Piwik Robots List Update
#2393 Migration to new Cluster and SPARK for LOD
mantis#9691 Updating iis-ci-test.ocean.icm.edu.pl requirements
mantis#9699 Fixing random documents classification failures
mantis#9710 Considering upgrade to 5.10.0, assessing potential risks
mantis#9711 Fixing virtual name node name resolution on CDH5 IIS cluster
mantis#9716 Dealing with Internal Server Error on IIS OCEAN CDH5 cluster master nodes

January 2017:

#2524 SSLHandshakeException when updating BETA stats caches
#2565 Security problem: SOAP methods are not protected
#2558 ldap permissions to access test.openaire.eu
mantis#9695 Fixing DataNode unavailable to NameNode on iis-ci-test.ocean.icm.edu.pl

December 2016:

#2494 Stats update job failed on BETA
#2511 IIS CDH4 cluster resource re-allocation
#2521 Secured access to mining.openaire.eu
#2523 wild tomcat running on services.openaire.eu
mantis#9688 Updating CDH5 cluster versions to 5.9.0
mantis#9687 Purging and reallocating old CDH4 cluster resources
mantis#9636 Fixing CDH5 integration test cluster jobs deadlock
mantis#9631 Upgrading CDH5 integration test cluster

1st Period (01/01/2015 - 13/12/2016)¶

December 13, 2016:
  - Updated Oozie sharelibs on ci-test.hadoop5.iis.openaire.eu, see Mantis #9691
  - Generated SSL cert signed by Let's Encrypt for mining.openaire.eu and enabled HTTPS for that site, see Redmine #2521
  - Removed old and unused LVM volumes on Xen openaire-dm-hadoop⁴⁵-dom0 machines (they replaced openaire-iis-hadoop¹²-dom0)
  - Created LVM volumes for openaire-dm-hadoop-s[11-16] on openaire-dm-hadoop⁴⁵-dom0, see Redmine #2511
December 12, 2016:
  - Deployed iis-ci-test.ocean.icm.edu.pl, a new pseudo Hadoop cluster for running IIS integration tests on CDH5, see Mantis #9688
  - Created a new pseudo Hadoop cluster (IIS CI Test) for running IIS integration tests on CDH5 maintained by Cloudera Manager on eos-cr-mgmt1.ocean.icm.edu.pl, see above
December 09, 2016:
  - Upgraded CDH (from 5.7.5 to 5.9.0) also on ci-test.hadoop5.iis.openaire.eu, see Mantis #9681
  - Removed HBase from ci-test.hadoop5.iis.openaire.eu, see above
  - Shutdowned all machines of the old IIS CDH4 Hadoop cluster, see Redmine #2512
  - Shutdowned openaire-iis-hadoop4-dom0 machine
  - Renamed openaire-iis-hadoop1-dom0 machine to openaire-dm-hadoop4-dom0 and openaire-iis-hadoop2-dom0 machine to openaire-dm-hadoop5-dom0
December 08, 2016:
  - Upgraded CDH (from 5.7.5 to 5.9.0) on OCEAN IIS CDH5 Test cluster, see Mantis #9681
December 07, 2016:
  - Created a ticket for announcing shutdown of IIS CDH4 Hadoop cluster (see Redmine #2511)
  - Upgraded Cloudera Manager agents for OCEAN IIS CDH5 Test cluser from 5.7.0 to 5.9.0 because of Cloudera Manager server upgrade
December 06, 2016:
  - Closed the Redmine tickets #2344, #2431, #2463
  - Upgraded Piwik for OpenAIRE (https://analytics.openaire.eu/)) from version 2.16.5 to 2.17.1.
December 02, 2016:
  - Added missing index.html files in components/, modules/ and plugins/ subdirectories of the portal Joomla, see Redmine #2484
  - Removed Disallow: lines from robots.txt of the portal Joomla for those subdirectories, see above
December 01, 2016:
  - Removed HBase from OCEAN IIS CDH5 Test cluster, see Mantis #9678
  - Tuned some OCEAN IIS CDH5 Test settings for YARN (yarn.nodemanager.resource.cpu-vcores=40, yarn.nodemanager.resource.memory-mb=92GB), see above
  - Started ntpd on openaire-www-(proxy|front|back)2, because it seems it was inactive (dead), hint: "Could not send report: SSL_connect returned=1 errno=0 state=error: certificate verify failed. This is often because the time is out of sync on the server or client"
November 30, 2016:
  - Fixed a broken backup of MongoDB on openaire-services.vls, because it failed a few days ago (November 26)
  - Enabled a backup of MongoDB on openaire-services-mongodb.vls
  - Removed an old & stale backup of MongoDB on openaire-services-beta.vls
  - Enabled a backup of MongoDB & PostgreSQL on openaire-services-beta.vls
  - Recreated a broken SSL cert for Puppet on openaire-www-(proxy|front|back)2.vls (Could not request certificate: stack level too deep)
  - Upgraded CDH (from 5.5.2 to 5.7.5) on OCEAN IIS CDH5 Test cluster because of CVE-2016-5393 Apache Hadoop Privilege escalation vulnerability, see Mantis #9681
  - Fixed timezone on OCEAN IIS CDH5 Test cluster machines (now we use GMT everywhere), see above
  - Fixed /proc/sys/vm/swappiness on iis-cdh5-test-gw & iis-cdh5-test-m[1-3] machines, see above
  - Fixed /sys/kernel/mm/transparent_hugepage/defrag on eos-m2-sn40 machine, see above
  - Upgraded CDH (from 5.5.2 to 5.7.5) also on ci-test.hadoop5.iis.openaire.eu, see above
November 29, 2016:
  - Fixed Apache configuration of http://guidelines.openaire.eu/ for DocumenRoot
November 28, 2016:
  - Purged the old logs of HDFS NameNode, balancer and Zookeeper on namenode1.hadoop.dm.openaire.eu because of a near full-filled / partition
  - Fixed Exim4 aliases on namenode1.hadoop.dm.openaire.eu & master1.hadoop.iis.openaire.eu in order to send to me the messages from HDFS balancer script launched by Cron (Mail delivery failed: returning message to sender: hdfs@namenode1.hadoop.dm.openaire.eu retry timeout exceeded)
  - Tuned yarn.scheduler.maximum-allocation-mb (16GB) YARN setting of OCEAN IIS CDH5 Test cluster, see Mantis #9678
  - Stopped HBase of OCEAN IIS CDH5 Test cluster, see above
  - Updated and restarted openaire-www-(proxy|front|back)1.vls & openaire-www-front3 machines
  - Started, updated and restarted again openaire-www-(proxy|front|back)2.vls machines
November 25, 2016:
  - Tuned yarn.scheduler.maximum-allocation-vcores (6) YARN setting of OCEAN IIS CDH5 Test cluster, see Mantis #9678
November 24, 2016:
  - Recreated a broken SSL cert for Puppet on openaire-iis-hadoop3-dom0.vls (Could not send report: SSL_CTX_use_PrivateKey:: key values mismatch)
November 23, 2016:
  - Fixed Apache configuration for a reverse proxy for CM on eos-cr-mgmt1.ocean.icm.edu.pl, because the clusters settings were changable only in a classic layout of CM, see Cloudera Case #117145
  - Fixed robots.txt of OpenAIRE portal, because Googlebot couldn't grab images/CSS/JavaScript/etc, see Redmine #2484
November 22, 2016:
  - Recreated a broken SSL cert for Puppet on openaire-www-test.vls (Could not send report: SSL_CTX_use_PrivateKey:: key values mismatch)
  - Tuned some OCEAN IIS CDH5 Test settings for YARN (mapreduce.map.memory.mb=0, mapreduce.map.java.opts.max.heap=1638MB, mapreduce.reduce.memory.mb=0, mapreduce.reduce.java.opts.max.heap=3276MB), see Mantis #9678
November 21, 2016:
  - Started container 8480 on openaire-services.vls, because it crashed by night, see Redmine #2477
  - Fixed owner of /var/log/nginx/ directory (from www-data: to root:) on furkot2, because of CVE-2016-1247 vulnerability against Nginx, for more details see https://legalhackers.com/advisories/Nginx-Exploit-Deb-Root-PrivEsc-CVE-2016-1247.html
  - Fixed Apache configuration of http://guidelines.openaire.eu/ for /robots.txt & /favicon.ico requests
  - Allowed connections to OpenAIRE Piwik from Dimitris Pierrakos machine without authentication against CNR LDAP server, because he needs it for sending requests to Piwik API
November 18, 2016:
  - michal.politowski added to groups "driver" & "joomla" on openaire.vls, because he wants to see the Apache logs
  - Moved /etc/apache2/sites-enabled/joomla.conf to /etc/apache2/sites-enabled/000-joomla.conf after Michal Politowski email about OpenAIRE portal SEO (hint SNI)
  - Fixed Google Analytics JavaScript code URL (from HTTP to HTTPS) on the portal using Apache's mod_substitute directive, see Redmine #2449
November 17, 2016:
  - Started HBase-Master on openaire-dm-hadoop-hb-m1.vls because it was dead
  - Created "usage_openaire" account in Piwik and granted him the view access for all current and future websites we truck by Piwik, see Redmine #2431
  - Created HDFS home directory for giorgos.alexiou on OCEAN IIS CDH5 Test cluster, see Redmine #2393
November 16, 2016:
  - Tested the PHP script addsite.php for creating a new Piwik websites, see Redmine #2463
  - Added the trusted NKUA/NTUA/Athena RC networks on the firewall of OCEAN IIS CDH5 Test cluster gateway, see Redmine #2393
  - Added giorgos.alexiou to "driver" group there, see above
November 15, 2016:
  - Installed Java 8 on openaire-virtuoso-beta, asked by Shirin Ameri, see Redmine #2344
  - Discussion about generating automatically $idsite of Piwik for a new aggregated repos, see Redmine #2463
  - Started working on a PHP script for creating a new Piwik websites, see above
November 14, 2016:
  - Upgraded and restarted the production machines (furkot2, openaire-www, openaire-stats, openaire-solr, openaire-services, openaire-services-mongodb, openaire-services-postgresql, openaire-virtuoso) with a new kernel fixed against Dirty COW bug, see Redmine #2423
November 10, 2016:
  - Added shirin.ameri to "driver" group on openaire-virtuoso-beta.vls, see Redmine #2344
November 08, 2016:
  - Closed the Redmine tickets #2100, #2342
November 07, 2016:
  - Further cleaning of infected Wordpress on https://blogs.openaire.eu/:: removed infected themes (guardian, twentythirteen & twentytwelve), restored infected wp-content/themes/wp-pinup/header.php file
  - Fixed sudo config on gateway.hadoop.iis.openaire.eu (now `sudo su - dnet ...` & `sudo su dnet ...` commands are defined separately), reported by K.Zamlynska
  - Fixed also sudo on gateway.hadoop.dm.openaire.eu, see above
  - Increased a NFS volume openaire_beta_objectstore_pdfs mounted on beta.services.openaire.eu as /mnt/objectstore_pdfs from 6TB to 8TB, because it was filled in 100%
  - Restarted Spark History Server of OCEAN CDH5 Test cluster, because it seemed to be dead, reported by M.Horst, see Mantis #9675
November 04, 2016:
  - Cleaned infected Wordpress on https://blogs.openaire.eu/
  - Sent to Claudio the current numbers of the hardware resources at ICM
  - Downgraded Java 8u91 to 8u66 on ci-test.hadoop5.iis.openaire.eu, because of a problem with Oozie, reported by M.Horst, see Redmine #2423
November 03, 2016:
  - Renewed SSL cert of Puppet for all OpenAIRE machines at ICM because of expired Puppet master SSL cert
  - Fixed a network issue on eos-m2-gw machine where iis-cdh5-test-gw is hosted after its reboot, reported by M.Horst, see Redmine #2423
  - Enabled IP forwarding on the OCEAN IIS CDH5 Test cluster gateway, but it won't be used because the cluster slaves get IP address and a default gateway from DHCP server...
  - Fixed broken routing on a default gateway for the OCEAN IIS CDH5 Test cluster after its reboot (hint: net.ipv4.ip_forward=1 & net.ipv4.conf.all.rp_filter=0), reported by M.Horst, see Redmine #2423
  - Purged the old logs of HDFS, YARN and Oozie on ci-test.hadoop5.iis.openaire.eu because of a near full-filled / partition
November 02, 2016:
  - Closed the Redmine tickets #2249, #2284, #2311, #2323, #2330, #2344, #2375
  - Configured clustershell for OCEAN IIS CDH5 Test cluster on the cluster gateway
  - Upgraded and restarted OCEAN IIS CDH5 Test cluster related machines (iis-cdh5-test-gw, iis-cdh5-test-m[1-3], eos-m2-gw, eos-m2-mn[1-3], eos-m2-sn0[1-8], eos-m2-sn[35-42]) with a new kernel fixed against Dirty COW bug, see Redmine #2423
  - Erased "chrony" package from some OCEAN IIS CDH5 Test cluster related machines, because there was in a conflict with "ntpd" service
  - Upgraded and restarted openaire-beta-(www|stats|solr|services).vls and openaire-services-pilot with a new kernel fixed against Dirty COW bug, see Redmine #2423
October 28, 2016:
  - Upgraded and restarted openaire-www-test.vls with a new kernel fixed against Dirty COW bug, see Redmine #2423
  - Upgraded and restarted openaire-dm-hadoop[1-3]-dom0 with a new kernel fixed against Dirty COW bug, see above
  - Upgraded and restarted openaire-iis-hadoop[1-4]-dom0 with a new kernel fixed against Dirty COW bug, see above
October 27, 2016:
  - Started 8380 on openaire-services-pilot.vls, because it was dead, reported by Natalia
  - Added forgotten monitoring of 8380 on openaire-services-pilot.vls, see above
October 24, 2016:
  - Fixed sudo for becoming "dnet" user on DM/IIS Hadoop cluster gateway, see Redmine #2413
  - Increased size (from 400GB to 500GB) of /srv partition on beta.solr.openaire.eu
October 21, 2016:
  - Allowed connections from DM Hadoop cluster to MongoDB on mongodb.services.openaire.eu, see Redmine #2175
October 19, 2016:
  - Upgraded Piwik for OpenAIRE (https://analytics.openaire.eu/)) from version 2.16.2 to 2.16.5.
  - Upgraded ownCloud on box.openaire.eu from 8.2.2 to 8.2.7
October 18, 2016:
  - Fixed a content of /var/run/tomcat7.pid file on openaire-virtuoso-beta.vls, because Monit unmonitored Tomcat there, but it was running well
  - Generated Let's Encrypt SSL certificates for (beta.)virtuoso.openaire.eu
  - Closed Redmine #2390 ticket
  - Upgraded ownCloud on box-test.openaire.eu from 8.2.2 to 8.2.7
October 17, 2016:
  - Generated Let's Encrypt SSL certificates for (analytics|api|beta|beta.services|box|box-test|goldoapilot|lareferencia|old|services|validator).openaire.eu
October 7, 2016:
  - Generated and installed a test SSL cert from Let's Encrypt CA on test.openaire.eu
  - Generated only a new SSL cert from Let's Encrypt CA on www.openaire.eu, because Natalia don't want to buy a new one (see Redmine #2291)
October 6, 2016:
  - Started Spark Worker on ci-test.hadoop5.iis.openaire.eu
  - Firewall rules for Hadoop daemons web UIs cleanup, see Mantis #9567
October 5, 2016:
  - Enabled an access to port 8980 on openaire-services.vls and openaire-services-beta.vls from OCEAN IIS Hadoop, see Redmine #2352
October 4, 2016:
  - Enabled an access to openaire-services-(mongodb|postgresql).vls via SSH from CNR IPv6 network, see Redmine #2175
  - Enabled IPv6 support for MongoDB on openaire-services-mongodb.vls, see above
October 3, 2016:
  - Enabled an access from openaire-services.vls to 27017 port (MongoDB) on openaire-services-mongodb.vls, see Redmine #2175
September 30, 2016:
  - Investigated an issue with Oozie and MySQL timeouts, see Redmine #2324
SEPTEMBER 29, 2016:
  - Restarted 8580 container on openaire-services-pilot.vls, because it seems that it hanged (the process was running, but it wasn't listening at 8580 port)
September 28, 2016:
  - Upgraded CDH (from 4.3.1 to 4.7.0) on DM Hadoop cluster, see Redmine #2324
  - Upgraded Java (from 7u80 to 8u101) there, see above
September 27, 2016:
  - Removed the old logs (from 2012 to 2014) of koprowy on the logserver on mnich because of a lack of free disk space on / there
September 26, 2016:
  - Reported an error with sh404sef plugin of a production Joomla (Sh404sefHelperLanguage::getUrlCodeFromTag::182: Language tag en-GB not in installed language list), see Redmine #2342
  - Purged the old logs of HDFS, YARN and Oozie on ci-test.hadoop5.iis.openaire.eu because of a near full-filled / partition
  - Investigated Alessia's issue with accessing 8980 on openaire-services.vls and openaire-services-beta.vls, see Redmine #2323
  - Added Alessia's IP address to a list of IPs allowed to access 8980 on both services machines, see above
  - Created a Cron job for "postgres" user on openaire-services-postgresql.vls and openaire-services-beta.vls for refreshing some materialized views of dnet_openaireplus database, asked by Alessia, see Redmine #2343
  - Set Nginx rules for authentication of https://services.openaire.eu/is/mvc/api/publications/, see Redmine #2323
September 24, 2016:
  - Removed the old Joomla backups and gzipped sh404sef logs because of lack of free disk space on /srv on openaire-www.vls
September 23, 2016:
  - Investigating an issue with offline regions of HBase on DM Hadoop cluster, reported by Claudio, Alessia and Tsampikos
  - Restarted HBase masters of DM Hadoop cluster
September 21, 2016:
  - Helping with an issue with Jenkins for IIS, see Mantis #9656
  - Restarted openaire-www.vls in order to increase its resources (8GB ~~> 18GB RAM, 4 CPU cores -> 8 CPU cores), see Redmine #2330~~
   Decreased CPU of openaire-www.vls (8 CPU cores ~~> 6 CPU cores) and restarted it again because of issue with a big load, the stolen CPUs etc~~
   Shutdowned openaire-www-(proxy|front|back)2 on prod2-dom0, see above
September 20, 2016:
  - Announced restart of openaire-www.vls in order to increase the resources, see Redmine #2330
September 19, 2016:
  - Set Nginx rules for authentication of https://beta.services.openaire.eu/is/mvc/api/publications/, see Redmine #2323
  - Investigating Sahar's troubles with access to beta.lod.openaire.eu, see Redmine #1748
  - Bumped a limit of the requests per minute in Fail2Ban (apache-badhosts jail) on www.openaire.eu, because it was making the troubles with the portal access (ERR_CONNECTION_REFUSED reported by Katerina, Paolo & Natalia)
September 16, 2016:
  - Set .deb packages source for CDH 4.7.0 on DM Hadoop cluster machines, see Redmine #2324
September 15, 2016:
  - Prepared .tar file with the Apache logs of the production Joomla for Dimitris Pierrakos, he needs them for any NKUA student
September 14, 2016:
  - Fixed .htaccess for (www|beta|test).openaire.eu, because it didn't support index.html for DirectoryIndex, see Redmine #2231
  - Fixed Apache auth configuration for /administration requests
September 13, 2016:
  - Fixed monitoring of 8380 on openaire-services-beta.vls, added monitoring of 8480 and switched monitoring of the search to 8480, see Redmine #2274
  - Blocked permanently communication between beta.services.openaire.eu and services.openaire.eu, asked by Alessia, see above
  - Investigating an issue with distcp from OCEAN IIS CDH5 Test cluster to OpenAIRE DM cluster, seecluster to OpenAIRE DM cluster, see Mantis #9621
  - Fixed the firewall rules on the gateway of OpenAIRE DM cluster (added 213.135.60.145 for 50071 port), see above
  - Decreased Apache2 limits on openaire-www.vls (ServerLimit=MaxRequestWorkers=120)
September 12, 2016:
  - Investigating an issue with Monit alerts about the failed beta search, asked by Katerina, see Redmine #2274
  - Fixed checking the search on beta.services.openaire.eu, see above
  - Copied 8480 from the production services to the beta one, asked by Katerina, see above
  - Investigating ERR_CONNECTION_REFUSED errors for some portal pages, see Redmine #2284
September 8, 2016:
  - Enabled access to OCEAN CDH5 Hadoop Test cluster from OpenAIRE services machines (both production & beta), asked by Claudio, see Redmine #2297
  - Granted access to OCEAN CDH5 Hadoop Test cluster web UIs for Claudio, see above
September 7, 2016:
  - Installed "siege" package on openaire-services-beta.vls, asked by Katerina, see Redmine #2274
  - Checking what Piwik sites can be accessed by Dimitris Pierrakos, asked by Dimitris
  - Granted the admin privileges for OpenAIRE portal beta site in Piwik to Dimitris, see above
  - Filled a ticket about renewal of SSL cert for .openaire.eu, see Redmine #2291
September 5, 2016:
  - Created backup of Joomla on www.openaire.eu, asked by Argiro
September 1, 2016:
  - Fixed home & siteurl parameters for blogs.openaire.eu Wordpress (was HTTPS instead of HTTP there and caused a redirection loop for /wp-login.php), reported by Ross & Pedro
August 31, 2016:
  - Reported issue with stats.openaire.eu murdered by the stats SELECTs from test.openaire.eu, see Redmine #2282
  - Restarted Apache on test.openaire.eu and killed SELECT processes coming from test.openaire.eu on stats.openaire.eu, see above
August 30, 2016:
  - Investigated issues with test.openaire.eu (failed workflows, internal errors etc), reported by Claudio, Tsmapikos & Katerina
  - Production Joomla redirect from HTTP to HTTPS returns now status code 301 (Moved Permanently) instead of 302 (Moved Temporarily)
August 29, 2016:
  - Created a ticket for investigating last issues with the search (see Redmine #2274)
  - Investigated 500 errors for jobtracker1.hadoop.dm.openaire.eu (reported by Claudio again, see Redmine #2249)
  - Restarted preemptively both JobTrackers of DM Hadoop cluster (see above)
  - Replaced Eri Katsari by Tsampikos Livisianos in the Monit alert lists, asked by Eri
  - Purged the old logs of HBase and Zookeeper on hbase-master1.hadoop.dm.openaire.eu because of a near full-filled / partition
  - Purged the old logs of HDFS and YARN on ci-test.hadoop5.iis.openaire.eu because of a near full-filled / partition
  - Restarted Virtuoso on virtuoso-beta.openaire.eu, because SPARQL endpoint was down (reported by Sahar, see Redmine #2277)
  - Investigated issues with Googlebot requests for /robots.txt, see Redmine #2280
  - Configured Apache on openaire-www.vls for serving also http://www.openaire.eu/sitemap.xml without redirecting to HTTPS (see Redmine #2065)
August 12, 2016:
  - Restarted a few machines of OCEAN IIS CDH5 Hadoop cluster in order to fix YARN NodeManager issue, see Mantis #9621
  - Fixed OpenAIRE blogs Wordpress settings broken by Pedro
  - Polished some Apache rules for fixing URLs on blogs.openaire.eu
August 11, 2016:
  - Removed a default gateway for IB network interface (mlx4_ib0.8002) from a routing table of OCEAN IIS CDH5 Test Hadoop cluster slave nodes
  - Upgraded Piwik for OpenAIRE (https://analytics.openaire.eu/)) from version 2.16.1 to 2.16.2.
August 10, 2016:
  - Fixed also HTTPS links for blogs.openaire.eu, see Redmine #1888
  - Investigated an issue with HDFS distcp invoked by Ooozie for OCEAN IIS CDH5 Hadoop cluster, see Mantis #9621
August 9, 2016:
  - Investigated MongoDB backup issue on services.openaire.eu which failed a few days ago because of "Failed: error reading collection: EOF" error
  - Deployed monitoring of memory used by JobTracker of IIS Hadoop cluster (helpful together with the script for a preventive restart, see below)
  - Fixed JavaScript HTTP links for blogs.openaire.eu, reported by Pedro Principe, see Redmine #1888
August 4, 2016:
  - "driver" group can manage Redis on stats.openaire.eu & beta.stats.openaire.eu now (asked by Antonis)
  - "driver" group can manage Tomcat7 on beta.stats.openaire.eu now (the same like for stats.openaire.eu)
  - "driver" group can become "www-data" user on stats.openaire.eu now (the same like for beta.stats.openaire.eu)
  - Enabled a preventive restart of JobTracker by Cron on master¹².hadoop.iis.openaire.eu (asked by Marek Horst)
  - Synced Redis database the beta and the production stats machines (asked by Antonis)
  - Enabled monitoring of Redis server on the beta stats machine
August 3, 2016:
  - Fixed a replica placement policy issue (block should be additionally replicated on 1 more rack(s)) for HDFS of DM Hadoop cluster
  - Moved a reverse proxy for blogs.openaire.eu from furkot2.vls to openaire-www.vls (see Redmine #1888)
  - Moved a reverse proxy for mining.openaire.eu from furkot2.vls to openaire-www.vls
August 2, 2016:
  - Removed Nikon Gasparis from Monit alert list on openaire-services-pilot.vls
  - Investigated an unsucessfully finished MongoDB backup on services.openaire.eu
  - Investigated an unbalanced HDFS of DM Hadoop cluster (from 44.19% for slave9 to 54.09% for slave5)
  - tsampikos.livisianos added to "driver" group on beta.stats.openaire.eu (see Redmine #2152)
  - Enabled monitoring of nscd & nslcd daemons on the machines which authenticate users against CNR LDAP server (see Redmine #2258)
August 1, 2016:
  - Deployed Webalizer for beta.virtuoso.openaire.eu
July 29, 2016:
  - Continued getting the search stats from Apache logs of the portal (see Redmine #2144)
  - tsampikos.livisianos added to "driver" & "joomla" groups on www.openaire.eu, beta.openaire.eu & test.openaire.eu (see Redmine #2152)
  - Fixed Nginx monitoring on furkot2.vls (now it checks old.openaire.eu, not www.openaire.eu)
July 28, 2016:
  - Investigated 500 errors for jobtracker1.hadoop.dm.openaire.eu (reported by Claudio, see Redmine #2249)
  - Switched http://api.openaire.eu/search request from 8380 to 8480 on services.openaire.eu
  - Getting the search stats from Apache logs of the portal (see Redmine #2144)
July 27, 2016:
  - Continued investigating usage of api.openaire.research-infrastructures.eu (see Redmine #2246)
  - Deployed Webalizer for beta.lod.openaire.eu (see Redmine #2197)
July 26, 2016:
  - Started YARN ResourceManager on ci-test.hadoop5.iis.openaire.eu, because it was dead and was reason of restaring YARN NodeManager there
  - Fixed hostname for CNR LDAP server (asked by Claudio, see Redmine #2245)
  - Investigated usage of api.openaire.research-infrastructures.eu (reported by Alessia, see Redmine #2246)
July 25, 2016:
  - ioannis.foufoulas added to "driver" group on beta.services.openaire.eu (see Redmine #2232)
  - Started Nginx at furkot2.vls, because it was down because of invalid domain name (kiczora.vls.icm.edu.pl) for Highcharts proxy (see Redmine #2239 & #2240)
  - Fixed Monit monitoring of Apache2 at openaire-www.vls (now restarts MySQL using Monit commands stop & start)
July 14, 2016:
  - Created a new LUN on the Dell MD3620F disk array and mounted it as /srv on openaire-services-mongodb
  - andrea.dellamico added to "driver" group on services-postgresql.openaire.eu (asked by Alessia, see Redmine #2174)
  - Dumped PostgreSQL of openaire-services.vls and restored it on openaire-services-postgresql.vls, see above
  - Increased /srv on ci-test.hadoop5.iis.openaire.eu to 60GB (from 40GB) because it was near full-filled by HDFS data
July 12, 2016:
  - Bumped memory usage limit for Monit on services.openaire.eu (from 85% to 90%), because of a lot of the Monit alerts
July 11, 2016:
  - Purged the old logs of HDFS and Zookeeper on namenode2.hadoop.dm.openaire.eu because of a near full-filled / partition
  - Granted a view access for ALL websites tracked by OpenAIRE Piwik to Jochen Schirrwagen
  - Mirrored http://search.driver.research-infrastructures.eu on the portal machine, asked by Claudio, see Redmine #2208
  - Purged the old logs of HBase on hbase-master2.hadoop.dm.openaire.eu because of a near full-filled / partition
June 29, 2016:
  - Created dnet.production and dnet.beta accounts on OCEAN IIS CDH5 Test gateway and HDFS, see Mantis #9621
  - Configured an user's authentication against CNR LDAP on OCEAN IIS CDH5 Test gateway
June 28, 2016:
  - Configured in DNS the aliases (mongodb|postgresql).services.openaire.eu and services-(mongodb|postgresql).openaire.eu pointing to openaire-services-(mongodb|postgresql).vls
  - Mounted via NFS netapp2-nfs:/vol/skalny_mongodb_tmp/ (as /mnt/mongodb_tmp) and netapp2-nfs:/vol/openaire_mongodb_backup/ (as /mnt/mongodb_backup) on openaire-services-mongodb.vls
  - Enabled monitoring of MongoDB on openaire-services-mongodb.vls
  - Configuration of MongoDB on openaire-services-mongodb.vls synced with openaire-services.vls
  - Configuration of PostgreSQL on openaire-services-postgresql.vls synced with openaire-services.vls
June 27, 2016:
  - Deployed openaire-services-postgresql.vls machine as a PostgreSQL backend of services.openaire.eu, asked by CNR, see Redmine #2174
  - Deployed openaire-services-mongodb.vls machine as a MongoDB backend of services.openaire.eu, asked by CNR, see Redmine #2175
  - Upgraded Java 8 (from 8u66 to 8u91) on ci-test.hadoop5.iis.openaire.eu
June 24, 2016:
  - Purged the old logs of Mapreduce and Zookeeper on jobtracker1.hadoop.dm.openaire.eu because of a near full-filled / partition
  - Fixed 8480 monitoring on services.openaire.eu and the search monitoring on www.openaire.eu
June 23, 2016:
  - Restarted HBase Masters of DM Hadoop cluster because of hanged tasks, see Redmine #2167
June 22, 2016:
  - Created a FTP account "dovepress" on openaire-www.vls (asked by Jochen and Tracy Cleary from Dove Medical Press)
  - Increased a NFS volume openaire_objectstore_pdfs mounted on services.openaire.eu as /mnt/objectstore_pdfs from 6TB to 8TB, because it was filled in 91%
June 21, 2016:
  - Fixed PostgreSQL monitoring by Monit on services.openaire.eu, see Redmine #2155
June 20, 2016:
  - tsampikos.livisianos added to "driver" group on services.openaire.eu, beta.services.openaire.eu, stats.openaire.eu & gateway.hadoop.dm.openaire.eu (see Redmine #2152)
  - Fixed one column in TABLE_PARAMS table of hive_metastore database of Hive for OCEAN IIS CDH5 Test cluster (see Mantis #7911)
  - Created on openaire-www-back2.vls the shell scripts for checking slapd master-slave replication status and showing a difference if the slapd servers are not synchronized
June 18, 2016:
  - Fixed monitoring tomcat-instance-8480 & openaire-search (can't be restarted in the same time, now Monit alerts if 8480 fails and restarts 8480 if openaire-search fails)
June 17, 2016:
  - Configured Monit for restarting 8380 on services.openaire.eu if openaire-search issue occurs
  - Replaced a failed disk in openaire-iis-hadoop2-dom0, see Mantis #9637
  - Enabled monitoring of 8480 on services.openaire.eu, see Redmine #2144
  - Configured Monit for restarting 8480 on services.openaire.eu if openaire-search issue occurs (see above)
June 15, 2016:
  - Synced LDAP on openaire-www-back¹².vls with configuration and content of LDAP on openaire-www.vls
  - Configured LDAP master-slave replication between openaire-www-back1.vls (provider) and openaire-www-back2.vls (consumer)
  - Configured Apache redirection from indexed by Bing http://openaire.eu/index.php?lang=en-GB to https://www.openaire.eu/, asked by Argiro
June 14, 2016:
  - Purged the old logs of HDFS and Oozie on ci-test.hadoop5.iis.openaire.eu because of a near full-filled / partition
  - Deployed openaire-www-back2.vls, a second (backup) back-end machine for OpenAIRE portal cluster
  - Configured MySQL replication between openaire-www-back1.vls (master) and openaire-www-back2.vls (slave)
  - Deployed 8480 container on services.openaire.eu, see Redmine #2144
June 13, 2016:
  - Restarted 8380 on services.openaire.eu because the portal was down
  - Decreased Apache2 limits on openaire-www.vls (ServerLimit=MaxRequestWorkers=200), see above
  - Deployed openaire-www-proxy2.vls, a second (backup) proxy machine for OpenAIRE portal cluster
June 09, 2016:
  - Increased resources for dm_import pool of OCEAN IIS CDH5 Test cluster (max 30 vcores & max 180GB of memory), see Mantis #9621
June 07, 2016:
  - Patched s3cmd 1.0.0 on gateway.hadoop.iis.openaire.eu, asked by Kasia Zamlynska
  - Changed a mount point for the PDFs uploaded via FTP from /mnt/ftp to /mnt/uploaded_dumps, see Redmine #1929
June 04, 2016:
  - Restarted 8380 on services.openaire.eu because of OOM error
  - Increased / on services.openaire.eu to 40GB because it was near full-filled by .hprof files of 8380
  - Set ~~Xmx10G for 8380 on services.openaire.eu because of too many OOM errors last time~~
   Removed option XX:HeapDumpOnOutOfMemoryError (because we don't want unused .hprof file anymore) and set -XX:OnOutOfMemoryError="/bin/kill -9 p" instead of -XX:OnOutOfMemoryError="/etc/init.d/tomcat-instance-8380 restart" (because "driver" is not allowed for restarting container) for all D-Net containers on services.openaire.eu
June 03, 2016:
   Tuned YARN resources settings on OCEAN IIS CDH5 Test cluster
  - Unbanned IP 161.53.254.14 on a firewall of the OpenAIRE portal (see Redmine #2130)
  - Unbanned all blocked IPs on a firewall of the OpenAIRE portal
  - Installed awscli x%x s3cmd tools on gateway.hadoop.iis.openaire.eu, asked by Kasia Zamlynska
  - Restarted 8380 on services.openaire.eu because of OOM error
June 02, 2016:
  - Installed pydoop 1.0.0 on iis-cdh5-test-gw.ocean.icm.edu.pl (see Mantis #9621)
  - Increased size of / partition of OCEAN IIS CDH5 Test gateway from 20GB to 100GB (see above)
  - Increased memory for YARN applications on OCEAN IIS CDH5 Test cluster: 6GB for mappers & 8GB for reducers (see above)
  - Downgraded Java on ci-test.hadoop5.iis.openaire.eu from version 8u91 to 8u66 in order to fix Oozie issue with JSP (see Mantis #9631)
  - Updated Oozie sharelibs on ci-test.hadoop5.iis.openaire.eu (see above)
  - Tuned YARN resources settings on OCEAN IIS CDH5 Test cluster
June 01, 2016:
  - Increased size of / partition of OCEAN IIS CDH5 Test master machines from 20GB to 100GB
  - Installed python-pip & avroknife on iis-cdh5-test-gw.ocean.icm.edu.pl (see Mantis #9621)
  - Installed Hive on OCEAN IIS CDH5 Test cluster (see above)
  - Changed a password for FTP account "jochen" on openaire-www.vls (asked by Jochen Schirrwagen)
  - Mounted via NFS openaire_ftp NetApp volume on both services machines (prod & beta)
May 31, 2016:
  - Group "driver" can check status of Solr on solr.openaire.eu and beta.solr.openaire.eu now, see Redmine #2120
  - Updated packages on eos-cr-mgmt*, eos-m2-gw, iis-cdh5-test-* and eos-m2-sn* machines
  - Updated Cloudera Manager (CM) server on eos-cr-mgmt1 and Cloudera Manager agents on eos-cr-mgmt1 and all OCEAN IIS CDH5 Test cluster from version 5.5.0 to 5.7.0, see Mantis #9621
  - Updated CDH on ci-test.hadoop5.iis.openaire.eu from version 5.4.7 to 5.2.2 (see Mantis #9631)
May 30, 2016:
  - Granted permissions to "driver" group for launching `tcpflow` with any arguments on services.openaire.eu and beta.services.openaire.eu (see Redmine #2114)
  - Removed the logs for job_201605162019_0683 and job_201605162019_0687, because they consumed most of the disk space of DM Hadoop cluster data disks (see Redmine #2115)
  - Added the new INFN CA SSL certificate to Java keyring on services.openaire.eu and beta.services.openaire.eu (see Redmine #2117)
  - Restarted openaire-virtuoso-beta.vls, asked by Giorgos Alexiou, see Redmine #2118)
  - Now a dynamic pool "dm_import" on OCEAN IIS CDH5 Test cluster can use only 20 VCPUs and 40GB RAM (see Mantis #9621)
  - Created a new dynamic pools "marek_dot_horst" and "michal_dot_politkowski" on OCEAN IIS CDH5 Test cluster, see above
  - Changed a password for FTP account "copernicus" on openaire-www.vls (asked by Vitalij Mik from Copernicus)
May 25, 2016:
  - Fixed typo in HAProxy config for hdfs-namenode on gateway.hadoop.(dm|iis).openaire.eu
  - Installed apsw on OCEAN IIS CDH5 Test slaves (see Mantis #9621)
  - Set ~~Xmx512m fot Tomcat 7 on stats.openaire.eu (see Redmine #2100)~~
   Configured cache for WiredTiger engine of MongoDB on beta.services.openaire.eu, see Redmine #2062
May 24, 2016:
  - Configured a smart proxy for 50070 of the NameNodes using HAProxy cool health check feature on the IIS Hadoop cluster gateway
  - "driver" group can manage Tomcat7 on stats.openaire.eu now (see Redmine #2100)
  - Created a dynamic pool "dm_import" for managing resources on OCEAN IIS CDH5 Test cluster (see Mantis #9621)
May 23, 2016:
  - Fixed an issue with the permissions of /tmp/logs direcory owned by YARN on OCEAN IIS CDH5 Test cluster (see Mantis #9621)
  - Fixed an issue with a file browsing in DM Hadoop Hue by changing an active HDFS NameNode (see Redmine #2094)
  - Configured a smart proxy for 50070 of the NameNodes using HAProxy cool health check feature on the DM Hadoop cluster gateway (see above)
May 20, 2016:
  - Installed Ext JS for Ooozie on iis-cdh5-test-m3.ocean.icm.edu.pl (see Mantis #9621)
  - Installed "mc" package on iis-cdh5-test-gw.ocean.icm.edu.pl, asked by Marek Horst (see above)
  - Installed Oozie ShareLib on OCEAN IIS CDH5 Test Hadoop cluster (see above)
  - Stopped and disabled in systemd wpa_supplicant on all machines in module #2 of OCEAN Hadoop infrastructure
  - Alessia Bardi will be receiving Monit alerts from DM Hadoop slaves as well (see Redmine #2090)
  - aenne.loehden added to "driver" group on services.openaire.eu & beta.services.openaire.eu (asked by Jochen)
  - Open Solr port (8983) for UNIBI networks (asked by Jochen too)
  - Granted a "view" access to 26 Portugal repos to Jochen, see Redmine #2059
  - Purged the old logs of HDFS and Zookeeper on namenode1.hadoop.dm.openaire.eu because of a near full-filled / partition
  - Installed LaTeX packages on openaire-www.vls for generating PDF of the OpenAIRE Guidelines (see Redmine #2067)
May 19, 2016:
  - Configured Apache on iis-cdh5-test-gw.ocean.icm.edu.pl as a reverse proxy for OCEAN IIS CDH5 Test master slaves (see Mantis #[9621>>https://mantis.icm.edu.pl/view.php?id=9621)
  - Restarted HDFS DataNodes on slave[9-10].hadoop.dm.openaire.eu to fix an issue with the missing blocks (see Redmine #2086)
  - Configured an user's authentication against CNR LDAP for all web services of OCEAN IIS CDH5 Test cluster behind a reverse proxy on iis-cdh5-test-gw.ocean.icm.edu.pl
  - Increased (from 60s to 180s) a proxy timeout of Nginx for api.openaire.eu (see Redmine #2085)
  - iis-cdh5-test-gw.ocean.icm.edu.pl (instead eos-m2-sn01.ocean.icm.edu.pl) is now allowed to access gateway.hadoop.(dm|iis).openaire.eu without LDAP authentication (see Mantis #9621)
  - Installed "screen" package on iis-cdh5-test-gw.ocean.icm.edu.pl, asked by Marek Horst (see above)
May 18, 2016:
  - katarzyna.zamlynska added to group driver on gateway.hadoop.iis.openaire.eu (see Mantis #9630)
  - Configured Apache on iis-cdh5-test-gw.ocean.icm.edu.pl as a reverse proxy for OCEAN IIS CDH5 Test master machines (see Mantis #[9621>>https://mantis.icm.edu.pl/view.php?id=9621)
May 17, 2016:
  - katerina.iatropoulou added to group "driver" on gateway.hadoop.dm.openaire.eu (see Redmine #2078)
  - Created an account for Katerina Iatropoulou & Eri Katsari on all slaves of DM Hadoop cluster (see above)
  - Granted permissions to Katerina & Eri for becoming "mapred" user on all slaves of DM Hadoop cluster (see above)
  - Moved YARN, Zookeeper & HBase master daemons of OCEAN IIS CDH5 Test cluster to iis-cdh5-test-m[1-3] machines (see Mantis #[9621>>https://mantis.icm.edu.pl/view.php?id=9621)
  - Deployed HDFS, YARN & HBase services on eos-m2-sn01 & eos-m2-sn35 machines (see above)
May 16, 2016:
  - Deployed KVM virtual machines iis-cdh5-test-m[1-3] for OCEAN IIS CDH5 Test cluster
  - Moved HDFS master daemons of OCEAN IIS CDH5 Test cluster to iis-cdh5-test-m[1-3] machines (see Mantis #[9621>>https://mantis.icm.edu.pl/view.php?id=9621)
May 13, 2016:
  - MongoDB on openaire-services.vls and openaire-services-beta.vls now listens at all network interfaces, because it needs to be reachable from DM Hadoop cluster (see Redmine #2076)
  - Purged the old logs of HBase and Zookeeper on hbase-master1.hadoop.dm.openaire.eu because of a near full-filled / partition
  - Created a new website in Piwik for Spanish E-Archivo repo, asked by Dimitris (see Redmine #2069)
  - Created the new website in Piwik for 2 Croatian repos (FULIR & HRCAK), asked by Dimitris (see Redmine #2073)
  - Created a new website in Piwik for Bulgarian OpenAIRE repo, asked by Dimitris (see Redmine #2074)
  - Granted a "view" access to 26 Portugal repos to the users who has an account in Piwik, see Redmine #2059
  - Created a shell script /usr/local/bin/build-guidelines-html.sh for building HTML code of http://guidelines.openaire.eu/ site on openaire-www.vls (see Redmine #2067)
  - http://guidelines.openaire.eu/ now handled by Apache on openaire-www.vls, see above and Redmine #1889
  - Increased size (from 200GB to 250GB) of /srv partition on beta.stats.openaire.eu
May 12, 2016:
  - Retarted Apache and MySQL on openaire-www.vls because of memory lack (Error displaying the error page: Application Instantiation Error: Could not connect to MySQL.), thanks to doe.isti.gov
  - Enabled fail2ban apache-badhosts jail on openaire-www.vls for protection against too many requests from clients, for example doe.isti.gov
  - Installed Sphinx on openaire-www.vls in order to build HTML code for http://guidelines.openaire.eu (cloned from https://github.com/openaire/guidelines)) and configured Apache for serving it (see Redmine #2067)
May 11, 2016:
  - Retarted Apache and MySQL on openaire-www.vls because of memory lack (Error displaying the error page: Application Instantiation Error: Could not connect to MySQL.), thanks to doe.isti.gov
  - Configured Apache on openaire-www.vls for serving http://www.openaire.eu/sitemap without redirecting to HTTPS (see Redmine #2065)
  - Built HTML code for http://guidelines.openaire.eu (cloned from https://github.com/openaire/guidelines)) using Sphinx installed on my laptop and uploaded it on futkot2.vls, because we had an issue with ReadTheDocs service (see Redmine #2067)
May 10, 2016:
  - Investigating the memory issues on services.openaire.eu (see Redmine #2062)
  - "driver" group can monitor/unmonitor Java containers on services.openaire.eu now
  - Helping Sahar to find DocumentRoot of http://beta.lod.openaire.eu (see Redmine #2063)
  - Created the Piwik websites for 26 Portugal repositories, asked by Pedro, see Redmine #2059
May 09, 2016:
  - Restarted ci-test.hadoop.iis.openaire.eu and removed the old MapReduce & Oozie logs, because the machine was unreacheable (see Mantis #[9141>>https://mantis.icm.edu.pl/view.php?id=9141)
  - Restared slave2.hadoop.dm.openaire.eu and replaced a failed disk in openaire-dm-hadoop1-dom0.vls (see Redmine #2057)
  - Restarted openaire-iis-hadoop4-dom0.vls Xen Dom0 machine because of hanged Dell DRAC, see Mantis #[9628>>https://mantis.icm.edu.pl/view.php?id=9628]]
May 06, 2016:
  - Upgraded MongoDB on openaire-services.vls & openaire-services-beta.vls from version 3.2.5 to 3.2.6 (see Redmine #2043)
May 05, 2016:
  - Restored all MongoDB databases of openaire-services.vls on ciemniak2.vls (see Redmine #2043)
  - Updated CA root cert of CNR LDAP server on piwik.vls (see Redmine #2049)
  - Created dump of MongoDB mdstore, objectStore & dnet_logs databases on openaire-services.vls and restored them on ciemniak2.vls
  - Increased size (from 150GB to 200GB) of /srv partition on beta.stats.openaire.eu (see Redmine #1999)
  - Transfered MongoDB 3.5.2 databases from ciemniak2.vls on openaire-services.vls (see Redmine #2043)
May 04, 2016:
  - Fixed `facter` on openaire-virtuoso.vls (/usr/lib/ruby/1.8/facter/vls/exim4.rb for checking Exim4's keep_environment was missing)
  - Enabled Puppet agent there, because it was disabled
  - Fixed a huge CPU load on openaire-services-beta.vls because of MongoDB 3.2 (`sudo date ~~s now` did the magic), see Redmine #2019~~
   Restored MongoDB oaistore* databases of openaire-services.vls on openaire-services-pilot.vls (see Redmine #2043
  - Disabled temporarily verification of SSL cert of CNR LDAP in Apache2 on the OpenAIRE portal machines (production, beta & test) and Hadoop gateways, because it has expired today (see Redmine #2049)
  - Updated CA root cert of CNR LDAP server and enabled its verification again (see above)
April 28, 2016:
  - Upgraded Piwik for OpenAIRE (https://analytics.openaire.eu/)) from version 2.16.0 to 2.16.1.
  - Copied logrotate config file for MongoDB from openaire-services.vls to openaire-services-beta.vls, because it seems that mongodb-org-server package from 10gen doesn't include it
April 27, 2016:
  - Deployed MongoDB 3.5.2 on ciemniak2.vls and restored there mdstore database of MongoDB 2.6 from openaire-services-beta.vls (see Redmine #2019)
  - Created an account for Claudio and Alessia on ciemniak2.vls (see above)
  - Transfered MongoDB 3.5.2 databases from ciemniak2.vls on openaire-services-beta.vls (see above)
April 26, 2016:
  - Restored mdstore database of MongoDB 2.6 on openaire-services-beta.vls (see Redmine #2019)
  - Fixed Puppet template of MongoDB for Monit, because MongoDB 3.x (from 10gen) listens at a different port (21017) now and it's managed only by Upstart init script
April 25, 2016:
  - Upgraded MongoDB on openaire-services-beta.vls from version 2.4 to 2.6 (see Redmine #2019)
  - Upgraded MongoDB on openaire-services-beta.vls from version 2.6 to 3.0 (see above)
April 22, 2016:
  - Configured MySQL for Joomla on openaire-www-back1.vls machine
  - Tuned YARN on OCEAN IIS CDH5 Test cluster (see Mantis #[9621>>https://mantis.icm.edu.pl/view.php?id=9621)
April 21, 2016:
  - Restarted Zookeeper service & HBase Masters of DM Hadoop cluster (see Redmine #1998)
  - Configured Nginx for Joomla on openaire-www-proxy1.vls machine
  - Configured Apache2 for Joomla on openaire-www-front[1-3].vls machines
April 20, 2016:
  - Disabled and removed snapshots of openaire_mongodb_backup volume on netapp1 because of lack of free disk space for MongoDB backup on openaire-services-beta.vls (see below)
  - Created a backup of MongoDB on openaire-services-beta.vls (see Redmine #2019)
  - Deployed openaire-www-proxy1.vls machine
  - Deployed openaire-www-front[1-3].vls machines
  - Deployed openaire-www-back1.vls machine
April 19, 2016:
  - Increased CPU (from 4 to 8) and RAM (from 8GB to 16GB) for openaire-www.vls
  - Decreased CPU (from 18 to 4) and RAM (from 32GB to 8GB) for furkot2.vls
  - Restarted furkot2.vls (it had 830 days uptime!!!)
April 18, 2016:
  - Tuned YARN on OCEAN IIS CDH5 Test cluster (see Mantis #[9621>>https://mantis.icm.edu.pl/view.php?id=9621)
April 15, 2016:
  - Shutdowned slave6.hadoop.dm.openaire.eu because it has hanged up because of OOMs, see Redmine #2008
April 14, 2016:
  - Enabled /etc/cron.d/cleanup-mnt-tmp Cron job which removes the files older then 3 weeks from /mnt/tmp on gateway.hadoop.(dm|iis).openaire.eu (the files located under preserved/ subdirectory are still preserved), see Redmine #1768
April 13, 2016:
  - Installed unzipped version of Python APSW 2.11.1-r1 on ci-test.hadoop5.iis.openaire.eu (see Mantis #[9622>>https://mantis.icm.edu.pl/view.php?id=9622)
  - Restarted HBase RegionServer on slave4.hadoop.iis.openaire.eu, because HBase Master couldn't see it
  - Tuned YARN on OCEAN IIS CDH5 Test cluster (see Mantis #[9621>>https://mantis.icm.edu.pl/view.php?id=9621)
  - Created an account for Marek Horst for all slaves of OCEAN IIS CDH5 Test cluster (see above)
  - Installed Tomcat7 on openaire-virtuoso-beta.vls & configured sudo for "driver" group (see Redmine #2005)
  - Configured Apache2 on openaire-virtuoso-beta.vls for handling http://beta.lod.openaire.eu/api requests (see above)
  - Added 4 new slaves to OCEAN IIS CDH5 Test cluster (eos-m2-sn0⁷⁸ & eos-m2-sn4¹²)
April 7, 2016:
  - Installed php5-psql package on beta.stats.openaire.eu, asked by Ioannis Diplas (see Redmine #1970)
  - Granted admin rights to Dimitris Pierrakos for OpenAIRE portal (idSite=5) and API website (idSite=18) in Piwik
April 6, 2016:
  - 10.0.6.0/26 network (VLAN 406) is now allowed to access gateway.hadoop.(dm|iis).openaire.eu without LDAP authentication (see Mantis #7593)
  - Granted admin rights to Natalia for OpenAIRE portal web site (idSite=5) in Piwik
March 31, 2016:
  - ioannis.diplas added to group driver on beta.stats.openaire.eu & beta.services.openaire.eu (see Redmine #1970)
April 5, 2016:
  - Grabbed 504 return code lines of Nginx logs for api.openaire.eu, asked by Alessia
  - Installed the latest version of Mikoomi's MongoDB plugin/template for Zabbix on services.openaire.eu and beta.services.openaire.eu
  - Configured Nginx on beta.services.openaire.eu for handling beta.repomanager.openaire.eu (see Redmine #1970)
  - Configured Apache2 on beta.stats.openaire.eu for handling /stats locally and passing the rest requests to http://beta.services.openaire.eu:8380/statsConfig/ (see above)
  - Tuned YARN on ci-test.hadoop5.iis.openaire.eu (see Mantis #[9567>>https://mantis.icm.edu.pl/view.php?id=9567)
April 4, 2016:
  - Memory tuning of IIS CDH5 Test (OCEAN) Hadoop cluster (see Mantis #9621)
  - Created an account for Frederic Merceur in OpenAIRE Piwik (see Redmine #1972)
April 1, 2016:
  - CIS IP 213.135.59.178 (default gateway for the slave machines of OCEAN IIS CDH5 Test cluster) is now allowed to access gateway.hadoop.(dm|iis).openaire.eu without LDAP authentication (see Mantis #9621)
March 31, 2016:
  - Enabled monitoring of /mnt/mongodb_tmp on services.openaire.eu
  - Increased size of /mnt/objectstore_pdfs from 4TB to 6TB on services.openaire.eu
  - Configured Apache2 on openaire-stats-beta.vls for handling beta.stats.openaire.eu & beta.topics.openaire.eu domains (see Redmine #1970)
  - nikolaos.begetis added to group driver on beta.stats.openaire.eu & beta.services.openaire.eu (see above)
  - Upgraded Piwik for OpenAIRE (https://analytics.openaire.eu/)) from version 2.15.0 to 2.16.0.
  - eos-m2-sn01.ocean.icm.edu.pl is now allowed to access gateway.hadoop.(dm|iis).openaire.eu without LDAP authentication (see Mantis #9621)
March 30, 2016:
  - Set JAVA_HOME=/usr/java/jdk1.8.0_77 on IIS CDH5 Test (OCEAN) Hadoop cluster (see Mantis #9621)
  - Created an account krzysztof.madry on eos-m2-sn(01|35).ocean.icm.edu.pl (see Mantis #9621)
  - Installed PHP5 and Sqlite 3.8.2 on beta.stats.openaire.eu (see Redmine #1970)
  - Configured beta.topics.openaire.eu & beta.repomanager.openaire.eu domain names in DNS (see above)
  - spark-gw.vls.icm.edu.pl is now allowed to access gateway.hadoop.iis.openaire.eu without LDAP authentication (see Mantis #9487)
  - Granted access to Claudio for reading logs in /srv/data/*/mapred/local/userlogs/job_201602171053_1254/ of DM cluster worker nodes
  - Tuned IIS CDH5 Test (OCEAN) Hadoop cluster (oozie.action.max.output.data=8192, oozie.subworkflow.classpath.inheritance=true), see Mantis #9621
  - Increased size of /mnt/objectstore_pdfs from 4TB to 6TB on beta.services.openaire.eu (see Redmine #1973)
  - Enabled monitoring of /mnt/objectstore_pdfs on services.openaire.eu & beta.services.openaire.eu (see above)
March 29, 2016:
  - Removed Java 6 and installed Java 8 on IIS CDH5 Test (OCEAN) Hadoop cluster (see Mantis #9621)
  - Create a new website in Piwik for French Archimer Repository (see Redmine #1972)
March 24, 2016:
  - Purged python-apsw 3.7.6.3-r1 package on ci-test.hadoop.iis.openaire.eu & ci-test.hadoop5.iis.openaire.eu and installed there Python APSW 2.11.1-r1 built on the gateway of IIS Hadoop cluster (see Mantis #[9622>>https://mantis.icm.edu.pl/view.php?id=9622)
  - Restarted HBase Master & all RegionServers of DM Hadoop cluster
March 22, 2016:
  - Removed some .hprof files of 8380 on services.openaire.eu, because of lack of free disk space on / partition (see Redmine #1967)
  - Deployed IIS CDH5 Test (OCEAN) Hadoop cluster (see Mantis #9621)
March 21, 2016:
  - Purged python-apsw 3.7.6.3-r1 package on the slaves of IIS Hadoop cluster and installed there Python APSW 2.11.1-r1 built on the gateway (see Redmine #1212)
March 18, 2016:
  - Purged the old logs of Oozie on master3.hadoop.iis.openaire.eu because of a near full-filled / partition
  - Created backup of Joomla on www.openaire.eu, asked by Argiro
March 16, 2016:
  - ioannis.foufoulas added to group driver on gateway.hadoop.iis.openaire.eu (see Mantis #9620)
  - Deleted unused arxiv_pdf-old/ directory from /mnt/downloaded_dumps volume (see Redmine #1950)
March 11, 2016:
  - Enabled monitoring of usage of /home and /mnt/tmp on gateway.hadoop.iis.openaire.eu
  - Enabled monitoring of usage of /mnt/downloaded_dumps on services.openaire.eu
March 10, 2016:
  - Disabled forgotten commands `chown ~~R dnet:monkeys /srv/dnet/` & `chown -R root:monkeys /usr/share/dnet/bundles/` in the root's crontab both on services.openaire.eu and beta.services.openaire.eu~~
   Now Monit on services.openaire.eu and beta.services.openaire.eu will be restarting Tomcat containers (if neccessary) instead only sending e-mail alerts, asked by Claudio
March 09, 2016:
  - michal.politowski added to driver group at gateway.hadoop.iis.openaire.eu (see Redmine #1936)
  - Moved MongoDB mdstore & oaistore* on services.openaire.eu from /srv & /srv3 to /srv2 (see Redmine #1940)
  - Fixed HDFS balancer on master1.hadoop.iis.openaire.eu which was causing the load peaks there
March 08, 2016:
  - Replaced a failed disk in RAID-0 (available as /dev/sdc on openaire-iis-hadoop2-dom0.vls) for HDFS in slave3.hadoop.iis.openaire.eu (available as /dev/xvde and mounted as /srv/data/3)
  - Replaced a failed disk in RAID-1 (available as /dev/sdg) for LVM on openaire-iis-hadoop2-dom0.vls
  - Started again slave3.hadoop.iis.openaire.eu
  - Created a testing account for Jochen in the FTP service on ftp.openaire.eu (alias for www.openaire.eu), see Redmine #1929
March 07, 2016:
  - Shutdowned slave3.hadoop.iis.openaire.eu because of HDFS disk failure
  - Restarted DataNodes of IIS Hadoop cluster, because of HBase Master issue ("org.apache.hadoop.hbase.util.FSUtils: Waiting for dfs to exit safe mode...", reported by Claudio, see Redmine #1931)
  - Launched manually /usr/local/sbin/get-piwik-users-actions.sh script of gateway.hadoop.iis.openaire.eu, because it failed because of IIS Hadoop HDFS issue ("Name node is in safe mode", see above)
  - Increased size (from 100GB to 150GB) of /srv partition on beta.stats.openaire.eu
  - Eri Katsari & Andrea Mannocci added to a recipient list of Monit alerts for /srv on stats.openaire.eu & beta.stats.openaire.eu
March 04, 2016:
  - Installed MongoDB on pilot.services.openaire.eu and restored there objectStore db for a testing purpose (see Redmine #1747)
  - Dropped objectStore db of MongoDB on services.openaire.eu and restored it from a backup (see above)
  - Purged the old logs of Zookeeper of {namenode1,jobtracker1,hbase-master1}.hadoop.dm.openaire.eu because of a near full-filled / partition
March 03, 2016:
  - Unsuccessful attempt to repair MongoDB on services.openaire.eu (see Redmine #1747)
March 02, 2016:
  - Installed python-pip and python-libxml2 Debian packages and also beautifulsoup4 Python package on services.openaire.eu and beta.services.openaire.eu (asked by Sandro La Bruzzo, see Redmine #1930)
February 29, 2016:
  - Restarted MySQL on openaire-www.vls, because Joomla had an issue with connecting its database
February 27, 2016:
  - Restarted openaire-virtuoso-beta.vls, asked by Giorgos Alexiou
February 26, 2016:
  - Deleted the local user logs of MapReduce on all slaves of the DM Hadoop cluster (asked by Giorgos, see Redmine #1917)
  - Increased a number of CPU cores (from 12 to 16) and memory (from 32GB to 48GB) on openaire-virtuoso-beta.vls (see Redmine #1866)
  - Launched 8580 on pilot.services.openaire.eu, because it was dead
February 24, 2016:
  - Deployed the new Xen DomUs called slave9.hadoop.dm.openaire.eu and slave10.hadoop.dm.openaire.eu for the DM Hadoop cluster
  - Deployed Ganglia on the master servers of DM Hadoop cluster
February 23, 2016:
  - Created a new Xen DomUs called slave9.hadoop.dm.openaire.eu and slave10.hadoop.dm.openaire.eu for the DM Hadoop cluster, because HDFS is near full-filled (see Redmine #1904)
  - Deleted the old local user logs of MapReduce on all slaves of the DM Hadoop cluster (see Redmine #1904)
February 22, 2016:
  - Created a new Xen Dom0 called openaire-dm-hadoop3-dom0.vls for the DM Hadoop cluster slaves, because HDFS is near full-filled (see Redmine #1904)
February 19, 2016:
  - Removed doe.osti.gov (IP 192.107.175.11) from a list of DROPed IP addresses on the openaire-www.vls firewall, after Natalia request
  - Bumped LIMIT (from 30 to 50) of /usr/local/sbin/antiloris.sh on openaire-www.vls
  - Increased a number of CPU cores (from 8 to 12) on openaire-virtuoso-beta.vls (see Redmine #1866)
  - Installed Ganglia on the master servers of DM Hadoop cluster
  - HDFS balancer threshold set to 3 in cron.d/hadoop-hdfs-balancer at DM Hadoop cluster
February 17, 2016:
  - Updated Java (from version 6 to 7) on DM Hadoop cluster (see Redmine #1892)
  - Restarted openaire-virtuoso-beta.vls, asked by Giorgos Alexiou
February 16, 2016:
  - Fixed an ownership of the files under /etc (in particular /etc/default/tomcat-instance-8*80) on openaire-services-beta.vls which belong now to "driver" group instead of an old and inactive "monkeys" group, reported by Nikon Gasparis who couldn't change the memory settings for 8380 container
  - The same action on openaire-services.vls (see Redmine #1891)
February 15, 2016:
  - Created backup of Joomla on www.openaire.eu, asked by Argiro
  - Increased a number of CPU cores (from 4 to 8) and amount of memory (from 16GB to 32GB) on openaire-virtuoso-beta.vls (see Redmine #1866)
  - Created backup of Joomla on test.openaire.eu, asked by Argiro
  - Synchronised test.openaire.eu with www.openaire.eu, asked by Argiro
  - Fixed Apache2 config for blogs.openaire.eu site on openaire-www.vls (enabled and used proxy_html module for fixing http://blogs.openaire.eu/ links)
February 12, 2016:
  - Activated and configured LoginLDAP plugin of Piwik (see Redmine #1828)
  - Opened access to port 1111 on openaire-virtuoso-beta.vls from openaire-dm-hadoop-gw (asked by Giorgos Alexiou, see Redmine #1866)
February 11, 2016:
  - Imported to Piwik the latest Nginx logs of api.openaire.eu starting from January 16 until to February 11 (see Redmine #1828)
  - Created a script import-api-logs.sh on piwik.vls for importing API logs to Piwik via Cron nightly (see above)
  - Installed LoginLDAP plugin of Piwik (see above)
February 10, 2016:
  - Upgraded ownCloud of box.openaire.eu from version 8.2.1 to 8.2.2
February 3, 2016:
  - Purged the old logs of Hadoop HDFS/MapReduce and Oozie of ci-test.hadoop.iis.openaire.eu because of a near full-filled / partition
  - Athena RC networks are allowed now to log in at gateway.hadoop.dm.openaire.eu (see Redmine #1866)
  - Added more CPU cores (4) and memory (16GB) for beta.virtuoso.openaire.eu (see above)
February 2, 2016:
  - Increased Xmx size (from 2GB to 4GB) for JobTracker HA on master¹².hadoop.iis.openaire.eu (see Redmine #1859)
January 29, 2016:
   Created a new instance of ownCloud (http://box-test.openaire.eu/)) on farm-www.vls for the update testing
January 28, 2016:
  - Created backup of Joomla on www.openaire.eu, asked by Argiro
January 27, 2016:
  - Assisting Claudio with resolving his issue with accessing the DM Hadoop slaves in order to check the job logs (changed his SSH public key, fixed the job directory permissions from 700 to 755)
January 25, 2016:
  - Disabled a shell script for cross-backup with CERN on orli, because our account at CERN wasn't working starting from May 2015
January 21, 2016:
  - Removed the old .hprof files of 8380 container on services.openaire.eu, because of a near full-filled / partition
January 20, 2016:
  - Reported to Pisa team that their 8290 container is dead on services.openaire.eu and beta.services.openaire.eu (see Redmine #1836)
January 19, 2016:
  - Purged the old logs of Zookeeper of hbase-master1.hadoop.dm.openaire.eu because of a near full-filled / partition
  - Create a new website in Piwik for Portugal Cocum Repository (see Redmine #1671)
  - Open access to port 6379 (Redis) on test.openaire.eu from beta.services.openaire.eu (see Redmine #1833)
January 18, 2016:
  - Set rotate 730 for rotating the Nginx logs on furkot2.vls, because Natalia wanted the numbers of api.openaire.eu for a report for EC
  - Created a new website for api.openaire.eu in Piwik (see Redmine #1828)
  - Imported Nginx logs of api.openaire.eu to Piwik (see above)
January 11, 2016:
  - Purged the old logs of Hadoop HDFS/MapReduce and Oozie of ci-test.hadoop.iis.openaire.eu because of a near full-filled / partition
  - Increased size of /mnt/mongodb_backup partition of services.openaire.eu from 5TB to 6TB, changed a backup policy (dump.old/ directory for a previous dump and dump/ for a current dump), enabled the snapshots feature of NetApp for the appropriate volume (see Redmine #1806)
  - Launched JobTracker on master1.hadoop.iis.openaire.eu, because it was dead
January 9, 2016:
  - Purged the old dumps of MongoDB of services.openaire.eu because of a near full-filled /mnt/mongodb_backup partition (see Redmine #1806)
January 8, 2016:
  - Launched 8380 on services.openaire.eu, because it was dead (see Redmine #1801)
  - Investigating a timeout issue reported by Dimitris Pierrakos which was trying to download UMINHO data from Piwik
January 7, 2016:
  - Launched 8580 on pilot.services.openaire.eu, because it was dead
  - Re-processed all data in Piwik for OpenAIRE Post-grant Pilot site, because a report for a year period was broken (reported by Natalia)
  - Enabled processing the unique visitors for a year period in Piwik and re-processed all data for every OpenAIRE related site (enabled by Natalia)
January 4, 2016:
  - Installed Java 7 on virtuoso.openaire.eu and virtuoso-beta.openaire.eu and deployed Tarql 1.0a there (see Redmine #1390)
December 26, 2015:
  - Purged the old logs of Hadoop HDFS/MapReduce and Oozie of ci-test.hadoop.iis.openaire.eu because of a near full-filled / partition
December 24, 2015:
  - Increased size of /srv partition on ci-test.hadoop.iis.openaire.eu from 20GB to 40GB
December 21, 2015:
  - Renamed /mnt/harvested_pdfs to /mnt/downloaded_dumps on services.openaire.eu, beta.services.openaire.eu and gateway.hadoop.iis.openaire.eu (see Redmine #1776)
December 18, 2015:
  - Making beta.lod.openaire.eu and virtuoso-beta.openaire.eu ready for testing (see Redmine #1748)
  - Configured a redirection from lod.openaire.eu to beta.lod.openaire.eu
December 15, 2015:
  - Installed python-apsw package on pyszny.vls & nosal.vls, because it's necessary for running IIS jobs on Jenkins (asked by Lukasz Dumiszewski)
  - Increased size of NetApp volume mounted via NFS on gw.hadoop.iis.openaire.eu as /mnt/harversed_pdfs from 1TB to 1.5TB (Mantis #9595)
  - Resolved Claudio's problem with launching `sudo su - dnet` on gw.hadoop.dm.openaire.eu (see Redmine #1765)
December 11, 2015:
  - Fixed a content of /var/run/mongodb/mongodb.pid file on services.openaire.eu which causes an issue with Monit, because it contaided an invalid PID number of MongoDB process (see Redmine #1746)
December 9, 2015:
  - Fixed an issue with an invalid SSH key on solr.openaire.eu which caused that making of a system backup was impossible
  - Removed an old and unused logrotate config for MongoDB on services.openaire.eu which caused "duplicate log entry for /var/log/mongodb/mongodb.log" error
  - Purged an old and unused postgresql-common package on furkot2.vls which caused "error accessing /var/log/postgresql: No such file or directory" error
December 7, 2015:
  - Upgraded ownCloud of box.openaire.eu from version 7.0.4 to 8.2.1
  - Created a new VM beta.virtuoso.openaire.eu for Virtuoso server (see Redmine #1748)
December 5, 2015:
  - Increased /srv from 400GB to 500GB on services.openaire.eu (see Redmine #1746)
December 2, 2015:
  - Created an account for Dimitris Pierrakos in OpenAIRE Piwik (see Redmine #1695)
  - Fixed the write rights for some HDFS directories on ci-test.hadoop5.iis.openaire.eu (see Mantis #9567)
  - Opened ports on the firewall for the web interfaces of HDFS, MapReduce, YARN and Spark (see above)
December 1, 2015:
  - Installed and configured Spark on ci-test.hadoop5.iis.openaire.eu (see Mantis #9567)
  - Created a website and an user for Zenodo in Piwik on analytics.openaire.eu (see Redmine #1695)
November 30, 2015:
  - Enabled monitoring of YARN {Node,Resource}Manager and Mapreduce HistoryServer on ci-test.hadoop5.iis.openaire.eu
November 26, 2015:
  - Installed Java 8 on ci-test.hadoop5.iis.openaire.eu (see Mantis #9567)
  - Fixed file https://www.openaire.eu/cerif_schema/cerif-1.6-2_openaire-1.0.xsd (copied from furkot2.vls to openaire-www.vls), reported by Jan Dvorak
  - Made box8.openaire.eu accessible for testing for Natalia and Pedro
November 25, 2015:
  - Removed old and unused Zookeeper files on jobtracker1.hadoop.dm.openaire.eu because of near full-filled / partition
  - Enabled /usr/local/sbin/make-hdfs-user-home script in PAM on ci-test.hadoop5.iis.openaire.eu (see Mantis #9567)
November 24, 2015:
  - Increased quota for Natalia (from 1GB to 5GB) in box.openaire.eu
  - Installed YARN on ci-test.hadoop5.iis.openaire.eu (see Mantis #9567)
November 19, 2015:
  - Investigating an issue with near full-filled / partition on services.openaire.eu where 8380 container created a few huge .hprof files (removed my Nikon after my request)
  - Created backup of Joomla on www.openaire.eu, asked by Argiro
  - Assisting during Joomla Security update on www.openaire.eu, done by Argiro
November 18, 2015:
  - Killed a hunged process for 8586 port on pilot.services.openaire.eu and started 8580 again (reported by Nikon)
November 17, 2015:
  - Investigating an issue with a bad encoding of the validator messages (reported by Nikon, see Redmine #921)
  - Investigating an issue with uploading files to box.openaire.eu: "Your storage is full, files can not be updated or synchronized anymore!" (reported by Sahar)
  - Set scheduled execution of VACUUM ANALYZE on the dnet_openaireplus database on services.openaire.eu and beta.services.openaire.eu (see Redmine #1694)
November 13, 2015:
  - Configured authorized SSH keys of jenkins at ci-test.hadoop(5).iis.openaire.eu for SSH connections from ci.ceon.pl (see Mantis #9573) for running IIS integration tests
November 5:
  - increased client_max_body_size (to 10M) for postgrantoapilot.openaire.eu (asked by Antonis)
November 4:
  - Created Piwik accounts for Portugal pilot repositories (asked by Pedro)
  - Fixed broken LDAP configuration of box.openaire.eu
November 1, 2015:
  - Security upgrade of PHP and MySQL on {beta,test,www}.openaire.eu
October 27, 2015:
  - Installed php5-gmp package on www.openaire.eu (asked by Eri)
  - Copied Joomla directory and database from www.openaire.eu to {beta,test}.openaire.eu (asked by Argiro)
  - Upgraded Piwik for OpenAIRE (https://analytics.openaire.eu/)) from version 2.14.3 to 2.15.0.
October 26, 2015:
  - Installed composer.phar (https://getcomposer.org/)) on www.openaire.eu (asked by Eri)
October 23, 2015:
  - Security upgrade of PostgreSQL 9.3 on beta.services.openaire.eu (see Redmine #1646)
October 22, 2015:
  - Deployed ci-test.hadoop5.iis.openaire.eu machine for a new pseudo Hadoop CDH5 cluster for running IIS integration tests (see Mantis #9567)
  - Security upgrade of PostgreSQL 9.3 on services.openaire.eu (see Redmine #1646)
  - Security upgrade of PostgreSQL 9.3 on stats.openaire.eu & beta.stats.openaire.eu (see Redmine #1645)
October 21, 2015:
  - Security upgrade of PostgreSQL 9.3 on pilot.services.openaire.eu (see Redmine #1646)
October 16, 2015:
  - Helping Dimitris Pierrakos with his trouble with accessing the portal Joomla logs stored on HDFS of IIS Hadoop cluster
  - Helping Natalia & Antonis with their trouble with accessing Piwik stats for blogs.openaire.eu and Post-Grant Pilot
  - Answering Antonis who asked about the backups of Post-Grant Pilot DB
  - Removed old and unused logs of HDFS, MapReduce and Oozie on ci-test.hadoop.iis.openaire.eu, because they near full-filled / partition
October 15, 2015:
  - Reported the portal Apache errors to Argiro Kokogiannaki
  - Copied Redis server content from stats.openaire.eu to beta.stats.openaire.eu (asked by Eri Katsari)
  - Investigating the issues with wrong URLs (http://localhost:8890/)) for lod.openaire.eu (see Redmine #1632)
October 14, 2015:
  - andrea.mannocci added to driver group at stats.openaire.eu (see Redmine #1441)
  - Reported the portal Apache errors to Argiro Kokogiannaki
October 13, 2015:
  - Created Piwik account for Pedro Principe and given him an access to the stats for all Portugal pilot services (see Redmine #1483)
October 12, 2015:
  - Fixed LDAP authentication for https://analytics.openaire.eu/ (see Redmine #1622)
  - Investigating the issue with unreachable MySQL server of the portal ("/usr/sbin/mysqld: Got an error writing communication packets")
October 09, 2015:
  - Copied 1.0/ and 0.3/ schema from furkot2.vls to openaire-www.vls (see Redmine #1617)
  - alessia.bardi added to driver & joomla groups at www.openaire.eu (see Redmine #1617)
October 07, 2015:
  - Removed the old and unused ZooKeeper/Hadoop logs on hbase-master¹².hadoop.dm.openaire.eu
  - Investigating the issue with LDAP/DAV authentication for http://virtuoso.openaire.eu/ (see Redmine #1391)
  - Enabled access to HFTP (50070 port) on gateway.hadoop.dm.openaire.eu from spark-gw.vls.icm.edu.pl (see Mantis #9561)
October 06, 2015:
  - Continued polishing the rewrite/proxy rules for lod.openaire.eu (see Redmine #1408)
October 02, 2015:
  - Fixed an issue of Anja OberlÃ¤nder (<Anja.Oberlaender@uni-konstanz.de>) who couldn't see some documents under http://box.openaire.eu, because she didn't belong to "Reviewers" group (forwarded by Claudio Atzori)
  - Polished the rewrite/proxy rules for lod.openaire.eu (see Redmine #1408)
October 01, 2015:
  - Ported the missing reverse proxy rules for /validator-{admin,services} from furkot2.vls (old.openaire.eu, Nginx) to openaire-www.vls (www.openaire.eu, Apache2) (reported by Nikon Gasparis, see Redmine #1577)
  - Created /mnt/virtuoso/upload directory writable by driver group on virtuoso.openaire.eu (asked by Giorgos Alexiou, see Redmine #1408)
  - A new proxy rules for ^/data/ requests on lod.openaire.eu (see above)
September 30, 2015:
  - Enabled a redirection from https://guidelines.openaire.eu/wiki/OpenAIRE_Guidelines:_For_Literature_repositories to https://guidelines.openaire.eu/en/latest/literature/index.html (asked by Nikon Gasparis, see Redmine #1571)
  - Updated avroknife from 0.9.3 to 1.0.5 on gateway.hadoop.iis.openaire.eu (asked by Mateusz Kobos, see Mantis #9559)
September 29, 2015:
  - Open access to virtuoso.openaire.eu via SSH from Athena RC network (asked by Giorgos Alexiou)
  - Open access to lod.openaire.eu via HTTP from the whole world (asked by Christoph Lange)
  - /var/www/lod directory made writable for driver group at virtuoso.openaire.eu (see Redmine #1569)
  - christoph.lange & george.papastefanatos added to driver group at virtuoso.openaire.eu (see Redmine #1568)
September 28, 2015:
  - Installed missing packages for Virtuoso, e.g. iSPARQL, Conductor, BPEL, etc
  - Configured a reverse proxy for http://lod.openaire.eu/sparql
  - Added permissions for managing Virtuoso server and editing virtuoso.ini file for "driver" group
September 25, 2015:
  - Restored a backup dump of dnet_openaireplus database of PostgreSQL on services.openaire.eu (asked by Alessia Bardi & Michele Artini)
September 24, 2015:
  - Increased /srv from 400GB to 500GB on solr.openaire.eu
  - Configured lod.openaire.eu (see Redmine #1408)
September 23, 2015:
  - Granted the privileges for Dimitris Pierrakos to an access Piwik statistics of pilot Portugal repositories (see Redmine #1483)
September 22, 2015:
  - Now Virtuoso server on virtuoso.openaire.eu runs as an unprivileged "virtuoso" user
  - Configured a reverse proxy for virtuoso.openaire.eu
September 21, 2015:
  - Created a new VM virtuoso.openaire.eu for Virtuoso server (see Redmine #1391)
  - Deployed Virtuoso Opensource 7 server there
September 17, 2015:
  - Removed .hprof file of 8380 (it had 4.8GB) on services.openaire.eu, because its / partition was near full-filled
  - Asked Alessia & Claudio for removing old versions of is.war on services.openaire.eu, because of the same issue (see above)
  - Removed old logs of HDFS, MapReduce and Oozie on ci-test.hadoop.ci.openaire.eu, because its / partition was near full-filled
  - Fixed DNS settings for openaire.eu (see Redmine #1532)
September 16, 2015:
  - Increased /srv on stats.openaire.eu from 150GB to 200GB
  - Configured Joomla for Piwik tracking on www.openaire.eu (we forgot about it yesterday)
  - Debugging an issue with a big load and poor I/O performance on stats.openaire.eu, because of a lot of long running SELECTs of PostgreSQL
September 15, 2015:
  - Migrated Joomla portal from furkot2.vls to openaire-www.vls
  - Stopped and uninstalled Redis on openaire-www.vls
September 14, 2015:
  - Created backup of Joomla on new.openaire.eu, asked by Argiro
September 10, 2015:
  - Created backup of Joomla on new.openaire.eu, asked by Argiro
  - Created backup of Joomla on www.openaire.eu, asked by Katerina
September 8, 2015:
  - Installed and configured Fail2Ban on test.openaire.eu (after Eri's email)
  - Patched apache-{auth,common}.conf rules of Fail2Ban, because the maintainer rules don't work with Apache2 error logs
  - Deployed Fail2Ban also on new.openaire.eu and beta.openaire.eu
September 7, 2015:
  - Installed Redis server on stats.openaire.eu and moved there Redis database content from furkot2 (see Redmine #1515)
  - Stopped furkot2 Redis (see the Redmine ticket above)
  - Open access to port 6379 (Redis) on test.openaire.eu from services.openaire.eu (asked by Alessia, see the Redmine ticket above)
  - Copied Redis database content from test.openaire.eu to new.openaire.eu (see the Redmine ticket above)
September 3, 2015:
  - Reporting that a production portal search is not working, because Thrift on services.openaire.eu (7911 port) is dead
  - Added a new recipients of DNet monitoring on services.openaire.eu and beta.services.openaire.eu
  - Enabled monitoring of a beta portal search by Monit at beta.services.openaire.eu
September 2, 2015:
  - Reconfigured Hive on hue.hadoop.dm.openaire.eu and created MySQL database for it on hbase-master2.hadoop.dm.openaire.eu, deployed Java connector for MySQL
  - Enabled web console of DM Oozie
  - Installed hue-plugins on DM Hadoop JobTrackers
September 1, 2015:
  - Debugging an issue with validating repositories available only via HTTPS (see Redmine #1479)
  - Debugging an issue with Hue of DM Hadoop (reported by Eri)
August 31, 2015:
  - Restarted Hue server on hue.hadoop.dm.openaire.eu, because it was unresponsible (reported by Eri)
  - Synchronised beta.openaire.eu with new.openaire.eu (asked by Katerina)
  - Increased the actions limits for OpenAIRE Piwik (asked by Dimitris Pierrakos)
  - Updated Ubuntu from Precise to Trusty on beta.openaire.eu (asked by Katerina)
August 28, 2015:
  - Debugging I/O issues on beta.services.openaire.eu (see Redmine #1502)
  - lukasz.dumiszewski and krzysztof.madry added to "driver" group on gateway.hadoop.iis.openaire.eu (see Mantis #[9545>>https://mantis.icm.edu.pl/view.php?id=9545)
  - Restarted solr.openaire.eu in order to bump memory there
August 27, 2015:
  - Increased memory from 24GB to 32GB on openaire-solr
  - Changed permissions for /etc/apache2/ssl/STAR_openaire_eu.key on new.openaire.eu, because SimpleSAMLphp (asked by Eri) couldn't read it
August 26, 2015:
  - Started nslcd daemon on test.openaire.eu, because it was dead and Eri & Argiro couldn't access there via SSH
  - Synchronised test.openaire.eu with new.openaire.eu (asked by Katerina & Argiro)
  - Moved some NKUA stuff from /srv to /srv2 on services.openaire.eu and gzipped some .csv files under /srv/dnet, because /srv partition was a near full-filled
  - Added self-signed CNR certificate to Java keystore on services.openaire.eu and beta.services.openaire.eu (asked by Andrea Mannocci, see Redmine #1442)
August 25, 2015:
  - Configured Nginx proxy rules for /dfm on services.openaire.eu (see Redmine #1387)
August 24, 2015:
  - Created data_flow_monitoring database owned by dfm user in PostgreSQL and /var/lib/dfm directory on openaire-services.vls (asked by Alessia Bardi)
  - Added a new IP address of Glykeria Katsari in Puppet and on a local PostgreSQL instances for OpenAIRE stats & services
  - Enabled Piwik tracking in Wordpress for http://blogs.openaire.eu (asked by Pedro Miguel Oliveira Bento PrÃncipe)
August 19, 2015:
  - Updated Ubuntu from Precise to Trusty on test.openaire.eu (asked by Katerina)
August 17, 2015:
  - Created backup of Joomla on beta.openaire.eu, asked by Katerina
  - Synchronised beta.openaire.eu with new.openaire.eu (asked by Katerina)
August 12, 2015:
  - Added an user's authentication against a local .htpasswd file on https://analytics.openaire.eu/ (Dimitris Pierrakos asked for a remote access to Piwik data for UMINHO Pilot website)
August 6, 2015:
  - Created a new Piwik websites for 3 Portugal repositories (see Redmine #1483)
  - Upgraded Piwik for OpenAIRE+ (https://analytics.openaire.eu/)) from version 2.14.1 to 2.14.3.
August 5, 2015:
  - Updated Ubuntu from Precise to Trusty on services.openaire.eu (see Redmine #1481)
  - Also migrated there PostgreSQL 9.1 to 9.3
  - Also upgraded there MongoDB to 2.4.9
August 3, 2015:
  - Created a reverse proxy rules for http://api.openaire.eu/provision/mvc/resources (see Redmine #[844>>https://issue.openaire.research-infrastructures.eu/issues/844)
  - Created a reverse proxy rules for http://api.openaire.eu/search/openSearchDescriptor (see Redmine #[348>>https://issue.openaire.research-infrastructures.eu/issues/348)
  - Created an account openaire-blogs in OpenAIRE's Piwik (https://analytics.openaire.eu/)) for accessing by Wordpress on blogs.openaire.eu (asked by Pedro Miguel Oliveira Bento PrÃncipe)
  - Synced with Zenodo https://www.openaire.eu/cerif_schema/cerif-1.6-2_openaire-1.0.xsd file (asked by Jan.Dvorak@ff.cuni.cz)
July 17, 2015:
  - Removed the old logs of Zookeeper on the masters of DM Hadoop cluster in order to take some free disk space on / partition
  - Removed the old logs of Zookeeper, Hadoop HDFS and Mapreduce on the masters of test-ci.hadoop.iis.openaire.eu because of the same reason like above
  - Upgraded Piwik for OpenAIRE+ (https://analytics.openaire.eu/)) from version 2.13.1 to 2.14.1.
July 16, 2015:
  - Increased memory from 24GB to 32GB on openaire-solr-beta
  - Enabled basic authentication in Nginx for https://beta.services.openaire.eu/dfm/ (see Redmine #1442)
July 15, 2015:
  - Enabled the users authentication against CNR LDAP on solr.openaire.eu and beta.solr.openaire.eu (see Redmine #1438)
  - Disabled "monkeys" account on solr.openaire.eu and beta.solr.openaire.eu (see above)
  - alessia.bardi and claudio.atzori added to "driver" group on solr.openaire.eu and beta.solr.openaire.eu (see above)
  - Enabled the users authentication against CNR LDAP on solr.openaire.eu and beta.solr.openaire.eu (see Redmine #1439)
  - Disabled "monkeys" and "joomla" accounts on solr.openaire.eu and beta.solr.openaire.eu (see above)
  - eri.katsari, katerina.iatropoulou and antonis.lempesis added to "driver" group on solr.openaire.eu and beta.solr.openaire.eu (see above)
  - Synchronised beta.openaire.eu with new.openaire.eu (asked by Argiro)
  - andrea.mannocci and alessia.bardi added to "driver" group on beta.stats.openaire.eu (see Redmine #1441)
July 14, 2015:
  - Disabled "monkeys" account on services.openaire.eu (see Redmine #1434)
  - christoph.broschinski added to "driver" group on services.openaire.eu (see Redmine #1434)
July 13, 2015:
  - Added a public SSH key of Argiro Kokogiannaki to keyring of "monkeys" user, because she needs an access to the claims database on services.openaire.eu
  - Open access to port 7911 (Thrift) on services.openaire.eu from scoobydoo.di.uoa.gr (asked by Argiro)
  - Enabled the users authentication against CNR LDAP on services.openaire.eu (see Redmine #1434)
July 10, 2015:
  - Debugging an issue (504 Gateway Timeout) for guidelines.openaire.eu (see Redmine #1425)
  - Fixed UID/GID for "dnet" user on gateway.hadoop.iis.openaire.eu (see Redmine #903)
  - Untared the newest bunch of arXiv PDFs: arXiv_pdf_150⁴⁵⁶_*.tar (see Redmine #903)
July 09, 2015:
  - Intervention on beta.services.openaire.eu because of a full-filed / partition because of a huge 8380 catelina.out log file (Katerina fixed it quickly and punished a guilty person)
  - Deployed vsftpd on openaire-www.vls (asked by Jochen), because it can be helpful for uploading metadata by enternal publishers (e.g. F-1000)
July 08, 2015:
  - Configured a reverse proxy for /search/api on beta.services.openaire.eu (asked by Katerina for an external parties)
July 07, 2015:
  - Modified a reverse proxy for guidelines.openaire.eu (asked by Jochen Schirrwagen). Now we use guidelines.readthedocs.org site instead of MediaWiki instance on posredni.vls
  - Made a backup of MediaWiki for guidelines.openaire.eu
  - Create a redirect rule for old URLs for MediaWiki for guidelines.openaire.eu
July 06, 2015:
  - jochen.schirrwagen added to "driver" group on beta.services.openaire.eu (see Redmine #1378)
July 02, 2015:
  - Restarted HBase on DM Hadoop cluster after nightly issues
  - Enabled CNR LDAP authentication for the users of {beta,test}.openaire.eu (see Redmine #1405)
July 01, 2015: Rescued OpenAIRE2020 machines from a leap second armaggeddon
June 30, 2015: Replaced a failed disk in slave4.hadoop.dm.openaire.eu
June 29, 2015:
  - Created a database for DFM on beta.services.openaire.eu (see Redmine #1355)
  - Configured Nginx on pilot.services.openaire.eu for supporting domain postgrantoapilot.openaire.eu instead of goldoapilot.openaire.eu (asked byStefania Martziou)
June 26, 2015:
  - Changed permissions for /var/lib/dfm on beta.services.openaire.eu on dnet:dnet (see Redmine #1355)
  - Disabled "monkeys" user on beta.services.openaire.eu (see Redmine #1387)
June 25, 2015:
  - Open access to Redis & PostgreSQL port on beta.stats.openaire.eu from new.openaire.eu (asked by Eri)
  - stefania.martziou added to "driver" group on pilot.services.openaire.eu and granted the same permissions like for Antonis to her (asked by Antonis)
  - Created backup of Joomla on beta.openaire.eu, asked by Eri
  - Synchronised beta.openaire.eu with new.openaire.eu (asked by Eri)
  - Configured Nginx for a reverse proxy for DFM on beta.services.openaire.eu (see Redmine #1387)
June 19, 2015:
  - eri.katsari, nikon.gasparis also added to "driver" group on beta.services.openaire.eu (see Redmine #1378)
June 18, 2015:
  - andrea.manocci, sandro.labruzzo also added to "driver" group on beta.services.openaire.eu (see Redmine #1378)
  - Set the same rights in /etc/sudoers for "driver" group like for "monkeys" user
  - Removed the rights from /etc/sudoers for the commands delivered by dnet-* packages, because they have been uninstalled during the distribution upgrade
June 17, 2015:
  - Enabled CNR LDAP authentication for the users of beta.services.openaire.eu (see Redmine #1378)
  - alessia.bardi, claudio.atzori, michele.artini, antonis.lempesis, katerina.iatropoulou added to "driver" group there
  - Mounted /home via NFS there
June 16, 2015:
  - Upgraded beta.services.openaire.eu from Ubuntu Precise to Trusty (see Redmine #1355)
June 15, 2015:
  - michele.artini (CNR) added to "driver" group on gateway.hadoop.dm.openaire.eu (see Redmine #1023)
  - Open access to port 6379 (Redis) on new.openaire.eu from test.openaire.eu (asked by Eri)
  - Open access to port 389 (Slapd) on new.openaire.eu from services.openaire.eu
June 10, 2015:
  - Restarted DataNodes & RegionServers of DM Hadoop cluster (see Redmine #1367)
  - Created backup of Joomla 3 on openaire-www.vls (aka new.openaire.eu), asked by Argiro
June 9, 2015: Removed old logs of Oozie on master3.hadoop.iis.openaire.eu because of a lack of a free disk space on / partition
June 8, 2015:
  - Uploaded a fixed versions of Java SSL related .jar files (local_policy.jar & US_export_policy.jar) on services.openaire.eu and beta.services.openaire.eu (see Redmine #1351)
June 3, 2015:
  - Fixed logrotate's config for rotating catalina.log and localhost_access.log on services.openaire.eu and beta.services.openaire.eu
  - Killed hunged mongodump process on services.openaire.eu and restarted MongoDB there (see Redmine #1342)
  - giorgos.alexiou (ATHENA) added to "driver" group on gateway.hadoop.{dm,iis}.openaire.eu (asked by Eri)
June 2, 2015: Open access to port 7911 (Thrift) on beta.services.openaire.eu from scoobydoo.di.uoa.gr (asked by Argiro)
June 1, 2015:
  - Set client_max_body_size 200M for Nginx on furkot2.vls (see Redmine #1343)
  - Set ssl = false for PostgreSQL on openaire-services-{beta,pilot}.vls, because we don't need SSL there
May 31, 2015:
  - Started PostgreSQL server on openaire-services.vls, because it was dead because of issues with permissions to a private SSL certificate
  - Set ssl = false, because we don't need SSL there
May 29, 2015:
  - Untared and next bzipped the newest bunch of arXiv PDFs: arXiv_pdf_1412_*.tar & arXiv_pdf_150[1-3]_*.tar (see Redmine #904)
May 28, 2015:
  - Started container 8580 on openaire-services-pilot.vls, because it was dead (asked by Nikon)
  - Disabled an user's authentication for goldoapilot.openaire.eu, because it's a public available now (asked by Antonis)
May 28, 2015: Debuged an issue with sending messages from openaire-services-pilot.vls (asked by Nikon)
May 26, 2015: Created a directory (/srv/www/static/openaire2020/wp10/deadlines/) on oscypek.vls for OpenAIRE2020 WP10 deadlines pages (asked by Mateusz Kobos)
May 25, 2015: Installed php5-mcrypt package on openaire-www.vls (asked by Eri)
May 22, 2015:
  - Removed some old logs of IS (8280) and 8380's .hprof files on services.openaire.eu, because / partition was full-filled.
  - Created backup of Joomla 3 on openaire-www.vls (aka new.openaire.eu), asked by Argiro
  - Restored Joomla from furkot2 on test.openaire.eu, asked by Argiro
May 21, 2015:
  - Migrated a proxy rules for {blogs,box,guidelines,mining,validator}.openaire.eu from furkot2 (Nginx) to openaire-www.vls (Apache2)
May 20, 2015:
  - jyunyao.huang (UBONN) added to "driver" group on gateway.hadoop.dm.openaire.eu
  - Migrated a proxy rules for api.openaire.eu from furkot2 (Nginx) to openaire-www.vls (Apache2)
May 19, 2015:
  - Created a new Piwik website for tracking UMINHO repository (asked by Dimitris)
  - Restarted slave8.hadoop.dm.openaire.eu because of issue with Java zombie processes (see Redmine #1326)
May 14, 2015: Disabled authentication for /simplesaml requests on new.openaire.eu (asked by Eri, see Redmine #1251)
May 12, 2015:
  - Installed python-pip and pymongo on beta.services.openaire.eu (asked by Sandro La Bruzzo, see Redmine #1271)
  - Upgraded Piwik for OpenAIRE+ (https://analytics.openaire.eu/)) from version 2.12.0 to 2.13.1.
May 11, 2015:
  - Created NetApp volumes (4TB) for objectStore's PDFs files (see Redmine #1271) adnd mounted via NFS on services.openaire.eu and beta.services.openaire.eu
  - Deployed a new .war files of 8380 container on services.openaire.eu which includes a new settings of webapps log4j.properties
May 8, 2015:
  - Prepared Nginx rules (/validator-admin/ & /validator-service/) for a new production validator (see Redmine #1303)
  - Disabled the last changes for an access policy for goldoapilot.openaire.eu, now it's protected by a basic authentication again
  - Installed python-pip and pymongo on services.openaire.eu (asked by Sandro La Bruzzo, see Redmine #1271)
May 7, 2015: Enabled an open access to goldoapilot.openaire.eu for NKUA's networks, the rest can see a static page
May 6, 2015: Enabled a basic authentication for goldoapilot.openaire.eu, because it's still not ready for a public access (asked by Natalia)
May 4, 2015: Applied a new logging.properties for 8280 and 8980 on beta.services.openaire.eu (see Redmine #1289)
April 30, 2015: Configured goldoa-pilot.openaire.eu as an alias for goldoapilot.openaire.eu
April 29, 2015:
  - Antonis Lampesis added to the user aliases POSTGRES_ADMINS and DNET_ADMINS in /etc/sudoers on pilot.services.openaire.eu
  - Configured goldoapilot.openaire.eu domain in DNS and Nginx for it on pilot.services.openaire.eu
  - Removed unused data folder jenshaarld.aasheim/ on box.openaire.eu (Jens Harald Aasheim changed his username to j.aasheim)
April 28, 2015:
  - Created backup of Joomla 3 on openaire-www.vls (aka new.openaire.eu), asked by Argiro
  - Changed symlinks for the logs of 8280 (/var/log/dnet/is instead of /var/log/tomcat_dnet/8280) and 8980 (/var/log/dnet/provision instead of /var/log/tomcat_dnet/8980) containers on beta.services.openaire.eu and services.openaire.eu (see Redmine #1289)
  - Applied a new logging.properties for 8380 on beta.services.openaire.eu (see Redmine #1289)
April 24, 2015:
  - Created backup of Joomla 3 on openaire-www.vls (aka new.openaire.eu), asked by Argiro
  - Allowed access of new.openaire.eu to 7911 port on beta.services.openaire.eu, asked by Argiro
April 19, 2015: Set log_min_duration_statement = 10000 in /etc/postgresql/9.1/main/postgresql.conf on stats.openaire.eu and beta.stats.openaire.eu
April 17, 2015: Set log_min_duration_statement = 100 in /etc/postgresql/9.1/main/postgresql.conf on stats.openaire.eu and beta.stats.openaire.eu, because postgresql.log was too big because of many logged INSERT statements
April 16, 2015: Created group "joomla" on openaire-www.vls (new.openaire.eu) in order to give NKUA people an access to Apache server logs
April 14, 2015: Increased size of /srv2 partition (MongoDB's objectStore) on beta.services.openaire.eu from 1.5TB to 1.7TB (see Redmine #1271)
April 15, 2015: Deployed ci-test.hadoop.iis.openaire.eu, a new pseudo Hadoop cluster for running IIS integration tests (see Redmine #781)
April 3, 2015: Deployed SimpleSAMLphp on new.openaire.eu (see Redmine #1251)
March 30, 2015: Removed unused Nginx proxy rules for /usagestats on furkot2 (asked by Eri)
March 30, 2015: Restored /var/www/templates/yoo_solar/styles/openaire/css/*.css files from a weekly snapshot of the backup (asked by Argiro)
March 30, 2015: Created /var/log/dnet/validator/ directory for the validator logs on openaire-services, openaire-services-beta and openaire-services-pilot (asked by Nikon)
March 26, 2015: Created a new Piwik website for tracking the beta validator (asked by Nikon)
March 25, 2015: Restored Joomla from furkot2 on new.openaire.eu, because it has been damaged by Natalia
March 24, 2015: Upgraded Piwik for OpenAIRE+ (https://analytics.openaire.eu/)) from version 2.11.0 to 2.12.0.
March 24, 2015: Moved analytics.openaire.eu to piwik.vls.icm.edu.pl which is running under control of Debian Jessie (Apache 2.4.10 & PHP 5.6.6)
March 24, 2015: Configured a reverse proxy (by Nginx) for La Referencia (http://lareferencia.openaire.eu/))
March 24, 2015: Assisting during the upgrade of the openaire Joomla component on the production and 8380 container on services.openaire.eu (asked by Katerina Iatropoulou).
March 23, 2015: Deployed Tomcat instances for Gold Open Access (see Redmine #1168 and La Referencia (see Redmine #1138), added permissions to Nikon Gasparis for managing PostgreSQL server and Tomcat services on openaire-services-pilot machine
March 20, 2015: Fixed proxy rules for http://api.openaire.eu/projects/ requests, now handled by provision.war container at services.openaire.eu:8980 (see Redmine #1136)
March 17, 2015:
  - Unpacked NKUA's .tar files with ArXiv PDF files (arXiv_pdf_1409_*.tar & arXiv_pdf_141⁰¹_*.tar)
  - Created README file in /mnt/harvested_pdfs/ on openaire-services
  - Compressed NKUA's .tar files (arXiv_pdf_130⁸⁹_*.tar, arXiv_pdf_1409_*.tar & arXiv_pdf_141⁰¹_*.tar)
March 9-11, 2015: Investigation on why home page of production portal (http://www.openaire.eu/)) is so slowly (noticed by Natalia). The reason was a huge k0wx9_sh404sef_urls table of Joomla and long running SELECT queries which needed a few seconds to finish and get hundreds of thousands of records and many mega bytes of data.
March 10, 2015: Configured on beta.openaire.eu a reverse proxy to new separated validator services (/validator, /validator-admin, /validator-service) (asked Nikon Gasparis)
March 10, 2015: Directory /var/www/images/ writable for joomla user on furkot2, openaire-www, openaire-www-{beta,test} (asked Katerina)
March 9, 2015: Created dnet.beta and dnet.production users on IIS Hadoop cluster (asked by Claudio and Marek Horst)
March 9, 2015: Installed advancecomp package (which delivers advpng utility) for PNG optimization on furkot2, openaire-www, openaire-www-{beta,test} (asked Katerina)
March 7, 2015: Increased (to 5GB) quota for najla.rettberg on http://box.openaire.eu/, removed doubled account Najla.Rettberg
March 6, 2015: Installed optipng & pngcrush packages for PNG optimization on furkot2, openaire-www, openaire-www-{beta,test} (asked Katerina)
March 5, 2015: Deployed a new VM openaire-services-pilot for handling OpenAIRE pilot services, for example LaReferencia (#1138) or Gold Open Access (see Redmine #1168)
March 4, 2015: Stopped PostgreSQL on furkot2, openaire-www, openaire-www-{beta,test}, because it's unused there
March 3, 2015: Deployed a new VM openaire-www for upgrading the portal Joomla to version 3.x (asked by Natalia). It will replaces furkot2 in a few months.
March 3, 2015: Integrated Piwik (analytics.openaire.eu) with guidelines.openaire.eu
March 3, 2015: Configured Piwik (analytics.openaire.eu) for validator.openaire.eu, guidelines.openaire.eu, blogs.openaire.eu and goldpilot.openaire.eu (asked by Natalia)
March 3, 2015: Created /user/dnet.beta and /user/dnet.production directories in DM HDFS (asked by Claudio and Marek Horst)
March 3, 2015: Created backup of /var/www/components/com_openaire/ and /var/www/templates/yoo_solar/ directories on openaire-www-beta and assisting during the upgrade of the openaire Joomla component
March 2, 2015: Created /user/dnet.beta and /user/dnet.production directories in IIS HDFS (asked by Claudio and Marek Horst)
March 2, 2015: Fixed LDAP settings on http://box.openaire.eu/ broken by Natalia
February 27, 2015: Removed doubled account Natalia.manola on http://box.openaire.eu/
February 20, 2015: Upgraded Piwik for OpenAIRE+ (https://analytics.openaire.eu/)) from version 2.10.0 to 2.11.0.
February 17, 2015: Increased memory from 16GB to 24GB on openaire-solr and openaire-solr-beta machines (see Redmine #1136)
February 16, 2015: Fixing a broken search feature of the portal (https://www.openaire.eu/search/find)) because of java.lang.OutOfMemoryError on openaire-solr machine with the index (please Redmine issue #1136)
February 11, 2015: Added SSH public key of Andrea Mannocci to authorized_keys of "monkeys" user
February 10, 2015: Deployed OwnCloud instance at https://box.openaire.eu/ (asked by Natalia)
February 6, 2015: Made /srv/www/joomla/api-docs/ on furkot2 writable for "www-data" group (and "joomla" user) (asked by Natalia who wanted to change CSS)
February 4, 2015: Created accounts on the gateway of DM Hadoop cluster for Sahar Vahdati and Christoph Lange from UBONN (please see Redmine issue #1089).
February 4, 2015: Configured a proxy rule for a new API exporting TSV of projects (please see Redmine issue #1106).
February 2, 2015: Renamed the mailing list openaireplus-wp7@bwnmail.icm.edu.pl to openaire-mining@bwnmail.icm.edu.pl (asked by Mateusz Kobos).
January 30, 2015: Increased size of /srv2 partition (from 2 TB to 2.4 TB) on services.openaire.eu.
January 27, 2015: Increased size of /srv3 partition (from 400 GB to 600 GB) on services.openaire.eu.
January 26, 2015: Increased value of Nginx timeouts on www.openaire.eu (please see Redmine issue #1079).
January 26, 2015: Increased size of /srv partition (from 450 GB to 500 GB) on services.openaire.eu.
January 23, 2015: Increased size of /srv partition (from 400 GB to 450 GB) on services.openaire.eu.
January 23, 2015: Installed xmlstarlet package on services.openaire.eu and beta.services.openaire.eu (asked by Alessia Bardi, please see Redmine issue #1076).
January 22, 2015: Assisting during the upgrade of the openaire Joomla component on the production and 8380 container on services.openaire.eu (asked by Katerina Iatropoulou). Fixed the write permissions to /var/www/joomla/components/com_openaire/ directory for joomla user on beta.openaire.eu.
January 16, 2015: Synchronised test.openaire.eu with the production (asked by Katerina Iatropoulou).
January 14, 2015: Assisting during the upgrade of 8380 container on services.openaire.eu (asked by Katerina Iatropoulou).
January 7, 2015: Upgraded Piwik for OpenAIRE+ (https://analytics.openaire.eu/)) from version 2.9.1 to 2.10.0.

Files (0)

Updated by Marek Horst over 6 years ago · 7 revisions

Project

General

Profile

OpenAIRE2020 - Wiki

Wiki

Change Log for servers administration operations¶

2nd Period (14/12/2016 - 30/06/2018)¶

1st Period (01/01/2015 - 13/12/2016)¶