IFCA Computing services

[Resolved] Databases error

Wed, 28 Jan 2026 07:59:19 +0000

Solved

[Resolved] Actualizacion de auth01

Tue, 02 Dec 2025 08:00:13 +0000

Mañana martes día 2/12/2025 se proceder a actualizar auth01 por motivos de seguridad. La interrupcin del servicio no deberaí llevar mas de 1 hora. Disculpen las molestias -– Tomorrow, Tuesday, December 2, 2025, auth01 will be updated for security reasons. The service interruption should not take more than 1 hour. We apologize for any inconvenience.

[Resolved] Corte de corriente en el CPD / Power outage in the datacenter (27-10-2025 10:00)

Mon, 27 Oct 2025 09:00:48 +0000

[English version below]

Estimado usuario:

Debido a tareas de mantenimiento de los transformadores generales del edificio se le informa que se realizará una parada parcial de nuestro CPD.

Aprovechando esta parada parcial se realizarán tareas de actualización y mantenimiento de diverso equipamiento de la sala.

- Parada de servicios (no básicos): 27 de Octubre de 2025 a las 10:00

- Los servicios básicos se mantendrán encendidos (almacenamiento, cloud, red y refrigeración).

- Los nodos de computo se apagarán para que el rearranque no les afecte, las colas se cerrarán el viernes 24 para que se drenen todos los jobs correctamente.

- Se aprovechará la parada para recablear al estado original el switch que se estropeó hace un par de meses.

- Duración estimada de la parada: 4h

En todo momento podrá seguir el estado de nuestros servicios en [1]

Si tiene cualquier duda, puede consultar con nosotros en [2]

Disculpen las molestias

IFCA Advanced Computing Service Team

-– — —

[English version]

Dear user:

Due to maintenance work on the building’s general transformers, we would like to inform you that our data center will be partial shut down.

We will take advantage of this partial shutdown to perform updates and maintenance on various equipment in the room.

- Shutdown of non-essential services: October 27, 2025, at 10:00 a.m.

- Basic services will remain online (storage, cloud, network, and cooling).

- The compute nodes will be shut down so that the restart does not affect them, and the queues will be closed on Friday, October 24, so that all jobs are drained correctly.

- The shutdown will be used to rewire the switch that broke down a couple of months ago.

- Estimated duration of the shutdown: 4 hours.

You can check the status of our services at any time at [1].

If you have any questions, please contact us at [2].

We apologize for any inconvenience.

IFCA Advanced Computing Service Team

-– — —

[Resolved] [IFCA Advanced Computing Service] Reemplazo de switch principal / Main switch replacement

Thu, 28 Aug 2025 19:45:49 +0000

23/9/2025 12:30

Debido a un problema, no se ha podido cambiar de momento el switch, estamos en espera de respuesta del servicio técnico.

Se actuará en cuanto tengamos mas información.

-–

Due to a problem, we have not been able to change the switch at this time. We are awaiting a response from technical support.

We will take action as soon as we have more information.

18/9/2025

[English version below]

Estimados/as usuarios/as,

Hace unos semanas tuvimos un problema con un switch principal del CPD y se logró parchear su ausencia.

Ya tenemos reemplazo preparado para volver a meterlo y se va a hacer el próximo martes 23 de Septiembre.

Les seguimos manteniendo informados de la actuación en nuestra página de status [1].

Para cualquier duda o información, no dude en contactarnos en [2].

Disculpen las molestias

Un saludo

IFCA Advanced Computing Service Team

[English version]

Dear users,

A few weeks ago, we had a problem with a main switch in the data center, and we managed to patch it up.

We already have a replacement ready to put back in, and it will be done next Tuesday, September 23.

We continue to keep you informed of the performance on our status page [1].

For any questions or information, please feel free to contact us at [2].

Sorry for the inconvenience

Best regards

IFCA Advanced Computing Service Team

19/9/2025

En el cambio del switch intentaremos que no pierda red lo máximo posible, pero alguna máquina perderá la conexión en algún momento determinado.

Disculpen las molestias

-–

When switching, we will try to minimize network loss as much as possible, but some machines will lose connection at some point.

Sorry for the inconvenience

29/8/2025

A switch that provides part of the network to our internal traffic is failing. We are investigating the cause of this failure and a possible solution to turn up the machines.

[Resolved] [Cloud] Disrupted issues in compute and networking services

Thu, 31 Jul 2025 10:00:49 +0000

[Update 31 July]

Cloud services are operative again.

-—————————————————–

Investigating - We are investigating a potential issue that might affect the cloud service when launching an instance o some other new action that affects networking or compute services. The access of the cloud machines might be operative except for some punctual interruptions.

We are sorry for any inconvenience this may cause you. This incident post will be updated once we have more information.

[Resolved] [HPC and Grid] Slurm upgrade

Tue, 29 Jul 2025 07:00:33 +0000

[Update 4 Aug 9am]

[English version below]

Estimados/as usuarios/as,

La pasada semana se procedió a actualizar Slurm a la última versión estable y no vulnerable.

Esta actualización nos ha conyevado a más cambios de los previstos inicialmente y estamos terminando algún otro:

Se han eliminado los nodos que teníamos en centos7, que se encuentra fuera de mantenimiento hace tiempo.
Todos los nuevos nodos están actualizados con Alma Linux 9.
Reestructuración de particiones de compute y res con los nuevos nodos en AL9
Eliminación del nodo antiguo de login1 en CentOS 7, estamos trabajando para que el nuevo login1 con AL9 funcione dentro de nada

Les seguimos manteniendo informados de la actuación en nuestra página de status [1].

Para cualquier duda o información, no dude en contactarnos en [2].

Disculpen las molestias

Un saludo

IFCA Advanced Computing Service Team

[English version]

Dear users,

Last week we proceeded to update Slurm to the latest stable and non-vulnerable version.

This update has led us to more changes than initially planned and we are finishing some others:

The nodes we had on centos7, which has been out of maintenance for some time, have been removed.
All new nodes are upgraded to Alma Linux 9.
Restructuring of compute and res partitions with the new nodes in AL9.
Removal of the old login1 node in CentOS 7, we are working to get the new login1 with AL9 up and running in no time.

We continue to keep you informed of the performance on our status page [1].

For any questions or information, please feel free to contact us at [2].

Sorry for the inconvenience

Best regards

IFCA Advanced Computing Service Team

[Update 1 Aug 9am]

Slurm controllers are failing. Altamira nodes were removed that cause some issues when slurmctld has to get up.

[Update 30 July 4pm]

Grid and HPC nodes are already upgraded to slurm 25.05 except altamira nodes. In the following days we will include new nodes to Altamira partition to replace old altamira nodes that are not compatible with the new version of slurm.

[Update 30 July 9am]

The last version of slurm (25.05) is already installed in the controllers and HPC nodes. Grid is missing but will be operative throughout the morning. There were a delay in the upgrade due to an issue in the WN.

-———————————————————————

El próximo martes 29 de Julio se procederá a la actualización del sistema de colas slurm del clúster de HPC. Esto es una medida que llevamos semanas posponiendo pero que debemos realizar aprovechando el periodo estival y que hay una menor actividad. Esta actualización es necesaria debido a una vulnerabilidad de seguridad en la versión actual.

La intervención está planificada para la mañana del martes donde todos los nodos de cómputo estarán apagados. También, para asegurar que los trabajos actuales terminan su ejecución correctamente se procederá a drenar todos los nodos hoy, 24 de julio (mañana y el lunes es fiesta en Santander). En principio, la intervención no debería durar más de la mañana del martes.

-—————————————————————–

Dear user,

Next Tuesday, July 29, the Slurm queueing system of the HPC cluster will be updated. This is a maintenance operation we have postponed for several weeks, but it must now be carried out, taking advantage of the summer period when system activity is lower. This update is necessary due to a security vulnerability present in the current version.

The intervention is scheduled for Tuesday morning, during which all compute nodes will be powered off. In order to ensure that currently running jobs complete properly, all nodes will be drained today, July 24 (since tomorrow and Monday are public holidays in Santander). The procedure should not take longer than Tuesday morning.

[Resolved] Redundancia red cloud / Cloud network redundancy

Tue, 24 Jun 2025 06:10:44 +0000

[English version below]

Estimados/as usuarios/as,

El próximo martes 24 de Junio se procederá a redundar el servicio de red de nuestro servicio cloud.

Esta redundancia lleva pendiente tiempo y conllevará un mejor servicio sobre nuestra red, pero la red hacia sus máquinas se irá y volverá en determinados momentos durante toda la mañana.

Les mantendremos informados de la actuación en nuestra página de status [1].

Para cualquier duda o información, no dude en contactarnos en [2].

Disculpen las molestias

Un saludo

IFCA Computing Service Team

-– — —

[English version]

Dear users,

Next Tuesday, June 24, we will proceed to redundant the network service of our cloud service.

This redundancy is long overdue and will lead to better service over our network, but the network to your machines will be down and back up at certain times throughout the morning.

We will keep you informed of the work on our status page [1].

For any questions or information, please do not hesitate to contact us at [2].

Apologize for any inconvenience this may cause.

Best regards

IFCA Computing Service Team

[1] https://ubuntu.com/blog/ubuntu-20-04-lts-end-of-life-standard-support-is-coming-to-an-end-heres-how-to-prepare

[Resolved] Electrical blackout

Mon, 28 Apr 2025 10:30:35 +0000

Apagón eléctrico general en España.

Estamos trabajando para que todo se levante con normalidad lo antes posible

Disculpen las molestias

-– — —

General power outage in Spain.

We are working to get everything up and running as soon as possible.

Sorry for the inconvenience

[Resolved] Cloud upgrade

Mon, 14 Apr 2025 07:31:17 +0000

Se va a proceder a actualizar cloud a una nueva versión, esto podrá tener algún corte de servicio momentaneo.

Disculpen las molestias

-– — —

We will proceed to update cloud to a new version, this may have some momentary service outage.

Sorry for the inconvenience

[Update 05-13]

One of the upcoming tasks involves upgrading the operating systems of all machines in the cloud ecosystem — including the core compute, network, and storage services (most of which have already been updated), as well as the nodes hosting the instances. This step is necessary because the operating system currently in use will reach its end of standard support this month [1].

This upgrade process will begin tomorrow, Wednesday, May 14, and will involve a downtime of approximately 1 to 1.5 hours per instance-hosting machine.

If any of you have a justified need to postpone the upgrade for a few days, please let us know at computing.support@ifca.unican.es, and we will schedule the upgrade of those specific machines for the end of the process. We apologize for the short notice.

[Update 05-09 15pm]

After the investigatin, we got to make the network metadata service working. You can launch an instance and access it without any problem.

[Update 05-08 10:00 am]

Second upgrade of the main controllers is done. The Operating system update of the nodes is missing, it is scheduled for the following weeks grou by group.

A problem in the metadata service of the network or the compute components makes that the launching or rebooting of the instances is not working well. We are investigating the reason of the issue.

[Update 04-25 13:30 pm]

After the first upgrade of all the Openstack services and some problems with cinder and nova, everything should be stabilized. If you see some issue, do not hesitate to send us a ticket to take a look if something is misconfigured.

We wil continue with the next update that hopefully, would be lasted until next week.

[Resolved] System authentication failing

Fri, 04 Apr 2025 10:05:51 +0000

Some errors in the FREEIPA database are causing issues in the authentication process.

[Resolved] Ceph Upgrade

Wed, 02 Apr 2025 08:06:17 +0000

The Ceph storage system, which provides external storage volumes for the cloud system, is about to be updated. Therefore, there may be some timeouts at some point."

[Resolved] Login2 - No login

Tue, 01 Apr 2025 09:07:41 +0000

Buenos días,

Tenemos un problema entre sssd y login2 que no autentica algunos usuarios.

Estamos trabajando en solucionarlo

Disculpen las molestias.

Un saludo

-– — —

Good morning,

We have a problem between sssd and login2 that does not authenticate some users.

We are working on fixing it

Sorry for the inconvenience.

Best regards

[Resolved] Ampliacion de potencia del CPD / Datacenter power upgrade

Sun, 23 Feb 2025 11:28:24 +0000

[ Update Mar 11, 11:00 a.m.]

Buenos días,

Cloud, HPC y Grid ya estná operativos.

El almacenamiento GPFS ya está ampliado.

Disculpen las molestias

Un saludo

-– —-

Good morning,

Cloud, HPC and Grid are now operational.

GPFS storage is now extended.

Apologies for the inconvenience

Best regards

[ Update Mar 5, 11:30 a.m.]

Buenos días,

Estamos teniendo algún problema con una base de datos y estamos trabajando en solucionarlo.

Rebalanceo de GPFS al 98%, está tardando más de lo que esperábamos por los millones de ficheros pequeños.

Disculpen las molestias, les mantendremos informados.

Un saludo

-–

Good morning,

We are having some problem with a database and we are working on fixing it.

GPFS rebalancing at 98%, it is taking longer than expected due to the millions of small files.

Sorry for the inconvenience, we will keep you informed.

Best regards

[ Update Mar 3, 13:40 a.m.]

GPFS status:

NVR scrub 14d (41%)

SSD scrub 14d (87%)

GPFS rebalance (96%)

[ Update Feb 28, 8:40 a.m.]

Todos los servicios cloud estn levantados, si tiene alguna VM con un volumen asociado y se encuentra en read-only reinicie la máquina.

Incremento de almacenamiento GPFS contina rebalanceando, se arrancará en cuanto termine este proceso.

[English version]

All cloud services are up, if you have any VM with an associated volume and it is in read-only restart the machine.

GPFS storage incrementing continues to rebalance, it will be started as soon as this process is finished.

Regards

[ Update Feb 27, 12:30 p.m.]

Trabajos terminados.

Arrancando servicios generales

[English version]

Finished works.

Starting up general services

Best regards

[ Update Feb 25, 05:30 p.m.]

After several unforeseen events in the power supply intervention, the estimated datetime for system power-on will be Feb 26th afternoon instead of the morning as it was expected.

[English version below]

Estimados/as usuarios/as,

El próximo domingo 23 de Febrero se procederá a realizar una parada técnica total del CPD sobre las 18:00

En esta parada técnica se realizarán las siguientes tareas:

Ampliación de potencia eléctrica del CPD.
Cambio de cuadro eléctrico general.
Ampliación de almacenamiento GPFS.
Renovación de refrigeradora

Ya que estas tareas serán costosas en tiempo, se prevee que comencemos a arrancar el CPD el miércoles a lo largo de la mañana.

Les mantendremos informados de la actuación en nuestra página de status [1].

Para cualquier duda o información, no dude en contactarnos en [2].

Disculpen las molestias

Un saludo

IFCA Computing Service Team

— — —

[English version]

Dear users,

Next Sunday, February 23rd, a total technical shutdown of the Datacenter will be carried out at 18:00.

In this technical shutdown the following tasks will be performed:
Extension of electrical power of the datacenter.
Change of the general electrical panel.
GPFS storage expansion.
Renewal of refrigerator.

Since these tasks will be time-consuming, it is expected that we recover the services on Wednesday morning.

We will keep you informed of the work on our status page [1].

For any questions or information, please do not hesitate to contact us at [2].

Apologize for any inconvenience this may cause.

Best regards

IFCA Computing Service Team

[1] https://status.ifca.es

[2] computing.support@ifca.unican.es

[3] https://advancedcomputing.ifca.es

[Resolved] Authentication system migration

Fri, 14 Feb 2025 11:35:41 +0000

During this week, we are updating the authentication system. We will proceed with the auth01 machine, this will not be available for a while, connect to the replica, auth02.ifca.es.

[Resolved] Authentication to login2 failing using password+otp

Tue, 19 Nov 2024 00:09:59 +0000

We are investigating a potential issue to authenticate users in login2 when ssh with password and otp is used.

Login2.ifca.es is already working

[Resolved] [Gitlab] Upgrade the service to solve security vulnerabilities

Thu, 10 Oct 2024 19:44:49 +0000

Several security vulnerabilities are needed to be solved in GitLab service [1]. We need to update GitLab and its dependencies to the latest versions but it will take more time than expected.

[Resolved] Incidencia eléctrica urgente / Urgent electrical incident

Sat, 28 Sep 2024 11:31:12 +0000

[Update 1 oct. 10:45]

The intervention was performed well and in time. The cloud services (nextcloud, confluence) are already available.

The rest of the machines (grid or HPC) are restarting.

-———————————————————————————————————————————————–

Nota: Corte de corriente eléctrica del CPD mañana 01/10/2024 martes a las 9:00

Note: Datacenter power down supply tomorrow 01/10/2024 Tuesday at 9:00 a.m.

[English version below]

Estimados/as usuarios/as,

Debido a una incidencia eléctrica detectada en un diferencial general del CPD, tenemos que realizar una intervención urgente sobre el.

Estamos pendientes de la decisión de la empresa de mantenimiento, si podemos cortar una parte o todo el CPD, pero os queremos informar que por seguridad el corte se realizará lo antes posible.

Disculpad las molestias que se puedan ocasionar.

Para cualquier duda, no dude en contactarnos en [2].

Un saludo

IFCA Advanced Computing Team

[English version]

Dear users,

Due to an electrical incident detected in a general differential of the Datacenter, we have to perform an urgent intervention on it.

We are awaiting the decision of the maintenance company, if we can cut a part or all of the Datacenter, but we want to inform you that for safety reasons the cut will be made as soon as possible.

We apologize for any inconvenience this may cause.

If you have any questions, please do not hesitate to contact us at [2].

Best regards

IFCA Advanced Computing Team