Installing and running HEPCloud’s Decision Engine on EL9

Currently the only version supporting EL9 is the development version, DE 2.0.x, which corresponds to the master branch in Git.

Decision Engine uses a PostgreSQL database back-end and Redis as message broker and cache.

The first one is installed as RPM requirement, the second is used as container. You need to install the pre-requisites RPM and then the Python packages for Decision Engine framework (decisionengine) and install and add the standard channels (decisionengine_modules).

The following instructions assume Alma Linux 9. You may need to adapt them slightly for other EL9 flavors.

These also assume a system installation, performed as root. decisionengine will run as the decisionengine user.

Install Decision Engine and the standard modules

RPM installation

Prerequisites setup. Make sure that the required yum repositories and some required packages (python3, gcc, …) are installed and up to date.

# Possible OSG versions: 24, 23, 24-upcoming
OSG_VERSION=24
# YUM repo for Decision Engine
GWMS_REPO=osg-development
dnf install -y epel-release yum-utils sed
dnf config-manager --set-enabled crb
/bin/sed -i '/^enabled=1/a priority=99' /etc/yum.repos.d/epel.repo
dnf -y install "https://repo.osg-htc.org/osg/$OSG_VERSION-main/osg-$OSG_VERSION-main-el9-release-latest.rpm"

Install the decision engine (add –enablerepo=osg-development for the latest development version)

DE_REPO=osg
dnf install -y --enablerepo="$DE_REPO" decisionengine-onenode
# Individual packages are: decisionengine-deps (framework req) decisionengine-modules-deps (modules req) decisionengine-standalone (2 deps+httpd)

Install the required Python packages

decisionengine-install-python
# This will install the latest version in PyPI (hepcloud-de, hepcloud-de-modules, and all requirements)
# To install an arbitrary version from GitHub you can use the desired tag:
decisionengine-install-python --remote --de-git-ref 2.0.4
# This shell script (included in decisionengine-deps) installs the Decision Engine Python code.
# You can run it as root or as the decisionengine user
# To see all the options:  decisionengine-install-python --help
# Double check that pip added $HOME/.local/bin to the PATH of user decisionengine

Start and enable HTCondor and httpd:

systemctl start condor
systemctl enable condor

systemctl start httpd
systemctl enable httpd

Optionally install these extra packages

# htgettoken - if you need it to generate SciTokens
dnf -y install htgettoken

Fix the GlideinWMS Frontend installation

We will make HEPCloud’s Decision Engine using some GlideinWMS libraries but independent from the Frontend. The codebases, though, are still intertwined, so there are some adjustments needed to the GlideinWMS installation.

Change to decisionengine the ownership of the frontend directories:

chown -R decisionengine: /etc/gwms-frontend

Set up PostgreSQL

PostgreSQL is installed by the requirements RPM, Postgresql 13:

Enable postgresql
```
systemctl enable postgresql
```
Init the database
```
postgresql-setup --initdb
```

Use the sed command or manually edit /var/lib/pgsql/data/pg_hba.conf like the following:

# Option1 - sed:
sed -e '/^local   all             all/s/peer/trust/' -e '/^host    all             all/s/ident/trust/' -i /var/lib/pgsql/data/pg_hba.conf
# Option2 - manual edit:
[root@fermicloud371 ~]# diff  /var/lib/pgsql/data/pg_hba.conf~ /var/lib/pgsql/data/pg_hba.conf
80c80
< local   all             all                                     peer
---
> local   all             all                                     trust
82c82
< host    all             all             127.0.0.1/32            ident
---
> host    all             all             127.0.0.1/32            trust
84c84
< host    all             all             ::1/128                 ident
---
> host    all             all             ::1/128                 trust

(difference of the correct file from the default one - pg_hba.conf~) This is setting the authentication method to trust

Fix the PostgreSQL installation. Not sure why, but the run directory was missing and causing the startup to fail.
```
# Without this the systemctl start was failing and the error was in /var/lib/pgsql/data/log/postgresql-*.log
```
mkdir -p /var/run/postgresql chown postgres: /var/run/postgresql
Start the database server
```
systemctl start postgresql
```
Create the decisionengine database
```
createdb -U postgres decisionengine
```

The schema and the connection will be created and configured during the Decision Engine framework initialization.

RHEL also provides other PostgreSQL versions via streams. These may require changes to environment variables like PG_VERSION and PATH to use the database.

Install Redis

Install and start the message broker (Redis) container on your system. You can find more details on the redis document

You may need to fix the firewall used removing the one incompatible with Podman

dnf rm iptables-legacy dnf install iptables-nft
Install Padman
```
dnf install -y podman
```

Run the Redis container

podman run --name decisionengine-redis -p 127.0.0.1:6379:6379 -d redis:6 --loglevel warning
# When prompted to select an image, pick "docker.io/library/redis:6".

Test

Now you can type decisionengine --help as root or while logged in as decisionengine to print the help message. To do more you need first to configure Decision Engine.

Remember that all the times that you start a new shell as decisionengine you may need to add the PIP binary directory to the PATH

export PATH="~/.local/bin:$PATH"

Configure Decision Engine

The default configuration file lives in /etc/decisionengine/decision_engine.jsonnet.

A number of defaults are set for you.

Selecting your datasource

You need a datasource to store in the database the channel’s data (datablocks). Each datasource has its own unique schema and cannot be used with a different datasource.

The SQLAlchemy Data Source

SQLAlchemy is the default Data Source and is setup with a configuration like:

"datasource": {
  "module": "decisionengine.framework.dataspace.datasources.sqlalchemy_ds",
  "name": "SQLAlchemyDS",
  "config": {
    "url": "postgresql://{db_user}:{db_password}@{db_host}:{db_port}/{db_dbname}",
    }
  }

Any extra keywords you can pass to the sqlalchemy.engine.Engine constructor may be set under config.

SQLAlchemy will create any tablespace objects it requires automatically.

The PostgreSQL data source, used until v1.7, is no more supported.

Start decision engine

Start the service

# For the RPM install, as root:
systemctl start decisionengine
# For the PIP install, as decisionengine user (Python packages are installed in ~decisionengine/.local/bin/):
export PATH="~/.local/bin:$PATH"
decisionengine --no-webserver &

Stop decision engine

To stop the service and remove the Redis container once you are done run the following:

# If you are in a RPM installation, as root:
systemctl stop decisionengine
# If you installed via PIP, as decisionengine:
export PATH="~/.local/bin:$PATH"
de-client --stop
# Run the following as root (root started the container)
podman stop decisionengine-redis | xargs podman rm

Add channels to decision engine

Decision engine decision cycles happen in channels. You can add channels by adding configuration files in /etc/decisionengine/config.d/ and restarting the decision engine.

Here is a simple test channel configuration. This test channel is using some NOP classes currently defined in the unit tests and not distributed.

The following configuration has been added as an example to /etc/decisionengine/config.d/test_channel.jsonnet during the installation process:

{
  sources: {
    source1: {
      module: "decisionengine.framework.tests.SourceNOP",
      parameters: {},
      schedule: 1,
    }
  },
  transforms: {
    transform1: {
      module: "decisionengine.framework.tests.TransformNOP",
      parameters: {},
      schedule: 1
    }
  },
  logicengines: {
    le1: {
      module: "decisionengine.framework.logicengine.LogicEngine",
      parameters: {
        facts: {
          pass_all: "True"
        },
        rules: {
          r1: {
            expression: 'pass_all',
            actions: ['publisher1']
          }
        }
      }
    }
  },
  publishers: {
    publisher1: {
      module: "decisionengine.framework.tests.PublisherNOP",
      parameters: {}
    }
  }
}

Finally, start or restart decision engine to start the new channel:

# For the RPM install:
systemctl restart decisionengine
# For the PIP install, as decisionengine user
decisionengine --no-webserver &

Once the decisionengine is running, de-client --status should show the active test channel.

Setup pressure-based pilot submission

After the previous steps, Decision Engine, the GlideinWMS libraries and HTCondor are supposed to be installed and able to run. We assume that the IDTOKEN to talk to the Factory and the VO credentials to submit to the computing resources (e.g. SciToken) are already available.

Configure the pressure-based submission

Write the configuration for the Decision Engine glideinwms module.

To ease the process you can use the templates available in the config_template contrib repo.

Copy the files from the EL9 folder into /etc/decisionengine, and the files in EL9/config.d/ into /etc/decisionengine/config.d.
If you made changes to decision_engine.jsonnet please merge it with the version form the repository.
The important part from the is the glideinwms import decision_engine.jsonnet template is the line: glideinwms: import 'glideinwms.libsonnet',.
Those configuration files have placeholder fields @TEMPLATE...@ that needs to be replaced with the proper parameters according to your specific system setup. The README file has some suggestions.

Once those configuration files have been updated, we are ready to finalize the Decision Engine configuration.

Setup Redis

Start the message broker (Redis) as pod container:

podman run --name decisionengine-redis -p 127.0.0.1:6379:6379 -d redis:6 --loglevel warning

Create GWMS frontend configuration

For this step you need first to restart the Decision Engine and then to run a configuration script. To do so, run:

# as root (fix the ownership of the frontend library files)
chown -R decisionengine: /var/lib/gwms-frontend
# for RPM installation as root
systemctl stop decisionengine
systemctl start decisionengine
ksu decisionengine -e /usr/bin/python3 /usr/lib/python3.9/site-packages/decisionengine_modules/glideinwms/configure_gwms_frontend.py
# for PIP installation as decisionengine
de-client --stop
decisionengine --no-webserver &
python3 ~decisionengine/decisionengine_modules/src/decisionengine_modules/glideinwms/configure_gwms_frontend.py

This command will create the file /var/lib/gwms-frontend/vofrontend/de_frontend_config

Stop, reset, and restart everything

To allow a fresh start stop and reset everything:

Stop the decisionengine (service), redis, drop and recreate the database

# If you are in a RPM installation, as root:
systemctl stop decisionengine
# If you installed via PIP, as decisionengine:
de-client --stop
# Run the following as root (root started the container)
podman stop decisionengine-redis | xargs podman rm
# Drop and recreate the DB
dropdb -U postgres decisionengine
createdb -U postgres decisionengine

Restart redis and the decisionengine

podman run --name decisionengine-redis -p 127.0.0.1:6379:6379 -d redis:6 --loglevel warning
# For RPM installations as root:
systemctl start decisionengine
sleep 5
systemctl status decisionengine
# For PIP installations as decisionengine:
decisionengine --no-webserver &
sleep 5
de-client --status

Submit a test job

Finally you can submit a test job to trigger Glidein requests and test the system. All Decision Engine commands can be run as root or the decisionengine user.

Make sure channel and sources are STEADY:
```
de-client --status
```

Prepare a HTCondor submission file mytest.submit with the following content

#  Content of the test HTCondor submission file - mytest.submit
executable = /bin/hostname
universe = vanilla
+DESIRED_Sites = "@CHANGEME@"
log = test.log
output = test.out.$(Cluster).$(Process)
error = test.err.$(Cluster).$(Process)
queue 1

Submit the test job
```
condor_submit mytest.submit
```
Check jobs in the queue
```
condor_q
```
Check for available glideins
```
condor_status
```

After the test jobs are submitted, DE and the Factory will take a few minutes (usually no more than 10 minutes) to get some Glideins and then get the job running. You can use also condor_status -any and condor_q on the Factory to monitor the DE requests and the Glidein submitted to the resources.

Stop Decision Engine service

Finally you can stop Decision Engine service and remove the Redis container (same commands as above):

# If you installed via RPMs run
systemctl stop decisionengine.service
# Run de-client --stop as decisionengine if you installed w/ PIP
podman stop decisionengine-redis | xargs podman rm

Troubleshooting

There is a known podman bug. podman is leaking volumes each time it starts a container, in the long run this is exhausting system resources. To check current volumes used by podman user can run podman volume list. To clean up volumes user can run podman volume prune -f after all podman container have been stopped and removed.