.. SPDX-FileCopyrightText: 2017 Fermi Research Alliance, LLC .. SPDX-License-Identifier: Apache-2.0 Release 2.0.6 ------------- This improves separation and compatibility with the GlideinWMS Frontend package. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Enhancements: - `DE 721 `_: Update Dockerfile and entrypoint to be able to build RPMs and wheels on Jenkins (@vitodb) - `DE 720 `_: Added option to prepare for OSG release to make_release (@mambelli) - `DE 719 `_: Update CI related Dockerfile and Jenkins pipeline (@vitodb) - `DE 718 `_: Release 2.0.4 (@mambelli) Full list of commits since version 2.0.4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `3b183afb2 `_: Added glidein group to coordinate with GWMS Frontend installation `765f66059 `_: Update EL9/python3-entrypoint.sh to use python venv from framework container. Add the option to call make-release script to build RPMs and wheels. Update Jenkinsfile_EL9 to fit these updates. `13d03d7df `_: Update EL9/framework/Dockerfile container to use a python venv to handle python modules `dc4c2cbbd `_: Added option to prepare for OSG release (make specfile, checksummed source tar ball and upload to OSG library) `c24b74ed5 `_: Merge pull request #719 from vitodb/feature/vito_update_ci_build `b4951eeea `_: In Jenkinsfile_EL9 pipeline replace rpmbuild stage with DEbuild and use make-release.sh script to build DE/DEM RPMs/wheels `ac9e23823 `_: In Dockerfile add dependencies and requirements to use mock `3f4fa209b `_: Fixed release notes and typo in make-release.sh `4add02381 `_: Remove use of build, not installed `f48a31839 `_: Install sphinx as user, not as root `ce1f28584 `_: Updated actions in gh-pages workflow Release 2.0.4 ------------- This release brings a new packaging with RPM packages for dependencies and system files, and a Python wheel for the Python code. There are also new Prometheus metrics and a few bug fixes like the source logging set incorrectly. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Bugs fixed: - `DE 670 `_: changes to fix source logging (@goodenou) - `DE 674 `_: Upgrading isort version to fix pre-config install error w/ poetry (@mambelli) - `DE 683 `_: Fix unit tests (@vitodb) - `DE 696 `_: Re-enable flake8 linter (@vitodb) - `DE 701 `_: Fixed bugs in metrics with labels (@IlyaBaburashvili) - `DE 700 `_: For flake8 skip build folder (@vitodb) - `DE 713 `_: Set /metrics Content-type header to text/plain (@shreyb) - `DE 710 `_: add invocation in the child processes for sources/channels logging (@namrathaurs) Enhancements: - `DE 668 `_: Introduce publisher status data product (@knoepfel) - `DE 671 `_: explicitly select a container with all our required python versions (@jcpunk) - `DE 676 `_: Added EL9 instructions (@mambelli) - `DE 678 `_: Updated and tested instructions, PIP installation working (@mambelli) - `DE 679 `_: Disabling unit tests for python 3.6 (@vitodb) - `DE 680 `_: Make default database init use stronger data protections (@jcpunk) - `DE 681 `_: Update to pytest 7 with pytest-postgresql 5 (@jcpunk) - `DE 682 `_: Updated EL9 instruction: PIP installation and GWMS config tested (@mambelli) - `DE 685 `_: Enable some tests only on DE 1.7 branch (@vitodb) - `DE 687 `_: Update GH actions (@vitodb) - `DE 688 `_: Added two metrics on the Source - de_source_status and de_source_acquire_seconds (@skylerfoster67) - `DE 690 `_: Adding DE EL9 containers based on AlmaLinux9 (@vitodb) - `DE 691 `_: Adding Jenkinsfile for EL9 (@vitodb) - `DE 689 `_: Added new de-client metrics for duration (@IlyaBaburashvili) - `DE 694 `_: Decision Engine Components Data (@skylerfoster67) - `DE 697 `_: Redis Exporter Data (@skylerfoster67) - `DE 703 `_: Create Redis mock for unit tests (@shreyb) - `DE 706 `_: In Jenkins pipeline config use podman instead of docker (@vitodb) - `DE 712 `_: Rpm pip packaging with uv and pyproject.toml (@mambelli) - `DE 714 `_: Fixed spec file and added release script (@mambelli) - `DE 715 `_: Added wrapper to run the decisionengine commands also as root. Fixed installation glitches. (@mambelli) - `DE 716 `_: Added a check for the Python code being installed and improved Python code install (@mambelli) - `DE 717 `_: Added codespell in pre-commit and fixed files to compliance (@mambelli) Full list of commits since version 2.0.2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `34ae84f15 `_: Added ability to build Python (wheel and sdist) packages to make-release.sh `8192524ba `_: Added codespell in pre-commit and fixed files to compliance `e045cb960 `_: Added a check for the Python code being installed. Improved the Python install script, adding the ability to clone the repo `7093d5b12 `_: Added wrapper to run the decisionengine commands also as root. Fixed installation glitches. `59cd65825 `_: Fixed spec file and added release script `8174106dd `_: Adding dependencies RPM package and packaging with uv and pyproject.toml `22951dc58 `_: Set /metrics Content-type header to text/plain `aea4fd7ad `_: logging fix for sources and channels `30fb83dc5 `_: In Jenkins pipeline config use podman instead of docker `0aca4d926 `_: Added .vscode/ to .gitignore `489d45abf `_: Dynamically check for redis server availability when a test is marked as redis `ff674229e `_: Remove unit test marker `6b1799f1b `_: Check if redis server is running. If not, skip integration test. Also marked applicable unit tests in this file `6879f72c5 `_: Added unit marker for pytest `22488d720 `_: For flake8 skip build folder `19a163a14 `_: Fixed bugs in metrics with labels (#701) `0a38b170b `_: Update test to be compliant with flake8 linter `22162226c `_: Add licence info to flake8 config file `1da6bd8df `_: Re-enable flak8 linter `65d5ccd9a `_: Dashboard on Redis Exporter information `daaac113c `_: Added json files containing the dashboards for Source and Channel Data in decisionengine/dashboards `d21728923 `_: Added license compliance for json files `ba29da2af `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `d79908778 `_: Configure buckets for de-client metrics `14826e839 `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `87457a1e0 `_: Fix de-client --metrics description `2cc376a60 `_: Added decorators for all rpc_ methods `7fe99a642 `_: Added directory for Grafana dashboards `a9d1590c3 `_: Adding Jenkinsfile for EL9 `018c8ab47 `_: Updated documentation for 1.7.5 release `ebfd6d4a0 `_: Numpy security update requires >= 1.22 (not available for Python 3.6) `aa474a9d8 `_: Fixed misspelling `9e9088418 `_: Adding DE EL9 containers based on AlmaLinux9 `438621a0a `_: Added two metrics on the Source - de_source_status and de_source_acquire_seconds `f54b92a59 `_: Update GH actions `f842996df `_: enable rpmbuild_el7, run_flake8 and pytest_el7 tests only on DE branch 1.7 `97915e0be `_: Note of possible future change `1c23be195 `_: update regexp in _expected_circularity `aa12938f9 `_: Updated instruction after testing PIP installation and pressure-based condifuraition on EL9 `d302e74bf `_: Update to pytest 7 with pytest-postgresql 5 `e9372585e `_: Make default database init use stronger data protections `aed2d7666 `_: Disabling unit tests for python 3.6 `87b3f6e92 `_: Updated and tested instructions, PIP installation working `84ebaa528 `_: Fixed setup for EL9 `0053e5963 `_: Added EL9 instructions `d23db48f2 `_: Upgrading isort version to fix install error w/ poetry `fd3c5bda7 `_: explicitly select a container with all our required python versions `85617b14d `_: fix source logging by defining logger in Sources correctly `669555cc7 `_: Add docstrings. `1681f211d `_: Add timing information for when publisher was disabled. `33e4f699e `_: Test publisher status w/in publisher. `4d3d9100b `_: Some renaming. Release 2.0.3 ------------- Skipping this tag Release 2.0.2 ------------- This is mainly a bug fix and documentation release. Instructions to run on EL8 have been added. Also a UP/DOWN status metric was added via Prometheus. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - `428 `_ : Decision engine 1.7.3 bug too many open file descriptors in glide_frontend_element.py - `427 `_ : Set CONTINUE_IF_NO_PROXY to False to allow hybrid configuration Full list of commits since version 2.0.1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `7ec132e9 `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `b942241a `_: Add installation instructions for CentOS 8 `4f6fc134 `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `e8d1922e `_: Fix docstrings errors and warnings `fc6aefd5 `_: Docker container and test setup for EL8 `51d5293f `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `0c15d3bd `_: Added UP/DOWN status metric of the decision engine `fc76a1f0 `_: Fixup coverage for new version `04b18750 `_: Set upper limit version for flake8. This is needed to have pytest-flake8 and flake8 versions working together. `98797411 `_: Add 'Setup pressure-based pilot submission' section to install document `0165183c `_: make RPM requires more flexible `28e2a0d4 `_: Updated release notes for 2.0.1 and porting of 1.7.3 Release 2.0.1 ------------- Patch level (bug fix) release. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Bugs fixed: - `DE 639 `_: de-client --status stalls whenever channels are not yet in STEADY state - `DE 638 `_: Sources should go offline if the client channel offline - `DE 634 `_: de-client --stop-channel / --start-channel doesn't work in 2.0rc2 - `DE 626 `_: New DE 2.0rc2 regularly takes 2-3 minutes to shut down - `DE 599 `_: Clarify timeout variable in block_while() - `DE 522 `_: Decision engine log files get split between several different processes with several different versions open - `DE 236 `_: New race condition in de-client Enhancements: - `DE 650 `_: Added separate log files for Sources Full list of commits since version 2.0.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `b5e56ab8 `_: Remove signal handler. `0fb6814b `_: Prevent blocking (if possible) during service actions. `bb68fc31 `_: Add logging handler to client-message receiver. `53fefbc5 `_: Update kombu version. `009cdd95 `_: Use kombu queues for server/client communication. `29a1ee25 `_: add distinct logging for sources `e44e9210 `_: Update GitHub actions; pylint workaround. `d192f8fb `_: Lock typing_extensions for Python 3.6 compat `2b946043 `_: Fix pre-commit node version to 17.9.0, the last to support SL7. `76f3ddfb `_: lock pyupgrade to python3.6 support `c9c7cb3e `_: Include psutil as part of runtime requirements. `df8a3941 `_: Make sure to kill worker process. `69924d0c `_: Do not block de-client calls during startup. `ddb18d7c `_: Minor cleanups. `f4dc7da7 `_: Do not take source offline more than once during detach. `cbffa992 `_: Update Docker entrypoint script for DE 2.0 branch `e10fe5af `_: Fixed cross-package link in the documentation `9da1eac8 `_: Added cross-package link in the documentation `d278726b `_: Updated 2.0 release notes and indexes, ready for 2.0.0 Release 2.0.0 ------------- This release series follows 1.7. A lot started to happen in 1.7.0 and has happened since, so we felt it was proper to change the major version number. We are proud to introduce Decision Engine 2.0.0 to outside users: it provides a friendlier installation procedure and configuration samples to test it on all resources supported by the GlideinWMS Factory, like OSG, some HPC resources and commercial cloud providers. - New architecture with redesigned source system using Kombu message passing with a Redis backend. - Token support via DE modules: support for SciToken, WlcgToken (for CE authentication) and HTCondor Idtokens (for Glideins and Factory communication) - Separation from the GlideinWMS Frontend. Decision Engine still shares some libraries with GlideinWMS but you don't need any more to install and configure the Frontend. - Structured logging. Improved python logging and adoption of structured logs format that will increase the semantinc content of the messages and ease the export of information for dashboards and Elastic Search. - Monitoring via Prometheus. - SQLAlchemy object-relational mapper to increase the testability of DB interactions and to allow different database backends. - Packaging via setuptools for both decisionengine and decisionengine_modules: Dependencies are not yet fully listed in the RPMs. - Added support of CentOS8 (RHEL7 is still out main platform) - Configuration example using HTC resources via GlideinWMS Factory - Decision Engine is distributed under the Apache 2.0 license - We increased our CI tests including also code auto-formatting and license compliance. We introduced integration tests and we are proud of our over 95% unit test coverage. .. note:: SQLAlchemy is required and is now the only datasource backend supported. Upgrading from a different datasource backend (1.6 or earlier were using direct PostgreSQL, 1.7 was supporting both) is a one-way change with a migration tool. We suggest dropping all objects if you wish to reuse the tablespace. You can preserve a copy of the old database to query historical information. .. note:: Added requirement on the Kombu library and a Redis server. We suggest to `install Redis using a container `_. .. note:: Added requirement on prometheus-client. Prometheus is be used as optional monitoring component. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - `528 `_: Update license and add copyright notices - `207 `_: Under certain circumstances the fetch of the "consumes" information fails but the channel does not go offline operations - `547 `_: Update DE client libs to pgsql-12 - `459 `_: Setuptools issues in decisionengine rpm - `546 `_: Request CentOS8 Stream support for Decision Engine - `453 `_: Struct Logging Self test errors with pytest-xdist - `418 `_: Add auto-formatting of the code - `134 `_: Yum update on decisionengine rpm doesn't restart the service - `480 `_: Request: Make postgresql migration script to migrate from old postgresql schema to new sqlalchemy schema Full list of commits since version 1.7.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `685a3a8e `_: Added changelog file for developers curated list of changes `044f4463 `_: Updated 1.7 and 2.0 release notes, ready for 2.0.0 RC4 `19994fb5 `_: Convert timeout program options to floats. `e2055f92 `_: Address Marco's review comments. `abdf35ad `_: Restore multiple queues but purge source queue after each publish. `52936cb5 `_: Improve error-handling. `aad20744 `_: Change to multiprocessing.Lock for protecting channel/source workers. `24bbee41 `_: Adjust launching of source workers in attempt to avoid deadlock. `6d13a392 `_: Remove unnecessary (and perhaps harmful) external updating of channel states. `5456f32f `_: Improve test coverage. `1afabb70 `_: Use service_actions to disable sources whenever client channels fail. `7f67a172 `_: Various naming and logging adjustments `e6e49184 `_: Adjust de-client --status and add --product-dependencies program option. `a7c1f351 `_: Apply block-while timeout to all channels, not each channel. `3d739ec7 `_: Update ci workflow to include workflow_dispatch mechanism and to customize artifact file name `c5a05650 `_: Archive unit test logs in case of unit test failure and make them available as artifacts `e94c2abb `_: Update Python 3.6-compatible pre-commit hooks. `aeb6b974 `_: Update Countdown docstrings. `525eb3a8 `_: Add Countdown class to address global timeout problem. `4c458e0c `_: Updated release notes for 2.0.0 RC3 (1.7.99.post3) `137b574a `_: Add a minimal container image more suited to production usage `9d7f6875 `_: Provide de-client --queue-status program option. `a7dcc30d `_: Ensure that channels and sources shared the same queues. `49a316e0 `_: Restore pyupgrade to v2.30, which works on Python 3.6 `2ce5ccb6 `_: [pre-commit.ci] pre-commit autoupdate `7bd41851 `_: Print number of pickled bytes of source-produced data. `97aed846 `_: Protect tests from Redis DB/routing-key collisions. `4d3abab7 `_: Flush the Redis DB once the DE server stops. `e36c2150 `_: Remove unnecessary @pytest.mark.usefixtures(...) decorations. `30d68610 `_: More unit testing `7850995d `_: We should have one path where we test without -v. `a81a52cc `_: A simple test to ensure the metrics can run `7547720c `_: Logger tests are a bit unstable at high parallelization. `56516df7 `_: Add missing test to ensure we can change the channel level twice `abde7d0f `_: Add missing tests for inherited functions `6522ed37 `_: Note lines we are not testing `de7829a4 `_: Remove the unit test log directory if it got created `28fbd599 `_: pin jsonnet 0.17.0 `9c5c827e `_: Metrics seems to want the channels setup to complete `b8829997 `_: Pin pytest version `b348d6f7 `_: Fix deadlock starting cherrypy metrics server `7697e6c1 `_: Log invocation of random port `9e7e4813 `_: Clarify note on xdist, run more workers `0b495fbf `_: Leave note to remember to cleanup temp files `ca5ddf6f `_: Ensure we are calling the cherrypy shutdown methods `e60efe78 `_: Move metrics fixtures to the fixtures file `9c717cc5 `_: Log finished with DB init `55965f9e `_: Prep the server fixture to permit the metrics webserver `732ff99b `_: Add a 'ping' method `6117cc95 `_: [pre-commit.ci] pre-commit autoupdate `b5af73ca `_: More logging about cherrypy state `dfe4278f `_: Added unlinked release notes for DE 2.0.0 `7d6484ad `_: Test source shared between two channels. `ae29d9d1 `_: Test same source types, separate channels `6095d33f `_: Test LatestMessages utility. `dfbf3e06 `_: Separate sources from channels. `2c10391e `_: Remove source proxy `afcc7cff `_: Add some more logging to try and trace startup state `dbd49a66 `_: Explicitly pass .coveragerc to pytest. `e6b03216 `_: Set max retry timeout for sqlite in unit tests. `51bed3d6 `_: Updated documentation for 1.7.1 release `3829151f `_: Allow duplicate keys if their values are the same. `1ea288e0 `_: [pre-commit.ci] pre-commit autoupdate `6b6611e5 `_: Use pre-commit.ci rather than local actions `dedbe4bd `_: Use local time for structlog timestamp `461c506e `_: Make sure de_std.libsonnet is provided when packaged. `f93b5963 `_: Update pre-commit hook versions and accommodate python-debian issue. `bba51609 `_: Reduce number of fixtures. `a4510cb1 `_: Segment the update for setuptools so it gets cached correctly `40098f35 `_: Merge pull request #584 from jcpunk/user-pip `4e1b79a1 `_: Merge pull request #583 from jcpunk/drop-dbutils `72c8db4a `_: Recommend using the site user pip dir instead. `ff604495 `_: Drop unneeded module `b203e2c4 `_: remove extraneous 'import gc' `ee2278e7 `_: replace needed import `4b7dedf2 `_: add licensing info `e5a56816 `_: add licensing info `a114abba `_: add licensing info `c2d511cd `_: adding queue logging to de_logger `77dd8d5a `_: Also run checks on backports to 1.7 `7c029578 `_: Updated developers instructions w/ license maintenance via REUSE information `e66b985d `_: Fix faulty tests. `d1a86c57 `_: Set Apache 2.0 license and added REUSE compliance `e488030e `_: Ensure that redis is running. `6c982c11 `_: Report PID for source process. `3f844ca4 `_: Further flesh out the documentation `1d750001 `_: Simplifications and rearrangements. `b85dca45 `_: Set state to error for exceptions caught before the thread start. `c4727acb `_: Changed summaries to histograms in DecisionEngine and TaskManager modules `6fa0bf4d `_: Added install document and updated the index and development instructions accordingly `e4de391e `_: Do the build of the wheel as not-root per our requirements `24ba5272 `_: Add a redis server to the CI testing containers `c939a6ed `_: Address Pat's comments. `1925a7b0 `_: First implementation using Kombu/redis to communicate data from sources to cycles. `82faa271 `_: Don't try to package obsolete sql file `9cbffe94 `_: Drop redundant tests. `ab0de9a5 `_: Drop obsolete raw postgresql interface `164b36d3 `_: Removed unnecessary comment `91f7a76f `_: Fixed rebase errors `e475fbd5 `_: Added import statement to fix MultiProcessCollector `a409f126 `_: Add no-webserver setting to all DE Test Workers `39cca32e `_: Moved multiprocess import to metrics to clean up imports. `73762e90 `_: Added --no-webserver to invocations of DEServer `303ee4be `_: Added __all__ global to control what is exported. `5170224b `_: Allow for metrics disabling from systemd unit file `2cacef4f `_: Added check for proper metrics environment and associated unit tests `a637a088 `_: Make webserver operation configurable `b3d6445a `_: Changed set_to_function calls to set() calls for metrics `5dccc7fa `_: Changed metric names to match prometheus convention `7371c2e8 `_: Added cherrypy requirement `2c511cea `_: Added metrics to record time to run Modules and DecisionEngine rpc calls `c24d33bc `_: Renamed prometheus.py to metrics.py `2335134d `_: Moved TaskManager metrics to util/prometheus.py to avoid duplicates `3c1b790c `_: Added metrics endpoint to RPC server, changed prometheus to multiprocess mode, and added CherryPy webserver for prometheus metrics `d8972de0 `_: Added unit tests for metrics API `7b0f641b `_: Add instructions for running the Redis container. `8d0c4919 `_: Block pytest-postgresql 4 `d988f1a0 `_: Lower timeout for actions. `4f920dcc `_: Simplifications in preparation for Kombu. `eb9f4292 `_: Make TaskManager not executable `00c8f6e6 `_: Remove unused files. `dd990d2c `_: Adding de-logparser, a tool to help parsing Decision Engine semi-structured logs `1da0d61e `_: Added a comment to help developers with incomplete installation `1cbc7334 `_: Drop testing/support for PyPy `3ba3e8e6 `_: Ignoring E203, whitespace after ':', since black is adding the whitespace `814669d5 `_: Disable PyPy test that fails for PG_DE_DB_WITH_SCHEMA fixture value. `8d68c287 `_: Fix debug message `33db6425 `_: Test composite workflows using source proxies and configuration combination. `30951a5b `_: EL7 doesn't ship with a new enough golang for jsonnetfmt `e72eb3fd `_: Forbid inheritance from SourceProxy. `e95071fd `_: Automatically format jsonnet files with jsonnetfmt `355ccd45 `_: Correct tests for python 3.10 `64119161 `_: Start testing python 3.10 `c1cb8258 `_: add dummy source and test `31b0f30b `_: Check for duplicate keys after source proxies have been removed. `140a4c47 `_: Fix configuration-combination function signature. `d4a05299 `_: Remove now unnecessary blocking. `1e78a889 `_: Don't run setup.py as root `a71d5b0a `_: Add error for running server as root `cd345701 `_: Increase coverage in LogicEngine `6c132924 `_: Fix out of sync devel requirements `8bfab003 `_: Start running tests with xdist `aebe7d49 `_: Remove unnecessary conversion to Pandas dataframe. `28919b16 `_: Allow channels to boot in parallel. `c4fc5997 `_: Improve parameter and variable names. `e021419b `_: Encourage use of automatic nag hook `cc4e469a `_: Update hooks to latest via `pre-commit autoupgrade` `cef30b69 `_: Further simplify some cases `590bea3f `_: Add channel-combination facilities. `a4a7938c `_: Various simplifications recommended by flake8-simple `d5157416 `_: Possible simplification to logging. `81e3d1ee `_: Fix pylint error on `create_runner` and `ProcessingState` `2a328c25 `_: Added missing init file to make managers a package `d7f44015 `_: Rework tests for #454 `9521d3ce `_: Add debug statement when default logic-engine configuration is used. `c6dc778c `_: Unconditionally execute publishers with default configured logic engine. `a48dd7d8 `_: Remove now-unnecessary Python-to-Jsonnet conversion. `a6a81ce7 `_: Run autoformatters `49dac1ec `_: Setup pre-commit hooks for autoformatters `3800cc2a `_: Run the code style/standards checks early. `1d42eb0d `_: TaskManager now inherits from ComponentManager. Also added SourceManager, ChannelManager, and SourceSubscriptionManager files for future integration. `85a16f3b `_: Python optimised byte code removes assert under some conditions `bed2f5d9 `_: Support latest setuptools_scm release