.. SPDX-FileCopyrightText: 2017 Fermi Research Alliance, LLC .. SPDX-License-Identifier: Apache-2.0 Release 2.0.6 ------------- Improved separation and compatibility with GlideinWMS Frontend package. Improved GCE billing and set hep-bill-calculator to 0.2.3. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -`DEM 513 `_: Updated documentation for release 2.0.4 -`DEM 516 `_: Refactor Existing Test Script and Add New Unit Tests for GCE Billing Full list of commits since version 2.0.4 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `46bfc52 `_: Refactor Existing Test Script and Add New Unit Tests for GCE Billing (#516) `b7accf0 `_: Updated setuptools in gh-pages workflow `993e380 `_: Update gh-pages workflow and install sphinx as user, not as root Release 2.0.4 ------------- This release brings a new packaging with RPM packages for dependencies and system files, and a Python wheel for the Python code. There are also new Prometheus metrics and a few bug fixes like the superfacility API and AWS and GCE billing. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -`DEM 444 `_: Add CodeQL workflow for GitHub code scanning (@lgtm-com) -`DEM 445 `_: Added CPU Metrics. (@Redjacket45) -`DEM 456 `_: Upgrading isort version to fix pre-config install error w/ poetry (@mambelli) -`DEM 461 `_: Eliminate escaping of double quotation marks to conform with NEWT API's changed behavior (@namrathaurs) -`DEM 459 `_: Disabling Python 3.6 and Fixing 3.9 Unit Tests in CI Workflow (@namrathaurs) -`DEM 462 `_: Feature/vito pandas2 updates (@vitodb) -`DEM 463 `_: Fixed Google dependency to work fine in Python 3.9 (@mambelli) -`DEM 457 `_: Making CONTINUE_IF_NO_PROXY attribute configurable (@namrathaurs) -`DEM 465 `_: Update tests (@vitodb) -`DEM 472 `_: Adding DE modules EL9 CI container based on AlmaLinux9 (@vitodb) -`DEM 468 `_: New Nersc Source Module using SuperFacilityApi (@hyunwoo18) -`DEM 474 `_: fix test_refresh_entry_token unit test (@vitodb) -`DEM 473 `_: Adding Jenkinsfile for EL9 (@vitodb) -`DEM 477 `_: Adding one line (@hyunwoo18) -`DEM 481 `_: Refactor DEConfigSource and configure_gwms_frontend (@BrunoCoimbra) -`DEM 482 `_: Components of decisionengine dashboards (@skylerfoster67) -`DEM 484 `_: Change glideclient advertisement logic (@BrunoCoimbra) -`DEM 485 `_: Fix a typo in glideids_to_advertise (@BrunoCoimbra) -`DEM 487 `_: Fix configure_gwms_frontend tests (@BrunoCoimbra) -`DEM 486 `_: Re-enable flak8 linter (@vitodb) -`DEM 492 `_: Do not test external dependencies with flake8 (@vitodb) -`DEM 489 `_: Added new metrics to job_q and source (@IlyaBaburashvili) -`DEM 493 `_: Add dem_htcondor_cores_count and dem_htcondor_cores_histogram metrics to source.py (@IlyaBaburashvili) -`DEM 498 `_: Added unit tests for dem_htcondor_slots_status_count, dem_htcondor_cores_count, dem_htcondor_memory_count. Fixed issues in source.py. (@IlyaBaburashvili) -`DEM 496 `_: New Figure of Merit Metric (@skylerfoster67) -`DEM 499 `_: Two dashboards for cores, slots, and runtimes (@IlyaBaburashvili) -`DEM 503 `_: In Jenkins pipeline config use podman instead of docker (@vitodb) -`DEM 502 `_: add new method of determining token expiration instead of relying on jwt.decode error code (@StevenCTimm) -`DEM 507 `_: Refactor time_left in security module (@BrunoCoimbra) -`DEM 505 `_: Migrating GCE Billing to BigQuery based Bill Calculations (@namrathaurs) -`DEM 506 `_: DEConfigSource converts nested dicts to OrderedDicts (@BrunoCoimbra) -`DEM 510 `_: Add retry functionality to NerscSFApi (@vitodb) -`DEM 511 `_: Packaging with uv and pyproject.toml (@mambelli) -`DEM 512 `_: Added codespell in pre-commit and fixed files to compliance (@mambelli) Full list of commits since version 2.0.2 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `c481f85 `_: Added codespell in pre-commit and fixed files to compliance `3bf56ab `_: Packaging with uv and pyproject.toml `5577e9e `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `230627f `_: Add retry functionality to NerscSFApi `b820998 `_: DEConfigSource converts nested dicts to OrderedDicts `8e78900 `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `a85edeb `_: migrating to using bigquery for GCE billing `c4d0bc1 `_: Refactor time_left in security module `a42d3bd `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `5305f59 `_: remove the extraneous print statements `b57507b `_: changing the way that the NERSC SFApi access token expiration is calculated to not rely on jwt.decode() exit value `c500a73 `_: In Jenkins pipeline config use podman instead of docker `aa975b8 `_: Added dashboards for cores, slots, and runtimes `af1003b `_: New Figure of Merit Metric (#496) `fb4c3f9 `_: Remove extra line from dep5 `63adb86 `_: Add unit tests for metrics, fix issue in source.py `4f590a6 `_: Fixed commit history `21ac8c4 `_: Remove include Histogram `aae8638 `_: Remove debugging statements `b673c4a `_: Fix PR 489 `7601451 `_: For flake8 skip build folder `9980e3a `_: Make the test flake8 compliant `85e6487 `_: Do not test external dependencies `8a07b0e `_: Add licence info to flake8 config file `97f6f63 `_: Re-enable flak8 linter `a28c97d `_: Update test_configure_gwms_frontend.py `01c9c40 `_: Fix configure_gwms_frontend tests `5eb2f2b `_: Fix a typo in glideids_to_advertise `cb19a5b `_: Fix glideclient advertisement logic `9c1f8e5 `_: Added json files on Channel and Source Data dashboards with specific naming `ea00e39 `_: Adding license information for dashboard files because JSON does not allow comments `c11a7e2 `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `92b72b6 `_: Refactor config source for configure_gwms_frontend `02ab367 `_: Adding one line `32d6114 `_: Added directory for Grafana dashboards `5a5fb8b `_: Adding Jenkinsfile for EL9 `aa59c49 `_: required jwt functionalities are provided by PyJWT `2df1fc5 `_: Updated documentation for 1.7.5 release `cf81d56 `_: Last update to setup.py `8bc6d24 `_: Last update to add KeyError `85604ef `_: Removed remaining hardcoded strings from the code `8117652 `_: Implementing Marco requests in the review `1d59d49 `_: Fixed pre-commit issues and jwt package `b107c06 `_: New Nersc Source Module using SuperFacilityApi `e917838 `_: Adding DE modules EL9 CI container based on AlmaLinux9 `a5a2bae `_: Update GH actions `bdb70b0 `_: Enable some GH actions only on DE 1.7 branch `6a22f44 `_: Update runtime and devel python module versions in setup.py `842fdc9 `_: CONTINUE_IF_NO_PROXY True by default `8906538 `_: added changes to make CONTINUE_IF_NO_PROXY configurable `5a7bc0d `_: Fixed Google dependency to work fine in Python 3.9. Install via PIP succeeds `d315736 `_: With pandas 2.x DataFrame.append() has been deprecated, use pandas.concat() instead `0484401 `_: Address a pandas deprecation warning: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(set.iloc[0]) instead. The warning is due to glidein_cpus that is a single element series. `07292a4 `_: updated as per second round of review comments `947e9b8 `_: updated as per PR review comments `e3608a0 `_: fixes for python 3.6 and 3.9 unit tests failing in the CI workflow `d8de2da `_: updated as per PR#461 comment `e6044f7 `_: removed escaping of quotes to conform with new API behavior `4a59acc `_: Upgrading isort version to fix install error w/ poetry `7b96d46 `_: Started to add startd_manifests (glidein) metrics (TotalCpus, TotalSlotCpus, TotalSlots) `2ab707f `_: Started moving metrics into global dict `58e62a4 `_: Added Number_of_jobs metric. `5a8a641 `_: Added license info Release 2.0.3 ------------- Release tag skipped Release 2.0.2 ------------- This is mainly a bug fix release. Some features were added: - Monotoring of the glideinwms module via Prometheus - Initial version of a Rigetti source - More flexible LogicEngine: Fsctory Entries categories are now configurable Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - `428 `_ : Decision engine 1.7.3 bug too many open file descriptors in glide_frontend_element.py - `427 `_ : Set CONTINUE_IF_NO_PROXY to False to allow hybrid configuration Full list of commits since version 2.0.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `16e1751 `_: update GWMS transforms This update improves handling of factory-entries data product. `0296667 `_: Add initial Rigetti source `9d199a5 `_: Update refresh_entry_token to use context managers `d45b799 `_: [pre-commit.ci] auto fixes from pre-commit.com hooks `7b37e83 `_: Renamed Variables (REQ_IDLE_GLIDEINS & REQ_MAX_GLIDEINS) and stored them into a global dictionary. `d7d0849 `_: Started moving metrics into global dict `b1e56f0 `_: added req_idle_glidens & req_max_glidens metrics `cee9652 `_: Started to add NUMBER_OF_JOBS, NAME_OF_GROUPS, STATUS_OF_JOB METRICS `893137c `_: Added Number_of_jobs metric. `e530c10 `_: Docker container and test setup for EL8 `ea02fc8 `_: Test case with no glideins. `4ae0bab `_: One monolithic factory-entries data product. `cc6b01a `_: Merge pull request #436 from vitodb/vito_fix_de_client_call `a6744f1 `_: Fix de_client call This is required to allow de_client to get its output returned as text string rather then using a logger. `f2057d1 `_: Merge pull request #433 from jcpunk/coverage `4f6dffd `_: Merge branch 'master' into coverage `0922c6d `_: No longer need to pin coverage version `57a5599 `_: Set upper limit version for flake8. This is needed to have pytest-flake8 and flake8 versions working together. `6bf9b48 `_: Set default glidein params on GlideFrontendElement `3ce646a `_: Set CONTINUE_IF_NO_PROXY to False `2290fb4 `_: Updated release notes for 2.0.1 and porting of 1.7.3 Release 2.0.1 ------------- Patch level (bug fix) release. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Full list of commits since version 2.0.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `6f2db85 `_: Fix coverage reporting. `a764562 `_: Silence warnings that are beyond our control or are irrelevant. `cc481e9 `_: Test that duplicate column names and empty dataframes are properly handled. `76e1085 `_: Fix NERSC FOM test. `f67df9d `_: Cleanup tests. `51c6e75 `_: Work around pylint bug. `3eb1e92 `_: Ensure setuptools is up-to-date in GitHub actions; also update actions. `bfe45e5 `_: in dataframe_for_entrytype take care of empty datablocks `5223daf `_: Enable token proxy hybrid GWMS configuration `265dc8e `_: Fix pre-commit node version to 17.9.0, the last to support SL7. `7f5922f `_: Single point of maintenance for entry types. `0125962 `_: Support additional queries according to the entrytype/logic-engine fact. `f685f24 `_: Use Pandas concepts. `331a284 `_: Testing cleanups. `05b605e `_: Remove unnecessary code. `0b9e6c7 `_: Fix pyupgrade version for python 3.6 compatibility `9f8dddb `_: Further cleanups for testing glidein frontend element. `0292842 `_: Remove unnecessary testing classes. `fcd1a40 `_: Added cross-package link in the documentation `a35c2e8 `_: Updated 2.0 release notes and indexes, ready for 2.0.0 Release 2.0.0 ------------- This release series follows 1.7. A lot started to happen in 1.7.0 and has happened since, so we felt it was proper to change the major version number. We are proud to introduce Decision Engine 2.0.0 to outside users: it provides a friendlier installation procedure and configuration samples to test it on all resources supported by the GlideinWMS Factory, like OSG, some HPC resources and commercial cloud providers. The decisionengine_modules is the standard library of modules distributed to support the main sources, transformations and publishers of the Decision Engine. See also the Release Notes of decisionengine, the Decision Engine framework. This release features: - Adopt new Framework architecture for the sources - Separation from the GlideinWMS Frontend. Decision Engine still shares some libraries with GlideinWMS but you don't need any more to install and configure the Frontend. - Structured logging. Improved python logging and adoption of structured logs format that will increase the semantinc content of the messages and ease the export of information for dashboards and Elastic Search. - Monitoring via Prometheus. - Added support of CentOS8 (RHEL7 is still out main platform) - Configuration example using HTC resources via GlideinWMS Factory - Decision Engine is distributed under the Apache 2.0 license - We increased our CI tests including also code auto-formatting and license compliance. We introduced integration tests and we are proud of our over 95% unit test coverage. .. note:: decisionengine_modules now requires GlideinWMS 3.9.4. The vofrontend installation and configuration is no more needed, decisionengine_modules now depends from the GlideinWMS vofrontend-lib. Check the new instructions for the changed configuration of the glideinwms module. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - `353 `_ : yum update decisionengine-standard-library doesn't detect and pull in decisionengine_modules - `282 `_ : Decision Engine classad encoding problems make multiple exceptions in the factory - `349 `_ : Minor issue setup tools decisionengine_modules - `363 `_ : Replace AWSOccupancy and AWSSpotPrice classes with SourceProxy + transforms - `253 `_ : Different modules are using different retry parameters in config - `348 `_ : Name convention on bill_calculator_hep vs. billing_calculator_hep - `361 `_ : glide_frontend_element.py throws logger exception in 1.7.0rc3 Full list of commits since version 1.7.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `d605ab0 `_: Added changelog file for developers curated list of changes `cc0b18a `_: Adding 1.7.2 release information `eb40d1e `_: Updated release notes, ready for 2.0.0 RC4 `53b2fe4 `_: The rpm-build workflow has been updated to include the workflow_dispatch action me chanism. This allows to trigger this action through a GitHub API or directly through the action dashboard, This PR also customize the RPM artifact file name. `cd52b39 `_: Catch exceptions when no VMs running in GCE `9044b64 `_: Add token authentication to the glideinwms module `97ca32a `_: Modified pre-commit bot suggestions. Avoid pyupgrade that requires python 3.7 `95a7d7b `_: Test the arguments used to call publish_to_htcondor. `27f0f4b `_: Adjust code-coverage configuration to display code in browser. `3addf20 `_: Make sure variables are defined before they're used. `75381df `_: Improve fe_group_classads test. `ed2a8b4 `_: Check DecisionEngineMonitorManifests for empty DFs `e7a247b `_: Fix GlideClientGlobal create_invalidate_constraint `f0819b1 `_: pin pytest version `ad59fed `_: [pre-commit.ci] pre-commit autoupdate `0fc20e5 `_: Added unlinked release notes for DEM 2.0.0 `123dd2b `_: Drop extra lines `0177318 `_: Rename product so that translation is not necessary. `94e334c `_: Explicitly pass .coveragerc to pytest. `04456a9 `_: Updated documentation for 1.7.1 release `240d937 `_: [pre-commit.ci] pre-commit autoupdate `f054532 `_: Use pre-commit.ci rather than local actions `e2bc8d8 `_: Update GlideinWMS dependencies `4dfefb1 `_: Add SPDX blurbs. `a0889aa `_: Update pre-commit hook versions and accommodate python-debian issue. `7fd1c9c `_: Change gwmw module to read cfg from DE framework `d57ac6e `_: Set Apache 2.0 license and added REUSE compliance `cf43119 `_: Set fail-fast to false to allow py3.6 tests to complete also when higher version fail. Temporarily downgraded from py3.10 to py3.9 because the condor dependency is not available yet for py3.10. `e2707e5 `_: Ignoring E203, whitespace after ':', since black is adding the whitespace `b0e3526 `_: Remove modules that inherit from SourceProxy. Inheritance from SourceProxy has been forbidden by PR#526. In this commit it has been removed the SourceProxy module GCEResourceLimits that is not used. `208a660 `_: Revert "Add jsonnetfmt hook" `efbf6b9 `_: Add jsonnetfmt hook `9140330 `_: Recasting AWSOccupancy, AWSSpotPrice and associated unit tests as Transforms AWSOccupancy and AWSSpotPrice source proxies will be handled at config level `81adb29 `_: Start testing against python 3.10 `a9c1f3c `_: Support retries for contacting Graphite. `3bd68e4 `_: Fix some errors, most caught by lgtm. `89e21f0 `_: Simplify some code paths `63f6315 `_: Homogenize parameter-naming. `325ea37 `_: Update pre-commit hooks via `pre-commit autoupdate` `ee7d45c `_: More logging improvements. `901291c `_: Take advantage of simplified logging. `f4752ab `_: Fixup pre-commit hooks `ad66754 `_: Add Linters task `ca61ae3 `_: Update to latest setuptools_scm `3ad934b `_: Fixed project name `ec2a94e `_: Added 1.1 release notes