Release 2.0.4
This release brings a new packaging with RPM packages for dependencies and system files, and a Python wheel for the Python code. There are also new Prometheus metrics and a few bug fixes like the superfacility API and AWS and GCE billing.
Issues fixed in this release
-DEM 444: Add CodeQL workflow for GitHub code scanning (@lgtm-com) -DEM 445: Added CPU Metrics. (@Redjacket45) -DEM 456: Upgrading isort version to fix pre-config install error w/ poetry (@mambelli) -DEM 461: Eliminate escaping of double quotation marks to conform with NEWT API’s changed behavior (@namrathaurs) -DEM 459: Disabling Python 3.6 and Fixing 3.9 Unit Tests in CI Workflow (@namrathaurs) -DEM 462: Feature/vito pandas2 updates (@vitodb) -DEM 463: Fixed Google dependency to work fine in Python 3.9 (@mambelli) -DEM 457: Making CONTINUE_IF_NO_PROXY attribute configurable (@namrathaurs) -DEM 465: Update tests (@vitodb) -DEM 472: Adding DE modules EL9 CI container based on AlmaLinux9 (@vitodb) -DEM 468: New Nersc Source Module using SuperFacilityApi (@hyunwoo18) -DEM 474: fix test_refresh_entry_token unit test (@vitodb) -DEM 473: Adding Jenkinsfile for EL9 (@vitodb) -DEM 477: Adding one line (@hyunwoo18) -DEM 481: Refactor DEConfigSource and configure_gwms_frontend (@BrunoCoimbra) -DEM 482: Components of decisionengine dashboards (@skylerfoster67) -DEM 484: Change glideclient advertisement logic (@BrunoCoimbra) -DEM 485: Fix a typo in glideids_to_advertise (@BrunoCoimbra) -DEM 487: Fix configure_gwms_frontend tests (@BrunoCoimbra) -DEM 486: Re-enable flak8 linter (@vitodb) -DEM 492: Do not test external dependencies with flake8 (@vitodb) -DEM 489: Added new metrics to job_q and source (@IlyaBaburashvili) -DEM 493: Add dem_htcondor_cores_count and dem_htcondor_cores_histogram metrics to source.py (@IlyaBaburashvili) -DEM 498: Added unit tests for dem_htcondor_slots_status_count, dem_htcondor_cores_count, dem_htcondor_memory_count. Fixed issues in source.py. (@IlyaBaburashvili) -DEM 496: New Figure of Merit Metric (@skylerfoster67) -DEM 499: Two dashboards for cores, slots, and runtimes (@IlyaBaburashvili) -DEM 503: In Jenkins pipeline config use podman instead of docker (@vitodb) -DEM 502: add new method of determining token expiration instead of relying on jwt.decode error code (@StevenCTimm) -DEM 507: Refactor time_left in security module (@BrunoCoimbra) -DEM 505: Migrating GCE Billing to BigQuery based Bill Calculations (@namrathaurs) -DEM 506: DEConfigSource converts nested dicts to OrderedDicts (@BrunoCoimbra) -DEM 510: Add retry functionality to NerscSFApi (@vitodb) -DEM 511: Packaging with uv and pyproject.toml (@mambelli) -DEM 512: Added codespell in pre-commit and fixed files to compliance (@mambelli)
Full list of commits since version 2.0.2
c481f85: Added codespell in pre-commit and fixed files to compliance
3bf56ab: Packaging with uv and pyproject.toml
5577e9e: [pre-commit.ci] auto fixes from pre-commit.com hooks
230627f: Add retry functionality to NerscSFApi
b820998: DEConfigSource converts nested dicts to OrderedDicts
8e78900: [pre-commit.ci] auto fixes from pre-commit.com hooks
a85edeb: migrating to using bigquery for GCE billing
c4d0bc1: Refactor time_left in security module
a42d3bd: [pre-commit.ci] auto fixes from pre-commit.com hooks
5305f59: remove the extraneous print statements
b57507b: changing the way that the NERSC SFApi access token expiration is calculated to not rely on jwt.decode() exit value
c500a73: In Jenkins pipeline config use podman instead of docker
aa975b8: Added dashboards for cores, slots, and runtimes
af1003b: New Figure of Merit Metric (#496)
fb4c3f9: Remove extra line from dep5
63adb86: Add unit tests for metrics, fix issue in source.py
4f590a6: Fixed commit history
21ac8c4: Remove include Histogram
aae8638: Remove debugging statements
b673c4a: Fix PR 489
7601451: For flake8 skip build folder
9980e3a: Make the test flake8 compliant
85e6487: Do not test external dependencies
8a07b0e: Add licence info to flake8 config file
97f6f63: Re-enable flak8 linter
a28c97d: Update test_configure_gwms_frontend.py
01c9c40: Fix configure_gwms_frontend tests
5eb2f2b: Fix a typo in glideids_to_advertise
cb19a5b: Fix glideclient advertisement logic
9c1f8e5: Added json files on Channel and Source Data dashboards with specific naming
ea00e39: Adding license information for dashboard files because JSON does not allow comments
c11a7e2: [pre-commit.ci] auto fixes from pre-commit.com hooks
92b72b6: Refactor config source for configure_gwms_frontend
02ab367: Adding one line
32d6114: Added directory for Grafana dashboards
5a5fb8b: Adding Jenkinsfile for EL9
aa59c49: required jwt functionalities are provided by PyJWT
2df1fc5: Updated documentation for 1.7.5 release
cf81d56: Last update to setup.py
8bc6d24: Last update to add KeyError
85604ef: Removed remaining hardcoded strings from the code
8117652: Implementing Marco requests in the review
1d59d49: Fixed pre-commit issues and jwt package
b107c06: New Nersc Source Module using SuperFacilityApi
e917838: Adding DE modules EL9 CI container based on AlmaLinux9
a5a2bae: Update GH actions
bdb70b0: Enable some GH actions only on DE 1.7 branch
6a22f44: Update runtime and devel python module versions in setup.py
842fdc9: CONTINUE_IF_NO_PROXY True by default
8906538: added changes to make CONTINUE_IF_NO_PROXY configurable
5a7bc0d: Fixed Google dependency to work fine in Python 3.9. Install via PIP succeeds
d315736: With pandas 2.x DataFrame.append() has been deprecated, use pandas.concat() instead
0484401: Address a pandas deprecation warning: FutureWarning: Calling float on a single element Series is deprecated and will raise a TypeError in the future. Use float(set.iloc[0]) instead. The warning is due to glidein_cpus that is a single element series.
07292a4: updated as per second round of review comments
947e9b8: updated as per PR review comments
e3608a0: fixes for python 3.6 and 3.9 unit tests failing in the CI workflow
d8de2da: updated as per PR#461 comment
e6044f7: removed escaping of quotes to conform with new API behavior
4a59acc: Upgrading isort version to fix install error w/ poetry
7b96d46: Started to add startd_manifests (glidein) metrics (TotalCpus, TotalSlotCpus, TotalSlots)
2ab707f: Started moving metrics into global dict
58e62a4: Added Number_of_jobs metric.
5a8a641: Added license info
Release 2.0.3
Release tag skipped
Release 2.0.2
This is mainly a bug fix release.
Some features were added: - Monotoring of the glideinwms module via Prometheus - Initial version of a Rigetti source - More flexible LogicEngine: Fsctory Entries categories are now configurable
Issues fixed in this release
Full list of commits since version 2.0.0
16e1751: update GWMS transforms This update improves handling of factory-entries data product.
0296667: Add initial Rigetti source
9d199a5: Update refresh_entry_token to use context managers
d45b799: [pre-commit.ci] auto fixes from pre-commit.com hooks
7b37e83: Renamed Variables (REQ_IDLE_GLIDEINS & REQ_MAX_GLIDEINS) and stored them into a global dictionary.
d7d0849: Started moving metrics into global dict
b1e56f0: added req_idle_glidens & req_max_glidens metrics
cee9652: Started to add NUMBER_OF_JOBS, NAME_OF_GROUPS, STATUS_OF_JOB METRICS
893137c: Added Number_of_jobs metric.
e530c10: Docker container and test setup for EL8
ea02fc8: Test case with no glideins.
4ae0bab: One monolithic factory-entries data product.
cc6b01a: Merge pull request #436 from vitodb/vito_fix_de_client_call
a6744f1: Fix de_client call This is required to allow de_client to get its output returned as text string rather then using a logger.
f2057d1: Merge pull request #433 from jcpunk/coverage
4f6dffd: Merge branch ‘master’ into coverage
0922c6d: No longer need to pin coverage version
57a5599: Set upper limit version for flake8. This is needed to have pytest-flake8 and flake8 versions working together.
6bf9b48: Set default glidein params on GlideFrontendElement
3ce646a: Set CONTINUE_IF_NO_PROXY to False
2290fb4: Updated release notes for 2.0.1 and porting of 1.7.3
Release 2.0.1
Patch level (bug fix) release.
Issues fixed in this release
Full list of commits since version 2.0.0
6f2db85: Fix coverage reporting.
a764562: Silence warnings that are beyond our control or are irrelevant.
cc481e9: Test that duplicate column names and empty dataframes are properly handled.
76e1085: Fix NERSC FOM test.
f67df9d: Cleanup tests.
51c6e75: Work around pylint bug.
3eb1e92: Ensure setuptools is up-to-date in GitHub actions; also update actions.
bfe45e5: in dataframe_for_entrytype take care of empty datablocks
5223daf: Enable token proxy hybrid GWMS configuration
265dc8e: Fix pre-commit node version to 17.9.0, the last to support SL7.
7f5922f: Single point of maintenance for entry types.
0125962: Support additional queries according to the entrytype/logic-engine fact.
f685f24: Use Pandas concepts.
331a284: Testing cleanups.
05b605e: Remove unnecessary code.
0b9e6c7: Fix pyupgrade version for python 3.6 compatibility
9f8dddb: Further cleanups for testing glidein frontend element.
0292842: Remove unnecessary testing classes.
fcd1a40: Added cross-package link in the documentation
a35c2e8: Updated 2.0 release notes and indexes, ready for 2.0.0
Release 2.0.0
This release series follows 1.7. A lot started to happen in 1.7.0 and has happened since, so we felt it was proper to change the major version number. We are proud to introduce Decision Engine 2.0.0 to outside users: it provides a friendlier installation procedure and configuration samples to test it on all resources supported by the GlideinWMS Factory, like OSG, some HPC resources and commercial cloud providers. The decisionengine_modules is the standard library of modules distributed to support the main sources, transformations and publishers of the Decision Engine. See also the Release Notes of decisionengine, the Decision Engine framework.
This release features:
Adopt new Framework architecture for the sources
Separation from the GlideinWMS Frontend. Decision Engine still shares some libraries with GlideinWMS but you don’t need any more to install and configure the Frontend.
Structured logging. Improved python logging and adoption of structured logs format that will increase the semantinc content of the messages and ease the export of information for dashboards and Elastic Search.
Monitoring via Prometheus.
Added support of CentOS8 (RHEL7 is still out main platform)
Configuration example using HTC resources via GlideinWMS Factory
Decision Engine is distributed under the Apache 2.0 license
We increased our CI tests including also code auto-formatting and license compliance. We introduced integration tests and we are proud of our over 95% unit test coverage.
Note
decisionengine_modules now requires GlideinWMS 3.9.4. The vofrontend installation and configuration is no more needed, decisionengine_modules now depends from the GlideinWMS vofrontend-lib. Check the new instructions for the changed configuration of the glideinwms module.
Issues fixed in this release
353 : yum update decisionengine-standard-library doesn’t detect and pull in decisionengine_modules
282 : Decision Engine classad encoding problems make multiple exceptions in the factory
349 : Minor issue setup tools decisionengine_modules
363 : Replace AWSOccupancy and AWSSpotPrice classes with SourceProxy + transforms
253 : Different modules are using different retry parameters in config
348 : Name convention on bill_calculator_hep vs. billing_calculator_hep
361 : glide_frontend_element.py throws logger exception in 1.7.0rc3
Full list of commits since version 1.7.0
d605ab0: Added changelog file for developers curated list of changes
cc0b18a: Adding 1.7.2 release information
eb40d1e: Updated release notes, ready for 2.0.0 RC4
53b2fe4: The rpm-build workflow has been updated to include the workflow_dispatch action me chanism. This allows to trigger this action through a GitHub API or directly through the action dashboard, This PR also customize the RPM artifact file name.
cd52b39: Catch exceptions when no VMs running in GCE
9044b64: Add token authentication to the glideinwms module
97ca32a: Modified pre-commit bot suggestions. Avoid pyupgrade that requires python 3.7
95a7d7b: Test the arguments used to call publish_to_htcondor.
27f0f4b: Adjust code-coverage configuration to display code in browser.
3addf20: Make sure variables are defined before they’re used.
75381df: Improve fe_group_classads test.
ed2a8b4: Check DecisionEngineMonitorManifests for empty DFs
e7a247b: Fix GlideClientGlobal create_invalidate_constraint
f0819b1: pin pytest version
ad59fed: [pre-commit.ci] pre-commit autoupdate
0fc20e5: Added unlinked release notes for DEM 2.0.0
123dd2b: Drop extra lines
0177318: Rename product so that translation is not necessary.
94e334c: Explicitly pass .coveragerc to pytest.
04456a9: Updated documentation for 1.7.1 release
240d937: [pre-commit.ci] pre-commit autoupdate
f054532: Use pre-commit.ci rather than local actions
e2bc8d8: Update GlideinWMS dependencies
4dfefb1: Add SPDX blurbs.
a0889aa: Update pre-commit hook versions and accommodate python-debian issue.
7fd1c9c: Change gwmw module to read cfg from DE framework
d57ac6e: Set Apache 2.0 license and added REUSE compliance
cf43119: Set fail-fast to false to allow py3.6 tests to complete also when higher version fail. Temporarily downgraded from py3.10 to py3.9 because the condor dependency is not available yet for py3.10.
e2707e5: Ignoring E203, whitespace after ‘:’, since black is adding the whitespace
b0e3526: Remove modules that inherit from SourceProxy. Inheritance from SourceProxy has been forbidden by PR#526. In this commit it has been removed the SourceProxy module GCEResourceLimits that is not used.
208a660: Revert “Add jsonnetfmt hook”
efbf6b9: Add jsonnetfmt hook
9140330: Recasting AWSOccupancy, AWSSpotPrice and associated unit tests as Transforms AWSOccupancy and AWSSpotPrice source proxies will be handled at config level
81adb29: Start testing against python 3.10
a9c1f3c: Support retries for contacting Graphite.
3bd68e4: Fix some errors, most caught by lgtm.
89e21f0: Simplify some code paths
63f6315: Homogenize parameter-naming.
325ea37: Update pre-commit hooks via pre-commit autoupdate
ee7d45c: More logging improvements.
901291c: Take advantage of simplified logging.
f4752ab: Fixup pre-commit hooks
ad66754: Add Linters task
ca61ae3: Update to latest setuptools_scm
3ad934b: Fixed project name
ec2a94e: Added 1.1 release notes