.. SPDX-FileCopyrightText: 2017 Fermi Research Alliance, LLC .. SPDX-License-Identifier: Apache-2.0 Release 1.7.5 ------------- Fixed source logging. Pinned some dependencies to maintain Python 3.6 compatibility. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~ Bugs fixed - `DE 522 `_: Decision engine log files get split between several different processes with several different versions open: (`fd1e99ce `_) Full list of commits since version 1.7.1 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `0c90cdfb6 `_: fix source logging by defining logger in Sources after PR670 plus missing adjustments `670a618f2 `_: Updated release notes for 1.7.5 `ea6ef79d `_: pin ubuntu version to 20.04 to get python versions we use to run DE 1.7 tests and set upper limit for python modules to be used by tests `a1af36f5 `_: For branch 1.7 pin pytest version to 6.2.5 `352eab54 `_: For branch 1.7 pin jsonnet version to 0.17.0 `bfcfef2f `_: Updated release notes for 1.7.4 `4ff9db91 `_: Updated release notes for 1.7.3 `53aba118 `_: Updated release notes, ready for 1.7.2 `a461a8f9 `_: Updated documentation for 1.7.1 release Release 1.7.4 ------------- Same as 1.7.1 release. Dome to maintain the same version number as decisionengine_modules. Release 1.7.3 ------------- Same as 1.7.1 release. Dome to maintain the same version number as decisionengine_modules. Release 1.7.2 ------------- Same as 1.7.1 release. Done to maintain the same version number as decisionengine_modules. Release 1.7.1 ------------- Patch level (bug fix) release. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Bugs fixed - `DE 522 `_: Decision engine log files get split between several different processes with several different versions open: (`fd1e99ce `_) Enhancements: - Added dummy sources (`de1536fa `_) Full list of commits since version 1.7.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `606e1e9f `_: Merge pull request #585 from vitodb/fix/1.7/vito_port_PR527 `538cf940 `_: Merge pull request #586 from HEPCloud/goodenou-patch-remove-gc `67febfd0 `_: remove unnecessary 'import gc' `9da797d3 `_: Improve parameter and variable names. `55a5b547 `_: porting #PR515 into 1.7 (simplifications to logging in Modules) cherry-picked from commit d515741 `fd1e99ce `_: porting #PR563 into 1.7 (adding queue logging into de_logger) `c75deef4 `_: Also run tests on PRs for backports to 1.7 `de1536fa `_: add dummy source and test Release 1.7.0 ------------- This release features: - New produces-consumes structure using decorators. This will improve the code quality, improving static checks and reducing the lines of code by removing repetitive boilerplates, especially in the modules. - Added structured logging. Improved python logging and adoption of structured logs format that will increase the semantinc content of the messages and ease the export of information for dashboards and Elastic Search. - Added SQLAlchemy object-relational mapper to increase the testability of DB interactions and to allow different database backends. Switching between datasource backends requires dropping all objects if you wish to reuse the tablespace. - Packaging via setuptools for both decisionengine and decisionengine_modules: Dependencies are not yet fully listed in the RPMs. - A new, optional, configuration parameter called "channel_name" is available. "channel_name" is one of the keys in the output dictionary of the structured logging and will be used in the upcoming monitoring. If the variable is not defined in the configuration file, then it is taken from the name of the file, e.g. the job_classification.jsonnet config file gives a default "channel_name" value of "job_classification". .. note:: Added requirement on SQLAlchemy (for new datasource backend). Non-SQLAlchemy users should ensure the indexes from `13c2f283 `_ are in their database. .. note:: Added requirement on prometheus-client. Prometheus will be used as optional monitoring component. .. note:: The "channel_name" key in the Source Proxy config dictionaries needs to be changed to "source_channel". "channel_name" is now being used to describe the name of the channel itself, not the name of the channel the Source Proxy is getting information from. Issues fixed in this release ~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - `481 `_: Channel name should be available to all worker types in TaskManager - `456 `_: Logic engine messages show in the main DE log (1.6.99 post4) prj_testing - `458 `_: Exception in new SQLAlchemy data source 1.6.99post4 - `455 `_: New postgresql exception in 1.6.99post4 (aka Fixed databese inconsistency silently ignored in v1.6) - `456 `_: Logic engine messages show in the main DE log (1.6.99 post4) - `451 `_: Transforms executed in wrong order in 1.6.99.post3 - `367 `_: Test race conditions bug - `406 `_: Taskmanager doesn't use/honor global log level - `379 `_: Add postgresql.sql to distributed decisionengine rpm - `329 `_: Docker container is missing pylint - `293 `_: Drop requirements.txt setup mode - `285 `_: Unify ProcessingState with Reaper state management code - `253 `_: Decision engine can sometimes start up at boot time before network name resolution is working (`ae04db5 `_) Full list of commits since version 1.6.0 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ `f42558df `_: Updated documentation for 1.7.0 release `029d118a `_: Updated release notes for 1.7.0 RC4 (1.6.99.post8) `0e19c754 `_: fix SP `810994af `_: Update release_notes_1.7.rst `fbee95e7 `_: Update release_notes_1.7.rst `68b955b0 `_: Make sure product is a string `ef7a8b96 `_: Automatically adjust PYTHONPATH for tests `e292d388 `_: Updated release notes for 1.7.0 RC3 (1.6.99.post7) `d60b6e4e `_: new changes for logging with common logger name "channel" `8cdeb67e `_: Simplify return expression `8fb128d3 `_: Ensure file is "flushed" so name is fully established `7806aa00 `_: Add github CodeQL analysis `9f09bca9 `_: removed modules/LogicEngine.py and corresponding test `b9d28fbf `_: Cleaner check for `Any` `cc91aa24 `_: Switch to fstring formatting `7bb5b64f `_: Just return created value rather than store then return `f4847fbe `_: Combine nested `with` blocks `4ba38bcd `_: Drop redundant brackets `bdcfe8c9 `_: By convention, pandas is usually imported as `pd` `1dd904ff `_: Use more traditional expression order `cccd31bc `_: Unused loop vars should start with `_` `c055a5cd `_: Drop `_keys` in favor of DB backed `keys` `e8c689b4 `_: Moved prometheus-client requirement to proper place in list `5391500d `_: Added metrics API module `c2d7835c `_: Drop unnecessary timeout `c167fc50 `_: Add tests for de-query-tool entry point `efabfeb3 `_: Updated release notes for 1.7.0 RC2 (1.6.99.post6) `b2739c14 `_: moved logging of LogicEngine from decisionengine logger to channel loggers `0c0532f3 `_: Add locks to help ensure data changes are "atomic" `ae63c6ee `_: Use DB generated known keys so it always matches DB state `b2259e9e `_: Use public .keys() rather than internal implementation `85b6c3ba `_: Real world data shows the defaults are fine `95fb3fdf `_: Further constrain tablespace `3ebe8619 `_: Finish implementation of get_datablock `edbb3568 `_: Add entry point for de-query-tool `fed95c62 `_: adding logging of importlib imports of modules `53e62f03 `_: Sometimes pypy times out on the cleanup. `a44d4bc4 `_: Don't test sqlite on pypy it isn't necessary `b13aa8a9 `_: Some corrections `94c14110 `_: Fix missing defines `5f102095 `_: More detailed testing of datablock `b6c99021 `_: Make sure our sqlite tests have ForeignKeyConditional support `6b76ba7c `_: Fix typo `6694369d `_: Ensure dbutils uses transactions `1df400ae `_: Fix spaces `5278fd99 `_: Raise timeout for numpy on pypy `6d0a1a74 `_: Release notes ready for v1.7.0 `084f74e1 `_: Initial SQLAlchemy Datasource `3353aa00 `_: Make sure our jsonnet is json syntax valid `402b1c26 `_: Fix transform-ordering problem. `49297573 `_: Fix incorrect packaging of tests at top level `fbfae499 `_: The test_channel loads data once per second. `33f9ade1 `_: Rename taskmanager test nodb `308343e9 `_: Initial modifications for addition of structured logging `6f337b75 `_: Add missing error message `23a4b770 `_: Call fixtures in a cleaner manner for xdist `1f2fe8c4 `_: Add self.config so I can introspect the fixtures later `689c0020 `_: Add missing `config` attrib test `d2732816 `_: Best practices are for fixtues to `yield` vs `return` `accef50a `_: Seed SQLAlchemy fixtures for later activation `31002bc5 `_: Help define the fixture interlocking `0f5fb129 `_: The pandas 1.3.0 doesn't build against PyPy any longer `a7d18a41 `_: Correctly test datablock construction paths `9af4c144 `_: the `mock` package was a backport for python2. `5ddaff8f `_: Add another constructor test `9ae9ad13 `_: Make sure if the client says to stop we don't override it `a581cd2b `_: run pyupgrade against codebase for python3.6 `09e4e79c `_: Handle reaper duplicate shutdowns more cleanly `64d29dc5 `_: Drop pointless cache restore `1c6b2588 `_: Update PyPy to 3.7 for testing `2bae173e `_: Increase wait for overloaded test workers, update log messages `b67c185c `_: When aborting CI builds cleanup all processes `6c5d6306 `_: Trim pytest fast functions, add required plugin `8c63ca6b `_: note why we're ignoring this line `2bd4ecbc `_: Add a syntax check for the toml files `e2dca404 `_: Sometimes these get stuck `6d012fab `_: Add in Jenkinsfile pipeline configuration a timeout at stage level `baf07973 `_: Add timeout option to block-while/until `970faf92 `_: Make pre-commit happy `0cea2285 `_: Fix alignment issue `5620c65b `_: List why we aren't checking `88611d90 `_: Ensure fixtures are cleaned up between invocations `0ba135d2 `_: Setup blank DB for SQLAlchemy tests and prep fixtures `3793e674 `_: Setup pre-commit `9e6d1317 `_: Migrate test_Reaper to pytest fixtures `51df43bf `_: Cleanup a bunch of pointless whitespace `96e5d069 `_: Fix typo `9f96f418 `_: Setup datablock to use our parameterized fixture `36ebc66c `_: Add config for LGTM `c6032e5f `_: Use topologically sorted transforms to remove some multi-threading. `e063f82a `_: Drop pointless comma `bfd6689e `_: Begin prepwork for PEP517 `72c5725f `_: Stub out null source rather than more complex mocking `3b65e5e2 `_: Push Singleton into its own space `fb5b177e `_: Put fixtures in central location `5ab3cbaa `_: Add more details to channel startup logs `afe7f7d7 `_: Add log about what DB we are hitting `38034b2c `_: Let the datasource handle the connections internally `5e03b6fe `_: Since we are opening an IPv4 socket, just use 127.0.0.1 to check `cac2bef3 `_: Fix missing version requirements `3be8f84f `_: Add line length for autoformater `90e2baad `_: Protect against inappropriate wait under error condition. `943a17a7 `_: Fix de-client typo and adjust tests accordingly. `3b104eba `_: Set the logs to DEBUG for testing `4c5564d4 `_: Add another sync method to try and make tests less spotty `66bd81f2 `_: Make sure to encourage updates to tools `d16f04cc `_: Put postgresql datasource schema into RPM `62b97e79 `_: Fix __str__ so it includes all the data `611ef1f8 `_: Drop pointless lines `5b9e2fb6 `_: Drop unreachable excepts `6991f65f `_: Restore product-name translation required for some source-proxy cases. `f6258c09 `_: Fixed formatting and updated content `104a0446 `_: Update index.rst `2ed61289 `_: Update index.rst `cb687150 `_: Create release_notes.rst `3b57d4a2 `_: Note new requirement `871af08b `_: Added 1.7.0 release notes `ce42b802 `_: improved 1.6 release note `583c10fb `_: fixed rst error `96d4dc1e `_: Added 1.6.2 release notes, from branch 1.6 `13c2f283 `_: Add some helpful indexes to our default schema `29c32571 `_: Log as workers are started `619021c2 `_: One of these tests seems to be spotty, break them out to find which one `29a2c72d `_: Run the test in a way that gives us colors `4e36bfd2 `_: Drop unused table create logic `5511f69e `_: Stronger notify state for when we've a lot of watchers. `b6cc7a46 `_: Test the dataspace abstractions `e3b1f594 `_: Better messages about our state `2d2feab9 `_: Drop duplicate tests, leave specifics `8e737329 `_: Add parameter based datasource api tests `5c023aa5 `_: Don't do debug logs for flake8, they aren't helpful `f5d1a12f `_: Setup list of public exports for dataspace.py `7158b422 `_: Merge pull request #365 from jcpunk/bad-update-is-error `cd98cc4a `_: Update should error out if you try to do it wrongly `eb7907fe `_: Add option to set taskmanager datestamp and sample usage `e124532c `_: Make sure the fixture uses the production flow `a8241b6e `_: Make sure RPM also owns the .egg-info so we don't confuse the namespaces `da87376e `_: Ensure the DE server is fully started before running query `622bfacf `_: Simplify use of our PG fixtures `df98ecdf `_: Fixed flake8 issue `061ff6cf `_: decisionengine/framework: stop_channel runs Publisher shutdown methods `3727b80b `_: Fixup comment to avoid assuming this test uses the DB `d45aaf6b `_: Fix script path typo `a25a4a30 `_: Fix ABC to match our actual usage `1510b2d1 `_: Address minor linting issues `945e4b16 `_: Fix missing attribute insert `5eace9d5 `_: Add note for how to get modules in place `50a8e268 `_: Add list of packages in the CI env to output `b9cb197d `_: Sanity check the home directory `cd17223c `_: Have client provide a hint when you ask for no behavior `95b02365 `_: Fix de-query-tool to support produce/consume model `e660ca72 `_: Update required versions for bugfixes `6863cb81 `_: Fix path error `bb52e8b1 `_: Merge pull request #340 from jcpunk/service-stop `6d7aba95 `_: Drop obsolete files `168ae7aa `_: Name the tests better `0f60c4e3 `_: Support new produces/consumes/configuration-description infrastructure. `81912469 `_: Add de-query-tool `2a26c944 `_: ExecStopPre is not supported on all systemd instances `67a54d5c `_: Merge pull request #338 from jcpunk/fix-pytest-postgres `70ab133f `_: Fixup use of pytest_postgresql for version 3.0.0 `f8f4255e `_: Merge pull request #337 from jcpunk/thread-names `5f49a4f6 `_: Set names for the various parallel code `64da77c6 `_: Merge pull request #327 from jcpunk/datablock-expire `de33a60a `_: Merge pull request #336 from knoepfel/use-toposort `31a8a905 `_: Merge pull request #328 from knoepfel/de-class-inference `410e383d `_: Merge pull request #331 from jcpunk/reaper-interval-tests `719ff0c8 `_: Test datablock expire functions `e14c49d8 `_: The 'name' parameter is optional. `7846c9f3 `_: Enable DE class inference based on configuration. `32ab7e44 `_: Use third-party topological sort. `01aa8ae6 `_: Merge pull request #325 from jcpunk/channel-tests `52b48479 `_: Merge pull request #326 from jcpunk/valid-config-tests `8c4749e7 `_: Merge pull request #330 from jcpunk/pylint-actions `a37770c9 `_: Ensure validation testing is tested `d8ab5eb6 `_: Add missing test to ensure the run interval is actually used `0cd9c42b `_: Also run pylint for extra sanity checks `c5cf1fff `_: Ensure our errors error out `baf01700 `_: Merge pull request #324 from jcpunk/cleanup-trivial-tests `2a0133aa `_: Try to cleanup trivial missing coverage `44e0ad6f `_: Merge pull request #323 from jcpunk/about-coverage `d811f617 `_: Merge pull request #322 from knoepfel/fix-fail-on-error `cb426262 `_: Merge pull request #312 from jcpunk/finish-setuptools `8f6d407d `_: Merge pull request #316 from jcpunk/abc-coverage `4d0676bb `_: Merge pull request #317 from vitodb/pylint `d7c43b96 `_: Use regular expression to support fail_on_error feature. `ada66925 `_: add support to run pylint tests `efb1e57b `_: Finish migration to pure setuptools `bc4720cf `_: We aren't testing 'unversioned" releases `e4dc35e3 `_: Merge pull request #314 from jcpunk/jsonnet_syntax `87e32c22 `_: Merge pull request #294 from jcpunk/move-reaper `dec85d5e `_: Merge pull request #319 from jcpunk/task-loop `4108472a `_: Merge pull request #320 from jcpunk/container-swig `920af1c9 `_: Merge pull request #321 from knoepfel/include-init-files `650dffa7 `_: Don't forget __init__.py files. `1b412e03 `_: The latest m2crypto seems to need swig now `a6e3ab1c `_: Merge pull request #313 from jcpunk/conf-test `1205636a `_: Simplify run loop `30e59dc9 `_: fix test_client_with_no_server_verbose unit test for Jenkins CI (#315) `10384a8c `_: Move reaper into its own place and reuse state logic `940584e4 `_: No real way to test abstract base classes `250c14b1 `_: The `_validate` function doesn't permit missing 'PRODUCES' `5ae1ce9f `_: Make sure syntax error in config names the problem `b899fa23 `_: Add SourceProxy module test. (#307) `7b3df14c `_: Increase coverage of utils (#304) `ddba2a31 `_: Fix duplicate entry warning (#311) `915673fa `_: Test modules minimally (#298) `bc0c21a9 `_: Some repos may error out, don't let them kill the build (#297) `924a7047 `_: doc: add 1.6.1 release notes `b1ab4d31 `_: doc: fix typo `85e5d714 `_: postgresql: do not print stack trace for low level library (#309) `255c6415 `_: Setuptools uses entry return value as an error msg (#303) `2fd8db45 `_: Fix name to match expectations (#305) `9cddb70a `_: updated release notes `7fe0358e `_: Error in more clean methods (#300) `84aa506c `_: Fix a bug in setup.py parsing of requirements. (#301) `a58b61bb `_: fix typo in release notes