Release 1.7.5
Fixed source logging. Pinned some dependencies to maintain Python 3.6 compatibility.
Issues fixed in this release
Bugs fixed
Full list of commits since version 1.7.1
0c90cdfb6: fix source logging by defining logger in Sources after PR670 plus missing adjustments
670a618f2: Updated release notes for 1.7.5
ea6ef79d: pin ubuntu version to 20.04 to get python versions we use to run DE 1.7 tests and set upper limit for python modules to be used by tests
a1af36f5: For branch 1.7 pin pytest version to 6.2.5
352eab54: For branch 1.7 pin jsonnet version to 0.17.0
bfcfef2f: Updated release notes for 1.7.4
4ff9db91: Updated release notes for 1.7.3
53aba118: Updated release notes, ready for 1.7.2
a461a8f9: Updated documentation for 1.7.1 release
Release 1.7.4
Same as 1.7.1 release. Dome to maintain the same version number as decisionengine_modules.
Release 1.7.3
Same as 1.7.1 release. Dome to maintain the same version number as decisionengine_modules.
Release 1.7.2
Same as 1.7.1 release. Done to maintain the same version number as decisionengine_modules.
Release 1.7.1
Patch level (bug fix) release.
Issues fixed in this release
Bugs fixed
DE 522: Decision engine log files get split between several different processes with several different versions open: (fd1e99ce)
Enhancements:
Added dummy sources (de1536fa)
Full list of commits since version 1.7.0
606e1e9f: Merge pull request #585 from vitodb/fix/1.7/vito_port_PR527
538cf940: Merge pull request #586 from HEPCloud/goodenou-patch-remove-gc
67febfd0: remove unnecessary ‘import gc’
9da797d3: Improve parameter and variable names.
55a5b547: porting #PR515 into 1.7 (simplifications to logging in Modules) cherry-picked from commit d515741
fd1e99ce: porting #PR563 into 1.7 (adding queue logging into de_logger)
c75deef4: Also run tests on PRs for backports to 1.7
de1536fa: add dummy source and test
Release 1.7.0
This release features:
New produces-consumes structure using decorators. This will improve the code quality, improving static checks and reducing the lines of code by removing repetitive boilerplates, especially in the modules.
Added structured logging. Improved python logging and adoption of structured logs format that will increase the semantinc content of the messages and ease the export of information for dashboards and Elastic Search.
Added SQLAlchemy object-relational mapper to increase the testability of DB interactions and to allow different database backends. Switching between datasource backends requires dropping all objects if you wish to reuse the tablespace.
Packaging via setuptools for both decisionengine and decisionengine_modules: Dependencies are not yet fully listed in the RPMs.
A new, optional, configuration parameter called “channel_name” is available. “channel_name” is one of the keys in the output dictionary of the structured logging and will be used in the upcoming monitoring. If the variable is not defined in the configuration file, then it is taken from the name of the file, e.g. the job_classification.jsonnet config file gives a default “channel_name” value of “job_classification”.
Note
Added requirement on SQLAlchemy (for new datasource backend). Non-SQLAlchemy users should ensure the indexes from 13c2f283 are in their database.
Note
Added requirement on prometheus-client. Prometheus will be used as optional monitoring component.
Note
The “channel_name” key in the Source Proxy config dictionaries needs to be changed to “source_channel”. “channel_name” is now being used to describe the name of the channel itself, not the name of the channel the Source Proxy is getting information from.
Issues fixed in this release
481: Channel name should be available to all worker types in TaskManager
456: Logic engine messages show in the main DE log (1.6.99 post4) prj_testing
458: Exception in new SQLAlchemy data source 1.6.99post4
455: New postgresql exception in 1.6.99post4 (aka Fixed databese inconsistency silently ignored in v1.6)
456: Logic engine messages show in the main DE log (1.6.99 post4)
451: Transforms executed in wrong order in 1.6.99.post3
367: Test race conditions bug
406: Taskmanager doesn’t use/honor global log level
379: Add postgresql.sql to distributed decisionengine rpm
329: Docker container is missing pylint
293: Drop requirements.txt setup mode
285: Unify ProcessingState with Reaper state management code
253: Decision engine can sometimes start up at boot time before network name resolution is working (ae04db5)
Full list of commits since version 1.6.0
f42558df: Updated documentation for 1.7.0 release
029d118a: Updated release notes for 1.7.0 RC4 (1.6.99.post8)
0e19c754: fix SP
810994af: Update release_notes_1.7.rst
fbee95e7: Update release_notes_1.7.rst
68b955b0: Make sure product is a string
ef7a8b96: Automatically adjust PYTHONPATH for tests
e292d388: Updated release notes for 1.7.0 RC3 (1.6.99.post7)
d60b6e4e: new changes for logging with common logger name “channel”
8cdeb67e: Simplify return expression
8fb128d3: Ensure file is “flushed” so name is fully established
7806aa00: Add github CodeQL analysis
9f09bca9: removed modules/LogicEngine.py and corresponding test
b9d28fbf: Cleaner check for Any
cc91aa24: Switch to fstring formatting
7bb5b64f: Just return created value rather than store then return
f4847fbe: Combine nested with blocks
4ba38bcd: Drop redundant brackets
bdcfe8c9: By convention, pandas is usually imported as pd
1dd904ff: Use more traditional expression order
cccd31bc: Unused loop vars should start with _
c055a5cd: Drop _keys in favor of DB backed keys
e8c689b4: Moved prometheus-client requirement to proper place in list
5391500d: Added metrics API module
c2d7835c: Drop unnecessary timeout
c167fc50: Add tests for de-query-tool entry point
efabfeb3: Updated release notes for 1.7.0 RC2 (1.6.99.post6)
b2739c14: moved logging of LogicEngine from decisionengine logger to channel loggers
0c0532f3: Add locks to help ensure data changes are “atomic”
ae63c6ee: Use DB generated known keys so it always matches DB state
b2259e9e: Use public .keys() rather than internal implementation
85b6c3ba: Real world data shows the defaults are fine
95fb3fdf: Further constrain tablespace
3ebe8619: Finish implementation of get_datablock
edbb3568: Add entry point for de-query-tool
fed95c62: adding logging of importlib imports of modules
53e62f03: Sometimes pypy times out on the cleanup.
a44d4bc4: Don’t test sqlite on pypy it isn’t necessary
b13aa8a9: Some corrections
94c14110: Fix missing defines
5f102095: More detailed testing of datablock
b6c99021: Make sure our sqlite tests have ForeignKeyConditional support
6b76ba7c: Fix typo
6694369d: Ensure dbutils uses transactions
1df400ae: Fix spaces
5278fd99: Raise timeout for numpy on pypy
6d0a1a74: Release notes ready for v1.7.0
084f74e1: Initial SQLAlchemy Datasource
3353aa00: Make sure our jsonnet is json syntax valid
402b1c26: Fix transform-ordering problem.
49297573: Fix incorrect packaging of tests at top level
fbfae499: The test_channel loads data once per second.
33f9ade1: Rename taskmanager test nodb
308343e9: Initial modifications for addition of structured logging
6f337b75: Add missing error message
23a4b770: Call fixtures in a cleaner manner for xdist
1f2fe8c4: Add self.config so I can introspect the fixtures later
689c0020: Add missing config attrib test
d2732816: Best practices are for fixtues to yield vs return
accef50a: Seed SQLAlchemy fixtures for later activation
31002bc5: Help define the fixture interlocking
0f5fb129: The pandas 1.3.0 doesn’t build against PyPy any longer
a7d18a41: Correctly test datablock construction paths
9af4c144: the mock package was a backport for python2.
5ddaff8f: Add another constructor test
9ae9ad13: Make sure if the client says to stop we don’t override it
a581cd2b: run pyupgrade against codebase for python3.6
09e4e79c: Handle reaper duplicate shutdowns more cleanly
64d29dc5: Drop pointless cache restore
1c6b2588: Update PyPy to 3.7 for testing
2bae173e: Increase wait for overloaded test workers, update log messages
b67c185c: When aborting CI builds cleanup all processes
6c5d6306: Trim pytest fast functions, add required plugin
8c63ca6b: note why we’re ignoring this line
2bd4ecbc: Add a syntax check for the toml files
e2dca404: Sometimes these get stuck
6d012fab: Add in Jenkinsfile pipeline configuration a timeout at stage level
baf07973: Add timeout option to block-while/until
970faf92: Make pre-commit happy
0cea2285: Fix alignment issue
5620c65b: List why we aren’t checking
88611d90: Ensure fixtures are cleaned up between invocations
0ba135d2: Setup blank DB for SQLAlchemy tests and prep fixtures
3793e674: Setup pre-commit
9e6d1317: Migrate test_Reaper to pytest fixtures
51df43bf: Cleanup a bunch of pointless whitespace
96e5d069: Fix typo
9f96f418: Setup datablock to use our parameterized fixture
36ebc66c: Add config for LGTM
c6032e5f: Use topologically sorted transforms to remove some multi-threading.
e063f82a: Drop pointless comma
bfd6689e: Begin prepwork for PEP517
72c5725f: Stub out null source rather than more complex mocking
3b65e5e2: Push Singleton into its own space
fb5b177e: Put fixtures in central location
5ab3cbaa: Add more details to channel startup logs
afe7f7d7: Add log about what DB we are hitting
38034b2c: Let the datasource handle the connections internally
5e03b6fe: Since we are opening an IPv4 socket, just use 127.0.0.1 to check
cac2bef3: Fix missing version requirements
3be8f84f: Add line length for autoformater
90e2baad: Protect against inappropriate wait under error condition.
943a17a7: Fix de-client typo and adjust tests accordingly.
3b104eba: Set the logs to DEBUG for testing
4c5564d4: Add another sync method to try and make tests less spotty
66bd81f2: Make sure to encourage updates to tools
d16f04cc: Put postgresql datasource schema into RPM
62b97e79: Fix __str__ so it includes all the data
611ef1f8: Drop pointless lines
5b9e2fb6: Drop unreachable excepts
6991f65f: Restore product-name translation required for some source-proxy cases.
f6258c09: Fixed formatting and updated content
104a0446: Update index.rst
2ed61289: Update index.rst
cb687150: Create release_notes.rst
3b57d4a2: Note new requirement
871af08b: Added 1.7.0 release notes
ce42b802: improved 1.6 release note
583c10fb: fixed rst error
96d4dc1e: Added 1.6.2 release notes, from branch 1.6
13c2f283: Add some helpful indexes to our default schema
29c32571: Log as workers are started
619021c2: One of these tests seems to be spotty, break them out to find which one
29a2c72d: Run the test in a way that gives us colors
4e36bfd2: Drop unused table create logic
5511f69e: Stronger notify state for when we’ve a lot of watchers.
b6cc7a46: Test the dataspace abstractions
e3b1f594: Better messages about our state
2d2feab9: Drop duplicate tests, leave specifics
8e737329: Add parameter based datasource api tests
5c023aa5: Don’t do debug logs for flake8, they aren’t helpful
f5d1a12f: Setup list of public exports for dataspace.py
7158b422: Merge pull request #365 from jcpunk/bad-update-is-error
cd98cc4a: Update should error out if you try to do it wrongly
eb7907fe: Add option to set taskmanager datestamp and sample usage
e124532c: Make sure the fixture uses the production flow
a8241b6e: Make sure RPM also owns the .egg-info so we don’t confuse the namespaces
da87376e: Ensure the DE server is fully started before running query
622bfacf: Simplify use of our PG fixtures
df98ecdf: Fixed flake8 issue
061ff6cf: decisionengine/framework: stop_channel runs Publisher shutdown methods
3727b80b: Fixup comment to avoid assuming this test uses the DB
d45aaf6b: Fix script path typo
a25a4a30: Fix ABC to match our actual usage
1510b2d1: Address minor linting issues
945e4b16: Fix missing attribute insert
5eace9d5: Add note for how to get modules in place
50a8e268: Add list of packages in the CI env to output
b9cb197d: Sanity check the home directory
cd17223c: Have client provide a hint when you ask for no behavior
95b02365: Fix de-query-tool to support produce/consume model
e660ca72: Update required versions for bugfixes
6863cb81: Fix path error
bb52e8b1: Merge pull request #340 from jcpunk/service-stop
6d7aba95: Drop obsolete files
168ae7aa: Name the tests better
0f60c4e3: Support new produces/consumes/configuration-description infrastructure.
81912469: Add de-query-tool
2a26c944: ExecStopPre is not supported on all systemd instances
67a54d5c: Merge pull request #338 from jcpunk/fix-pytest-postgres
70ab133f: Fixup use of pytest_postgresql for version 3.0.0
f8f4255e: Merge pull request #337 from jcpunk/thread-names
5f49a4f6: Set names for the various parallel code
64da77c6: Merge pull request #327 from jcpunk/datablock-expire
de33a60a: Merge pull request #336 from knoepfel/use-toposort
31a8a905: Merge pull request #328 from knoepfel/de-class-inference
410e383d: Merge pull request #331 from jcpunk/reaper-interval-tests
719ff0c8: Test datablock expire functions
e14c49d8: The ‘name’ parameter is optional.
7846c9f3: Enable DE class inference based on configuration.
32ab7e44: Use third-party topological sort.
01aa8ae6: Merge pull request #325 from jcpunk/channel-tests
52b48479: Merge pull request #326 from jcpunk/valid-config-tests
8c4749e7: Merge pull request #330 from jcpunk/pylint-actions
a37770c9: Ensure validation testing is tested
d8ab5eb6: Add missing test to ensure the run interval is actually used
0cd9c42b: Also run pylint for extra sanity checks
c5cf1fff: Ensure our errors error out
baf01700: Merge pull request #324 from jcpunk/cleanup-trivial-tests
2a0133aa: Try to cleanup trivial missing coverage
44e0ad6f: Merge pull request #323 from jcpunk/about-coverage
d811f617: Merge pull request #322 from knoepfel/fix-fail-on-error
cb426262: Merge pull request #312 from jcpunk/finish-setuptools
8f6d407d: Merge pull request #316 from jcpunk/abc-coverage
4d0676bb: Merge pull request #317 from vitodb/pylint
d7c43b96: Use regular expression to support fail_on_error feature.
ada66925: add support to run pylint tests
efb1e57b: Finish migration to pure setuptools
bc4720cf: We aren’t testing ‘unversioned” releases
e4dc35e3: Merge pull request #314 from jcpunk/jsonnet_syntax
87e32c22: Merge pull request #294 from jcpunk/move-reaper
dec85d5e: Merge pull request #319 from jcpunk/task-loop
4108472a: Merge pull request #320 from jcpunk/container-swig
920af1c9: Merge pull request #321 from knoepfel/include-init-files
650dffa7: Don’t forget __init__.py files.
1b412e03: The latest m2crypto seems to need swig now
a6e3ab1c: Merge pull request #313 from jcpunk/conf-test
1205636a: Simplify run loop
30e59dc9: fix test_client_with_no_server_verbose unit test for Jenkins CI (#315)
10384a8c: Move reaper into its own place and reuse state logic
940584e4: No real way to test abstract base classes
250c14b1: The _validate function doesn’t permit missing ‘PRODUCES’
5ae1ce9f: Make sure syntax error in config names the problem
b899fa23: Add SourceProxy module test. (#307)
7b3df14c: Increase coverage of utils (#304)
ddba2a31: Fix duplicate entry warning (#311)
915673fa: Test modules minimally (#298)
bc0c21a9: Some repos may error out, don’t let them kill the build (#297)
924a7047: doc: add 1.6.1 release notes
b1ab4d31: doc: fix typo
85e5d714: postgresql: do not print stack trace for low level library (#309)
255c6415: Setuptools uses entry return value as an error msg (#303)
2fd8db45: Fix name to match expectations (#305)
9cddb70a: updated release notes
7fe0358e: Error in more clean methods (#300)
84aa506c: Fix a bug in setup.py parsing of requirements. (#301)
a58b61bb: fix typo in release notes