decisionengine.framework.dataspace.datasources package

Subpackages

• decisionengine.framework.dataspace.datasources.sqlalchemy_ds package
  • Submodules
  • decisionengine.framework.dataspace.datasources.sqlalchemy_ds.datasource_api module
    • SQLAlchemyDS
      • SQLAlchemyDS._abc_impl
      • SQLAlchemyDS.close()
      • SQLAlchemyDS.connect()
      • SQLAlchemyDS.create_tables()
      • SQLAlchemyDS.delete_data_older_than()
      • SQLAlchemyDS.duplicate_datablock()
      • SQLAlchemyDS.get_datablock()
      • SQLAlchemyDS.get_dataproduct()
      • SQLAlchemyDS.get_dataproducts()
      • SQLAlchemyDS.get_header()
      • SQLAlchemyDS.get_last_generation_id()
      • SQLAlchemyDS.get_metadata()
      • SQLAlchemyDS.get_schema()
      • SQLAlchemyDS.get_taskmanager()
      • SQLAlchemyDS.get_taskmanagers()
      • SQLAlchemyDS.insert()
      • SQLAlchemyDS.reset_connections()
      • SQLAlchemyDS.store_taskmanager()
      • SQLAlchemyDS.update()
  • decisionengine.framework.dataspace.datasources.sqlalchemy_ds.db_schema module
    • Base
      • Base._sa_registry
      • Base.metadata
      • Base.registry
    • Dataproduct
      • Dataproduct._sa_class_manager
      • Dataproduct.generation_id
      • Dataproduct.id
      • Dataproduct.key
      • Dataproduct.taskmanager
      • Dataproduct.taskmanager_id
      • Dataproduct.value
    • Header
      • Header._sa_class_manager
      • Header.create_time
      • Header.creator
      • Header.expiration_time
      • Header.generation_id
      • Header.id
      • Header.key
      • Header.scheduled_create_time
      • Header.schema_id
      • Header.taskmanager
      • Header.taskmanager_id
    • Metadata
      • Metadata._sa_class_manager
      • Metadata.generation_id
      • Metadata.generation_time
      • Metadata.id
      • Metadata.key
      • Metadata.missed_update_count
      • Metadata.state
      • Metadata.taskmanager
      • Metadata.taskmanager_id
    • Schema
      • Schema._sa_class_manager
      • Schema.schema
      • Schema.schema_id
    • Taskmanager
      • Taskmanager._sa_class_manager
      • Taskmanager.datestamp
      • Taskmanager.name
      • Taskmanager.sequence_id
      • Taskmanager.task_dataproduct
      • Taskmanager.task_header
      • Taskmanager.task_metadata
      • Taskmanager.taskmanager_id
  • decisionengine.framework.dataspace.datasources.sqlalchemy_ds.utils module
    • add_engine_pidguard()
    • clone_model()
    • orm_as_dict()
  • Module contents
    • SQLAlchemyDS
      • SQLAlchemyDS._abc_impl
      • SQLAlchemyDS.close()
      • SQLAlchemyDS.connect()
      • SQLAlchemyDS.create_tables()
      • SQLAlchemyDS.delete_data_older_than()
      • SQLAlchemyDS.duplicate_datablock()
      • SQLAlchemyDS.get_datablock()
      • SQLAlchemyDS.get_dataproduct()
      • SQLAlchemyDS.get_dataproducts()
      • SQLAlchemyDS.get_header()
      • SQLAlchemyDS.get_last_generation_id()
      • SQLAlchemyDS.get_metadata()
      • SQLAlchemyDS.get_schema()
      • SQLAlchemyDS.get_taskmanager()
      • SQLAlchemyDS.get_taskmanagers()
      • SQLAlchemyDS.insert()
      • SQLAlchemyDS.reset_connections()
      • SQLAlchemyDS.store_taskmanager()
      • SQLAlchemyDS.update()
• decisionengine.framework.dataspace.datasources.tests package
  • Submodules
  • decisionengine.framework.dataspace.datasources.tests.fixtures module
    • PG_DE_DB_WITHOUT_SCHEMA()
    • PG_PROG()
    • SQLALCHEMY_PG_WITH_SCHEMA()
    • SQLALCHEMY_TEMPFILE_SQLITE()
    • datasource()
    • mock_data_block()
  • decisionengine.framework.dataspace.datasources.tests.test_datasource_api module
    • test_create_tables()
    • test_delete_data_older_than_arg()
    • test_duplicate_datablock()
    • test_get_datablock()
    • test_get_dataproduct()
    • test_get_dataproduct_not_exist()
    • test_get_dataproducts()
    • test_get_dataproducts_not_exist()
    • test_get_header()
    • test_get_header_not_exist()
    • test_get_last_generation_id()
    • test_get_last_generation_id_not_exist()
    • test_get_metadata()
    • test_get_metadata_not_exist()
    • test_get_taskmanager_exists()
    • test_get_taskmanager_not_exists()
    • test_get_taskmanagers()
    • test_get_taskmanagers_not_exist()
    • test_has_config()
    • test_insert()
    • test_reset_connections()
    • test_store_taskmanager()
    • test_update()
    • test_update_bad()
  • Module contents

Submodules

decisionengine.framework.dataspace.datasources.null module

class decisionengine.framework.dataspace.datasources.null.NullDataSource(config_dict)

Bases: DataSource

Implementation of data source ABC that does nothing

_abc_impl = <_abc._abc_data object>

close()

Close all connections to the database

connect()

Create a pool of database connections

create_tables()

Create database tables

delete_data_older_than(days)

Delete data older that interval :type days: long :arg days: remove data older than interval

duplicate_datablock(taskmanager_id, generation_id, new_generation_id)

For the given taskmanager_id, make a copy of the datablock with given generation_id, set the generation_id for the datablock copy

Parameters:

• taskmanager_id (string) – taskmanager_id for generation to be retrieved

• generation_id (int) – generation_id of the data

• new_generation_id (int) – generation_id of the new datablock created

get_datablock(taskmanager_id, generation_id)

Return the entire datablock from the dataproduct table for the given taskmanager_id, generation_id

Parameters:

• taskmanager_id (string) – taskmanager_id for generation to be retrieved

• generation_id (int) – generation_id of the data

get_dataproduct(taskmanager_id, generation_id, key)

Return the data from the dataproduct table for the given taskmanager_id, generation_id, key

Parameters:

• taskmanager_id (string) – taskmanager_id for generation to be retrieved

• generation_id (int) – generation_id of the data

• key (string) – key for the value

get_dataproducts(taskmanager_id, key=None)

Return list of all data products associated with with taskmanager_id

Parameters:

key (string) – data product key

get_header(taskmanager_id, generation_id, key)

Return the header from the header table for the given taskmanager_id, generation_id, key

Parameters:

• taskmanager_id (string) – taskmanager_id for generation to be retrieved

• generation_id (int) – generation_id of the data

• key (string) – key for the value

get_last_generation_id(taskmanager_name, taskmanager_id=None)

Return last generation id for current task manager or taskmanager w/ task_manager_id.

Parameters:

• taskmanager_name (string) – task manager name

• taskmanager_id (string) – task manager id

get_metadata(taskmanager_id, generation_id, key)

Return the metadata from the metadata table for the given taskmanager_id, generation_id, key

Parameters:

• taskmanager_id (string) – taskmanager_id for generation to be retrieved

• generation_id (int) – generation_id of the data

• key (string) – key for the value

get_schema(table=None)

Given the table name return it’s schema

Parameters:

table (string) – Name of the table

get_taskmanager(taskmanager_name, taskmanager_id=None)

Retrieve TaskManager :type taskmanager_name: string :arg taskmanager_name: name of taskmanager to retrieve :type taskmanager_id: string :arg taskmanager_id: id of taskmanager to retrieve

get_taskmanagers(taskmanager_name=None, start_time=None, end_time=None)

Retrieve TaskManagers :type taskmanager_name: string :arg taskmanager_name: name of taskmanager to retrieve :type taskmanager_id: string :arg taskmanager_id: id of taskmanager to retrieve

insert(taskmanager_id, generation_id, key, value, header, metadata)

Insert data into respective tables for the given taskmanager_id, generation_id, key

Parameters:

• taskmanager_id (string) – taskmanager_id for generation to be retrieved

• generation_id (int) – generation_id of the data

• key (string) – key for the value

• value (object) – Value can be an object or dict

• header (Header) – Header for the value

• header – Metadata for the value

reset_connections()

Drop any cached connections and reconnect to the database

store_taskmanager(name, taskmanager_id, datestamp=None)

Store TaskManager :type taskmanager_name: string :arg taskmanager_name: name of taskmanager to retrieve :type taskmanager_id: string :arg taskmanager_id: id of taskmanager to retrieve :type datestamp: datetime :arg datestamp: datetime of created object, defaults to ‘now’

update(taskmanager_id, generation_id, key, value, header, metadata)

Update the data in respective tables for the given taskmanager_id, generation_id, key

Parameters:

• taskmanager_id (string) – taskmanager_id for generation to be retrieved

• generation_id (int) – generation_id of the data

• key (string) – key for the value

• value (object) – Value can be an object or dict

• header (Header) – Header for the value

• header – Metadata for the value

Module contents