api2db.app package

Submodules

api2db.app.api2db module

Contains the Api2Db class

api2db.app.api2db.DEV_SHRINK_DATA = 0

Library developer setting to shrink incoming data to the first DEV_SHRINK_DATA rows

Type

int

class api2db.app.api2db.Api2Db(collector: api2db.ingest.collector.Collector)

Bases: object

Performs data import, passes data to streams, and schedules data storage

__init__(collector: api2db.ingest.collector.Collector)

Creates a Api2Db object and attaches the collector

Parameters

collector – The collector object to attach to

wrap_start() → multiprocessing.context.Process

Starts the running loop of an Api2Db instance in a spawned process

Returns

The process spawned with target start

start() → None

The target for Api2Db main process running loop

Returns

None

schedule() → None

schedule starts the streams, schedules collector refresh, schedules storage refresh

Returns

None

Raises

NameError or ModuleNotFoundError if streams cannot be created –

static collect_wrap(import_target: Callable[], Optional[List[dict]]], api_form: Callable[], api2db.ingest.api_form.ApiForm], stream_qs: List[queue.Queue], stream_locks: List[_thread.allocate_lock]) → Optional[type]

Starts/restarts dead streams, and calls method collect to import data

Parameters

  • import_target – Function that returns data imported from an Api

  • api_form – Function that instantiates and returns an ApiForm object

  • stream_qs – A list of queues to pass the incoming data into to be handled by stream targets

  • stream_locks – A list of locks that become acquirable if their respective stream has died

Returns

CancelJob if stream has died, restarting the streams, None otherwise

static collect(import_target: Callable[], Optional[List[dict]]], api_form: Callable[], api2db.ingest.api_form.ApiForm], stream_qs: List[queue.Queue]) → None

Performs a data-import, cleans the data, and sends the data into its stream queues

Parameters

  • import_target – Function that returns data imported from an Api

  • api_form – Function that instantiates and returns an ApiForm object

  • stream_qs – A list of queues to pass the incoming data into to be handled by stream targets

Returns

None

static store_wrap(stores: Callable[], List[api2db.store.store.Store]]) → None

Checks to ensure that storage jobs are scheduled to run and schedules any jobs that have been unscheduled

Parameters

stores – Function that returns a list of Store subclassed objects

Returns

None

Raises

NameError or ModuleNotFoundError if stores cannot be created –

static store(store: api2db.store.store.Store) → None

Performs the data storage operation of a Store subclass

Parameters

store – The Store to perform storage on

Returns

None

static import_handle(e: Exception) → Exception

Handles import errors. Informs the user of libraries they need

Parameters

e – The raised Exception

Returns

ModuleNotFoundError if dependencies missing otherwise the original exception

api2db.app.auth_manager module

Contains the auth_manage function

api2db.app.auth_manager.auth_manage(path: str) → Optional[dict]

Loads authentication credentials from the specified path

Parameters

path – The path where the authentication file resides

Returns

Authentication credentials if file successfully loaded, None otherwise

api2db.app.log module

Contains the get_logger function

api2db.app.log.get_logger(filename: Optional[str] = None, q: Optional[multiprocessing.context.BaseContext.Queue] = None) → logging.Logger

Retrieves the logger for the current process for logging to the log file

If no filename is provided, the logger for the current process is assumed to already have handlers registered, and will be returned.

If a filename is provided an the logger has no handlers, a handler will be created and registered

Parameters

  • filename – The name of the file to log to

  • q – The queue used to pass messages if the collector is running in debug mode

Returns

A logger that can be used to log messages

api2db.app.run module

Contains the Run class

class api2db.app.run.Run(collectors: List[api2db.ingest.collector.Collector])

Bases: object

Serves as the main entry point for the application

__init__(collectors: List[api2db.ingest.collector.Collector])

The Run object is the application entry point

Parameters

collectors – A list of collector objects to collect data for

q

Used for message passing for collectors with debug mode enabled

Type

multiprocessing.Queue

run()

Starts the application

Returns

None

multiprocessing_start()

Starts each collector in it’s own process

Returns

None

Module contents

Original Author

Tristen Harr

Creation Date

04/27/2021

Revisions

None