panko.storage package¶

Subpackages¶

Submodules¶

panko.storage.base module¶

Base classes for storage engines

class panko.storage.base.Connection(conf)¶

Bases: object

Base class for event storage system connections.

CAPABILITIES = {'events': {'query': {'simple': False}}}¶

STORAGE_CAPABILITIES = {'storage': {'production_ready': False}}¶

static clear()¶

Clear database.

static clear_expired_data(ttl, max_count=None)¶

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live. :param ttl: Number of seconds to keep records for. :param max_count: Number of records to delete.

classmethod get_capabilities()¶

Return an dictionary with the capabilities of each driver.

static get_event_types()¶

Return all event types as an iterable of strings.

static get_events(event_filter, pagination=None)¶

Return an iterable of model.Event objects.

classmethod get_storage_capabilities()¶

Return a dictionary representing the performance capabilities.

This is needed to evaluate the performance of each driver.

static get_trait_types(event_type)¶

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned. :param event_type: the type of the Event

static get_traits(event_type, trait_type=None)¶

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

static record_events(events)¶

Write the events to the backend storage system.

Parameters

events – a list of model.Event objects.

static upgrade()¶

Migrate the database to version or the most recent version.

class panko.storage.base.Model(**kwds)¶

Bases: object

Base class for storage API models.

as_dict()¶

panko.storage.impl_elasticsearch module¶

class panko.storage.impl_elasticsearch.Connection(url, conf)¶

Bases: panko.storage.base.Connection

Put the event data into an ElasticSearch db.

Events in ElasticSearch are indexed by day and stored by event_type. An example document:

{"_index":"events_2014-10-21",
 "_type":"event_type0",
 "_id":"dc90e464-65ab-4a5d-bf66-ecb956b5d779",
 "_score":1.0,
 "_source":{"timestamp": "2014-10-21T20:02:09.274797"
            "traits": {"id4_0": "2014-10-21T20:02:09.274797",
                       "id3_0": 0.7510790937279408,
                       "id2_0": 5,
                       "id1_0": "18c97ba1-3b74-441a-b948-a702a30cbce2"}
           }
}

CAPABILITIES = {'events': {'query': {'simple': True}}}¶

STORAGE_CAPABILITIES = {'storage': {'production_ready': True}}¶

get_event_types()¶

Return all event types as an iterable of strings.

get_events(event_filter, pagination=None)¶

Return an iterable of model.Event objects.

get_trait_types(event_type)¶

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned. :param event_type: the type of the Event

get_traits(event_type, trait_type=None)¶

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

record_events(events)¶

Write the events to the backend storage system.

Parameters

events – a list of model.Event objects.

upgrade()¶

Migrate the database to version or the most recent version.

panko.storage.impl_log module¶

class panko.storage.impl_log.Connection(conf)¶

Bases: panko.storage.base.Connection

Log event data.

static clear_expired_data(ttl, max_count)¶

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live.

Parameters

• ttl – Number of seconds to keep records for.

• max_count – Number of records to delete.

panko.storage.impl_mongodb module¶

MongoDB storage backend

class panko.storage.impl_mongodb.Connection(url, conf)¶

Bases: panko.storage.pymongo_base.Connection

Put the event data into a MongoDB database.

CONNECTION_POOL = <panko.storage.mongo.utils.ConnectionPool object>¶

clear()¶

Clear database.

clear_expired_data(ttl, max_count=None)¶

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live.

Parameters

• ttl – Number of seconds to keep records for.

• max_count – Number of records to delete (not used for MongoDB).

static update_ttl(ttl, ttl_index_name, index_field, coll)¶

Update or create time_to_live indexes.

Parameters

• ttl – time to live in seconds.

• ttl_index_name – name of the index we want to update or create.

• index_field – field with the index that we need to update.

• coll – collection which indexes need to be updated.

upgrade()¶

Migrate the database to version or the most recent version.

panko.storage.impl_sqlalchemy module¶

SQLAlchemy storage backend.

class panko.storage.impl_sqlalchemy.Connection(url, conf)¶

Bases: panko.storage.base.Connection

Put the event data into a SQLAlchemy database.

Tables:

- EventType
  - event definition
  - { id: event type id
      desc: description of event
      }
- Event
  - event data
  - { id: event id
      message_id: message id
      generated = timestamp of event
      event_type_id = event type -> eventtype.id
      }
- TraitInt
  - int trait value
  - { event_id: event -> event.id
      key: trait name
      value: integer value
      }
- TraitDatetime
  - datetime trait value
  - { event_id: event -> event.id
      key: trait name
      value: datetime value
      }
- TraitText
  - text trait value
  - { event_id: event -> event.id
      key: trait name
      value: text value
      }
- TraitFloat
  - float trait value
  - { event_id: event -> event.id
      key: trait name
      value: float value
      }

CAPABILITIES = {'events': {'query': {'simple': True}}}¶

STORAGE_CAPABILITIES = {'storage': {'production_ready': True}}¶

clear()¶

Clear database.

clear_expired_data(ttl, max_count)¶

Clear expired data from the backend storage system.

Clearing occurs according to the time-to-live.

Parameters

• ttl – Number of seconds to keep records for.

• max_count – Number of records to delete.

static dress_url(url)¶

get_event_types()¶

Return all event types as an iterable of strings.

get_events(event_filter, pagination=None)¶

Return an iterable of model.Event objects.

Parameters

• event_filter – EventFilter instance

• pagination – Pagination parameters.

get_trait_types(event_type)¶

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned. :param event_type: the type of the Event

get_traits(event_type, trait_type=None)¶

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type. :param event_type: the type of the Event to filter by :param trait_type: the name of the Trait to filter by

record_events(event_models)¶

Write the events to SQL database via sqlalchemy.

Parameters

event_models – a list of model.Event objects.

upgrade()¶

Migrate the database to version or the most recent version.

panko.storage.models module¶

Model classes for use in the events storage API.

class panko.storage.models.Event(message_id, event_type, generated, traits, raw)¶

Bases: panko.storage.base.Model

A raw event from the source system. Events have Traits.

Metrics will be derived from one or more Events.

DEFAULT_DIR = 'asc'¶

DEFAULT_SORT = [('generated', 'asc'), ('message_id', 'asc')]¶

DUPLICATE = 1¶

INCOMPATIBLE_TRAIT = 3¶

PRIMARY_KEY = 'message_id'¶

SUPPORT_DIRS = ('asc', 'desc')¶

SUPPORT_SORT_KEYS = ('message_id', 'generated')¶

UNKNOWN_PROBLEM = 2¶

append_trait(trait_model)¶

serialize()¶

class panko.storage.models.Trait(name, dtype, value)¶

Bases: panko.storage.base.Model

A Trait is a key/value pair of data on an Event.

The value is variant record of basic data types (int, date, float, etc).

DATETIME_TYPE = 4¶

FLOAT_TYPE = 3¶

INT_TYPE = 2¶

NONE_TYPE = 0¶

TEXT_TYPE = 1¶

classmethod convert_value(trait_type, value)¶

classmethod get_name_by_type(type_id)¶

classmethod get_type_by_name(type_name)¶

get_type_name()¶

classmethod get_type_names()¶

serialize()¶

type_names = {0: 'none', 1: 'string', 2: 'integer', 3: 'float', 4: 'datetime'}¶

panko.storage.models.serialize_dt(value)¶

Serializes parameter if it is datetime.

panko.storage.pymongo_base module¶

Common functions for MongoDB backend

class panko.storage.pymongo_base.Connection(conf)¶

Bases: panko.storage.base.Connection

Base event Connection class for MongoDB driver.

CAPABILITIES = {'events': {'query': {'simple': True}}}¶

STORAGE_CAPABILITIES = {'storage': {'production_ready': True}}¶

get_event_types()¶

Return all event types as an iter of strings.

get_events(event_filter, pagination=None)¶

Return an iter of models.Event objects.

Parameters

• event_filter – storage.EventFilter object, consists of filters for events that are stored in database.

• pagination – Pagination parameters.

get_trait_types(event_type)¶

Return a dictionary containing the name and data type of the trait.

Only trait types for the provided event_type are returned.

Parameters

event_type – the type of the Event.

get_traits(event_type, trait_name=None)¶

Return all trait instances associated with an event_type.

If trait_type is specified, only return instances of that trait type.

Parameters

• event_type – the type of the Event to filter by

• trait_name – the name of the Trait to filter by

record_events(event_models)¶

Write the events to database.

Parameters

event_models – a list of models.Event objects.

Module contents¶

Storage backend management

class panko.storage.EventFilter(start_timestamp=None, end_timestamp=None, event_type=None, message_id=None, traits_filter=None, admin_proj=None)¶

Bases: object

Properties for building an Event query.

Parameters

• start_timestamp – UTC start datetime (mandatory)

• end_timestamp – UTC end datetime (mandatory)

• event_type – the name of the event. None for all.

• message_id – the message_id of the event. None for all.

• admin_proj – the project_id of admin role. None if non-admin user.

• traits_filter – the trait filter dicts, all of which are optional. This parameter is a list of dictionaries that specify trait values:

{'key': <key>,
'string': <value>,
'integer': <value>,
'datetime': <value>,
'float': <value>,
'op': <eq, lt, le, ne, gt or ge> }

exception panko.storage.InvalidMarker¶

Bases: Exception

Invalid pagination marker parameters

exception panko.storage.StorageBadAggregate¶

Bases: Exception

Error raised when an aggregate is unacceptable to storage backend.

code = 400¶

exception panko.storage.StorageBadVersion¶

Bases: Exception

Error raised when the storage backend version is not good enough.

exception panko.storage.StorageUnknownWriteError¶

Bases: Exception

Error raised when an unknown error occurs while recording.

panko.storage.get_connection(url, conf)¶

Return an open connection to the database.

panko.storage.get_connection_from_config(conf)¶