cloudmarker.stores package

A package for store plugins packaged with this project.

This package contains store plugins that are packaged as part of this project. The store plugins implement a function named write() that accepts input records and typically stores them into a persistent data store. The event plugins also implement and a function named done that perform cleanup work when called.

Submodules

cloudmarker.stores.esstore module

cloudmarker.stores.filestore module

Filesystem store plugin.

class cloudmarker.stores.filestore.FileStore(path='/tmp/cloudmarker')

Bases: object

A plugin to store records on the filesystem.

Create an instance of FileStore plugin.

Parameters:path (str) – Path of directory where files are written to.
done()

Perform final cleanup tasks.

This method is called after all records have been written. In this example implementation, we properly terminate the JSON array in the .tmp file. Then we rename the .tmp file to .json file.

Note that other implementations of a store may perform tasks like closing a connection to a remote store or flushing any remaining records in a buffer.

write(record)

Write JSON records to the file system.

This method is called once for every record read from a cloud. In this example implementation of a store, we simply write the record in JSON format to a file. The list of records is maintained as JSON array in the file. The origin worker name in record['com']['origin_worker'] is used to determine the filename.

The records are written to a .tmp file because we don’t want to delete the existing complete and useful .json file prematurely.

Note that other implementations of a store may choose to buffer the records in memory instead of writing each record to the store immediately. They may then flush the buffer to the store based on certain conditions such as buffer size, time interval, etc.

Parameters:record (dict) – Data to write to the file system.

cloudmarker.stores.mongodbstore module

cloudmarker.stores.splunkhecstore module

SplunkStore plugin to index data in Splunk using HEC token.

class cloudmarker.stores.splunkhecstore.SplunkHECStore(uri, token, index, ca_cert, buffer_size=1000)

Bases: object

SplunkHECStore plugin to index cloud data in Splunk using HEC token.

Create an instance of SplunkHECStore plugin.

Parameters:
  • uri (str) – Splunk collector service URI.
  • token (str) – Splunk HEC token.
  • index (str) – Splunk HEC token accessible index.
  • ca_cert (str) – Location of cetificate file to verify the identity of host in URI, or False to disable verification
  • buffer_size (int) – Maximum number of records to hold in in-memory buffer for each record type.
done()

Flush any remaining records.

write(record)

Save the record in a bulk-buffer.

Also, flush the buffer by saving its content to Splunk when the buffer size exceeds configured self._buffer_size

Parameters:record (dict) – Data to save to the Splunk.