cloudmarker.stores package¶
A package for store plugins packaged with this project.
This package contains store plugins that are packaged as part of this
project. The store plugins implement a function named write()
that
accepts input records and typically stores them into a persistent data
store. The event plugins also implement and a function named done
that perform cleanup work when called.
Submodules¶
cloudmarker.stores.esstore module¶
cloudmarker.stores.filestore module¶
Filesystem store plugin.
-
class
cloudmarker.stores.filestore.
FileStore
(path='/tmp/cloudmarker')¶ Bases:
object
A plugin to store records on the filesystem.
Create an instance of
FileStore
plugin.Parameters: path (str) – Path of directory where files are written to. -
done
()¶ Perform final cleanup tasks.
This method is called after all records have been written. In this example implementation, we properly terminate the JSON array in the .tmp file. Then we rename the .tmp file to .json file.
Note that other implementations of a store may perform tasks like closing a connection to a remote store or flushing any remaining records in a buffer.
-
write
(record)¶ Write JSON records to the file system.
This method is called once for every
record
read from a cloud. In this example implementation of a store, we simply write therecord
in JSON format to a file. The list of records is maintained as JSON array in the file. The origin worker name inrecord['com']['origin_worker']
is used to determine the filename.The records are written to a
.tmp
file because we don’t want to delete the existing complete and useful.json
file prematurely.Note that other implementations of a store may choose to buffer the records in memory instead of writing each record to the store immediately. They may then flush the buffer to the store based on certain conditions such as buffer size, time interval, etc.
Parameters: record (dict) – Data to write to the file system.
-
cloudmarker.stores.mongodbstore module¶
cloudmarker.stores.splunkhecstore module¶
SplunkStore plugin to index data in Splunk using HEC token.
-
class
cloudmarker.stores.splunkhecstore.
SplunkHECStore
(uri, token, index, ca_cert, buffer_size=1000)¶ Bases:
object
SplunkHECStore plugin to index cloud data in Splunk using HEC token.
Create an instance of
SplunkHECStore
plugin.Parameters: - uri (str) – Splunk collector service URI.
- token (str) – Splunk HEC token.
- index (str) – Splunk HEC token accessible index.
- ca_cert (str) – Location of cetificate file to verify the identity of host in URI, or False to disable verification
- buffer_size (int) – Maximum number of records to hold in in-memory buffer for each record type.
-
done
()¶ Flush any remaining records.