cloudmarker package¶

Cloudmarker - Cloud security monitoring framework.

Submodules¶

cloudmarker.baseconfig module¶

Base configuration.

cloudmarker.baseconfig.config_yaml¶

Base configuration as YAML code.

Type:	str

cloudmarker.baseconfig.config_dict¶

Base configuration as Python dictionary.

Type:	dict

Here is the complete base configuration present as a string in the config_yaml attribute:

# Base configuration
plugins:
  mockcloud:
    plugin: cloudmarker.clouds.mockcloud.MockCloud

  filestore:
    plugin: cloudmarker.stores.filestore.FileStore

  esstore:
    plugin: cloudmarker.stores.esstore.EsStore

  mongodbstore:
    plugin: cloudmarker.stores.mongodbstore.MongoDBStore

  firewallruleevent:
    plugin: cloudmarker.events.firewallruleevent.FirewallRuleEvent

  azvmosdiskencryptionevent:
    plugin: cloudmarker.events.azvmosdiskencryptionevent.AzVMOSDiskEncryptionEvent

  azvmdatadiskencryptionevent:
    plugin: cloudmarker.events.azvmdatadiskencryptionevent.AzVMDataDiskEncryptionEvent

  mockevent:
    plugin: cloudmarker.events.mockevent.MockEvent

audits:
  mockaudit:
    clouds:
      - mockcloud
    stores:
      - filestore
    events:
      - mockevent
    alerts:
      - filestore

run:
  - mockaudit

logger:
  version: 1

  disable_existing_loggers: false

  formatters:
    simple:
      format: >-
          %(asctime)s [%(process)s] %(levelname)s
          %(name)s:%(lineno)d - %(message)s
      datefmt: "%Y-%m-%d %H:%M:%S"

  handlers:
    console:
      class: logging.StreamHandler
      formatter: simple
      stream: ext://sys.stdout

    file:
      class: logging.handlers.TimedRotatingFileHandler
      formatter: simple
      filename: /tmp/cloudmarker.log
      when: midnight
      encoding: utf8
      backupCount: 5

  loggers:
    adal-python:
      level: WARNING

  root:
    level: INFO
    handlers:
      - console
      - file

schedule: "00:00"

cloudmarker.manager module¶

Manager of worker subprocesses.

This module invokes the worker subprocesses that perform the cloud security monitoring tasks. Each worker subprocess wraps around a cloud, store, event, or alert plugin and executes the plugin in a separate subprocess.

class cloudmarker.manager.Audit(audit_key, audit_version, config)¶

Bases: object

Audit manager.

This class encapsulates a set of worker subprocesses and worker input queues for a single audit configuration.

Create an instance of Audit from configuration.

A single audit definition (from a list of audit definitions under the audits key in the configuration) is instantiated. Each audit definition contains lists of cloud plugins, store plugins, event plugins, and alert plugins. These plugins are instantiated and multiprocessing queues are set up to take records from one plugin and feed them to another plugin as per the audit workflow.

Parameters:	audit_key (str) – Key name for an audit configuration. This key is looked for in `config['audits']`. audit_version (str) – Audit version string. config (dict) – Configuration dictionary. This is the entire configuration dictionary that contains top-level keys named `clouds`, `stores`, `events`, `alerts`, `audits`, `run`, etc.

join()¶: Wait until all workers terminate.

start()¶: Start audit by starting all workers.

cloudmarker.manager.main()¶: Run the framework based on the schedule.

cloudmarker.util module¶

Utility functions.

exception cloudmarker.util.PluginError¶

Bases: Exception

Represents an error while loading a plugin.

exception cloudmarker.util.PluralizeError¶

Bases: Exception

Represents an error while converting a word to plural form.

cloudmarker.util.expand_port_ranges(port_ranges)¶

Expand port_ranges to a set of ports.

Examples

Here is an example usage of this function:

>>> from cloudmarker import util
>>> ports = util.expand_port_ranges(['22', '3389', '8080-8085'])
>>> print(ports == {22, 3389, 8080, 8081, 8082, 8083, 8084, 8085})
True
>>> ports = util.expand_port_ranges(['8080-8084', '8082-8086'])
>>> print(ports == {8080, 8081, 8082, 8083, 8084, 8085, 8086})
True

Note that in a port range of the form m-n, both m and n are included in the expanded port set. If m > n, we get an empty port set.

>>> ports = util.expand_port_ranges(['8085-8080'])
>>> print(ports == set())
True

If an invalid port range is found, it is ignored.

>>> ports = util.expand_port_ranges(['8080', '8081a', '8082'])
>>> print(ports == {8080, 8082})
True
>>> ports = util.expand_port_ranges(['7070-7075', '8080a-8085'])
>>> print(ports == {7070, 7071, 7072, 7073, 7074, 7075})
True

Parameters:	port_ranges (list) – A list of strings where each string is a port number (e.g., `'80'`) or port range (e.g., `80-89`).
Returns:	A set of integers that represent the ports specified by `port_ranges`.
Return type:	set

cloudmarker.util.friendly_list(items, conjunction='and')¶

Translate a list of items to a human-friendly list of items.

Examples

Here are a few example usages of this function:

>>> from cloudmarker import util
>>> util.friendly_list([])
'none'
>>> util.friendly_list(['apple'])
'apple'
>>> util.friendly_list(['apple', 'ball'])
'apple and ball'
>>> util.friendly_list(['apple', 'ball', 'cat'])
'apple, ball, and cat'
>>> util.friendly_list(['apple', 'ball'], 'or')
'apple or ball'
>>> util.friendly_list(['apple', 'ball', 'cat'], 'or')
'apple, ball, or cat'

Parameters:	items (list) – List of items.
Returns:	Human-friendly list of items with correct placement of comma and conjunction.
Return type:	str

cloudmarker.util.friendly_string(technical_string)¶

Translate a technical string to a human-friendly phrase.

In most of our code, we use succint strings to express various technical details, e.g., 'gcp' to express Google Cloud Platform. However these technical strings are not ideal while writing human-friendly messages such as a description of a security issue detected or a recommendation to remediate such an issue.

This function helps in converting such technical strings into human-friendly phrases that can be used in strings intended to be read by end users (e.g., security analysts responsible for protecting their cloud infrastructure) of this project.

Examples

Here are a few example usages of this function:

>>> from cloudmarker import util
>>> util.friendly_string('azure')
'Azure'
>>> util.friendly_string('gcp')
'Google Cloud Platform (GCP)'

Parameters:	technical_string (str) – A technical string.
Returns:	Human-friendly string if a translation from a technical string to friendly string exists; the same string otherwise.
Return type:	str

cloudmarker.util.load_config(config_paths)¶

Load configuration from specified configuration paths.

Parameters:	config_paths (list) – Configuration paths.
Returns:	A dictionary of configuration key-value pairs.
Return type:	dict

cloudmarker.util.load_plugin(plugin_config)¶

Construct an object with specified plugin class and parameters.

The plugin_config parameter must be a dictionary with the following keys:

plugin: The value for this key must be a string that represents the fully qualified class name of the plugin. The fully qualified class name is in the dotted notation, e.g., pkg.module.ClassName.
params: The value for this key must be a dict that represents the parameters to be passed to the __init__ method of the plugin class. Each key in the dictionary represents the parameter name and each value represents the value of the parameter.

Example

Here is an example usage of this function:

>>> from cloudmarker import util
>>> plugin_config = {
...     'plugin': 'cloudmarker.clouds.mockcloud.MockCloud',
...     'params': {
...         'record_count': 4,
...         'record_types': ('baz', 'qux')
...     }
... }
...
>>> plugin = util.load_plugin(plugin_config)
>>> print(type(plugin))
<class 'cloudmarker.clouds.mockcloud.MockCloud'>
>>> for record in plugin.read():
...     print(record['raw']['data'],
...           record['ext']['record_type'],
...           record['com']['record_type'])
...
0 baz mock
1 qux mock
2 baz mock
3 qux mock

Parameters:	plugin_config (dict) – Plugin configuration dictionary.
Returns:	An object of type mentioned in the `plugin` parameter.
Return type:	object
Raises:	`PluginError` – If plugin class name is invalid.

cloudmarker.util.merge_dicts(*dicts)¶

Recursively merge dictionaries.

The input dictionaries are not modified. Given any number of dicts, deep copy and merge into a new dict, precedence goes to key value pairs in latter dicts.

Example

Here is an example usage of this function:

>>> from cloudmarker import util
>>> a = {'a': 'apple', 'b': 'ball'}
>>> b = {'b': 'bat', 'c': 'cat'}
>>> c = util.merge_dicts(a, b)
>>> print(c == {'a': 'apple', 'b': 'bat', 'c': 'cat'})
True

Parameters:	dicts (dict*) – Variable length dictionary list
Returns:	Merged dictionary
Return type:	dict

cloudmarker.util.parse_cli(args=None)¶

Parse command line arguments.

Parameters:	args (list) – List of command line arguments.
Returns:	Parsed command line arguments.
Return type:	argparse.Namespace

cloudmarker.util.pluralize(count, word, *suffixes)¶

Convert word to plural form if count is not 1.

Examples

In the simplest form usage, this function just adds an 's' to the input word when the plural form needs to be used.

>>> from cloudmarker import util
>>> util.pluralize(0, 'apple')
'apples'
>>> util.pluralize(1, 'apple')
'apple'
>>> util.pluralize(2, 'apple')
'apples'

The plural form of some words cannot be formed merely by adding an 's' to the word but requires adding a different suffix. For such cases, provide an additional argument that specifies the correct suffix.

>>> util.pluralize(0, 'potato', 'es')
'potatoes'
>>> util.pluralize(1, 'potato', 'es')
'potato'
>>> util.pluralize(2, 'potato', 'es')
'potatoes'

The plural form of some words cannot be formed merely by adding a suffix but requires removing a suffix and then adding a new suffix. For such cases, provide two additional arguments: one that specifies the suffix to remove from the input word and another to specify the suffix to add.

>>> util.pluralize(0, 'sky', 'y', 'ies')
'skies'
>>> util.pluralize(1, 'sky', 'y', 'ies')
'sky'
>>> util.pluralize(2, 'sky', 'y', 'ies')
'skies'

Returns:	The input `word` itself if `count` is `1`; plural form of the `word` otherwise.
Return type:	str

cloudmarker.util.send_email(from_addr, to_addrs, subject, content, host='', port=0, ssl_mode='ssl', username='', password='', debug=0)¶

Send email message.

When ssl_mode` is ``'ssl' and host is uspecified or specified as '' (the default), the local host is used. When ssl_mode is 'ssl' and port is unspecified or specified as 0, the standard SMTP-over-SSL port, i.e., port 465, is used. See smtplib.SMTP_SSL documentation for more details on this.

When ssl_mode is 'ssl'` and if ``host or port are unspecified, i.e., if host or port are '' and/or 0, respectively, the OS default behavior is used. See smtplib.SMTP documentation for more details on this.

We recommend these parameter values:

Leave ssl_mode unspecified (thus 'ssl' by default) if your SMTP server supports SSL.
Set ssl_mode to 'starttls' explicitly if your SMTP server does not support SSL but it supports STARTTLS.
Set ssl_mode to 'disable' explicitly if your SMTP server supports neither SSL nor STARTTLS.
Set host to the SMTP hostname or address explicitly.
Leave port unspecified (thus 0 by default), so that the appropriate port is chosen automatically.

With these recommendations, this function should do the right thing automatically, i.e., connect to port 465 if use_ssl is unspecified or False and port 25 if use_ssl is True.

Note that in case of SMTP, there are two different encryption protocols in use:

SSL/TLS (or implicit SSL/TLS): SSL/TLS is used from the beginning of the connection. This occurs typically on port 465. This is enabled by default (ssl_mode as 'ssl').
STARTTLS (or explicit SSL/TLS): The SMTP session begins as a plaintext session. Then the client (this function in this case) makes an explicit request to switch to SSL/TLS by sending the STARTTLS command to the server. This occurs typically on port 25 or port 587. Set ssl_mode to 'starttls' to enable this behaviour

If username is unspecified or specified as an empty string, no SMTP authentication is done. If username is specified as a non-empty string, then SMTP authentication is done.

Parameters:

from_addr (str) – Sender’s email address.
to_addrs (list) – A list of str objects where each str object is a recipient’s email address.
subject (str) – Email subject.
content (str) – Email content.
host (str) – SMTP host.
port (int) – SMTP port.
ssl_mode (str) – SSL mode to use: 'ssl' for SSL/TLS connection (the default), 'starttls' for STARTTLS, and 'disable' to disable SSL.
username (str) – SMTP username.
password (str) – SMTP password.
debug (int or bool) – Debug level to pass to SMTP.set_debuglevel() to debug an SMTP session. Set to 0 (the default) or False to disable debugging. Set to 1 or True to see SMTP messages. Set to 2 to see timestamped SMTP messages.

cloudmarker.util.wrap_paragraphs(text, width=70)¶

Wrap each paragraph in text to the specified width.

If the text is indented with any common leading whitespace, then that common leading whitespace is removed from every line in text. Further, any remaining leading and trailing whitespace is removed. Finally, each paragraph is wrapped to the specified width.

Parameters:	width (int) – Maximum length of wrapped lines.

cloudmarker.workers module¶

Worker functions.

The functions in this module wrap around plugin classes such that these worker functions can be specified as the target parameter while launching a new subprocess with multiprocessing.Process.

Each worker function can run as a separate subprocess. While wrapping around a plugin class, each worker function creates the multiprocessing queues necessary to pass records from one plugin class to another.

cloudmarker.workers.alert_worker(audit_key, audit_version, plugin_key, plugin, input_queue)¶

Worker function for alert plugins.

This function behaves like cloudmarker.workers.store_worker(). See its documentation for details.

Parameters:	audit_key (str) – Audit key name in configuration. audit_version (str) – Audit version string. plugin_key (str) – Plugin key name in configuration. plugin (object) – Alert plugin object. input_queue (multiprocessing.Queue) – Queue to read records from.

cloudmarker.workers.cloud_worker(audit_key, audit_version, plugin_key, plugin, output_queues)¶

Worker function for cloud plugins.

This function expects the plugin object to implement a read method that yields records. This function calls this read method to retrieve records and puts each record into each queue in output_queues.

Parameters:	audit_key (str) – Audit key name in configuration. audit_version (str) – Audit version string. plugin_key (str) – Plugin key name in configuration. plugin (object) – Cloud plugin object. output_queues (list) – List of `multiprocessing.Queue` objects to write records to.

cloudmarker.workers.event_worker(audit_key, audit_version, plugin_key, plugin, input_queue, output_queues)¶

Worker function for event plugins.

This function expects the plugin object to implement a eval method that accepts a single record as a parameter and yields one or more records, and a done method to perform cleanup work in the end.

This function gets records from input_queue and passes each record to the eval method of plugin. Then it puts each record yielded by the eval method into each queue in output_queues.

When there are no more records in the input_queue, i.e., once None is found in the input_queue, this function calls the done method of the plugin to indicate that record processing is over.

Parameters:	audit_key (str) – Audit key name in configuration. audit_version (str) – Audit version string. plugin_key (str) – Plugin key name in configuration. plugin (object) – Store plugin object. input_queue (multiprocessing.Queue) – Queue to read records from. output_queues (list) – List of `multiprocessing.Queue` objects to write records to.

cloudmarker.workers.store_worker(audit_key, audit_version, plugin_key, plugin, input_queue)¶

Worker function for store plugins.

This function expects the plugin object to implement a write method that accepts a single record as a parameter and a done method to perform cleanup work in the end.

This function gets records from input_queue and passes each record to the write method of plugin.

When there are no more records in the input_queue, i.e., once None is found in the input_queue, this function calls the done method of the plugin to indicate that record processing is over.

Parameters:	audit_key (str) – Audit key name in configuration. audit_version (str) – Audit version string. plugin_key (str) – Plugin key name in configuration. plugin (object) – Store plugin object. input_queue (multiprocessing.Queue) – Queue to read records from.

cloudmarker package¶

Subpackages¶

Submodules¶

cloudmarker.baseconfig module¶

cloudmarker.manager module¶

cloudmarker.util module¶

cloudmarker.workers module¶