cloudmarker.clouds package

A package for cloud plugins packaged with this project.

This package contains cloud plugins that are packaged as part of this project. The cloud plugins implement a function named read() that connects to remote data sources, typically cloud APIs, and yield data records.

Submodules

cloudmarker.clouds.azcloud module

Microsoft Azure cloud plugin to read Azure infrastructure data.

This module defines the AzCloud class that retrieves data from Microsoft Azure.

class cloudmarker.clouds.azcloud.AzCloud(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure cloud plugin.

Create an instance of AzCloud plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of processes to launch.
  • threads (int) – Number of threads to launch in each process.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of records of each type to fetch under each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure cloud infrastructure configuration record.

Yields:dict – An Azure cloud infrastructure configuration record.

cloudmarker.clouds.azdisk module

Microsoft Azure disk plugin to read Azure disk data.

This module defines the AzDisk class that retrieves disk from Microsoft Azure.

class cloudmarker.clouds.azdisk.AzDisk(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure disk plugin.

Create an instance of AzDisk plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of worker processes to run.
  • threads (int) – Number of worker threads to run.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of Postgres records to fetch for each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure disk record.

Yields:dict – An Azure disk record.

cloudmarker.clouds.azkv module

Microsoft Azure Key Vault plugin to read Key Vault and associated resources.

This module defines the AzKV class that retrieves Key Vault from Microsoft Azure. This module also retrieves the keys and secret attributes stored within a Key Vault.

class cloudmarker.clouds.azkv.AzKV(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure Key Vault plugin.

Create an instance of AzKV plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of worker processes to run.
  • threads (int) – Number of worker threads to run.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of Key Vault records to fetch for each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure Key Vault record.

Yields:dict – An Azure Key Vault and associated resource record.

cloudmarker.clouds.azmonitor module

Microsoft Azure monitor plugin to read Azure monitoring data.

This module defines the AzMonitor class that retrieves data from Microsoft Azure.

class cloudmarker.clouds.azmonitor.AzMonitor(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure monitor plugin.

Create an instance of AzMonitor plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of processes to launch.
  • threads (int) – Number of threads to launch in each process.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of records of each type to fetch under each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure monitor record.

Yields:dict – An Azure monitor record.

cloudmarker.clouds.azpostgres module

Microsoft Azure Postgres plugin to read Azure Postgres data.

This module defines the AzPostgres class that retrieves Postgre SQL data from Microsoft Azure.

class cloudmarker.clouds.azpostgres.AzPostgres(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure Postgres plugin.

Create an instance of AzPostgres plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of worker processes to run.
  • threads (int) – Number of worker threads to run.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of Postgres records to fetch for each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure Postgres record.

Yields:dict – An Azure Postgres record.

cloudmarker.clouds.azsql module

Microsoft Azure SQL Database plugin to read Azure SQL DB data.

This module defines the AzSQL class that retrieves SQL DB from Microsoft Azure. This module also retrieves the Transparent Data Encryption (TDE) configuration of the SQL database.

class cloudmarker.clouds.azsql.AzSQL(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure SQL Database plugin.

Create an instance of AzSQL plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of worker processes to run.
  • threads (int) – Number of worker threads to run.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of SQL server records to fetch for each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure SQL database record.

Yields:dict – An Azure SQL database record.

cloudmarker.clouds.azstorageaccount module

Microsoft Azure storage accounts plugin to read Azure storage accounts data.

This module defines the AzStorageAccount class that retrieves storage accounts data from Microsoft Azure.

class cloudmarker.clouds.azstorageaccount.AzStorageAccount(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure storage account plugin.

Create an instance of AzStorageAccount plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of worker processes to run.
  • threads (int) – Number of worker threads to run.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of storage accounts records to fetch for each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure storage account record.

Yields:dict – An Azure storage account record.

cloudmarker.clouds.azvm module

Microsoft Azure virtual machine plugin to read Azure virtual machine data.

This module defines the AzVM class that retrieves virtula machine data from Microsoft Azure.

class cloudmarker.clouds.azvm.AzVM(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure Virtual Machine plugin.

Create an instance of AzVM plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of worker processes to run.
  • threads (int) – Number of worker threads to run.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of virtual machines records to fetch for each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure virtual machine record.

Yields:dict – An Azure virtual machine record.

cloudmarker.clouds.azwebapp module

Microsoft Azure web apps plugin to read Azure web app data.

This module defines the AzWebApp class that retrieves web apps data from Microsoft Azure.

class cloudmarker.clouds.azwebapp.AzWebApp(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)

Bases: object

Azure web app plugin.

Create an instance of AzWebApp plugin.

Note: The _max_subs and _max_recs arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.
Parameters:
  • tenant (str) – Azure subscription tenant ID.
  • client (str) – Azure service principal application ID.
  • secret (str) – Azure service principal password.
  • processes (int) – Number of worker processes to run.
  • threads (int) – Number of worker threads to run.
  • _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
  • _max_recs (int) – Maximum number of web apps records to fetch for each subscription.
done()

Log a message that this plugin is done.

read()

Return an Azure web app record.

Yields:dict – An Azure web app record.

cloudmarker.clouds.gcpcloud module

Google Cloud Platform (GCP) plugin to read GCP infrastructure data.

This module defines the GCPCloud class that retrieves data from Google Cloud Platform.

class cloudmarker.clouds.gcpcloud.GCPCloud(key_file_path, processes=4, threads=30, _max_projects=0)

Bases: object

GCP cloud plugin.

Create an instance of GCPCloud plugin.

Note: The _max_projects argument should be used only in the development-test-debug phase. It should not be used in production environment. This is why we use the convention of beginning it’s name with underscore.
Parameters:
  • key_file_path (str) – Path of the service account key file for a project.
  • processes (int) – Number of processes to launch.
  • threads (int) – Number of threads to launch in each process.
  • _max_projects (int) – Maximum number of projects to fetch data for if the value is greater than 0.
done()

Log a message that this plugin is done.

read()

Return a GCP cloud infrastructure configuration record.

Yields:dict – A GCP cloud infrastructure configuration record.

cloudmarker.clouds.mockcloud module

Mock cloud plugin for testing purpose.

class cloudmarker.clouds.mockcloud.MockCloud(record_count=10, record_types=('foo', 'bar'))

Bases: object

Mock cloud plugin for testing purpose.

Create an instance of MockCloud plugin.

This plugin generates mock records. The records generated contains three fields under three top-level keys that we also call “bucket keys”: raw, data, and type, as shown in the example below:

Example

Here is an example that shows that the records generated by this plugin with the default initialization parameters:

>>> from cloudmarker.clouds import mockcloud
>>> cloud = mockcloud.MockCloud()
>>> for record in cloud.read():
...     print(record['raw']['data'],
...           record['ext']['record_type'],
...           record['com']['record_type'])
...
0 foo mock
1 bar mock
2 foo mock
3 bar mock
4 foo mock
5 bar mock
6 foo mock
7 bar mock
8 foo mock
9 bar mock

The three top-level keys, raw, ext, and com represent the names of the three buckets under which various data attributes are kept. While this is only a mock plugin, but in an actual cloud plugin implementation, the meaning of these buckets are as follows:

  • raw: The value for the raw key is a dict object that represents the actual data object obtained from a cloud in its original form. No modifications should be done to the object obtained from the cloud.
  • ext: The value for the ext key is a dict object which contains key-value pairs for any additional cloud-specific metadata that need to be stored. The data in this bucket is also known as extended metadata.
  • com: The value for the com key is a dict object which contains key-value pairs for any metadata that is common to all clouds.
Parameters:
  • record_count (int) – Number of mock records to generate.
  • record_types (tuple) – A tuple of strings that represent the different record types to be generated.
done()

Perform cleanup work.

Since this is a mock plugin, this method does nothing. However, a typical cloud plugin may or may not need to perform cleanup work in this method depending on its nature of work.

read()

Generate a record.

This method creates and yields mock records.

In actual cloud implementations, this method would typically connect to the cloud, retrieve JSON objects using the cloud API, and yield those objects as dict objects.

Yields:dict – Mock record.