cloudmarker.clouds package¶
A package for cloud plugins packaged with this project.
This package contains cloud plugins that are packaged as part of this
project. The cloud plugins implement a function named read()
that
connects to remote data sources, typically cloud APIs, and yield
data records.
Submodules¶
cloudmarker.clouds.azcloud module¶
Microsoft Azure cloud plugin to read Azure infrastructure data.
This module defines the AzCloud
class that retrieves data
from Microsoft Azure.
-
class
cloudmarker.clouds.azcloud.
AzCloud
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure cloud plugin.
Create an instance of
AzCloud
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of processes to launch.
- threads (int) – Number of threads to launch in each process.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of records of each type to fetch under each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure cloud infrastructure configuration record.
Yields: dict – An Azure cloud infrastructure configuration record.
cloudmarker.clouds.azdisk module¶
Microsoft Azure disk plugin to read Azure disk data.
This module defines the AzDisk
class that retrieves disk
from Microsoft Azure.
-
class
cloudmarker.clouds.azdisk.
AzDisk
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure disk plugin.
Create an instance of
AzDisk
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of worker processes to run.
- threads (int) – Number of worker threads to run.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of Postgres records to fetch for each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure disk record.
Yields: dict – An Azure disk record.
cloudmarker.clouds.azkv module¶
Microsoft Azure Key Vault plugin to read Key Vault and associated resources.
This module defines the AzKV
class that retrieves Key Vault
from Microsoft Azure. This module also retrieves the keys and secret
attributes stored within a Key Vault.
-
class
cloudmarker.clouds.azkv.
AzKV
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure Key Vault plugin.
Create an instance of
AzKV
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of worker processes to run.
- threads (int) – Number of worker threads to run.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of Key Vault records to fetch for each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure Key Vault record.
Yields: dict – An Azure Key Vault and associated resource record.
cloudmarker.clouds.azmonitor module¶
Microsoft Azure monitor plugin to read Azure monitoring data.
This module defines the AzMonitor
class that retrieves data
from Microsoft Azure.
-
class
cloudmarker.clouds.azmonitor.
AzMonitor
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure monitor plugin.
Create an instance of
AzMonitor
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of processes to launch.
- threads (int) – Number of threads to launch in each process.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of records of each type to fetch under each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure monitor record.
Yields: dict – An Azure monitor record.
cloudmarker.clouds.azpostgres module¶
Microsoft Azure Postgres plugin to read Azure Postgres data.
This module defines the AzPostgres
class that retrieves Postgre SQL
data from Microsoft Azure.
-
class
cloudmarker.clouds.azpostgres.
AzPostgres
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure Postgres plugin.
Create an instance of
AzPostgres
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of worker processes to run.
- threads (int) – Number of worker threads to run.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of Postgres records to fetch for each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure Postgres record.
Yields: dict – An Azure Postgres record.
cloudmarker.clouds.azsql module¶
Microsoft Azure SQL Database plugin to read Azure SQL DB data.
This module defines the AzSQL
class that retrieves SQL DB
from Microsoft Azure. This module also retrieves the Transparent
Data Encryption (TDE) configuration of the SQL database.
-
class
cloudmarker.clouds.azsql.
AzSQL
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure SQL Database plugin.
Create an instance of
AzSQL
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of worker processes to run.
- threads (int) – Number of worker threads to run.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of SQL server records to fetch for each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure SQL database record.
Yields: dict – An Azure SQL database record.
cloudmarker.clouds.azstorageaccount module¶
Microsoft Azure storage accounts plugin to read Azure storage accounts data.
This module defines the AzStorageAccount
class that retrieves storage
accounts data from Microsoft Azure.
-
class
cloudmarker.clouds.azstorageaccount.
AzStorageAccount
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure storage account plugin.
Create an instance of
AzStorageAccount
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of worker processes to run.
- threads (int) – Number of worker threads to run.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of storage accounts records to fetch for each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure storage account record.
Yields: dict – An Azure storage account record.
cloudmarker.clouds.azvm module¶
Microsoft Azure virtual machine plugin to read Azure virtual machine data.
This module defines the AzVM
class that retrieves virtula machine data
from Microsoft Azure.
-
class
cloudmarker.clouds.azvm.
AzVM
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure Virtual Machine plugin.
Create an instance of
AzVM
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of worker processes to run.
- threads (int) – Number of worker threads to run.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of virtual machines records to fetch for each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure virtual machine record.
Yields: dict – An Azure virtual machine record.
cloudmarker.clouds.azwebapp module¶
Microsoft Azure web apps plugin to read Azure web app data.
This module defines the AzWebApp
class that retrieves web apps
data from Microsoft Azure.
-
class
cloudmarker.clouds.azwebapp.
AzWebApp
(tenant, client, secret, processes=4, threads=30, _max_subs=0, _max_recs=0)¶ Bases:
object
Azure web app plugin.
Create an instance of
AzWebApp
plugin.Note: The_max_subs
and_max_recs
arguments should be used only in the development-test-debug phase. They should not be used in production environment. This is why we use the convention of beginning their names with underscore.Parameters: - tenant (str) – Azure subscription tenant ID.
- client (str) – Azure service principal application ID.
- secret (str) – Azure service principal password.
- processes (int) – Number of worker processes to run.
- threads (int) – Number of worker threads to run.
- _max_subs (int) – Maximum number of subscriptions to fetch data for if the value is greater than 0.
- _max_recs (int) – Maximum number of web apps records to fetch for each subscription.
-
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return an Azure web app record.
Yields: dict – An Azure web app record.
cloudmarker.clouds.gcpcloud module¶
Google Cloud Platform (GCP) plugin to read GCP infrastructure data.
This module defines the GCPCloud
class that retrieves data from
Google Cloud Platform.
-
class
cloudmarker.clouds.gcpcloud.
GCPCloud
(key_file_path, processes=4, threads=30, _max_projects=0)¶ Bases:
object
GCP cloud plugin.
Create an instance of
GCPCloud
plugin.Note: The_max_projects
argument should be used only in the development-test-debug phase. It should not be used in production environment. This is why we use the convention of beginning it’s name with underscore.Parameters: -
done
()¶ Log a message that this plugin is done.
-
read
()¶ Return a GCP cloud infrastructure configuration record.
Yields: dict – A GCP cloud infrastructure configuration record.
-
cloudmarker.clouds.mockcloud module¶
Mock cloud plugin for testing purpose.
-
class
cloudmarker.clouds.mockcloud.
MockCloud
(record_count=10, record_types=('foo', 'bar'))¶ Bases:
object
Mock cloud plugin for testing purpose.
Create an instance of
MockCloud
plugin.This plugin generates mock records. The records generated contains three fields under three top-level keys that we also call “bucket keys”:
raw
,data
, andtype
, as shown in the example below:Example
Here is an example that shows that the records generated by this plugin with the default initialization parameters:
>>> from cloudmarker.clouds import mockcloud >>> cloud = mockcloud.MockCloud() >>> for record in cloud.read(): ... print(record['raw']['data'], ... record['ext']['record_type'], ... record['com']['record_type']) ... 0 foo mock 1 bar mock 2 foo mock 3 bar mock 4 foo mock 5 bar mock 6 foo mock 7 bar mock 8 foo mock 9 bar mock
The three top-level keys,
raw
,ext
, andcom
represent the names of the three buckets under which various data attributes are kept. While this is only a mock plugin, but in an actual cloud plugin implementation, the meaning of these buckets are as follows:raw
: The value for theraw
key is adict
object that represents the actual data object obtained from a cloud in its original form. No modifications should be done to the object obtained from the cloud.ext
: The value for theext
key is adict
object which contains key-value pairs for any additional cloud-specific metadata that need to be stored. The data in this bucket is also known as extended metadata.com
: The value for thecom
key is adict
object which contains key-value pairs for any metadata that is common to all clouds.
Parameters: -
done
()¶ Perform cleanup work.
Since this is a mock plugin, this method does nothing. However, a typical cloud plugin may or may not need to perform cleanup work in this method depending on its nature of work.