SpecialistOff.NET / Вопросы / Статьи / Фрагменты кода / Резюме / Метки / Помощь / Файлы

Назад

Google Compute Engine Operators

Метки: airflow apache airflow

GceInstanceStartOperator

Use the GceInstanceStartOperator to start an existing Google Compute Engine instance.

Arguments

The following examples of OS environment variables used to pass arguments to the operator:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_INSTANCE = os.environ.get('GCE_INSTANCE', 'testinstance')

Using the operator

The code to create the operator:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

gce_instance_start = GceInstanceStartOperator(
    project_id=GCP_PROJECT_ID,
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    task_id='gcp_compute_start_task'
)

You can also create the operator without project id - project id will be retrieved from the GCP connection id used:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

gce_instance_start2 = GceInstanceStartOperator(
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    task_id='gcp_compute_start_task2'
)

Templating

template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')

More information

See Google Compute Engine API documentation to start an instance.

GceInstanceStopOperator

Use the operator to stop Google Compute Engine instance.

For parameter definition, take a look at GceInstanceStopOperator

Arguments

The following examples of OS environment variables used to pass arguments to the operator:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_INSTANCE = os.environ.get('GCE_INSTANCE', 'testinstance')

Using the operator

The code to create the operator:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

gce_instance_stop = GceInstanceStopOperator(
    project_id=GCP_PROJECT_ID,
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    task_id='gcp_compute_stop_task'
)

You can also create the operator without project id - project id will be retrieved from the GCP connection used:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

gce_instance_stop2 = GceInstanceStopOperator(
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    task_id='gcp_compute_stop_task2'
)

Templating

template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')

More information

See Google Compute Engine API documentation to stop an instance.

GceSetMachineTypeOperator

Use the operator to change machine type of a Google Compute Engine instance.

For parameter definition, take a look at GceSetMachineTypeOperator.

Arguments

The following examples of OS environment variables used to pass arguments to the operator:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')
GCE_INSTANCE = os.environ.get('GCE_INSTANCE', 'testinstance')

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

GCE_SHORT_MACHINE_TYPE_NAME = os.environ.get('GCE_SHORT_MACHINE_TYPE_NAME', 'n1-standard-1')
SET_MACHINE_TYPE_BODY = {
    'machineType': 'zones/{}/machineTypes/{}'.format(GCE_ZONE, GCE_SHORT_MACHINE_TYPE_NAME)
}

Using the operator

The code to create the operator:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

gce_set_machine_type = GceSetMachineTypeOperator(
    project_id=GCP_PROJECT_ID,
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body=SET_MACHINE_TYPE_BODY,
    task_id='gcp_compute_set_machine_type'
)

You can also create the operator without project id - project id will be retrieved from the GCP connection used:

airflow/contrib/example_dags/example_gcp_compute.pyVIEW SOURCE

gce_set_machine_type2 = GceSetMachineTypeOperator(
    zone=GCE_ZONE,
    resource_id=GCE_INSTANCE,
    body=SET_MACHINE_TYPE_BODY,
    task_id='gcp_compute_set_machine_type2'
)

Templating

template_fields = ('project_id', 'zone', 'resource_id', 'gcp_conn_id', 'api_version')

More information

See Google Compute Engine API documentation to set the machine type.

GceInstanceTemplateCopyOperator

Use the operator to copy an existing Google Compute Engine instance template applying a patch to it.

For parameter definition, take a look at GceInstanceTemplateCopyOperator.

Arguments

The following examples of OS environment variables used to pass arguments to the operator:

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

GCE_TEMPLATE_NAME = os.environ.get('GCE_TEMPLATE_NAME', 'instance-template-test')
GCE_NEW_TEMPLATE_NAME = os.environ.get('GCE_NEW_TEMPLATE_NAME',
                                       'instance-template-test-new')
GCE_NEW_DESCRIPTION = os.environ.get('GCE_NEW_DESCRIPTION', 'Test new description')
GCE_INSTANCE_TEMPLATE_BODY_UPDATE = {
    "name": GCE_NEW_TEMPLATE_NAME,
    "description": GCE_NEW_DESCRIPTION,
    "properties": {
        "machineType": "n1-standard-2"
    }
}

Using the operator

The code to create the operator:

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

gce_instance_template_copy = GceInstanceTemplateCopyOperator(
    project_id=GCP_PROJECT_ID,
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task'
)

You can also create the operator without project id - project id will be retrieved from the GCP connection used:

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

gce_instance_template_copy2 = GceInstanceTemplateCopyOperator(
    resource_id=GCE_TEMPLATE_NAME,
    body_patch=GCE_INSTANCE_TEMPLATE_BODY_UPDATE,
    task_id='gcp_compute_igm_copy_template_task_2'
)

Templating

template_fields = ('project_id', 'resource_id', 'request_id',
                   'gcp_conn_id', 'api_version')

More information

See Google Compute Engine API documentation to create a new instance with an existing template.

GceInstanceGroupManagerUpdateTemplateOperator

Use the operator to update a template in Google Compute Engine Instance Group Manager.

For parameter definition, take a look at GceInstanceGroupManagerUpdateTemplateOperator.

Arguments

The following examples of OS environment variables used to pass arguments to the operator:

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

GCP_PROJECT_ID = os.environ.get('GCP_PROJECT_ID', 'example-project')
GCE_ZONE = os.environ.get('GCE_ZONE', 'europe-west1-b')

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

GCE_INSTANCE_GROUP_MANAGER_NAME = os.environ.get('GCE_INSTANCE_GROUP_MANAGER_NAME',
                                                 'instance-group-test')

SOURCE_TEMPLATE_URL = os.environ.get(
    'SOURCE_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/" + GCP_PROJECT_ID +
    "/global/instanceTemplates/instance-template-test")

DESTINATION_TEMPLATE_URL = os.environ.get(
    'DESTINATION_TEMPLATE_URL',
    "https://www.googleapis.com/compute/beta/projects/" + GCP_PROJECT_ID +
    "/global/instanceTemplates/" + GCE_NEW_TEMPLATE_NAME)

UPDATE_POLICY = {
    "type": "OPPORTUNISTIC",
    "minimalAction": "RESTART",
    "maxSurge": {
        "fixed": 1
    },
    "minReadySec": 1800
}

Using the operator

The code to create the operator:

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

gce_instance_group_manager_update_template = \
    GceInstanceGroupManagerUpdateTemplateOperator(
        project_id=GCP_PROJECT_ID,
        resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
        zone=GCE_ZONE,
        source_template=SOURCE_TEMPLATE_URL,
        destination_template=DESTINATION_TEMPLATE_URL,
        update_policy=UPDATE_POLICY,
        task_id='gcp_compute_igm_group_manager_update_template'
    )

You can also create the operator without project id - project id will be retrieved from the GCP connection used:

airflow/contrib/example_dags/example_gcp_compute_igm.pyVIEW SOURCE

gce_instance_group_manager_update_template2 = \
    GceInstanceGroupManagerUpdateTemplateOperator(
        resource_id=GCE_INSTANCE_GROUP_MANAGER_NAME,
        zone=GCE_ZONE,
        source_template=SOURCE_TEMPLATE_URL,
        destination_template=DESTINATION_TEMPLATE_URL,
        task_id='gcp_compute_igm_group_manager_update_template_2'
    )

Templating

template_fields = ('project_id', 'resource_id', 'zone', 'request_id',
                   'source_template', 'destination_template',
                   'gcp_conn_id', 'api_version')

Troubleshooting

You might find that your GceInstanceGroupManagerUpdateTemplateOperator fails with missing permissions. To execute the operation, the service account requires the permissions that theService Account User role provides (assigned via Google Cloud IAM).

More information

See Google Compute Engine API documentation to manage a group instance.