Hack GCP Composer - Use AWS SES for the existing environment

Subscribe to my newsletter and never miss my upcoming articles

GCP Cloud Composer is a managed Airflow service. Its using GKE as the core component for the distributed worker. AppEngine will host the Web Server and the CloudSQL(MySQL) as the backend database. We'll see how to use AWS SES for sending email alerts from the GCP cloud composer.

ā˜ ļøšŸšØā˜ ļøšŸšØDisclaimer: This is just to show a small hack to use a plain text password for existing cloud composer environments. And it's not at all recommended to use it in production.

Airflow will send an email notification in case of a task failure. We can configure any working SMTP details. But in GCP Cloud Composer, we can integrate with SendGrid to send the emails. Its not good for the folks who don't use a third-party for email communication. But there is a way to setup and use our own SMTP details via Environment Variables. Again there is a glitch here. I tried to setup this env variable to an existing cloud composer environment, it was not working.

Attempt #1

We have an option to add the values to override the airflow configuration on the composer console. So I tried to add the smtp_password. But it was throwing its not allowed to set this variable.

image.png

Attempt #2

I wanted to play around with this and somehow I need to solve it. So my first attempt is override the password in the airflow.cfg file. Its available on the GCS bucket where our DAGs will be stored. Under the [smtp] block I had modified the smtp_password.

[smtp]
smtp_host = email-smtp.us-east-1.amazonaws.com
smtp_starttls = false
smtp_ssl = true
smtp_user = *******
smtp_port = 465
smtp_password = MY_PASSWORD
smtp_mail_from = ******

Then uploaded the file to GCS. But the webserver was showing the previous value only. Then I did the restart of the webserver. It's not possible directly, so just add a dummy variable.

gcloud composer environments update ${ENVIRONMENT_NAME} \
--location=${ENV_LOCATION} \
--update-airflow-configs=webserver-dummy=true

After the restart, still it was showing the old password. Then we referred to the documentation, and its clearly mentioned that its not allowed in Composer .

image.png

Attempt #3

From the same documentation page, it mentioned that, we can add the SMTP password as an Environment variable , so the airflow take that value while sending the email. But the documentation page refers that, the variable needs to be set while providing the composer environment. So I tried to add manually using the gcloud cli.

gcloud composer environments update staging-composer-frankfurt \
  --location europe-west3 \
  --update-env-variables=AIRFLOW__SMTP__SMTP_PASSWORD='xxxXXXXxxx'

ERROR: (gcloud.composer.environments.update) INVALID_ARGUMENT: Found 1 problem:
1) Environment variables [AIRFLOW__SMTP__SMTP_PASSWORD] may not be overridden.

Attempt #4

This time instead of work playing with the UI, I wanted to do something on the GKE level because all of these configurations are actually applied to the worker pods, so directly go and change the values over there. (use bastion or jump host or CloudShell)

Note: I have reverted everything to the default state(how was it before the changes including airflow.cfg file.

  1. Get the namespace for the airflow. It'll be like composer-1-12-5-airflow-1-10-10-xxxx
    kubectl get ns
    
  2. Set default ns to airflow namespace using
    kubectl config set-context --current --namespace=<insert-namespace-name-here>
    
  3. List the config maps. There should be a config map called airflow-configmap.
    kubectl get configmap
    
  4. Edit and update the smtp block values of the config map.
    kubectl edit configmap airflow-configmap
    
  5. List out airflow scheduler and worker pods, because we need to redeploy the worker pods.
    kubectl get pods
    
    6.Delete the worker pods.
    kubectl delete pod <worker-pod-name-1> <worker-pod-name-2><worker-pod-name-3>
    
    Wait for a few mins, and see the values on the Airflow UI. I was not lucky, still, it was showing the old password even after restarting the airflow webserver.

The final attempt #5

This time we combined actions from attempt #2 and attempt #4. Yes, it was the Hack. After restarting the webserver we were able to see the new SMTP details.

Demo time:

We created a new DAG for testing and added some wrong input. Then ran the DAG. Guess what? we got the email.

image.png

Credit:

I would like to thank Alfred Tommy who was helping me with the GKE part.šŸ™šŸ»

No Comments Yet