Exception Alert on Google Chat

Introduction

This tutorial is about setting up alerts on Google Chat for exceptions triggered by a python application running on Google Cloud.

Exception Alert on Google Chat is a module by which we can monitor the incidents that occur on the server based on logs. When an exception occurs in our instance or any other service falls under the customised policy. Then the filtered exceptions are reported as an incident. And a notification for the same is triggered to the Google Chat.

Terminologies :

• Exception: Any unusual behaviour that occurs while processing. Ex: A circle with a corner is an exception.
• Alert Monitoring: Raising and reporting alerts on specified exceptional behaviour while processing. Ex. raising an alert to inform a circle detected with a corner.
• Alert Policies: The policies to filter the exceptions can be customised in the monitoring window where we create a data alerts policy. Ex, report when unusual circle behaviour is detected.

Why

Previously, we used to track manually for the exceptions daily. And it is tedious to filter the required exceptions. It was a time-consuming process when there was no desired exception in the list. Hence there was a need for an automated process which helps us to notify whenever it occurs on G-Chat.

How

Assumptions:

Let’s assume we have a project that is hosted on Google Cloud Platform and the language of the project is Python.

Step 1 : Need to integrate GCP logger with our Python Logger.

• Create a logger.py file as shown below.

import os

import google.cloud.logging
from google.cloud.logging_v2.handlers import CloudLoggingHandler
from loguru import logger


def init_gcp_logging():
    log_level = os.getenv("LOGURU_LEVEL", "INFO")
    client = google.cloud.logging.Client()
    handler = CloudLoggingHandler(client)
    logger.add(handler, level=log_level)

• Initiate init_gcp_logging with main or to a starting point.

import os
from logger import init_gcp_logging

if os.getenv("GCP_LOGGING") == "true":
    init_gcp_logging()

• After this your default log handler will be changed to GCP logger, you can add below lines to main to test it.

logger.info("Hello Welcome to Our First Blog")
logger.warning("This is warning")
logger.error("Error occured")

• GCP Logging sample output :

Step 2 : Create a Pub/Sub topic by which filtered exceptions will be pushed to the Pub/Sub.

• Go to Pub/Sub.
• Create a topic (possibly default, if you don’t want customised settings).

Step 3 : Need to create a log based alert by setting up the policies.

• Go to Logging
• Click on Create Alert
• Fill all Details Ex. Alert Policy name, log query, etc.
• Align Notification channel as pub sub with created channel in previous step.

Note : Make sure, to allow your-service-id@gcp-sa-monitoring-notification.iam.gserviceaccount.com a permission of Pub/Sub Publisher.

Step 4 : Create a G-Chat webhook and a cloud function to which we can integrate our Pub/Sub to Google Chat.

• Open Apps And Integrations of your Google Chat Workspace.
• Set up a new webhook.
• Go to Cloud Function in Google Cloud Platform.
• Create an event-driven cloud function for Pub/Sub topic with the below-given code by using the webhook URL.

MESSAGE_TEMPLATE = """
{{
    "cards": [
        {{
            "header": {{
                "title": "<users/all> Google Cloud Monitoring Alert"
            }},
            "sections": [
                {{
                    "header": "{incident[policy_name]}",
                    "widgets": [
                        {{
                            "textParagraph": {{
                                "text": "{incident[summary]}"
                            }}
                        }}
                    ]
                }},
                {{
                    "widgets": [
                        {{
                            "keyValue": {{
                                "topLabel": "Environment",
                                "content": "{incident[environment]}"
                            }}
                        }}
                    ]
                }},
                {{
                    "widgets": [
                        {{
                            "buttons": [
                                {{
                                    "textButton": {{
                                        "text": "GO TO INCIDENT",
                                        "onClick": {{
                                            "openLink": {{
                                                "url": "{incident[url]}"
                                            }}
                                        }}
                                    }}
                                }}
                            ]
                        }}
                    ]
                }}
            ]
        }}
    ]
}}
"""
def hello_pubsub(event, context, **kwargs): 
  try:
         webhook_url = "Enter You G-Chat Webhook URL"
         alert = json.loads(base64.b64decode(event['data']).decode('utf-8'))
         alert["incident"]["environment"] = "TESTING"
         message = json.loads(MESSAGE_TEMPLATE.format(**alert))
         requests.post(url=webhook_url, headers={"Content-Type": "application/json; charset=UTF-8"}, json=message)
    except Exception as e:
         print("Unable to trigger alert on G-Chat", e)

• Output :

Usage of Alert Exception

When

1. The log Exceptions are rare, and we want to know when they occur and save time in manual monitoring.
2. When you want to monitor the specific behaviour of a particular service.
3. When there are exceptions in large numbers, and user needs to monitor specific occurrence areas for exceptions.

Where

1. For sending an Alert notification where the resources used exceed the limit.
2. For tracking operations performed by the authorised users.
3. To monitor the cost when we use an external platform.

This blog was written with collaborating with Dhruvi Gangawal.