By Arya Pramudika in Monitoring —

Alerting in Zabbix

Alerting in Zabbix

The Importance of Alerting

Alerting is a crucial aspect of monitoring; we can't spend 24/7 looking at a monitoring dashboard. With alerting, we receive notifications from platforms like Telegram, Discord, Opsgenie, or Jira. This provides an immediate indicator that a host or item we are monitoring is down or has reached a critical severity. In Zabbix, alerts are defined in Trigger actions, meaning alerts are generated by triggers.

What is a Zabbix Trigger?

Triggers are logical rules that evaluate data collected from monitored items and flag issues. For example, if I monitor a host's disk usage percentage, Zabbix collects this data every minute and stores it as item values. I can then create trigger rules based on this data:

Zabbix Trigger
  • Warning Severity Trigger: Activates if disk usage value is greater than or equal to 80%.
  • High Severity Trigger: Activates if disk usage value is greater than or equal to 90%.

After Zabbix creates these triggers, we need to configure their actions. For instance, I might want to send 'Warning' severity triggers to a Telegram chat and 'High' severity triggers to a Discord channel. Beyond just notifications, you can also configure actions to run scripts or perform other automated tasks.

Zabbix Trigger Severity

Configuration Guide: Step-by-Step

This guide will walk through the entire process based on a practical setup.

Summary of Steps:

  1. Create an Item to monitor disk usage.
  2. Create Triggers with 'Warning' and 'High' severity levels.
  3. Create Trigger Actions to send notifications to Telegram and Discord.

Step 1: Create the Monitoring Item

First, we need to create a new item because the default Zabbix Agent template for Linux does not include disk usage monitoring.

  1. Navigate to your Host's configuration and click Items.
  2. Select Create Item.
  3. Fill in the item details as follows:
    • Name: Disk Usage /
    • Type: Zabbix agent
    • Key: vfs.fs.size[/,pused]
    • Type of information: Numeric (float)
    • Host interface: 10.8.13.2:10050 (or your specific host)
    • Units: %
    • Update interval: 1m
Select Host Items
Create new Item
Key Explanation: The key vfs.fs.size[/,pused] is used based on Zabbix documentation, where pused stands for "used, percentage." The information type is float to handle percentage values.

Test the Item

Before saving, click the Test button at the bottom of the form and then Get value to ensure Zabbix can retrieve data. A successful test should return a numeric value (e.g., 42.92...).

Testing item

Step 2: Create Triggers with Severity Levels

With the item collecting data, we can now create triggers that react to that data.

  1. Navigate to your Host's configuration and select Triggers.
  2. Click Create trigger.
Create Trigger

Create the 'Warning' Trigger

  1. Name: Disk Usage / Warning
  2. Severity: Warning
  3. In the Expression field, click Add.
  4. Define the condition:
    • Item: Select the Disk Usage / item we created earlier.
    • Function: last() - This gets the most recent value.
    • Result: >= 80
  5. Check the box for Allow manual close. This allows an administrator to manually resolve the trigger from the Zabbix dashboard.
Warning Trigger
Select Condition
Select Host and Item
Define condition

To make this trigger fire only between 80% and 90%, we add a second condition. The final expression should be:

last(/blog.pramudika.my.id/vfs.fs.size[/,pused])>=80 and last(/blog.pramudika.my.id/vfs.fs.size[/,pused])<90
Complete warning triggers

Create the 'High' Severity Trigger

Follow the same process to create a trigger for high severity.

  1. Name: Disk Usage / High
  2. Severity: High
  3. Check Allow manual close.

Expression:

last(/blog.pramudika.my.id/vfs.fs.size[/,pused])>=90
Create High severity triggers

You should now have two new triggers associated with your host.

Step 3: Create Trigger Actions to Send Notifications

Now we'll configure Zabbix to send alerts to Telegram and Discord when the triggers fire.

Prerequisites

  1. Prepare a Telegram Bot: Follow instructions to create a bot using BotFather.
  2. Prepare a Discord Webhook: Create a webhook for the desired channel in your Discord server.
  3. Enable Media Types: In Zabbix, go to Alerts -> Media types and ensure both Telegram and Discord are enabled.

Configure Media Types in Zabbix

  • Telegram:
    1. Select the Telegram media type.
    2. Input your bot's API token in the api_token field and click Update.
    3. Go to Users -> Your User -> Media.
    4. Add a new media of type Telegram and enter your personal or group chat ID in the Send to field.
Insert Api Token
Insert Chat ID
  • Discord:
    1. Select the Discord media type and paste your webhook URL.
    2. To use the Discord media type, you must configure the {$ZABBIX.URL} macro. Go to Administration -> Macros, add a new macro {$ZABBIX.URL} with the value of your Zabbix frontend URL, and click Update.
Insert Webhook
Error if not configure {$ZABBIX.URL}
Configure global macros {$ZABBIX.URL}

Create the 'Warning' Trigger Action

  • Navigate to Alerts -> Actions -> Trigger Actions and click Create action.
  • Name: blog.pramudika.my.id Warning
  • Under the Conditions tab, set the following:
    • Host equals blog.pramudika.my.id
    • Trigger severity equals Warning
Create action condition based on host
Add condition trigger severity warning
Summary conditions
  • Switch to the Operations tab. Here you define what happens when the conditions are met.
    • Steps: 1-1 (This means the action runs only once).
    • Step duration: 0 (No delay).
    • Operations: Click Add to define the operation.
      • Send to users: Admin (or your desired user).
      • Send only to: Telegram.
Add operations
Select users and media (telegram)
  • Switch to the Recovery operations tab. This defines what happens when the issue is resolved.
    • To notify everyone involved that the problem is solved, simply check Notify all involved.
  • Click Add to save the action.
Recovery operations
Notify every involved
Summary operations

Create the 'High' Severity Trigger Action

Repeat the process above to create an action for high-severity alerts, but with these changes:

  • Name: blog.pramudika.my.id High
  • Condition: Trigger severity equals High
  • Operation: Send only to Discord.

Step 4: Testing the Alerts

To test the setup, we'll manually increase the disk usage on the monitored server.

Test the 'Warning' Alert (>=80%)

Create a large file to push usage over 80%.

root@blog:~# fallocate -l 6G file.iso
root@blog:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        15G   13G  2.2G  85% /

Check current disk usage.

root@blog:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        15G  6.2G  8.2G  44% /

After the next check (within 1 minute), a notification should be sent to Telegram, and the trigger will appear on the Zabbix Dashboard.

Telegram notification
Zabbix dashboard

Test the 'High' Alert (>=90%)

Create another file to push usage over 90%.

root@blog:~# fallocate -l 1G file2.iso
root@blog:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
/dev/vda1        15G   14G  1.2G  92% /

This will trigger the 'High' severity action, and a notification should be sent to Discord. The Zabbix dashboard will update to show the new, more critical problem.

Discord notificaton

Conclusion

This guide demonstrated how to set up Zabbix alerting from creating custom items and defining triggers with different severities to configuring notification actions for platforms like Telegram and Discord. By following these steps, you can effectively monitor your systems and receive timely alerts for critical events.

References