This article is contributed. See the original author and article here.

Hello blog readers :smile:


One of recurring questions during my customer engagements on Azure Monitor is: how do I set alert state to either Acknowledged or Closed with no manual intervention?


 


This question is broader and deeper than it appears. In fact, linked to the pure and simple alert state there are often ITSM processes coming along. State is just an alert property that can have only 1 of the 3 following values at a given time: New, Acknowledged or Closed. Should you want to read more about Azure Monitor alerts (including their states) you can find more information in the official Microsoft documentation at Overview of alerting and notification monitoring in Azure – Azure Monitor | Microsoft Docs.


 


Hence, when it comes to the state, we also need to consider other actors. In a simple scenario, where have notifications and no ITSM processes, we can automate alert state management using Azure Automation to fire a runbook that sets the alert state on schedules. Differently, on mature customers or high integrated IT environments, where alerts are part of the incident management process(es), we must consider that alert states have to be managed in line with the ITSM integration. The below diagram quickly describes the scenario for alerts lifecycle when the ITSM integration is in place:


 


Azure Monitor <--> ITSM integration flowAzure Monitor <–> ITSM integration flow


 


So, provided that you have evaluated the best scenario according to the company’s business needs, the idea shared here is very easy and works very well especially with metrics-based alerts where you have a stateful alert approach.


With log-search based alerts, the situation can become a bit more complex since these alerts are stateless.


Looking at the alerts from Azure Monitor – Alerts blade,


 


Azure Monitor Alert DashboardAzure Monitor Alert Dashboard


 


you may have noticed that among all the columns, we have one called Monitor condition, whose value is sometime set to Fired or Resolved, and one called Signal type.


Let us start with the Signal type one. This one stands for the repository (and hence the type of data we are going to use for the alert: Metrics or Logs) where the data is stored. It is important to understand that because the type of data is what drives the value in the Monitor condition column. This column is showing the status of the object/aspects we created the alert for.


But why it sometime shows as resolved and sometimes not? The answer is exactly in the value reported by the Signal type column. When Signal type is Metrics or Health, it means that we are using data whose certainty is guaranteed 100%. In other words, that type of data will always be produced, collected and stored in Azure, so we can check whether an issue has been resolved or not and set the Monitor condition property value accordingly. This certainty makes the alerts stateful. For more info you can check the Understand how metric alerts work in Azure Monitor documentation at https://docs.microsoft.com/en-us/azure/azure-monitor/alerts/alerts-metric-overview


Differently, when it is Log there is no assurance that we either collected or received data. Think about an on-prem environment in which we have several dependencies as part of the trip to Azure Log Analytics. Think about what happen when we lose Internet connectivity or the monitoring agent just stops, or the server is powered off. How can we make sure the issue is resolved if we have no data confirming it? This uncertainty makes the log-based alerts stateless. Should you need more info, you can refer to the Log alerts in Azure Monitor documentation, specifically looking at the State and resolving alerts paragraph.


With all that said we now have a better idea of what to do to set our alert state in both scenarios (Metrics/Health and logs).


Since we proved so far that using Metrics or Health as signal type we always have the correct and up-to-date condition, we can just look at that the MonitorCondition property value and set the alert state to Closed. In that case the simple automation runbook I am suggesting below can help:


 


 


 

<#
.SYNOPSIS 
    This sample automation runbook is designed to set the metric or health based alerts to Closed.
    

.DESCRIPTION
    This sample automation runbook is designed to set the metric or health based alerts to Closed. It looks for all the alerts in the provided time range and for each,
    it will check the value of the MonitorCondition property. Should it be equal to Resolved, we set the state property to Closed.
    This runbook requires the Az.AlertsManagement PowerShell module which can be found at https://docs.microsoft.com/en-us/powershell/module/az.alertsmanagement/?view=azps-5.6.0

    NOTE: TimeRange parameter only accepts the value reported in the ValidateSet. This is in line with the underlying API requirements that is 
    documented at https://docs.microsoft.com/en-us/rest/api/monitor/alertsmanagement/alerts/getall#timerange


.PARAMETER TimeRange
    Required. The TimeRange on which we query the alerts.


.EXAMPLE
    .Close-ResolvedAlerts.ps1 -TimeRange 1d

.NOTES
    AUTHOR:   Bruno Gabrielli
    VERSION:  1.0
    LASTEDIT: Dec 08th, 2020
#>

#Parameters
param(
    [ValidateSet('1h', '1d', '7d', '30d')]
    [string] $TimeRange = '1d'
)

#Inizialiting connection to the AutomationAccount
[String]$connectionName = "AzureRunAsConnection"
try
{
    #Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName

    #"Logging in to Azure..."
    $nullOut = (Add-AzAccount `
     -ServicePrincipal `
     -TenantId $servicePrincipalConnection.TenantId `
     -ApplicationId $servicePrincipalConnection.ApplicationId `
     -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint `
     -WarningAction:Ignore)
        
    #"Setting context to a specific subscription"  
    $nullOut = (Set-AzContext -SubscriptionId $servicePrincipalConnection.SubscriptionId -WarningAction:Ignore)

    $inactiveAlerts = (Get-AzAlert -MonitorCondition Resolved -State New -TimeRange $TimeRange)

    if($inactiveAlerts)
    {
        foreach($alert in $inactiveAlerts)
        {
            Write-Output "Setting state to 'Closed' for alert '$($alert.Name)' which had the monitor condition set to '$($alert.MonitorCondition)' and the state set to '$($alert.State)'"
            Update-AzAlertState -AlertId $alert.Id -State Closed
        }
    }
    else
    {
        Write-Output "No inactive (Resolved) alerts in the specified '$($TimeRange)' period."
    }
}
catch
{
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    }
    else
    {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
} 

 


 


 


As opposite to Metrics or Health based alerts, the Log based alerts need to be managed differently. Here we must look first for the MonitorService property value, making sure that it is equal to “Log Analytics”. After that we need to make some assumptions based on the LastModified property value. Based on the log-based alerts nature, we might assume that if an alert has not been changed later than the TimeRange parameter value we provided, we could close it. We will get a new one soon if the corresponding issue has not been resolved in the meantime. Here below you can find another sample runbook for that purpose:


 


 


 

<#
.SYNOPSIS 
    This sample automation runbook is designed to set the Log Analytics based alerts to Closed.
    

.DESCRIPTION
    This sample automation runbook is designed to set the Log Analytics based alerts to Closed. It looks for all the alerts in the provided time range and for each,
    it will check the value of the MonitorService property. Should it be equal to Log Analytics and last modified later than TimeRange, we set the state property to Closed.
    This runbook requires the Az.AlertsManagement PowerShell module which can be found at https://docs.microsoft.com/en-us/powershell/module/az.alertsmanagement/?view=azps-5.6.0

    NOTE: TimeRange parameter only accepts the value reported in the ValidateSet. This is inline with the underlying API requirements that is
    documented at https://docs.microsoft.com/en-us/rest/api/monitor/alertsmanagement/alerts/getall#timerange


.PARAMETER TimeRange
    Required. The TimeRange on which we query the alerts.


.EXAMPLE
    .Close-ResolvedAlerts.ps1 -TimeRange 1d

.NOTES
    AUTHOR:   Bruno Gabrielli
    VERSION:  1.0
    LASTEDIT: Jan 21st, 2021
#>

#Parameters
param(
    [ValidateSet('1h', '1d', '7d', '30d')]
    [string] $TimeRange = '1d'
)

#Inizialiting connection to the AutomationAccount
[String]$connectionName = "AzureRunAsConnection"
try
{
    #Get the connection "AzureRunAsConnection "
    $servicePrincipalConnection=Get-AutomationConnection -Name $connectionName

    #"Logging in to Azure..."
    $nullOut = (Add-AzAccount `
     -ServicePrincipal `
     -TenantId $servicePrincipalConnection.TenantId `
     -ApplicationId $servicePrincipalConnection.ApplicationId `
     -CertificateThumbprint $servicePrincipalConnection.CertificateThumbprint `
     -WarningAction:Ignore)
        
    #"Setting context to a specific subscription"  
    $nullOut = (Set-AzContext -SubscriptionId $servicePrincipalConnection.SubscriptionId -WarningAction:Ignore)

    $inactiveAlerts = (Get-AzAlert -MonitorService 'Log Analytics' -State New -TimeRange $TimeRange)

    if($inactiveAlerts)
    {
        foreach($alert in $inactiveAlerts)
        {
            if($alert.LastModified -le ((Get-date).add(-$TimeRange)))
            {
                Write-Output "Setting state to 'Closed' for alert '$($alert.Name)' which had the monitor service equal to $($alert.MonitorService), monitor condition set to '$($alert.MonitorCondition)' and the state set to '$($alert.State)'"
                Update-AzAlertState -AlertId $alert.Id -State Closed
            }
        }
    }
    else
    {
        Write-Output "No inactive (Resolved) alerts in the specified '$($TimeRange)' period."
    }
}
catch
{
    if (!$servicePrincipalConnection)
    {
        $ErrorMessage = "Connection $connectionName not found."
        throw $ErrorMessage
    }
    else
    {
        Write-Error -Message $_.Exception
        throw $_.Exception
    }
} 

 


 


 


Both sample runbook codes requires the Az.AlertsManagement PowerShell module to be imported into your Automation Account.


 


 


NOTE: As you can see from the script’ comments, TimeRange parameter only accepts the values reported in the ValidateSet. This is in line with the underlying API requirements that is documented at https://docs.microsoft.com/en-us/rest/api/monitor/alertsmanagement/alerts/getall#timerange


 


 


With all the ingredients and knowledge, you just have to import the 2 scripts as new runbooks:


 


Azure Automation RunbooksAzure Automation Runbooks


 


and schedule them to run on your preferred interval which can be different from the value you used as TimeRange parameter:


 


Azure Automation SchedulesAzure Automation Schedules


 


Thanks for reading this one till the end,


Bruno.


 


Disclaimer


The sample scripts are not supported under any Microsoft standard support program or service. The sample scripts are provided AS IS without warranty of any kind. Microsoft further disclaims all implied warranties including, without limitation, any implied warranties of merchantability or of fitness for a particular purpose. The entire risk arising out of the use or performance of the sample scripts and documentation remains with you. In no event shall Microsoft, its authors, or anyone else involved in the creation, production, or delivery of the scripts be liable for any damages whatsoever (including, without limitation, damages for loss of business profits, business interruption, loss of business information, or other pecuniary loss) arising out of the use of or inability to use the sample scripts or documentation, even if Microsoft has been advised of the possibility of such damages.

Brought to you by Dr. Ware, Microsoft Office 365 Silver Partner, Charleston SC.