Monitor AzureAD Conditional Access Policy changes with PowerShell (Scheduled Script)

When there are multiple administrators in an AzureAD tenant, it is inevitable that one may change settings in Conditional Access policies – without notifying everyone involved. To keep track of changes you could regualarly check the AzureAD audit logs, or have an automation for it. I may be a bit old-fashioned, but I prefer to have PowerShell scripts running on-premises with Scheduled Tasks, etc. So this blogpost covers monitoring Conditional Access policy changes with a scheduled PowerShell script.

TL;DR

• Create a self-signed cert in your local machine store (on which you plan to run the scheduled script), export the public key
• Create an App registration in your tenant, grant the following Microsoft Graph API permissions (type=Application):
  • AuditLog.Read.All
  • Directory.Read.All
• Add the exported certificate to the app registration
• Note the tenantID, clientID and certificate thumbprint
• Install Microsoft.Graph.Authentication, Microsoft.Graph.Reports Powershell modules on the computer which will run the script
• Modify the below script accordingly:
  • $scheduleMins = task repetition in minutes. It is used to filter audit records by timestamp. Eg.: when this value is set to 60 (run every hour) it will look for audit events generated in the past 60 minutes
  • $clientID = client ID/app ID of App registration created in your AzureAD tenant
  • $tenantID = your AzureAD tenant ID
  • $certThumbprint = thumbprint of the self signed certificate
  • modify Send-MailMessage parameters on the last line to your needs
• Schedule the task (with admin rights to read the local machine cert store, I use SYSTEM account for this purpose, but this may be a security concern)

The script:

$scheduleMins = 60

$clientID = "" #clientID of registered app
$tenantID = "" #your tenant ID
$certThumbprint = "" #thumbprint of certificate used to connect to Graph API

Connect-MgGraph -ClientId $clientID -TenantId $tenantID -Certificate (gci Cert:\LocalMachine\my | ? {$_.thumbprint -eq $certThumbprint }) -ContextScope Process

### Search audit logs
    $activitiesAfter = (Get-date).AddMinutes(-$scheduleMins).ToUniversalTime().ToString("yyyy-MM-ddTHH:mmZ")
    $CAAuditEvents = Get-MgAuditLogDirectoryAudit -filter "(activityDateTime ge $activitiesAfter) and (loggedbyservice eq 'Conditional Access')"
    #exit if no events found
    if (!($CAAuditEvents)){exit}

## Create report
$report_CAEvents = foreach ($event in $CAAuditEvents){
    [pscustomobject]@{
        Action = $event.ActivityDisplayName
        UTCTimeStamp = $event.ActivityDateTime
        Initiator = $event.InitiatedBy.User.UserPrincipalName
        TargetDisplayName = $event.TargetResources.DisplayName
        TargetID = $event.TargetResources.Id
        OldValue = $event.TargetResources.modifiedproperties.OldValue
        NewValue = $event.TargetResources.modifiedproperties.NewValue
        }
    }

$Header = @"
<style>
TABLE {border-width: 1px; border-style: solid; border-color: black; border-collapse: collapse;}
TD {border-width: 1px; padding: 3px; border-style: solid; border-color: black;}
TH {border-width: 1px; padding: 3px; border-style: solid; border-color: black;}
</style>
"@
[string]$html = $report_CAEvents | ConvertTo-Html -Title "Conditional Access policy modifications" -Head $Header

Send-MailMessage -From <sender> -to <recipient> -Subject "Conditional Access Policy change alert" -Body $html -BodyAsHtml -SmtpServer <smtpserver> -Port 25

The result will show the old and new JSON notation of each modified policy. When there is a new policy, OldValue column will be empty, while a deleted policy’s report will have an empty NewValue column (for logical reasons 🙂 )