Microsoft is rolling out the managed conditional access policies (link) gradually and I wanted to know how it is going to impact the users (which users to be exact). Apparently, if the Sign-in logs are not streamed to a Log Analytics Workspace, the options are limited – but if you have the AADSignInEventsBeta table under Advanced hunting on the Microsoft Defender portal, some extra info can be gathered.
Streaming Entra logs to Log Analytics gives wonderful insights (not only for Conditional Access), so it is recommended to set up the diagnostic settings. If it is not an option, but the AADSignInEventsBeta is available (typically organizations with E5 licences), then the following query will show those sign-ins that would have been impacted by a report-only Conditional Access policy:
AADSignInEventsBeta
| where LogonType has "InteractiveUser"
| mv-apply todynamic(ConditionalAccessPolicies) on (
where ConditionalAccessPolicies.result == "reportOnlyInterrupted" or ConditionalAccessPolicies.result == "reportOnlyFailure"
| where ConditionalAccessPolicies.displayName has "Microsoft-managed:" //filter for managed Conditional Access policies
| extend CADisplayName = tostring(ConditionalAccessPolicies.displayName)
| extend CAResult = tostring(ConditionalAccessPolicies.result))
| distinct Timestamp,RequestId,Application,ResourceDisplayName, AccountUpn, CADisplayName, CAResult
Note: in the AADSignInEventsBeta table, the ConditionalAccessPolicies is a JSON value stored as a string so the todynamic function is needed.
Note2: Since every Conditional Access policy is evaluated against each logon, the query first filters for those sign-ins where the report-only result is ‘Interrupted’ or ‘Failure’, then the policy displayname is used to narrow down the results. Starting the filter with displayName would be pointless.
Some example summarizations if you need to see the big picture (same query as above but the last line can be replaced with these ones):
View impacted users count by application:
| summarize AffectedUsersCount=dcount(AccountUpn) by Application, CADisplayName, CAResult
Same summarization in one day buckets:
| summarize AffectedUsers = dcount(AccountUpn) by bin(Timestamp,1d), CADisplayName, CAResult
List countries by result:
| summarize make_set(Country) by CADisplayName, CAResult
Other useful feature is the Monitoring (Preview) menu in Conditional Access – Overview:
Here we have a filter option called ‘Policy evaluated’ where report-only policies are grouped under the ‘Select individual report-only policies’ section. This gives an overview but unfortunately does not list the affected users.
When a Microsoft-managed policy is opened, this chart is presented under the policy info as well.