In old times Azure AD Conditional Access policies could be only enabled or disabled. That means that policy deployment have to be tested and planned very carefelly and still, you might face some issues with use cases you didn’t accounted for and as a consequence to have an impact on end users.
But fortunately we are not in “dark ages” anymore and Conditional Access policies can work in Report-Only mode now.
What is Report-Only mode?
Well, naming is quite sef explainatory. It is a mode with no* impact on user, but action would be evaluated and you will be able to find results of evaluation in logs.
*In speciefic cases users might see additional prompts to select certificate on macOS, iOS, Android. More details can be found here
How to use it?
To start, you need to create any policy and enable it with “report-only” mode.
After that you will be able to see Report-only section in Azure AD sign-in logs:
Most important here is the result of policy evaluation. In case of this screenshot, I had succesfull one, which means that in case of policy in “On” mode, access would be granted without any issues.
Crucial point here is that in this case it was just evaluation and not actual action. That gives me a way to test Conditional Access policy before deployment.
Obviously, you will not be checking logs manually. There are more convinient ways.
But to do that, you have to create Log Analytics Workspace first and integrate Azure AD logs with that. Step-by-step guide can be found here.
As a result, you will be able to use Microsoft standard built-in workbooks to analyse how your Conditional Access policies are working.
But I am not a fan of standard reports and preffer to look into Log Analytics directly, due to simple reason of having more flexibility.
For example, in screenshot below you can see results of testing policy change, which reduced number of succesful logins to almost zero. Without Report-only mode I would have some problems after that (it can be called “Oh, shit” moment), but in this case it was just me testing without impacting real users. Now I could just change policy settings and confirm that issue gone.
Should you use it?
Definetely!
If you are using Azure AD Conditional Access policies in your environment, you have to use Report-Only mode before changing or deploying anything with “On” mode.
The only downside, is that you have to pay for Log Analytics workspace, but it can be used for more things in the future.
Alex Vanyurikhin 