What is Security Mode Reject in LTE NAS?

It is the EPS NAS message the UE sends when it cannot accept the selected NAS security configuration from Security Mode Command.

What should I inspect first in Security Mode Reject?

Start with the EMM cause, then compare it with the selected algorithms and NAS key set identifier in Security Mode Command.

What usually happens after Security Mode Reject?

The current procedure usually fails, stops, or has to be retried with a fresh context.

Is Security Mode Reject the same as Authentication Failure?

No. Authentication Failure belongs to the authentication challenge stage, while Security Mode Reject belongs to the later NAS security activation stage.