I am using a third-party library and sometimes it pops up a dialog. Before I finish the current activity, I want to check whether there is a dialog popped up in the current con
I am assuming, you are dealing with third party library and you don't have access to dialog object.
You can get the root view from the activity,
Then you can use tree traversal algorithm to see if you can reach any of the child view. You should not reach any of your child view if alert box is displayed.
When alert view is displayed ( check with Ui Automator ), the only element present in UI tree are from DialogBox / DialogActivity. You can use this trick to see if dialog is displayed on the screen. Though it sounds expensive, it could be optimized.