Managed to resolve this issue yesterday. You need to detach from the main browser window, and reattach to the popup window. Once this is done, you can spy the popup.
You will need to perform this step in your business object as well. Detach from the main window, attach to the popup. Once you're finished with the popup and it is closed, you then need to detach from it again, and re-attach back to the main window.