Firstly - if this is a HTML appliction, have you looked at the Troubleshooting guide in the product help for browser integration? This will tell you about the TabProcGrowth registry setting that may make pop-ups and windows spawn in seperate/new windows process threads, effectily behave as seperate applications to the one you are attached to in Application Modeller.
If settiing TabProcGrowth to 0 still means you cannot interact with the IE pop-ups, you may need to create a seperate Object specifically for attaching and interacting with them. The browser integration guide on the portal does say for IE popups that ' To speed up development, it might be best to build a separate generic object to handle IE pop-up windows. '.
If you need more assistance with this, it could be a subject to raise with me on the weekly mentoring calls
Den