Tap coordinates are incorrect after device rotation

[pg8wood]

When using axe tap to automate taps on iOS devices, the specified -x and -y coordinates no longer map to the correct screen location after rotating the device. It looks like the coordinate system isn’t updated to reflect the new orientation. This causes taps to land in the wrong place.

Steps to Reproduce

1. Launch any iOS app on a simulator or device.
2. Rotate the device to landscape.
3. Run a command such as axe tap -x 600 -y 450 --udid <udid>
4. Observe that the tap does not land where expected.

Expected Behavior

Taps should land at the specified screen coordinates regardless of device orientation. The coordinate system should update automatically after rotation.

Actual Behavior

After rotation, the tap location is offset. In portrait orientation the coordinates are correct, but in landscape orientation the same coordinates tap a different spot.

Additional Info

• Verified with a minimal SwiftUI test app that draws a marker at tap locations. AXe taps line up correctly in portrait but are shifted after rotation.
• Attached video demonstrates the issue.

Environment

• AXe version: v1.1.1
• iOS: 18.5
• Device: iPad Pro 13-inch (M4)
• Xcode: 16.4.0
• macOS: Sequoia 15.6.1

Screen.Recording.2025-10-08.at.5.18.07.PM.mov

[paul-foreflight]

would definitely have preferred to figure out what the issue is here on the util side, but 90% of my experience is doing stuff in python.

so its not exactly trivial, but I was able to make a workaround. it assumes that you have the ability to parse elements from axe's describe ui functionality and an element that will always be on screen

the first method is a function of my AXeElement's class (you'll have to make your own or figure something else out), and this orientation correction will normally be applied when they are created (we're going to do a special use case of course to find the current orientation below), if you want, you could always stop here and force whichever orientation you want.

If you're like me and it just itches at your brain that in your CI/CD env. or when other people run the tests you don't want to say "it has to be in the orientation that you set in the .env file" then you'll want to look at the other methods.

You'll want to find an element that is always on screen, and you can select, or something that causes another event to happen on screen (lock and unlock are the same button, but change names when clicked)

create your equivalent of find_orientation script, try each possible position you can be in, find it, click it, validate click did something (or not) and return the orientation. Now we know our orientation. Yay.

Also for what its worth I did look into other ways of checking for the sim's orientation, and came up with nothing besides installing a separate app? I dunno maybe I'll ask my app side people if there's some way they can report that from the app itself, shove it in some element or other

  def _apply_orientation_correction(self, orientation):
        # Get the original frame data (assumed to be from PORTRAIT mode)
        self.frame = self.element_data.get('frame', {})
        x_orig = self.frame.get('x')
        y_orig = self.frame.get('y')
        w_orig = self.frame.get('width')
        h_orig = self.frame.get('height')

        screen_info = AXeUtils.get_screen_info(self.udid, logger=self.logger)
        p_width = screen_info.get('width')  # Portrait Width (shorter side)
        p_height = screen_info.get('height') # Portrait Height (longer side)

        # Start with new values defaulting to original values
        x_new = x_orig
        y_new = y_orig
        w_new = w_orig
        h_new = h_orig

        try:
            if orientation == 'LANDSCAPE':
                # 90° clockwise (home button right)
                # The new top-left corner is the old top-right corner, rotated.
                x_new = y_orig
                y_new = p_width - (x_orig + w_orig)
                w_new = h_orig
                h_new = w_orig

            elif orientation == 'PORTRAIT_UPSIDE_DOWN':
                # 180° rotation
                # The new top-left corner is the old bottom-right corner.
                x_new = p_width - (x_orig + w_orig)
                y_new = p_height - (y_orig + h_orig)
                # w_new and h_new are unchanged
                
            elif orientation == 'LANDSCAPE_FLIPPED':
                # 270° clockwise / 90° counter-clockwise (home button left)
                # The new top-left corner is the old bottom-left corner, rotated.
                x_new = p_height - (y_orig + h_orig)
                y_new = x_orig
                w_new = h_orig
                h_new = w_orig
            
            # else: orientation is 'PORTRAIT', no changes needed
        # Apply the new, corrected frame values
            self.frame['x'] = x_new
            self.frame['y'] = y_new
            self.frame['width'] = w_new
            self.frame['height'] = h_new

        except Exception as e:
            if self.logger:
                self.logger.debug(f"Orientation correction failed, using original coordinates: {e}")
            # Ensure frame is reset to original values if any math fails
            self.frame['x'] = x_orig
            self.frame['y'] = y_orig
            self.frame['width'] = w_orig
            self.frame['height'] = h_orig


    def get_screen_info(self) -> str:
        """Determine device orientation by checking lock/unlock button visibility."""
        screen_info = self.get_screen_dimensions()
        orientation = 'UNKNOWN'
        self.logger.info(f"Screen info for orientation check: {screen_info}")
        orientations_to_check = ['PORTRAIT', 'LANDSCAPE', 'PORTRAIT_UPSIDE_DOWN', 'LANDSCAPE_FLIPPED']
        
        for orientation in orientations_to_check:
            screen_info['orientation'] = orientation
            element_data = axe_utils.find_element_in_ui(self.udid, lock_button, logger=self.logger)
            lock_element = AXeElement(self.udid, element_data, screen_info, self.logger)
            lock_element.tap()
            self.logger.info(f'{orientation}: {lock_element.frame}')
            try:
                if unlock_element_data := axe_utils.find_element_in_ui(self.udid, unlock_button, logger=self.logger):
                    unlock_element = AXeElement(self.udid, unlock_element_data, screen_info, self.logger)
                    unlock_element.tap()
                    return screen_info
            except Exception:
                pass
    
    @staticmethod
    def get_screen_info(udid: str, logger=None) -> Dict[str, int]:
        """Get screen information using AXe CLI."""
        app_element = AXeUtils.find_element_in_ui(udid, AXeUtils.application, logger=logger)
        if app_element and 'frame' in app_element:
            return app_element['frame']
        else:
            if logger:
                logger.warning("Could not find application element or frame data, using default screen info")
            # Return default iPad Pro dimensions if we can't get actual screen info
            return {'x': 0, 'y': 0, 'width': 834, 'height': 1194}