Fix: Unity Input System Mouse Position Wrong With RenderTexture

Here is how to fix Unity picking that works in the main view but breaks when you display a Camera through a RawImage of a RenderTexture (split-screen, security camera, mirror). Mouse.position reports main-display pixels; the RenderTexture is positioned and scaled by the RawImage; the two coordinate systems do not match.

The Symptom

You set up a camera with a target texture, display via RawImage in UI. Click on objects shown in the RawImage; the picking ray is in the wrong place. Worse, scaling the RawImage breaks alignment further.

What Causes This

Mouse.position is screen-space. Pixels on the main display. The camera no longer renders to that display.

RawImage transforms the texture. Position, rotation, scale, anchoring all warp the on-screen coordinates relative to the texture.

ScreenPointToRay assumes screen space. Calling it on the RenderTexture camera with main-display Mouse.position uses the wrong coordinate system.

The Fix

Step 1: Convert mouse to RawImage local coords.

using UnityEngine;
using UnityEngine.UI;
using UnityEngine.InputSystem;

public class RTPicker : MonoBehaviour
{
    [SerializeField] private RawImage rawImage;   // displays the RT
    [SerializeField] private Camera rtCamera;     // renders to RT

    public bool PickFromMouse(out RaycastHit hit)
    {
        Vector2 screenPos = Mouse.current.position.ReadValue();
        RectTransformUtility.ScreenPointToLocalPointInRectangle(
            rawImage.rectTransform, screenPos, null, out Vector2 local);

        // local is centered; convert to 0-1 UV
        Rect r = rawImage.rectTransform.rect;
        Vector2 uv = new Vector2(
            (local.x - r.x) / r.width,
            (local.y - r.y) / r.height);

        if (uv.x < 0 || uv.x > 1 || uv.y < 0 || uv.y > 1)
        {
            hit = default;
            return false;   // outside the RawImage
        }

        Vector2 rtPixel = new Vector2(uv.x * rtCamera.pixelWidth, uv.y * rtCamera.pixelHeight);
        Ray ray = rtCamera.ScreenPointToRay(rtPixel);
        return Physics.Raycast(ray, out hit);
    }
}

Pipeline: screen mouse → RawImage local → UV → RT pixel → ScreenPointToRay.

Step 2: Handle uses Camera = null in ScreenPointToLocal. If the RawImage is on a Screen Space Overlay canvas, pass null as camera. For ScreenSpace - Camera or WorldSpace, pass the canvas’s render camera.

Step 3: For UV-mapped picking, sample the RT directly. If you only need to know what world position is under the mouse on the RT (for tooltip text), reading rt pixels is unnecessary — the ScreenPointToRay already gives you a 3D ray. Only the conversion is the work.

Step 4: Verify with Debug.DrawRay.

Debug.DrawRay(ray.origin, ray.direction * 100f, Color.red, 0.1f);

Compare visually to expectations. If the ray is way off, your UV math has a flip or scale error.

Step 5: For Y-flipped textures. Some RenderTexture setups have flipped Y; if rays are vertically inverted, swap uv.y with 1 - uv.y.

“Screen mouse is not RT mouse. Convert through RawImage local coords. ScreenPointToRay on the RT camera does the rest.”

Related Issues

For touch input, see Touch Not Firing. For control scheme, see Control Scheme Switching.

RectTransformUtility.ScreenPointToLocal then UV then RT pixel then ScreenPointToRay. Picking works.