Fix: Unity Input System Mouse Position Wrong With RenderTexture

Here is how to fix Unity picking that works in the main view but breaks when you display a Camera through a RawImage of a RenderTexture (split-screen, security camera, mirror). Mouse.position reports main-display pixels; the RenderTexture is positioned and scaled by the RawImage; the two coordinate systems do not match.

The Symptom

You set up a camera with a target texture, display via RawImage in UI. Click on objects shown in the RawImage; the picking ray is in the wrong place. Worse, scaling the RawImage breaks alignment further.

What Causes This

Mouse.position is screen-space. Pixels on the main display. The camera no longer renders to that display.

RawImage transforms the texture. Position, rotation, scale, anchoring all warp the on-screen coordinates relative to the texture.

ScreenPointToRay assumes screen space. Calling it on the RenderTexture camera with main-display Mouse.position uses the wrong coordinate system.

The Fix

Step 1: Convert mouse to RawImage local coords.

using UnityEngine;
using UnityEngine.UI;
using UnityEngine.InputSystem;

public class RTPicker : MonoBehaviour
{
    [SerializeField] private RawImage rawImage;   // displays the RT
    [SerializeField] private Camera rtCamera;     // renders to RT

    public bool PickFromMouse(out RaycastHit hit)
    {
        Vector2 screenPos = Mouse.current.position.ReadValue();
        RectTransformUtility.ScreenPointToLocalPointInRectangle(
            rawImage.rectTransform, screenPos, null, out Vector2 local);

        // local is centered; convert to 0-1 UV
        Rect r = rawImage.rectTransform.rect;
        Vector2 uv = new Vector2(
            (local.x - r.x) / r.width,
            (local.y - r.y) / r.height);

        if (uv.x < 0 || uv.x > 1 || uv.y < 0 || uv.y > 1)
        {
            hit = default;
            return false;   // outside the RawImage
        }

        Vector2 rtPixel = new Vector2(uv.x * rtCamera.pixelWidth, uv.y * rtCamera.pixelHeight);
        Ray ray = rtCamera.ScreenPointToRay(rtPixel);
        return Physics.Raycast(ray, out hit);
    }
}

Pipeline: screen mouse → RawImage local → UV → RT pixel → ScreenPointToRay.

Step 2: Handle uses Camera = null in ScreenPointToLocal. If the RawImage is on a Screen Space Overlay canvas, pass null as camera. For ScreenSpace - Camera or WorldSpace, pass the canvas’s render camera.

Step 3: For UV-mapped picking, sample the RT directly. If you only need to know what world position is under the mouse on the RT (for tooltip text), reading rt pixels is unnecessary — the ScreenPointToRay already gives you a 3D ray. Only the conversion is the work.

Step 4: Verify with Debug.DrawRay.

Debug.DrawRay(ray.origin, ray.direction * 100f, Color.red, 0.1f);

Compare visually to expectations. If the ray is way off, your UV math has a flip or scale error.

Step 5: For Y-flipped textures. Some RenderTexture setups have flipped Y; if rays are vertically inverted, swap uv.y with 1 - uv.y.

Understanding the issue

Input handling sits between hardware and gameplay. Hardware has its own protocol; gameplay has its own model. When these don't agree, the player perceives unresponsiveness even though every layer is technically functional.

The specific bug described above is the kind that surfaces during integration rather than unit testing. It depends on a combination of factors: the asset configuration, the runtime state, the platform's specific behavior. In isolation, each piece looks correct; in combination, the bug emerges. This is why thorough integration testing - playing the actual game in realistic conditions - catches things that automated tests miss.

Why this happens

This bug class disproportionately affects late-stage development. The work to surface it is interactive testing in realistic conditions, which only really happens after the gameplay is in place and assets are populated. Catching it early requires deliberate testing of conditions that look unimportant.

At the engine level, the behavior comes from a deliberate design decision in Unity. The engine team chose a particular trade-off - usually performance versus convenience, or generality versus specificity - and that trade-off has consequences when you push against it. Understanding the trade-off is what turns 'this bug is mysterious' into 'this bug is the expected consequence of this design'.

Verifying the fix

Verifying this fix in isolation is straightforward: reproduce the bug, apply the change, confirm the bug no longer reproduces. The harder verification is regression - did this fix introduce a new bug elsewhere? Run your standard regression suite, plus any tests that exercise the same code path with different inputs.

Reproducibility is the prerequisite for verification. If you can't reliably reproduce the bug pre-fix, you can't reliably verify it post-fix. Spend time getting a clean reproduction before you write any fix code. The fix is fast once you understand the reproduction; the reproduction is the slow part.

Variations to watch for

There's almost always a less obvious case where the same problem applies. The reported case is the one a player hit; the related cases hide because they're rarer or affect fewer players. After fixing the reported case, search the codebase for the pattern - one fix often unlocks several.

Adjacent bugs often share a root cause. After fixing the case you've found, spend an hour searching the codebase for similar patterns. What's the same call with different arguments? The same data flow with a different entity type? The same lifecycle issue in a sibling system? Each match is a candidate for the same fix, or a related fix that prevents future bugs of the same class.

In production

Live games surface this bug class at scale. What's a rare edge case in development becomes a daily occurrence once you have a few thousand concurrent players. The class isn't 'this player has a unique setup'; it's 'one in N thousand sessions will trigger this exact combination'.

When triaging a similar issue in production, prioritize gathering data over hypothesizing causes. A player report describes a symptom; what you need is a build SHA, a session timestamp, and ideally a screen recording or session replay. With those, the bug becomes tractable. Without them, you're guessing at hypothetical reproductions that may not match what the player actually hit.

Performance considerations

If this issue manifests under high load (many actors, many particles, many network connections), profile the post-fix code path with realistic counts. The original cost was a bug; the new cost is real work, and real work has a budget.

Diagnostic approach

The diagnostic tools available depend on your engine and platform. Use the engine's native profilers and debug overlays before reaching for external tools. The native tools have context that external tools lack - they know which subsystem owns the code, which assets are loaded, and what state the engine is in.

For Unity-specific diagnostics, the editor's profiler is the canonical starting point. Capture a representative frame with the symptom present; compare against a frame without the symptom; the diff often points directly at the cause. If the symptom is non-deterministic, capture multiple frames and look for the pattern - the cause is usually a state transition or a specific input value rather than a continuous effect.

Tooling and ecosystem

The tooling around this bug class matters as much as the fix itself. Good logging, accessible profilers, and clear error messages turn 30-minute investigations into 5-minute ones. If your project doesn't have visibility into this code path, the first fix should add the visibility - the second fix uses it.

Within Unity, the relevant diagnostic surfaces include the standard frame debugger, memory profiler, and engine-specific debug overlays. Each one shows a different facet of what's happening. The frame debugger reveals draw call ordering and state transitions; the memory profiler shows allocation patterns; the debug overlay reveals per-system state. Bugs that resist one tool usually surrender to another - the trick is knowing which tool to reach for first.

Edge cases and pitfalls

Platform-specific edge cases are worth enumerating explicitly. iOS handles backgrounding differently than Android; Windows handles focus changes differently than macOS. A fix that works on the development platform may not work on every target. Test on each shipping platform deliberately.

When writing a regression test for this fix, focus on the boundary conditions that surfaced the original bug. Tests that exercise the happy path catch obvious regressions; tests that exercise the boundary catch the subtler regressions that look like new bugs but are really the original returning. The latter are the tests that earn their keep over the long life of the project.

Team communication

When this bug class affects multiple teams (often the case for cross-system issues), early communication prevents duplicate work. The team that owns the symptom may not own the cause. A 15-minute conversation at the start of triage often saves hours of independent investigation.

If this fix touches a system several engineers work in, a short writeup in the team's engineering channel helps. Not a full design doc - a paragraph explaining what was wrong, what's fixed, and what to watch for. Future engineers encountering similar symptoms will search for the fix; making it findable is a small investment that pays back later.

“Screen mouse is not RT mouse. Convert through RawImage local coords. ScreenPointToRay on the RT camera does the rest.”

Related Issues

For touch input, see Touch Not Firing. For control scheme, see Control Scheme Switching.

RectTransformUtility.ScreenPointToLocal then UV then RT pixel then ScreenPointToRay. Picking works.