Quick answer: Disable Raycast Target on every UI Image or Text that doesn’t need to be clicked. For gameplay input, gate it with EventSystem.current.IsPointerOverGameObject().

A player click on a 3D enemy does nothing. The same click on empty space — same. You remove the HUD and clicks register again. The HUD was eating the click without obviously intercepting it because Unity treats every Image as a click target by default.

The Default That Causes the Problem

When you add an Image, Text, or RawImage to a Canvas, the component has Raycast Target ticked on. This tells the Canvas’s GraphicRaycaster to consider this element when resolving pointer events. A full-screen background panel, a static title bar, a decorative frame — all of them claim hits even though they have no associated click handler.

The result: EventSystem.current.IsPointerOverGameObject() returns true everywhere the panel covers; your custom 3D click handler sees no events because the UI consumed them.

Fix 1: Disable Raycast Target on Decorations

Open every Image and Text in your Canvas. If the element is purely visual — backgrounds, title text, decorative icons, scoreboard counters — uncheck Raycast Target. Keep it checked for Buttons, Toggles, Sliders, and any other interactable.

A practical default is to disable it project-wide and only enable on interactables. Add this Editor menu command to scan for excessive raycast targets:

[MenuItem("Tools/Disable Raycast Targets on Decorations")]
static void Disable()
{
    foreach (var img in FindObjectsOfType<Image>())
        if (img.GetComponent<Button>() == null && img.GetComponent<Toggle>() == null)
            img.raycastTarget = false;
}

Fix 2: Gate Gameplay Input on Pointer-Over-UI

Even with raycasts trimmed, you still want Buttons to swallow clicks while letting world clicks through on bare canvas regions. In your gameplay click handler:

using UnityEngine.EventSystems;

void Update()
{
    if (Mouse.current.leftButton.wasPressedThisFrame())
    {
        if (EventSystem.current != null &&
            EventSystem.current.IsPointerOverGameObject())
            return;   // click was over UI; UI handles it

        DoWorldClick();
    }
}

For touch input, pass the finger ID:

foreach (var touch in Touchscreen.current.touches)
{
    if (touch.press.wasPressedThisFrame() &&
        !EventSystem.current.IsPointerOverGameObject(touch.touchId.ReadValue()))
    {
        DoWorldClick();
    }
}

Fix 3: Layer Filtering on a Canvas

If you need certain UI to never block raycasts at all — for example, a floating damage number that shouldn’t intercept clicks — place it on a separate Canvas with the GraphicRaycaster component removed entirely. Pointer events still flow through that Canvas to whatever is below.

Performance Side Effect

Every Raycast Target costs a per-frame check during pointer events. A Canvas with 500 raycast-target Images forces the raycaster to test all 500 each frame the mouse moves. Disabling Raycast Target on non-interactives can save a measurable chunk of UI processing on busy HUDs.

Verifying

Use Window → Analysis → UI Debugger (or the Frame Debugger on a canvas-heavy scene) to see active raycast targets. Toggle the HUD Canvas off in play mode — if 3D clicks register only with the Canvas disabled, you still have a stray Raycast Target enabled somewhere. Use the menu command above to bulk-disable.

“Every Raycast Target is a stolen click waiting to happen. Default it off, enable it where you need it.”

A full-screen Image is the single most common cause — check your background panel first.