I will cite the official Windows documentation. Although the doc is about Native API, Winforms uses the same underlying technology:

An application invalidates a portion of a window and sets the update region by using the InvalidateRect or InvalidateRgn function. These functions add the specified rectangle or region to the update region, combining the rectangle or region with anything the system or the application may have previously added to the update region.

The InvalidateRect and InvalidateRgn functions do not generate WM_PAINT messages. Instead, the system accumulates the changes made by these functions and its own changes. By accumulating changes, a window processes all changes at once instead of updating bits and pieces one step at a time.

Calling .NET Refresh() is equivalent of calling InvalidateAll() + Update(). InvalidateAll marks the whole screen as invalid, and Update() forces the process of redrawing what's invalid, so the whole screen. You can optimize your program if you just invalidate "manually" what you know has changed.

That's what I did in my modified sample. Instead of calling Refresh(), I added a new oldRcRect variable to be able to invalidate old state and new state, and a RefreshOnMove() method like this (and I replaced most Refresh calls by a RefreshOnMove call) :

    private void RefreshOnMove()
    {
        // invalidate the old rect (+ size of red box)
        var rc = oldRcSelect;
        rc.Inflate(3, 3);
        Invalidate(rc);

        // invalidate the new rect (+ size of red box)
        // note you can almost omit this second one, but if you move the mouse really fast, you'll see some red box not fully displayed
        // but the benefit is small, something like a 3 x width/height rectangle
        rc = rcSelect;
        rc.Inflate(3, 3);
        Invalidate(rc);

        // each time you call invalidate, you just accumulate a change
        // to the change region, nothing actually changes on the screen

        // now, ask Windows to process the combination of changes
        Update();
    }

PS: about my comment about inner region, I just mean it may be possible to avoid invalidating the content of the white box each time as well, but it's more complex.