See the question and my original answer on StackOverflow

You say you're using Windows 10, but DWMWA_WINDOW_CORNER_PREFERENCE is supported only starting with Windows 11 Build 22000 as official doc says:

DWMWA_WINDOW_CORNER_PREFERENCE Use with DwmSetWindowAttribute. Specifies the rounded corner preference for a window. The pvAttribute parameter points to a value of type DWM_WINDOW_CORNER_PREFERENCE.

This value is supported starting with Windows 11 Build 22000.