I wonder if it's actually the desktop process crashing when you right click, rather than a refresh - it would appear the same. Do you have any old shell extensions that may not be compatible with Windows 11 installed?
Was there anything added to your right click menu can you remember, software/plugins can often add items to the rightclick menu and I think one of these is causing a crash.
If you check "event viewer" (search for this on start search bar), you may be able to find the error/alert notification which will track down the offending application.