A Solution Folder is a project in a given solution like any other project.

To remove a project from a solution, you cannot call proj.Delete() which is not implemented as per official documentation:

Removes the project from the current solution.

Note that this method is not currently implemented.

Instead, you must must use dte.Solution.Remove(proj); and that should work for solution folders as well.