Once you have defined your project, let's suppose the custom action is a .NET installer project (it's not mandatory, but it's like that in my sample).

Just go the the list of Custom Actions (right-click on Project/View/Custom Actions), select your special custom action, and in the "Condition" property in the property grid, you can use the following test:

  OriginalDatabase >< "Debug"

The funny syntax (>< means "contains") is documented in Conditional Statement Syntax, OriginalDatabase (case-sensitive) is a property that contains the full path name of the .msi file path that's being installed.

This is documented in Windows Installer's Property Reference. You can use this type of syntax in all Condition properties that are defined on various items (custom actions, files, etc.). Note "Debug" may be too simple as a discriminant piece of text, but you get the idea.

Of course, that supposes you've changed the .msi output name to include the "Debug" text, in Debug configuration, something like this:

Another option would be to always embed the custom action, but do nothing in it if some property is set (to indicate release mode for example). You can use CustomActionData to pass information to custom actions.