How does COM/Automation do IPC under the hood?
See the question and my original answer on StackOverflowDCOM was originally added as an extension to COM, precisely for cross apartment calls. Note cross apartment calls are not always from process to process. A process can have many apartments (0 or 1 MTA and/or 0 to n STAs, etc.) . There is at least one apartment per process, etc.
DCOM, some kind of a "middleware", needed a technology for all this low-level work: data representation, caller/callee convention, memory management, wire marshaling, session handling, security, error handling, etc. so Microsoft naturally used the in-house implementation of DCE/RPC: MSRPC. Note that as Microsoft says on its site,
"With the exception of some of its advanced features, Microsoft RPC is interoperable with other vendors’ implementations of OSF RPC."
There was some tentative work to have all this implemented by other vendors, but they were basically killed by the rise of the internet and HTTP.
Also, note this RPC uses Windows Messages for STA apartement messages. I suggest you read carefully this document (not available any more on Microsoft site, shame on them :-) for more details: DCOM Architecture by Markus Horstmann and Mary Kirtland - July 23, 1997 .
See also this interesting case study about a DCOM/RCP issue that should tell you a lot of how RPC over Windows message works under the scene: Troubleshooting a DCOM issue: Case Study