Technically, a "server" is just a program that delivers a service. So, in COM, you have clients and servers. Of course, today we think HTTP/REST/SOA/micro service, etc. But COM existed well before that, even before an obvious concept such as "the network" (not even speaking about inter network/internet) was not so obvious.

Let's quote the official doc: COM Clients and Servers

A COM client is whatever code or object gets a pointer to a COM server and uses its services by calling the methods of its interfaces.

A COM server is any object that provides services to clients; these services are in the form of COM interface implementations that can be called by any client that is able to get a pointer to one of the interfaces on the server object.

There are two main types of servers, in-process and out-of-process. In-process servers are implemented in a dynamic linked library (DLL), and out-of-process servers are implemented in an executable file (EXE). Out-of-process servers can reside either on the local computer or on a remote computer.

In addition, COM provides a mechanism that allows an in-process server (a DLL) to run in a surrogate EXE process to gain the advantage of being able to run the process on a remote computer. For more information

It's true that many COM servers are implemented in DLL, but it's not always the case. Examples of well-known out-of-process servers: Word, Excel, etc.

Plus, as said in the last phrase, you can (almost always) turn an in-process server (so a DLL), into an out-of-process server quite easily just through configuration using surrogates. COM supplies a defaut surrogate called "dllhost.exe" that you can see sometimes in the task manager. For example, Windows Explorer uses this to protect itself from rogue in-process extensions.