exactly, mostly they force to use GUI installation by default, for example: centos 7 and recent rhel.I always keep all my servers as slim as possible - I think one has a full GUI install because of a requirement that some users could get VNC sessions occasionally but otherwise they are all command line only. I use core edition on windows servers where possible as well.
One other option for remote GUI management of linux servers is to use X forwarding over an SSH connection. You can run the odd GUI application from the server without running an X server on the server. It's kind of like VNC or RDP but for just a single application.
when installation is done, I just switch to text based screen. when needed, I can run X on command line by typing startX .