From time to time, I've needed to remove hosts from and AppFabric cache cluster and always end up having to google what to do. Here are the steps to follow on the host you wish to remove from the cache cluster:

  1. Open up a Windows Powershell (preferably Caching Administration Windows Powershell) 
  2. If you simply openened up Windows Powershell run the following command: Import-Module DistributedCacheAdministration
  3. Run: Import-Module DistributedCacheConfiguration
  4. Run: Stop-CacheHost -HostName <host to remove> -CachePort <default is 22233>
  5. Run: Unregister-CacheHost
  6. Run: Rmove-CacheHost

Finally, run Restart-CacheCluster on another machine in the cache cluster, to make sure your changes take effect properly and to ensure the host one you wanted to remove, no longer appears. I am pretty confident that the steps above are correct, but I have had issues with not being able to connect to the cache cluster after removing a host. CurrPorts was a great help to troubleshoot what was happening as it shows what applications are listening on what ports.