I've noticed a strange thing: the PowerShell remoting works much faster if you use the remote computer's IP address for connection rather than a DNS name. Well, not everything is faster but the re-connections. PowerShell uses the HTTP (or HTTPS) protocol to send the data. This connection is cached between the successive commands but if you don't enter anything for a few minutes, this connection gets dropped and then re-established when you enter the next command. This re-establishment, and also the initial connection, works much faster with the IP address than with the DNS name. The effect is most pronounced if connecting over a WAN, such as to a VM in Azure. Don't know why. There shouldn't be that much delay with the DNS. It's a mystery.
Of course, this won't work with HTTPS connections, HTTPS connections require that the machine name matches the name in the certificate.