We don’t seem to be able to successfully use the Remote Control > Machine Commands recently, getting a “No connection could be made because the target machine actively refused it” error, this also affects the power management, so Pulse isn’t able to shut idle machines down.
It worked until recently, we tried updating to deadline 10.1.9.2, but no change.
The Render Nodes don’t have any firewall enabled, and all are running the latest Windows 10 2004, Win Build 19041.vb_release.191206-1406
Ok, some progress, it seems to be related to Pulse, currently on 10.1.9.2
When the machine has been running for some time any machine commands that are channeled through Pulse are being refused by the server itself, that’s what the message ‘No connection could be made because the target machine actively refused it’ relates to.
However if the server is restarted then the commands are accepted and transmitted to the render slave?
If Pulse is restarted I’m getting the following in the log…
Listener Thread - Received message: PulseRemoteControl render54 LongOperation ShutdownMachine
And the output in the Remote Command window…
Failure: InitiateShutdown failed: Access is denied.
(System.Exception)
And could you try opening a Monitor on one of the render nodes and trying to send it remote commands? Just to see if it’s a hiccup in the [your computer]->Pulse->Render node path by changing it to Render node->Pulse->Render Node.
I haven’t included the whole log here as it gets quite large, but I have included below the lines that are created as soon as the remote shutdown is created…
OK, I’m pretty sure I’ve tracked down the problem, so for the record, and anybody else who encounters the same issue.
It looks like it was caused by one of the recent Windows 10 updates, we don’t update the render slaves too often to reduce issues and avoid problem like this one, however we did go from Windows 10 1709 to 2004 recently, via the intermediate updates.
It looks to be related to the local security policy setting “force shutdown from a remote machine”, and although as far I can determine this has been around for some time and we have not had the problem previously, at least until the recent windows updates, it seems to now be essential to set it correctly since one of the recent Windows 10 updates (Sorry can be specific, but my suspicion is 1804).
You can see this setting by running secpol.msc, and in Local Policies >> User Rights Assignment >> Force Shutdown from Remote System, you have to add the users or groups who are allowed to remotely shutdown/restart a machine. I also set “Shutdown the system” in the same section for good measure, not sure if this step is essential.
To resolve the issue I enforced a Group Policy Setting on the domain to set these policies, under Computer Configuration >> Policies >> Windows Settings >> Security Settings >> Local Policies >> User Rights Assignment >> Force shutdown from a remote system
Once the machines have been restarted the issue seems to be resolved
This error is a network-related error occurred while establishing a connection to the Server. It means that the error is occurring because there is no server listening at the hostname and port you assigned. It literally means that the machine exists but that it has no services listening on the specified port . So, no connection can be established.
Try running netstat -anb from the command line to see if there’s anything listening on the port you were entered. If you get nothing, try changing your port number and see if that works for you. In Windows operating systems, you can use the netstat services via the command line (cmd.exe) . On Linux you may need to do netstat -anp instead.
The target machine actively refused it occasionally , it is likely because the server has a full ‘backlog’ . Regardless of whether you can increase the server backlog , you do need retry logic in your client code, sometimes it cope with this issue; as even with a long backlog the server might be receiving lots of other requests on that port at that time.
