If there was a way to set Blender’s GPU settings from the command line we could inherit the GPU affinity from the Worker, just like we do with Redshift.
However I’m failing to find anything on this page of blender docs and the command line flags only mention flags for troubleshooting GPU rendering. 
Otherwise, I think @danielskovli has got a solid plan here!
Edit: I bet if you were particularly clever with the script here Forcing Blender to use GPU - #16 by karpreet you could set each Worker to use a preconfigured GPU. Maybe even read the Worker’s GPU affinity and pass that along to the script.