Houdini 20.5 and worker GPU affinity is not working

I was trying to use both concurrent tasks and limiting to one GPU, as well as having 2 workers on one machine with GPU affinity set for each to use 1 GPU, but it seems that karma does not support that option at the moment and I can see both GPUs fully utilized even when running 1 GPU limit or worker with single GPU selected.
Any way to get this working properly?