AWS Thinkbox Discussion Forums

Custom Apps/Script in Deadline? Maya / T-Rex Miner

Deadline
#1

Hi there, We have a small network of GPU rendering machines using deadline (love it) to control them. Around 45 GPU’s total.

We are also Crypto mining on them now for additional profit when they aren’t being used.

Currently I am manually enabling/disabling them via teamviewer which is quite tedious.
Miners override any GPU render’s so no deadline tasks get through.

I am looking to add the miner task to deadline, so maya jobs will take priority and the miners will stop when a rendering job starts.

I have alot of questions, but is it feasable even to stop/start a simple batch from deadline?

Currently the miner is started via a .bat file with simple tags for speed/fanspeed/memory settings etc.

Thanks for any help

#2

Because the mining task(s) will take a very long time to complete, you might not be able to simply submit this as a normal job with less priority than your Maya jobs.

I think my personal inclination is to run a custom Event plugin, triggering via the OnHouseCleaningCallback, which happens every minute by default (configurable in the repository I think).

In this hook, you could query the repository to see if it had any pending jobs. If not, you could send the start-mining command to each worker and also mark it as disabled (preventing it from picking up jobs). Then whenever you detect pending jobs, reverse this process by sending the stop-mining signal and enable the worker again.

If that sounds like something you’re interested in, we could help you get started with an Event-plugin, or you could just dissect one of the ones that come with the Repository install.

2 Likes
#3

Oh wow, thanks so Much Daniel,
Your plan sounds perfect. I looked through the custom plugins section of the documentation, but as a non-coder, I’m not sure how much progress I will make. I would love some help with creating it if you have any spare time to assist. If not I can look into hiring someone (Eth/Ergo) Let me know. In the meantime I’ll read up on how plugins work on deadline to get a better overview of the process. I don’t have any python experience, but I can sure have a go at modifying something and testing.
Thanks again

#4

This is completely untested, but hopefully enough to get you started.

Please be warned though, I can already feel it in my bones that you’ll have some race conditions here, so you may want to elaborate heavily on this code going forward – but you won’t really know until you start seeing some issues with mining/rendering overlaps, etc.

Both of these files should be placed in a folder inside <DeadlineRepo>/custom/events. For instance <DeadlineRepo>/custom/events/FancyMiningEvent.

FancyMiningEvent.param:

[State]
Type=Enum
Items=Global Enabled;Disabled
Label=State
Category=Options
CategoryOrder=0
Default=Global Enabled
Description=How this event plug-in should respond to events. Eg. enabled or disabled

FancyMiningEvent.py

# -*- coding: utf-8 -*-
'''Fancy Mining Event'''

from Deadline.Events import DeadlineEventListener
from Deadline.Scripting import RepositoryUtils


def GetDeadlineEventListener():
    '''Return the main DeadlineEventListener class

    Deadline calls this to get an instance of the main event listener.

    Returns:
        DeadlineEventListener: main event listener
    '''

    return FancyMiningEvent()


def CleanupDeadlineEventListener(deadlineEventPlugin):
    '''Called when deadline stops using the plugin.

    This is the function that Deadline calls when the event plugin is
    no longer in use so that it can get cleaned up.

    Args:
        deadlineEventPlugin (subclass of DeadlineEventListener): Plugin to clean up
    '''

    deadlineEventPlugin.Cleanup()


class FancyMiningEvent(DeadlineEventListener):
    '''Main event plugin entry'''

    def __init__(self):
        '''Initialize the callbacks and class members'''

        self.OnHouseCleaningCallback += self.OnHouseCleaning

        # Not at all sure if this is correct or in any way a complete list
        self.activeJobStatuses = [
            'queued',
            'pending',
            'rendering'
        ]

    def Cleanup(self):
        '''Remove the callbacks and other cleanup tasks'''

        del self.OnHouseCleaningCallback

    def OnHouseCleaning(self):
        '''Triggers on housecleaning intervals

        Usually every minute or so, but beware of race conditions!
        '''

        self.LogInfo('FancyMiningEvent->OnHouseCleaning triggered')

        havePendingJobs = False

        # Iterate jobs
        jobs = RepositoryUtils.GetJobs(True)
        for job in jobs:
            if job.JobStatus.lower() in self.activeJobStatuses:
                havePendingJobs = True
                break

        # Iterate workers
        workerSettings = RepositoryUtils.GetSlaveSettingsList(True)
        for worker in workerSettings:

            # Stop mining
            if havePendingJobs:
                
                # Worker is active, no action required
                if worker.SlaveEnabled:
                    continue

                # Worker is disabled: stop mining and enable
                # TODO: Send your STOP MINING command here
                # Maybe something like this? https://docs.thinkboxsoftware.com/products/deadline/10.1/2_Scripting%20Reference/class_deadline_1_1_scripting_1_1_slave_utils.html#a8ac33b07adbebb04c1941ecc42c819b1
                worker.SlaveEnabled = True
                RepositoryUtils.SaveSlaveSettings(worker)

            # Start mining
            else:
                
                # Worker is disabled, no action required
                if not worker.SlaveEnabled:
                    continue

                # Worker is enabled: start mining and disable
                # TODO: Send your START MINING command here
                # Maybe something like this? https://docs.thinkboxsoftware.com/products/deadline/10.1/2_Scripting%20Reference/class_deadline_1_1_scripting_1_1_slave_utils.html#a8ac33b07adbebb04c1941ecc42c819b1
                worker.SlaveEnabled = False
                RepositoryUtils.SaveSlaveSettings(worker)

Please read through the comments and see if it all makes sense to you. You could try running this without actually implementing the start/stop mining commands, which would let you monitor the performance of your script without risking crazy behavior.

1 Like
#5

Thankyou Daniel!!
I’ll dig into it now, so it all makes sense

#6

Hi Daniel, I have just now come back to this project.

Can pools/regions still apply? or will every machine with the custom script run whenever the deadline slave is open?

I was checking the log for my local slave and noticed:
“Skipping pending job scan because it is not required at this time” Is there a way to override to force housekeeping every interval? - 30 sec’s in my case.

I’m going to have so many questions getting this into an editable state.

I wonder… are there user’s here familiar with Python/Deadline that might be interested in customizing the script for my purposes? And also - is contracting allowed on your forums?

#7

As far as I know we don’t have any rules against contracting, though I’d rather that was all kept in direct messages.

For the problem at hand, I wonder if creating the mining job(s) as very low priority interruptible jobs might simplify things. The idea being that the Workers are always working on the mining job, but are constantly checking for more important work to do. And when they find a higher priority job they drop the mining job and instead go to work on the non-mining jobs.

I don’t know what that would look like long-term as I can’t remember if a Worker dropping a task in favor of another creates a log or an error. But it would be a much simpler test at the very least.

1 Like
#8

Hi Justin,
I have setup a simple command line submission (manual)
calling a .BAT file which in turn gives parameters to the miner.exe

This starts OK, but the window is visible. Do you have any idea how to hide the process? So it is just the worker visible, not the DOS window.

Oh also, one odd thing. My slave keeps changing to DISABLED when finishing or failing a task, even if I suspend the job, the slave stops, then switches to Disabled.

PS. the job interrupt idea works fine. Thanks for that
Thanks!

#9

I’m sure there’s a way to hide the DOS window but I don’t know it.

Something is marking your Worker as Disabled as they won’t do that to themselves. Right click on the Worker and check the Worker history to see if there’s any mention of some event scripts editing the Worker’s state.

#10

Thanks,
The disabled issues seems to have disappeared by itself. Just the hiding window issue now!

#11

Are you sure the disabled issue isn’t (wasn’t) related to the scaffolding Event plugin I posted above? Feels like they could be connected, if you messed around with that for a bit before moving on :slight_smile:

#12

Yeah could be Daniel, I global disabled that event at some point, not sure if that was the trigger for the disabling.

#13

I read Daniel’s script this time, that’s 100% what was going on there. :smiley:

As for hiding the command prompt, you could re-install the client as a service so that everything Deadline runs is run by a service user. Which has no ability to create GUIs, so the prompt window would disappear but so would the worker. And you’d have to launch the Monitor from the executable instead of through the Launcher.

1 Like
#14

Hey Just an update. this method of submitting and controlling the mining slaves has been working really well. The mining job gives up nodes when a rendering job appears.

We just switched to using a deadline user which runs the slave on each PC.
Other users log in and use the PC at the same time.

Some machines run fine and some give the error below. The machines are identical in terms of deadline privileges.
ps. If it helps - I know for a fact that a machine which previously gave the same error below is now fine, and vice versa, machines that were fine now have this error.

Any idea on the cause?

ERROR:

2022-04-20 09:56:42: 0: Got task!
2022-04-20 09:56:42: 0: Render Thread - Render State transition from = ‘ReceivedTask’ to = ‘Other’
2022-04-20 09:56:42: 0: Loading Job’s Plugin timeout is Disabled
2022-04-20 09:56:43: ERROR: 0: An exception occurred: Attempted to perform an unauthorized operation. (System.UnauthorizedAccessException)
2022-04-20 09:56:43: 0: Render Thread - Render State transition from = ‘Other’ to = ‘WaitingForTask’
2022-04-20 09:56:43: ERROR: Scheduler Thread - Render Thread 0 threw an unexpected error:
2022-04-20 09:56:43: >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
2022-04-20 09:56:43: Exception Details
2022-04-20 09:56:43: UnauthorizedAccessException – Attempted to perform an unauthorized operation.
2022-04-20 09:56:43: Exception.TargetSite: Int32 SetSecurityInfo(System.Security.AccessControl.ResourceType, System.String, System.Runtime.InteropServices.SafeHandle, System.Security.AccessControl.SecurityInfos, System.Security.Principal.SecurityIdentifier, System.Security.Principal.SecurityIdentifier, System.Security.AccessControl.GenericAcl, System.Security.AccessControl.GenericAcl)
2022-04-20 09:56:43: Exception.Data: ( )
2022-04-20 09:56:43: Exception.Source: System.Security.AccessControl
2022-04-20 09:56:43: Exception.HResult: -2147024891
2022-04-20 09:56:43: Exception.StackTrace:
2022-04-20 09:56:43: at System.Security.AccessControl.Win32.SetSecurityInfo(ResourceType type, String name, SafeHandle handle, SecurityInfos securityInformation, SecurityIdentifier owner, SecurityIdentifier group, GenericAcl sacl, GenericAcl dacl)
2022-04-20 09:56:43: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, SafeHandle handle, AccessControlSections includeSections, Object exceptionContext)
2022-04-20 09:56:43: at System.Security.AccessControl.NativeObjectSecurity.Persist(String name, AccessControlSections includeSections, Object exceptionContext)
2022-04-20 09:56:43: at System.Security.AccessControl.FileSystemSecurity.Persist(String fullPath)
2022-04-20 09:56:43: at FranticX.IO.Directory2.CreateWindowsDirectoryWithPermissions(String path, DirectorySecurity directorySecurity)
2022-04-20 09:56:43: at Deadline.IO.DeadlineClientPath.a(String bwj, UserInfo bwk)
2022-04-20 09:56:43: at Deadline.IO.DeadlineClientPath.CreateDirectoryWithMaxTwoUserAccess(String path, UserInfo additionalAllowedUser)
2022-04-20 09:56:43: at Deadline.IO.DeadlineClientPath.GetDeadlineClientSlaveJobPluginsFolder(String workerName, String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2022-04-20 09:56:43: at Deadline.Slaves.SlaveSettings.GetSlavePluginPath(String jobId, Boolean createIfMissing, Boolean updatePermissions, UserInfo jobUser)
2022-04-20 09:56:43: at Deadline.Slaves.SlaveRenderThread.e(String ajs, Job ajt, CancellationToken aju)
2022-04-20 09:56:43: at Deadline.Slaves.SlaveRenderThread.b(TaskLogWriter ajo, CancellationToken ajp)
2022-04-20 09:56:43: at Deadline.Slaves.SlaveRenderThread.a()
2022-04-20 09:56:43: <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<

Privacy | Site terms | Cookie preferences