Hqueue sim freezes after first frame exported

Igor · April 14, 2023, 6:28am

Hi,

I have 2 workstations Win 10 , Houdini 19.5.403 Py3.9 , Houdini Indie and tried to run distributed sim with Deadline. Unfortunately both tasks produced only one frame and hanged out infinitely. Destination folder is on one of the workstation’s shared folder without path expressions. Similar to this //work1/aaa/geo/fluid.$SLICE.$F.bgeo.sc . Hqueue rop connected to Deadline submitter.

Native Hqueue works fine and creates full bgeo sequences. Here is full log(some personal data changed to abstract).

Last couple lines starts to repeat infinitely so I cut it down.

P.s. this is path to simtracker file in plugin configuration.
C:\Program Files\Side Effects Software\Houdini 19.5.403\houdini\python3.9libs\simtracker.py

Would be really appreciate for any help. Thanks


Log
=======================================================
2023-04-13 22:50:41:  0: Loading Job's Plugin timeout is Disabled
2023-04-13 22:50:41:  0: SandboxedPlugin: Render Job As User disabled, running as current user 'user-b'
2023-04-13 22:50:44:  0: Executing plugin command of type 'Initialize Plugin'
2023-04-13 22:50:44:  0: DEBUG: InitializePlugin: debug logging enabled
2023-04-13 22:50:44:  0: DEBUG: InitializePlugin: m_pluginParamFilename = 'C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\Houdini.param'
2023-04-13 22:50:44:  0: DEBUG: InitializePlugin: m_pluginScriptFilename = 'C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\Houdini.py'
2023-04-13 22:50:44:  0: DEBUG: InitializePlugin: m_pluginPreLoadFilename = 'C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\PluginPreLoad.py'
2023-04-13 22:50:44:  0: DEBUG: InitializePlugin: m_jobPreLoadFilename = 'C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\JobPreLoad.py'
2023-04-13 22:50:44:  0: DEBUG: InitializePlugin: Checking for Plugin Pre-Load
2023-04-13 22:50:44:  0: DEBUG: InitializePlugin: Loading Plugin...
2023-04-13 22:50:44:  0: INFO: Executing plugin script 'C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\Houdini.py'
2023-04-13 22:50:45:  0: INFO: Plugin execution sandbox using Python version 3
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: getting job user
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: setting job filenames
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: Obtaining Deadline plugin object
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: Setting internal variables and delegates
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: Preparing Environment Variables
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: Obtaining network settings
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: Initializing Deadline plugin
2023-04-13 22:50:45:  0: DEBUG: This is a simple plugin job.
2023-04-13 22:50:45:  0: INFO: About: Houdini Plugin for Deadline
2023-04-13 22:50:45:  0: INFO: The job's environment will be merged with the current environment before rendering
2023-04-13 22:50:45:  0: DEBUG: InitializePlugin: returning
2023-04-13 22:50:45:  0: Done executing plugin command of type 'Initialize Plugin'
2023-04-13 22:50:45:  0: Start Job timeout is disabled.
2023-04-13 22:50:45:  0: Task timeout is disabled.
2023-04-13 22:50:45:  0: Loaded job: hqueu_test - /out/distributedsim (d4c437936cad6b1482f91153)
2023-04-13 22:50:45:  0: Executing plugin command of type 'Start Job'
2023-04-13 22:50:45:  0: DEBUG: StartJob: called
2023-04-13 22:50:45:  0: DEBUG: StartJob: Checking for Job Pre-Load
2023-04-13 22:50:45:  0: DEBUG: S3BackedCache Client is not installed.
2023-04-13 22:50:45:  0: DEBUG: GlobalAssetTransferPreLoadJob: called
2023-04-13 22:50:45:  0: INFO: Executing global asset transfer preload script 'C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\GlobalAssetTransferPreLoad.py'
2023-04-13 22:50:45:  0: INFO: Looking for legacy (pre-10.0.26) AWS Portal File Transfer...
2023-04-13 22:50:45:  0: INFO: Looking for legacy (pre-10.0.26) File Transfer controller in C:/Program Files/Thinkbox/S3BackedCache/bin/task.py...
2023-04-13 22:50:45:  0: INFO: Could not find legacy (pre-10.0.26) AWS Portal File Transfer.
2023-04-13 22:50:45:  0: INFO: Legacy (pre-10.0.26) AWS Portal File Transfer is not installed on the system.
2023-04-13 22:50:45:  0: DEBUG: GlobalAssetTransferPreLoadJob: returning
2023-04-13 22:50:45:  0: DEBUG: StartJob: Starting Job...
2023-04-13 22:50:45:  0: DEBUG: StartJob: returning
2023-04-13 22:50:45:  0: Done executing plugin command of type 'Start Job'
2023-04-13 22:50:45:  0: Plugin rendering frame(s): 1
2023-04-13 22:50:45:  0: Executing plugin command of type 'Render Task'
2023-04-13 22:50:45:  0: DEBUG: RenderTasks: called
2023-04-13 22:50:45:  0: DEBUG: RenderTasks: rendering frames 1 to 1
2023-04-13 22:50:45:  0: DEBUG: ManagedProcess.Execute: debug logging enabled
2023-04-13 22:50:45:  0: DEBUG: ManagedProcess.Execute: called
2023-04-13 22:50:45:  0: DEBUG: ManagedProcess.Execute: calling PreRenderTasks()
2023-04-13 22:50:45:  0: INFO: Starting Houdini Job
2023-04-13 22:50:45:  0: DEBUG: ManagedProcess.Execute: setting up child process
2023-04-13 22:50:45:  0: INFO: Stdout Redirection Enabled: True
2023-04-13 22:50:45:  0: INFO: Stdout Handling Enabled: True
2023-04-13 22:50:45:  0: INFO: Popup Handling Enabled: True
2023-04-13 22:50:45:  0: INFO: QT Popup Handling Enabled: False
2023-04-13 22:50:45:  0: INFO: WindowsForms10.Window.8.app.* Popup Handling Enabled: False
2023-04-13 22:50:45:  0: INFO: Using Process Tree: True
2023-04-13 22:50:45:  0: INFO: Hiding DOS Window: True
2023-04-13 22:50:45:  0: INFO: Creating New Console: False
2023-04-13 22:50:45:  0: INFO: Running as user: user-b
2023-04-13 22:50:45:  0: INFO: Executable: "C:\Program Files\Side Effects Software\Houdini 19.5.403\bin\hython.exe"
2023-04-13 22:50:45:  0: INFO: Sim Job: Checking which machine is running the tracking process
2023-04-13 22:50:45:  0: INFO: Sim Job: Worker "ironman" is running the tracking proccess
2023-04-13 22:50:45:  0: INFO: Sim Job: Connecting to Worker machine using IP address "5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e"
2023-04-13 22:50:45:  0: INFO: Argument: "C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\hrender_dl.py" -s 1 "5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e" 8000 -g -d /out/distributedsim -tempdir "C:\ProgramData\Thinkbox\Deadline10\workers\work1\jobsData\d4c437936cad6b1482f91153\0_tempruZnU0" -arnoldAbortOnLicenseFail 1 "C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc"
2023-04-13 22:50:45:  0: INFO: Full Command: "C:\Program Files\Side Effects Software\Houdini 19.5.403\bin\hython.exe" "C:\ProgramData\Thinkbox\Deadline10\workers\work1\plugins\d4c437936cad6b1482f91153\hrender_dl.py" -s 1 "5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e" 8000 -g -d /out/distributedsim -tempdir "C:\ProgramData\Thinkbox\Deadline10\workers\work1\jobsData\d4c437936cad6b1482f91153\0_tempruZnU0" -arnoldAbortOnLicenseFail 1 "C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc"
2023-04-13 22:50:45:  0: INFO: Startup Directory: "C:\Program Files\Side Effects Software\Houdini 19.5.403\bin"
2023-04-13 22:50:45:  0: DEBUG: ManagedProcess.Execute: launching the child process
2023-04-13 22:50:45:  0: INFO: Process Priority: BelowNormal
2023-04-13 22:50:45:  0: INFO: Process Affinity: default
2023-04-13 22:50:45:  0: INFO: Process is now running
2023-04-13 22:50:45:  0: DEBUG: ManagedProcess.Execute: waiting for child process to exit
2023-04-13 22:50:45:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:45:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:45:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:46:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:46:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:46:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:46:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:46:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:46:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:46:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:46:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:47:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:47:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:47:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:47:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:47:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:47:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:47:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:47:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Detected Houdini version: (19, 5, 403)
2023-04-13 22:50:48:  0: STDOUT: Detected Houdini version: (19, 5, 403)
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: ['C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\work1\\plugins\\d4c437936cad6b1482f91153\\hrender_dl.py', '-s', '1', '5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e', '8000', '-g', '-d', '/out/distributedsim', '-tempdir', 'C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\work1\\jobsData\\d4c437936cad6b1482f91153\\0_tempruZnU0', '-arnoldAbortOnLicenseFail', '1', 'C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc']
2023-04-13 22:50:48:  0: STDOUT: ['C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\work1\\plugins\\d4c437936cad6b1482f91153\\hrender_dl.py', '-s', '1', '5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e', '8000', '-g', '-d', '/out/distributedsim', '-tempdir', 'C:\\ProgramData\\Thinkbox\\Deadline10\\workers\\work1\\jobsData\\d4c437936cad6b1482f91153\\0_tempruZnU0', '-arnoldAbortOnLicenseFail', '1', 'C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc']
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Ignore Inputs: True
2023-04-13 22:50:48:  0: STDOUT: Ignore Inputs: True
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: No output specified. Output will be handled by the driver
2023-04-13 22:50:48:  0: STDOUT: No output specified. Output will be handled by the driver
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Driver: /out/distributedsim
2023-04-13 22:50:48:  0: STDOUT: Driver: /out/distributedsim
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Input File: C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc
2023-04-13 22:50:48:  0: STDOUT: Input File: C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:48:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:48:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:48:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:48:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:48:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:48:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Warnings were generated during load.
2023-04-13 22:50:48:  0: STDOUT: Warnings were generated during load.
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Error loading: C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc
2023-04-13 22:50:48:  0: STDOUT: Error loading: C:/ProgramData/Thinkbox/Deadline10/workers/work1/jobsData/d4c437936cad6b1482f91153/hqueu_test.hiplc
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Warning:     The following node types are using incomplete asset definitions:
2023-04-13 22:50:48:  0: STDOUT: Warning:     The following node types are using incomplete asset definitions:
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout:     Driver/deadline
2023-04-13 22:50:48:  0: STDOUT:     Driver/deadline
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Sim control node: /obj/billowysmoke_simulation/DISTRIBUTE_smokeobject1_CONTROLS
2023-04-13 22:50:48:  0: STDOUT: Sim control node: /obj/billowysmoke_simulation/DISTRIBUTE_smokeobject1_CONTROLS
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Sim slice: 1
2023-04-13 22:50:48:  0: STDOUT: Sim slice: 1
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Sim Tracker: 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e
2023-04-13 22:50:48:  0: STDOUT: Sim Tracker: 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Sim Port: 8000
2023-04-13 22:50:48:  0: STDOUT: Sim Port: 8000
2023-04-13 22:50:48:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Begin Path Mapping
2023-04-13 22:50:48:  0: STDOUT: Begin Path Mapping
2023-04-13 22:50:49:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:49:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:49:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:49:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:49:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:49:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:49:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:49:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:50:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:50:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:50:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:50:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:50:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: b''
2023-04-13 22:50:50:  0: STDOUT: b''
2023-04-13 22:50:50:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:50:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:50:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:50:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:51:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:51:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:51:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:51:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:51:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:51:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:51:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:51:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: b''
2023-04-13 22:50:52:  0: STDOUT: b''
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: End Path Mapping
2023-04-13 22:50:52:  0: STDOUT: End Path Mapping
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: ROP type: rop_geometry
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: line was matched by handler ".*ROP type: (.*)"
2023-04-13 22:50:52:  0: STDOUT: ROP type: rop_geometry
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Rendering frame 1 to 50
2023-04-13 22:50:52:  0: STDOUT: Rendering frame 1 to 50
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:52:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:52:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:52:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): True
2023-04-13 22:50:52:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:52:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:52:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: ALF_PROGRESS 0%
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: line was matched by handler ".*ALF_PROGRESS ([0-9]+)%.*"
2023-04-13 22:50:52:  0: STDOUT: ALF_PROGRESS 0%
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Write Connect failure.  code: Bad Address
2023-04-13 22:50:52:  0: STDOUT: Write Connect failure.  code: Bad Address
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My length 7956018259993780077
2023-04-13 22:50:52:  0: STDOUT: My length 7956018259993780077
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:52:  0: STDOUT: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Write Connect failure.  code: Bad Address
2023-04-13 22:50:52:  0: STDOUT: Write Connect failure.  code: Bad Address
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My length 31
2023-04-13 22:50:52:  0: STDOUT: My length 31
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:52:  0: STDOUT: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:52:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Message in error state after connection attempt!
2023-04-13 22:50:52:  0: STDOUT: Message in error state after connection attempt!
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: ---- Pump enters error status ----
2023-04-13 22:50:53:  0: STDOUT: ---- Pump enters error status ----
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Write Connect failure.  code: Bad Address
2023-04-13 22:50:53:  0: STDOUT: Write Connect failure.  code: Bad Address
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My length 2044286080
2023-04-13 22:50:53:  0: STDOUT: My length 2044286080
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:53:  0: STDOUT: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Write Connect failure.  code: Bad Address
2023-04-13 22:50:53:  0: STDOUT: Write Connect failure.  code: Bad Address
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My length 29
2023-04-13 22:50:53:  0: STDOUT: My length 29
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:53:  0: STDOUT: My address 5584:5c8:c-43:ff-4:ce7-:8df:3-39:11e by 8000
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.HandleStdoutLine: parsing stdout: Message in error state after connection attempt!
2023-04-13 22:50:53:  0: STDOUT: Message in error state after connection attempt!
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:53:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:53:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:53:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:53:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:53:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:53:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:53:  0: DEBUG: PopupHandler.CheckForPopups: Returning
2023-04-13 22:50:54:  0: DEBUG: ManagedProcess.Execute: m_process.IsRunning(): True    m_process.IsStdoutAvailable(): False
2023-04-13 22:50:54:  0: DEBUG: PopupHandler.CheckForPopups: Getting popup handles
2023-04-13 22:50:54:  0: DEBUG: PopupHandler.CheckForPopups: Got 0 handles
2023-04-13 22:50:54:  0: DEBUG: PopupHandler.CheckForPopups: Returning

zainali · April 14, 2023, 6:19pm

Hello @Igor

I checked the logs, seems like the debug logging for the plugin is turned on, so all those repetitive lines actually mean that is it looking for handles but could not find. It is a normal debug log. I suggest following this KBA: Troubleshooting render issues in Houdini and sharing logs.

Igor · April 15, 2023, 12:18am

Hi Zain,

Thanks for answer.
I looked at link you provided. I am wondering how It will work for distributed sim?
It requires more than 1 worker to run simultaneously and simtracker.py before that.
If I run only one “worker” for job that supposed to run on 2 workers - task will never ends because it will wait response from second machine.

Hqueue finished my job successfully (picture) so it is clean from Houdini side. Produced two caches sequences in specified folder. Scene that was used for Deadline is the same as for Hqueue - the difference only in Deadline submitter rop(pic) that was launched.

Also I sent one slice and it passed through deadline perfectly fine. I tested it on both machines. But for fluid sims it doesn’t make sense since whole idea is to slice it for communications on more than one machine.

From my log it mentioned “Bad Address” before moved in infinite loop. Also both tasks produced 1 bgeo cache so Houdini sim had initial start. Something happened after that.
Maybe some problem comes from Python 3.9 integration.

sent