Deadline Monitor 10.2 constantly crashing

jimboslice · March 14, 2023, 8:45pm

Ever since updating to 10.2.0.8 I’m seeing Deadline Monitor constantly crashing on Linux

Time to initialize: 56.000 ms
Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
Connecting to Deadline RCS 10.2 [v10.2.0.8 Release (42f3f857d)]
Auto Configuration: Picking configuration based on: bebop / 192.168.1.197
Auto Configuration: No auto configuration could be detected, using local configuration
Time to connect to Repository: 1.396 s
Time to check user account: 12.000 ms
Time to purge old logs and temp files: 4.000 ms
Time to synchronize plugin icons: 192.000 ms
Time to initialize main window: 388.000 ms
Main Window shown
Python 3.7.12 | packaged by Thinkbox Software | (default, Sep 30 2022, 18:35:20) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
Time to show main window: 16.000 ms
Creating default config file:System (ad/Thinkbox/Deadline10/deadline.ini
Creating default config file:534:nobody:/nonexistent:/usr/sb/Thinkbox/Deadline10/deadline.ini
[1]    17822 segmentation fault (core dumped)  ./deadlinemonitor -new

zainali · March 15, 2023, 5:24pm

Hello @jimboslice

Thanks for reaching out. Looks like you are using RHEL 4? It is not supported: System Requirements — Deadline 10.2.0.10 documentation

If you are not using RHEL 4 let us run the monitor in an strace like below:
strace -v -s 256 -D -o /tmp/log.txt deadlinemonitor
-v: print environment
-s string size: 256 bytes
-D run strace as detached granchild
-o logfile
Then share the log here.

Also check if /usr/sb/Thinkbox/Deadline10/deadline.ini this file exists, if yes what is the permission set on it for the user trying to run Deadline Monitor.

Silly question: Is this a headless node? if yes then Monitor will not run. Try running it like below:
./deadlinemonitor -nogui -new
Do you get the same error?

jimboslice · March 16, 2023, 1:30am

Running on Pop OS 22.04 LTS

/usr/sb/Thinkbox/Deadline10/deadline.ini
does not exist. However
/var/lib/Thinkbox/Deadline10/deadline.ini
does and im seeing
-rwxrwxrwx 1 nobody nogroup 821 Mar 14 12:56 /var/lib/Thinkbox/Deadline10/deadline.ini

Monitor will start with deadlinemonitor -new but will crash after a little while with the following

Auto Configuration: No auto configuration for Repository Path could be detected, using local configuration
Connecting to Deadline RCS 10.2 [v10.2.0.8 Release (42f3f857d)]
Auto Configuration: Picking configuration based on: bebop / 192.168.1.197
Auto Configuration: No auto configuration could be detected, using local configuration
Time to connect to Repository: 2.004 s
Time to check user account: 8.000 ms
Time to purge old logs and temp files: 12.000 ms
Time to synchronize plugin icons: 200.000 ms
Time to initialize main window: 396.000 ms
Main Window shown
Python 3.7.12 | packaged by Thinkbox Software | (default, Sep 30 2022, 18:35:20) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
Time to show main window: 12.000 ms
free(): unaligned chunk detected in tcache 2
[1]    1669869 IOT instruction (core dumped)  deadlinemonitor -new

zainali · March 17, 2023, 9:42pm

It is weird, for some reason the client configuration file is read from a wrong link.

The output you are sharing, is that run under the strace as I suggested?

We do not support Pop OS 22.04 LTS I might be like a missing library. strace will be able to tell though, can you share the strace logs? I have seen monitor not working when libffi.6.so is not on the system. Check my response here: Custom Deadline Events not executing - #8 by zainali

jimboslice · March 23, 2023, 3:56pm

There’s no way to upload text files on the forum so here’s a drive link.

What’s weird is that I’ve never had a problem on Pop before Deadline 10.2.

Thanks

jimboslice · April 18, 2023, 4:01pm

Tried updating to 10.2.1.0 and still seeing the same crashes
malloc(): unaligned tcache chunk detected

zainali · April 18, 2023, 7:11pm

Hello again!! @jimboslice

I went over the strace, thanks for sharing that it seems like a problem with Qt but I am not so sure.

Try running simpler UIs to more complex:

$DEADLINE_PATH\deadlinecommand changerepository - share your findings
Start Pulse (slightly more complex) - share your findings
Share the application logs. The application logs sit here: Logs — Deadline 10.2.1.0 documentation

Try running gdb

dnf install gdb
gdb --args /opt/Thinkbox/Deadline10/bin/deadlinemonitor

Also it may be possible to do this with the .net example here:

jimboslice · April 19, 2023, 3:20pm

Hey @zainali thanks for checking back in.
I’m seeing the crashes in the changerepository UI as well.
Running Monitor from gdb I’m seeing

malloc(): unaligned tcache chunk detected

Thread 54 ".NET ThreadPool" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffefdbff640 (LWP 1179505)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140733155636800) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.

jimboslice · April 29, 2023, 6:10pm

I’ve updated Popos to all of the latest updates. I then went to try and install 10.1.23.6 (which was the last working version). and noticed that it required .NET 6.0.14 not 6.0.16 which is currently the latest for the Ubuntu apt repo.
I downloaded .NET 6.0.14 and tried running monitor again and got a different error this time.

Thread 51 ".NET ThreadPool" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff13fff640 (LWP 52706)]
0x00007fff024027a0 in SHA256_Update () from /lib/x86_64-linux-gnu/libcrypto.so.3

jimboslice · April 29, 2023, 6:35pm

the command i used was
DOTNET_DbgEnableMiniDump=1 PATH=/home/jhb/Downloads/dotnet-runtime-6.0.14-linux-x64/shared/Microsoft.NETCore.App/6.0.14/:$PATH LD_LIBRARY_PATH=/home/jhb/Downloads/dotnet-runtime-6.0.14-linux-x64/shared/Microsoft.NETCore.App/6.0.14/:$LD_LIBRARY_PATH gdb --args /opt/Thinkbox/Deadline10/bin/deadlinemonitor.exe

jimboslice · May 4, 2023, 3:55pm

@zainali any thoughts?
Is it possible to run the deadline GUI apps inside of a virtual env and try switching the library dependency versions?
Thinking maybe it has something to do with the current version of Qt that’s shipping with Deadline
I wanted to try and see if I could get the app running with the default for Ubuntu 20.04 which is Python 3.8.2 and the default PyQt version 5.15.6

I found that the app is currently using Qt: v 5.12.12 and PyQt: v 5.14.2 and python 3.7.16

jimboslice · May 9, 2023, 3:49pm

I was able to trigger a segfault by running the following in gdb

gdb --args /opt/Thinkbox/Deadline10/bin/deadlinemonitor.exe --test-nogui

I assume this isn’t loading Qt so it seems to maybe be a bug with the OpenSSL library

Thread 22 ".NET ThreadPool" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff11ffb640 (LWP 2570904)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140733495359040) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
(gdb) backtrace
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140733495359040) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140733495359040) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140733495359040, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7842476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff78287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff78896f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff79dbb8c "%s\n") at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff78a0d7c in malloc_printerr (str=str@entry=0x7ffff79ded40 "malloc(): unaligned tcache chunk detected") at ./malloc/malloc.c:5664
#7  0x00007ffff78a545c in tcache_get (tc_idx=<optimized out>) at ./malloc/malloc.c:3195
#8  __GI___libc_malloc (bytes=112) at ./malloc/malloc.c:3313
#9  0x00007fff21bb741e in CRYPTO_zalloc () from /lib/x86_64-linux-gnu/libcrypto.so.3
#10 0x00007fff21b7e5bd in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#11 0x00007fff743f725c in CryptoNative_EvpDigestOneShot () from /opt/Thinkbox/Deadline10/bin/libSystem.Security.Cryptography.Native.OpenSsl.so
#12 0x00007fff80cb71f6 in ?? ()
#13 0x00000000320ee257 in ?? ()
#14 0x00007ffff76ce8f0 in ?? () from /opt/Thinkbox/Deadline10/bin/libcoreclr.so
#15 0x00007fff11ffa740 in ?? ()
#16 0x00007fff80751740 in ?? ()
#17 0x00007fff80751740 in ?? ()
#18 0x00007fff11ff9750 in ?? ()
#19 0x00007fff80cb71f6 in ?? ()
#20 0x00007fff11ff9870 in ?? ()
#21 0x00007fff11ff9840 in ?? ()
#22 0x00007fff80751740 in ?? ()
#23 0x00007fff11ff98d8 in ?? ()
#24 0x00007fff47fafbd0 in ?? ()
#25 0x0000000000000079 in ?? ()
#26 0x00007fff47fafb38 in ?? ()
#27 0x00007fff21df9120 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#28 0x00007fff11ff98d8 in ?? ()
#29 0x00007fff04001850 in ?? ()
#30 0x00007fff81aeac8f in ?? ()
#31 0x00007fff743f71f0 in ?? () from /opt/Thinkbox/Deadline10/bin/libSystem.Security.Cryptography.Native.OpenSsl.so
#32 0x00007ffff76ce8f0 in ?? () from /opt/Thinkbox/Deadline10/bin/libcoreclr.so
#33 0x00007fff11ff98d8 in ?? ()
#34 0x00007fff47fafbd0 in ?? ()
#35 0x0000000000000079 in ?? ()
#36 0x00007fff47fafb38 in ?? ()
#37 0x00007fff21df9120 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#38 0x0000000011ff9850 in ?? ()
#39 0x00007fff11ff98d8 in ?? ()
#40 0x00007fff47fafbd0 in ?? ()
#41 0x0000007900000020 in ?? ()
#42 0x00007fff47fafb38 in ?? ()
#43 0x00007fff21df9120 in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#44 0x0000000000000020 in ?? ()
#45 0x0000000000000020 in ?? ()
#46 0x00007fff04001850 in ?? ()
#47 0x00007fff47fafbd0 in ?? ()
#48 0x0000000000000079 in ?? ()
#49 0x00007fff11ff9920 in ?? ()
#50 0x00007fff82ad14b2 in ?? ()
#51 0x00000000320ee257 in ?? ()
#52 0x00007ffff76ce8f0 in ?? () from /opt/Thinkbox/Deadline10/bin/libcoreclr.so
#53 0x00007fff11ffa740 in ?? ()
#54 0x00007fff80751788 in ?? ()
#55 0x0000000000000000 in ?? ()

Thread 12 ".NET ThreadPool" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fff413fe640 (LWP 449688)]
__pthread_kill_implementation (no_tid=0, signo=6, threadid=140734288094784) at ./nptl/pthread_kill.c:44
44	./nptl/pthread_kill.c: No such file or directory.
(gdb) backtrace
#0  __pthread_kill_implementation (no_tid=0, signo=6, threadid=140734288094784) at ./nptl/pthread_kill.c:44
#1  __pthread_kill_internal (signo=6, threadid=140734288094784) at ./nptl/pthread_kill.c:78
#2  __GI___pthread_kill (threadid=140734288094784, signo=signo@entry=6) at ./nptl/pthread_kill.c:89
#3  0x00007ffff7842476 in __GI_raise (sig=sig@entry=6) at ../sysdeps/posix/raise.c:26
#4  0x00007ffff78287f3 in __GI_abort () at ./stdlib/abort.c:79
#5  0x00007ffff78896f6 in __libc_message (action=action@entry=do_abort, fmt=fmt@entry=0x7ffff79dbb8c "%s\n")
    at ../sysdeps/posix/libc_fatal.c:155
#6  0x00007ffff78a0d7c in malloc_printerr (
    str=str@entry=0x7ffff79ded40 "malloc(): unaligned tcache chunk detected") at ./malloc/malloc.c:5664
#7  0x00007ffff78a545c in tcache_get (tc_idx=<optimized out>) at ./malloc/malloc.c:3195
#8  __GI___libc_malloc (bytes=112) at ./malloc/malloc.c:3313
#9  0x00007fff203b741e in CRYPTO_zalloc () from /lib/x86_64-linux-gnu/libcrypto.so.3
#10 0x00007fff2037e5bd in ?? () from /lib/x86_64-linux-gnu/libcrypto.so.3
#11 0x00007fff743fd25c in CryptoNative_EvpDigestOneShot ()
   from /opt/Thinkbox/Deadline10/bin/libSystem.Security.Cryptography.Native.OpenSsl.so
#12 0x00007fff80ca18f6 in ?? ()
#13 0x0000000002f28f8d in ?? ()
#14 0x00007ffff76ce8f0 in ?? () from /opt/Thinkbox/Deadline10/bin/libcoreclr.so
#15 0x00007fff413fd740 in ?? ()
#16 0x00007fff80741fe8 in ?? ()
#17 0x00007fff80741fe8 in ?? ()
#18 0x00007fff413fc770 in ?? ()
#19 0x00007fff80ca18f6 in ?? ()
#20 0x00007fff413fc890 in ?? ()
#21 0x00007fff413fc860 in ?? ()
#22 0x00007fff80741fe8 in ?? ()