Got a fairly obscure one here.
After leaving the Monitor running overnight, I came in to find it had crashed hard. I tried again the next night, with the same results.
I’ve included the crash dump below, but in doing some searching, it seems that this maybe be an issue related to .NET applications (possibly C# specifically) running on Mono spawning too many threads that are never properly cleaned up. The scary thing is that it looks like a standard Mono build sets this limit at 16384, and a “small” build at 256.
Unfortunately there isn’t a whole lot more info out there besides “the application needs to clean up threads,” but I think this is something worth looking into. I’m going to try running a Slave long-term as well to see if it suffers from the same issue, but this seems like it could have serious implications for some of the processes that need to be running for weeks or months at a time.
[code]Main Window shown
- Assertion at threads.c:391, condition `id < HAZARD_TABLE_MAX_SIZE’ not met
Native stacktrace:
mono() [0x4917da]
/lib64/libpthread.so.0(+0xeef0) [0x7fe7b90f9ef0]
/lib64/libc.so.6(gsignal+0x35) [0x7fe7b8b712d5]
/lib64/libc.so.6(abort+0x17b) [0x7fe7b8b72beb]
mono() [0x5caa8f]
mono() [0x5cabb6]
mono() [0x58dcf2]
mono(mono_thread_attach+0xe8) [0x58f368]
mono(mono_jit_thread_attach+0x48) [0x416588]
[0x4108e0f0]
Debug info from gdb:
Mono support loaded.
Traceback (most recent call last):
File “/usr/share/gdb/auto-load/usr/lib64/libstdc++.so.6.0.16-gdb.py”, line 59, in
from libstdcxx.v6.printers import register_libstdcxx_printers
ImportError: No module named libstdcxx.v6.printers
[Thread debugging using libthread_db enabled]
[New Thread 0x7fe77ffff700 (LWP 2039)]
[New Thread 0x7fe785ccb700 (LWP 13013)]
[New Thread 0x7fe79e845700 (LWP 13011)]
[New Thread 0x7fe79ea8e700 (LWP 13010)]
[New Thread 0x7fe79fafb700 (LWP 13009)]
[New Thread 0x7fe7a257e700 (LWP 13008)]
[New Thread 0x7fe79fcfc700 (LWP 13007)]
[New Thread 0x7fe7aebe1700 (LWP 13006)]
[New Thread 0x7fe7af712700 (LWP 13005)]
[New Thread 0x7fe7aff13700 (LWP 13004)]
[New Thread 0x7fe7b0714700 (LWP 13003)]
[New Thread 0x7fe7b0f15700 (LWP 13002)]
[New Thread 0x7fe7b1716700 (LWP 13001)]
[New Thread 0x7fe7b1f17700 (LWP 13000)]
[New Thread 0x7fe7b2718700 (LWP 12999)]
0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
Id Target Id Frame
16 Thread 0x7fe7b2718700 (LWP 12999) “mono” 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
15 Thread 0x7fe7b1f17700 (LWP 13000) “mono” 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
14 Thread 0x7fe7b1716700 (LWP 13001) “mono” 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
13 Thread 0x7fe7b0f15700 (LWP 13002) “mono” 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
12 Thread 0x7fe7b0714700 (LWP 13003) “mono” 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
11 Thread 0x7fe7aff13700 (LWP 13004) “mono” 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
10 Thread 0x7fe7af712700 (LWP 13005) “mono” 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
9 Thread 0x7fe7aebe1700 (LWP 13006) “mono” 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
8 Thread 0x7fe79fcfc700 (LWP 13007) “deadlinemonitor” 0x00007fe7b90f681f in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
7 Thread 0x7fe7a257e700 (LWP 13008) “deadlinemonitor” 0x00007fe7b8c1c303 in epoll_wait () from /lib64/libc.so.6
6 Thread 0x7fe79fafb700 (LWP 13009) “deadlinemonitor” 0x00007fe7b90f8640 in sem_timedwait () from /lib64/libpthread.so.0
5 Thread 0x7fe79ea8e700 (LWP 13010) “deadlinemonitor” 0x00007fe7b90f976d in nanosleep () from /lib64/libpthread.so.0
4 Thread 0x7fe79e845700 (LWP 13011) “deadlinemonitor” 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
3 Thread 0x7fe785ccb700 (LWP 13013) “deadlinemonitor” 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
2 Thread 0x7fe77ffff700 (LWP 2039) “deadlinemonitor” 0x00007fe7b90f901d in read () from /lib64/libpthread.so.0
- 1 Thread 0x7fe7b9b90760 (LWP 12997) “deadlinemonitor” 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
Thread 16 (Thread 0x7fe7b2718700 (LWP 12999)):
#0 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005dd993 in ?? ()
#2 0x00000000005d7bac in ?? ()
#3 0x00000000005dc097 in ?? ()
#4 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 15 (Thread 0x7fe7b1f17700 (LWP 13000)):
#0 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005dd993 in ?? ()
#2 0x00000000005d7bac in ?? ()
#3 0x00000000005dc097 in ?? ()
#4 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 14 (Thread 0x7fe7b1716700 (LWP 13001)):
#0 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005dd993 in ?? ()
#2 0x00000000005d7bac in ?? ()
#3 0x00000000005dc097 in ?? ()
#4 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 13 (Thread 0x7fe7b0f15700 (LWP 13002)):
#0 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005dd993 in ?? ()
#2 0x00000000005d7bac in ?? ()
#3 0x00000000005dc097 in ?? ()
#4 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 12 (Thread 0x7fe7b0714700 (LWP 13003)):
#0 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005dd993 in ?? ()
#2 0x00000000005d7bac in ?? ()
#3 0x00000000005dc097 in ?? ()
#4 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 11 (Thread 0x7fe7aff13700 (LWP 13004)):
#0 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005dd993 in ?? ()
#2 0x00000000005d7bac in ?? ()
#3 0x00000000005dc097 in ?? ()
#4 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 10 (Thread 0x7fe7af712700 (LWP 13005)):
#0 0x00007fe7b90f64b5 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005dd993 in ?? ()
#2 0x00000000005d7bac in ?? ()
#3 0x00000000005dc097 in ?? ()
#4 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 9 (Thread 0x7fe7aebe1700 (LWP 13006)):
#0 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
#1 0x00000000005c2808 in mono_sem_wait ()
#2 0x000000000051068b in ?? ()
#3 0x0000000000592eb7 in ?? ()
#4 0x00000000005bd202 in ?? ()
#5 0x00000000005dd04e in ?? ()
#6 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 8 (Thread 0x7fe79fcfc700 (LWP 13007)):
#0 0x00007fe7b90f681f in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1 0x00000000005a9908 in ?? ()
#2 0x00000000005ba98c in ?? ()
#3 0x00000000005927fb in ?? ()
#4 0x0000000041f07758 in ?? ()
#5 0x00007fe798000c20 in ?? ()
#6 0xffffffffffffffff in ?? ()
#7 0x00007fe79fcfb99f in ?? ()
#8 0x00007fe7a5467910 in ?? ()
#9 0x0000000000000007 in ?? ()
#10 0x00007fe79fcfb9c0 in ?? ()
#11 0x00007fe79fcfb920 in ?? ()
#12 0x0000000000000000 in ?? ()
Thread 7 (Thread 0x7fe7a257e700 (LWP 13008)):
#0 0x00007fe7b8c1c303 in epoll_wait () from /lib64/libc.so.6
#1 0x000000000058a6e8 in ?? ()
#2 0x0000000000592eb7 in ?? ()
#3 0x00000000005bd202 in ?? ()
#4 0x00000000005dd04e in ?? ()
#5 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#6 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 6 (Thread 0x7fe79fafb700 (LWP 13009)):
#0 0x00007fe7b90f8640 in sem_timedwait () from /lib64/libpthread.so.0
#1 0x00000000005c28fb in mono_sem_timedwait ()
#2 0x000000000058c10f in ?? ()
#3 0x0000000000592eb7 in ?? ()
#4 0x00000000005bd202 in ?? ()
#5 0x00000000005dd04e in ?? ()
#6 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 5 (Thread 0x7fe79ea8e700 (LWP 13010)):
#0 0x00007fe7b90f976d in nanosleep () from /lib64/libpthread.so.0
#1 0x00000000005bc446 in ?? ()
#2 0x0000000000589cf8 in ?? ()
#3 0x0000000000592eb7 in ?? ()
#4 0x00000000005bd202 in ?? ()
#5 0x00000000005dd04e in ?? ()
#6 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#7 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 4 (Thread 0x7fe79e845700 (LWP 13011)):
#0 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007fe7ae3434f8 in PyThread_acquire_lock (lock=0x122adc0, waitflag=128) at Python/thread_pthread.h:349
#2 0x00007fe7ae312814 in PyEval_RestoreThread (tstate=0x7fe78c0e81d0) at Python/ceval.c:353
#3 0x00007fe7ae336936 in PyGILState_Ensure () at Python/pystate.c:610
#4 0x0000000041090825 in ?? ()
#5 0x00007fe78c000c20 in ?? ()
#6 0x00000000404da04d in ?? ()
#7 0x00007fe78c000c20 in ?? ()
#8 0xffffffffffffffff in ?? ()
#9 0x00007fe79e20bb80 in ?? ()
#10 0x00007fe79e844830 in ?? ()
#11 0x00007fe79e844780 in ?? ()
#12 0x00007fe7677d48c0 in ?? ()
#13 0x00007fe76bef0a00 in ?? ()
#14 0x00007fe784803690 in ?? ()
#15 0x0000000000000001 in ?? ()
#16 0x00000000410907bc in ?? ()
#17 0x00007fe79e844a70 in ?? ()
#18 0x00000000407f8250 in ?? ()
#19 0x00007fe78486ddc0 in ?? ()
#20 0x00007fe79dedefa0 in ?? ()
#21 0x00007fe76bef0a00 in ?? ()
#22 0x0000000000000000 in ?? ()
Thread 3 (Thread 0x7fe785ccb700 (LWP 13013)):
#0 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007fe7ae3434f8 in PyThread_acquire_lock (lock=0x122adc0, waitflag=128) at Python/thread_pthread.h:349
#2 0x00007fe7ae312814 in PyEval_RestoreThread (tstate=0x7fe7800012d0) at Python/ceval.c:353
#3 0x00007fe7a5da797d in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtCore.so
#4 0x00007fe7ae31729d in call_function (oparg=, pp_stack=) at Python/ceval.c:3750
#5 PyEval_EvalFrameEx (f=0x7fe7800013f0, throwflag=) at Python/ceval.c:2412
#6 0x00007fe7ae318fe9 in PyEval_EvalCodeEx (co=0x2f8dbe8, globals=, locals=, args=0x1256428, argcount=1, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3000
#7 0x00007fe7ae29dc5d in function_call (func=0x2f942a8, arg=0x1256410, kw=0x0) at Objects/funcobject.c:524
#8 0x00007fe7ae26f0a8 in PyObject_Call (func=0x2f942a8, arg=0x1256410, kw=0x0) at Objects/abstract.c:2492
#9 0x00007fe7ae27feb0 in instancemethod_call (func=0x2f942a8, arg=0x1256410, kw=0x0) at Objects/classobject.c:2579
#10 0x00007fe7ae26f0a8 in PyObject_Call (func=0x2de4140, arg=0x7fe7adfc7050, kw=0x0) at Objects/abstract.c:2492
#11 0x00007fe7ae311dd6 in PyEval_CallObjectWithKeywords (func=0x2de4140, arg=0x7fe7adfc7050, kw=0x0) at Python/ceval.c:3619
#12 0x00007fe7a61594a2 in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/sip.so
#13 0x00007fe7a5d72ff7 in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtCore.so
#14 0x00007fe7a5da8882 in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtCore.so
#15 0x00007fe7a7b8950f in ?? () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#16 0x00007fe7b90f2b31 in start_thread () from /lib64/libpthread.so.0
#17 0x00007fe7b8c1bd2d in clone () from /lib64/libc.so.6
Thread 2 (Thread 0x7fe77ffff700 (LWP 2039)):
#0 0x00007fe7b90f901d in read () from /lib64/libpthread.so.0
#1 0x0000000000491976 in ?? ()
#2
#3 0x00007fe7b8b712d5 in raise () from /lib64/libc.so.6
#4 0x00007fe7b8b72beb in abort () from /lib64/libc.so.6
#5 0x00000000005caa8f in ?? ()
#6 0x00000000005cabb6 in ?? ()
#7 0x000000000058dcf2 in ?? ()
#8 0x000000000058f368 in mono_thread_attach ()
#9 0x0000000000416588 in mono_jit_thread_attach ()
#10 0x000000004108e0f0 in ?? ()
#11 0x00007fe770001588 in ?? ()
#12 0x00000000020f70e0 in ?? ()
#13 0x0000000000000000 in ?? ()
Thread 1 (Thread 0x7fe7b9b90760 (LWP 12997)):
#0 0x00007fe7b90f8560 in sem_wait () from /lib64/libpthread.so.0
#1 0x00007fe7ae3434f8 in PyThread_acquire_lock (lock=0x122adc0, waitflag=128) at Python/thread_pthread.h:349
#2 0x00007fe7ae312814 in PyEval_RestoreThread (tstate=0x113fc90) at Python/ceval.c:353
#3 0x00007fe7ae336936 in PyGILState_Ensure () at Python/pystate.c:610
#4 0x00007fe7a5e8ff2e in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtCore.so
#5 0x00007fe7a5e90075 in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtCore.so
#6 0x00007fe7a7cb2e20 in QMetaObject::activate(QObject*, QMetaObject const*, int, void**) () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#7 0x00007fe7a5da3eef in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtCore.so
#8 0x00007fe7a7cae4b4 in QObject::event(QEvent*) () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#9 0x00007fe7a5da3e82 in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtCore.so
#10 0x00007fe7ac29bf6f in QApplicationPrivate::notify_helper(QObject*, QEvent*) () from /lumalocal/Deadline6/client/bin/libQtGui.so.4
#11 0x00007fe7ac2a25d0 in QApplication::notify(QObject*, QEvent*) () from /lumalocal/Deadline6/client/bin/libQtGui.so.4
#12 0x00007fe7ad318cc4 in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtGui.so
#13 0x00007fe7a7c9ab64 in QCoreApplication::notifyInternal(QObject*, QEvent*) () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#14 0x00007fe7a7ccde9d in ?? () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#15 0x00007fe7a7ccec00 in QEventDispatcherUNIX::processEvents(QFlagsQEventLoop::ProcessEventsFlag) () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#16 0x00007fe7ac349526 in ?? () from /lumalocal/Deadline6/client/bin/libQtGui.so.4
#17 0x00007fe7a7c99bc5 in QEventLoop::processEvents(QFlagsQEventLoop::ProcessEventsFlag) () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#18 0x00007fe7a7c99f98 in QEventLoop::exec(QFlagsQEventLoop::ProcessEventsFlag) () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#19 0x00007fe7a7ca00f4 in QCoreApplication::exec() () from /lumalocal/Deadline6/client/bin/libQtCore.so.4
#20 0x00007fe7ad3198fd in ?? () from /lumalocal/Deadline6/client/bin/python/lib/python2.6/site-packages/PyQt4/QtGui.so
#21 0x00007fe7ae31729d in call_function (oparg=, pp_stack=) at Python/ceval.c:3750
#22 PyEval_EvalFrameEx (f=0x1f3d1f0, throwflag=) at Python/ceval.c:2412
#23 0x00007fe7ae3184b4 in fast_function (nk=, na=, n=, pp_stack=, func=) at Python/ceval.c:3836
#24 call_function (oparg=, pp_stack=) at Python/ceval.c:3771
#25 PyEval_EvalFrameEx (f=0x3717100, throwflag=) at Python/ceval.c:2412
#26 0x00007fe7ae3184b4 in fast_function (nk=, na=, n=, pp_stack=, func=) at Python/ceval.c:3836
#27 call_function (oparg=, pp_stack=) at Python/ceval.c:3771
#28 PyEval_EvalFrameEx (f=0x1ba7060, throwflag=) at Python/ceval.c:2412
#29 0x00007fe7ae318fe9 in PyEval_EvalCodeEx (co=0x7fe7adf09210, globals=, locals=, args=0x0, argcount=0, kws=0x0, kwcount=0, defs=0x0, defcount=0, closure=0x0) at Python/ceval.c:3000
#30 0x00007fe7ae319162 in PyEval_EvalCode (co=0x122adc0, globals=0x80, locals=0x0) at Python/ceval.c:541
#31 0x00007fe7ae32b710 in PyImport_ExecCodeModuleEx (name=0x1f40950 “MainWindow”, co=0x7fe7adf09210, pathname=0x0) at Python/import.c:677
#32 0x000000004143e013 in ?? ()
#33 0x0000000000ea8850 in ?? ()
#34 0x0000000000000040 in ?? ()
#35 0x0000000000ea8850 in ?? ()
#36 0x0000000000000000 in ?? ()
=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
Aborted[/code]
Let me know if I can provide any more info.
-Nathan