Hi there,

We have attempted to roll out the beta10 slave binaries again, and it quickly resulted in pulse crashing (i didnt trigger an update on it, yet it crashed right away, nothing in the log).

The mongo db is now using 6-7 cores @ 100% and most of the slaves are idle, with this messge:

Scheduler - returning mx_render
Scheduler - returning nuke
Scheduler - returning 527ea366be332d148c847122
Scheduler Thread - Seconds before next job scan: 29
---- 2013/11/10 17:18 ----
Purging unsubmitted jobs
Slave ‘LAPRO0233’ has stalled because it has not updated its state in 2.016 hrs. Performing house cleaning…
---- 2013/11/10 17:20 ----
Slave - An error occurred while updating the slave’s info: An unexpected error occurred while interacting with the database (deadline.scanlinevfxla.com:27017):
Timeout waiting for a MongoConnection. (FranticX.Database.DocumentException)
---- 2013/11/10 17:21 ----
Connecting to slave log: LAPRO1290
---- 2013/11/10 17:22 ----
Slave - An error occurred while updating the slave’s info: An unexpected error occurred while interacting with the database (deadline.scanlinevfxla.com:27017):
Timeout waiting for a MongoConnection. (FranticX.Database.DocumentException)
---- 2013/11/10 17:23 ----
Listener Thread - 172.18.4.80 has connected
Listener Thread - Received message: StreamLog
Listener Thread - Responded with: Success
Slave - An error occurred while updating the slave’s info: An unexpected error occurred while interacting with the database (deadline.scanlinevfxla.com:27017):
Timeout waiting for a MongoConnection. (FranticX.Database.DocumentException)

Database stats:

Mongo Database Statistics
    name: deadline6db
    collections: 23
    objects: 7147382
    average object size: 415.0 Bytes
    data size:: 2.8 GB
    storage size: 5.8 GB
    extents: 192
    indexes: 42
    index size: 697.4 MB
    file size: 9.9 GB

BalancerInfo
    namespace: deadline6db.BalancerInfo
    objects: 0
    average object size:     error: Element 'avgObjSize' not found. (System.Collections.Generic.KeyNotFoundException)

DeadlineSettings
    namespace: deadline6db.DeadlineSettings
    objects: 7
    average object size: 354.6 KB
    data size: 2.4 MB
    storage size: 13.0 MB
    extents: 3
    last extent size: 12.8 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 8.0 KB
    index sizes: 
        _id_: 7.984 KB

DeletedDocuments
    namespace: deadline6db.DeletedDocuments
    objects: 202965
    average object size: 1.4 KB
    data size: 282.4 MB
    storage size: 320.3 MB
    extents: 14
    last extent size: 88.3 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 5
    total index size: 46.7 MB
    index sizes: 
        _id_: 10.339 MB
        DeletedTime_1: 5.505 MB
        DocID_1: 11.158 MB
        OrigCollection_1: 8.655 MB
        OrigCollection_1_DeletedTime_1: 11.033 MB

DeletedJobReportEntries
    namespace: deadline6db.DeletedJobReportEntries
    objects: 31
    average object size: 286.0 Bytes
    data size: 8.7 KB
    storage size: 409.1 MB
    extents: 16
    last extent size: 108.7 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 8.0 KB
    index sizes: 
        _id_: 7.984 KB

DeletedJobs
    namespace: deadline6db.DeletedJobs
    objects: 22
    average object size: 19.2 KB
    data size: 423.2 KB
    storage size: 44.3 MB
    extents: 7
    last extent size: 14.3 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 16.0 KB
    index sizes: 
        _id_: 7.984 KB
        LastWriteTime_1: 7.984 KB

JobReportEntries
    namespace: deadline6db.JobReportEntries
    objects: 748105
    average object size: 374.0 Bytes
    data size: 266.5 MB
    storage size: 817.7 MB
    extents: 17
    last extent size: 217.2 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 104.5 MB
    index sizes: 
        _id_: 54.924 MB
        Job_1: 49.528 MB

JobReports
    namespace: deadline6db.JobReports
    objects: 8016
    average object size: 11.3 KB
    data size: 88.5 MB
    storage size: 440.5 MB
    extents: 14
    last extent size: 121.4 MB
    padding factor: 1.10900000003483
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 662.7 KB
    index sizes: 
        _id_: 662.703 KB

Jobs
    namespace: deadline6db.Jobs
    objects: 18431
    average object size: 8.7 KB
    data size: 156.8 MB
    storage size: 260.3 MB
    extents: 13
    last extent size: 71.8 MB
    padding factor: 1.00000000000086
    system flags: HasIdIndex
    user flags: None
    indexes: 6
    total index size: 5.7 MB
    index sizes: 
        _id_: 1.302 MB
        LastWriteTime_1: 870.297 KB
        Props.Grp_1: 726.578 KB
        Props.Pool_1: 670.688 KB
        Stat_1_Props.Pool_1_Props.Grp_1: 1.162 MB
        Stat_1_Props.SecPool_1_Props.Grp_1: 1.014 MB

JobStatistics
    namespace: deadline6db.JobStatistics
    objects: 83171
    average object size: 1.4 KB
    data size: 112.2 MB
    storage size: 118.2 MB
    extents: 11
    last extent size: 35.9 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 7.1 MB
    index sizes: 
        _id_: 4.780 MB
        EntryTime_1: 2.363 MB

JobTasks
    namespace: deadline6db.JobTasks
    objects: 2152639
    average object size: 338.0 Bytes
    data size: 694.9 MB
    storage size: 1.2 GB
    extents: 17
    last extent size: 325.8 MB
    padding factor: 1.00400000053444
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 167.3 MB
    index sizes: 
        _id_: 167.321 MB

LimitGroups
    namespace: deadline6db.LimitGroups
    objects: 18461
    average object size: 1.2 KB
    data size: 22.2 MB
    storage size: 118.2 MB
    extents: 11
    last extent size: 35.9 MB
    padding factor: 1.00000000008433
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 2.4 MB
    index sizes: 
        _id_: 1.497 MB
        LastWriteTime_1: 886.266 KB

PermissionGroups
    namespace: deadline6db.PermissionGroups
    objects: 5
    average object size: 13.4 KB
    data size: 66.9 KB
    storage size: 140.0 KB
    extents: 2
    last extent size: 132.0 KB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 8.0 KB
    index sizes: 
        _id_: 7.984 KB

PulseInfo
    namespace: deadline6db.PulseInfo
    objects: 2
    average object size: 484.0 Bytes
    data size: 968.0 Bytes
    storage size: 8.0 KB
    extents: 1
    last extent size: 8.0 KB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 16.0 KB
    index sizes: 
        _id_: 7.984 KB
        LastWriteTime_1: 7.984 KB

RepositoryStatistics
    namespace: deadline6db.RepositoryStatistics
    objects: 12815
    average object size: 291.0 Bytes
    data size: 3.6 MB
    storage size: 10.7 MB
    extents: 6
    last extent size: 8.0 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 966.1 KB
    index sizes: 
        _id_: 598.828 KB
        EntryTime_1: 367.281 KB

ScriptInfos
    namespace: deadline6db.ScriptInfos
    objects: 89
    average object size: 239.0 Bytes
    data size: 20.8 KB
    storage size: 40.0 KB
    extents: 2
    last extent size: 32.0 KB
    padding factor: 1.00000000000001
    system flags: HasIdIndex
    user flags: None
    indexes: 3
    total index size: 24.0 KB
    index sizes: 
        _id_: 7.984 KB
        ScriptType_1_ScriptName_1: 7.984 KB
        EntryTime_1: 7.984 KB

SlaveInfo
    namespace: deadline6db.SlaveInfo
    objects: 1341
    average object size: 2.7 KB
    data size: 3.5 MB
    storage size: 20.0 MB
    extents: 6
    last extent size: 15.0 MB
    padding factor: 1.00000000000163
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 143.7 KB
    index sizes: 
        _id_: 71.859 KB
        LastWriteTime_1: 71.859 KB

SlaveReports
    namespace: deadline6db.SlaveReports
    objects: 1341
    average object size: 453.0 KB
    data size: 593.2 MB
    storage size: 1.0 GB
    extents: 18
    last extent size: 271.5 MB
    padding factor: 1.0000000000075
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 71.9 KB
    index sizes: 
        _id_: 71.859 KB

SlaveSettings
    namespace: deadline6db.SlaveSettings
    objects: 1341
    average object size: 585.0 Bytes
    data size: 766.6 KB
    storage size: 2.7 MB
    extents: 5
    last extent size: 2.0 MB
    padding factor: 1.0290000000003
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 127.8 KB
    index sizes: 
        _id_: 71.859 KB
        LastWriteTime_1: 55.891 KB

SlaveStatistics
    namespace: deadline6db.SlaveStatistics
    objects: 3898158
    average object size: 161.0 Bytes
    data size: 598.0 MB
    storage size: 1.1 GB
    extents: 18
    last extent size: 293.3 MB
    padding factor: 1
    system flags: HasIdIndex
    user flags: None
    indexes: 2
    total index size: 361.7 MB
    index sizes: 
        _id_: 245.473 MB
        EntryTime_1: 116.202 MB

system.indexes
    namespace: deadline6db.system.indexes
    objects: 42
    average object size: 93.0 Bytes
    data size: 3.8 KB
    storage size: 20.0 KB
    extents: 2
    last extent size: 16.0 KB
    padding factor: 1
    system flags: None
    user flags: None
    indexes: 0
    total index size: 0.0 Bytes
    index sizes: 

Tasks
    namespace: deadline6db.Tasks
    objects: 143
    average object size: 2.4 KB
    data size: 346.7 KB
    storage size: 948.0 KB
    extents: 3
    last extent size: 908.0 KB
    padding factor: 1.08200000000004
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 8.0 KB
    index sizes: 
        _id_: 7.984 KB

UserInfo
    namespace: deadline6db.UserInfo
    objects: 193
    average object size: 162.0 Bytes
    data size: 30.5 KB
    storage size: 168.0 KB
    extents: 3
    last extent size: 128.0 KB
    padding factor: 1.00000000000001
    system flags: HasIdIndex
    user flags: None
    indexes: 1
    total index size: 8.0 KB
    index sizes: 
        _id_: 7.984 KB

Server Status
    host: deadline.scanlinevfxla.com
    version: 2.4.1
    process: mongod
    pid: 28045
    uptime: 262956
    uptimeMillis: 262956548
    uptimeEstimate: 245097
    localTime: 2013-11-11T01:33:30.566Z
    asserts
        regular: 0
        warning: 0
        msg: 0
        user: 0
        rollovers: 0
    backgroundFlushing
        flushes: 4373
        total_ms: 10335365
        average_ms: 2363.4495769494624
        last_ms: 1012
        last_finished: 2013-11-11T01:32:56.214Z
    connections
        current: 2455
        available: 17545
        totalCreated: 55064427
    cursors
        totalOpen: 90
        clientCursors_size: 90
        timedOut: 630
        pinned: 2
        totalNoTimeout: 47
    dur
        commits: 26
        journaledMB: 0.909312
        writeToDataFilesMB: 2.596767
        compression: 0.34877384196185285
        commitsInWriteLock: 0
        earlyCommits: 0
        timeMs
            dt: 3057
            prepLogBuffer: 2
            writeToJournal: 348
            writeToDataFiles: 6
            remapPrivateView: 37
    extra_info
        note: fields vary by platform
        heap_usage_bytes: 680704264
        page_faults: 8838
    globalLock
        totalTime: 262956550000
        lockTime: 6437833731
        currentQueue
            total: 1354
            readers: 1354
            writers: 0
        activeClients
            total: 6
            readers: 6
            writers: 0
    indexCounters
        accesses: 1994433571
        hits: 1994398889
        misses: 0
        resets: 0
        missRatio: 0
    locks
        .
            timeLockedMicros
                R: 250986972
                W: 6437833731
            timeAcquiringMicros
                R: 653354629
                W: 50455428
        admin
            timeLockedMicros
            timeAcquiringMicros
        local
            timeLockedMicros
                r: 593807
                w: 0
            timeAcquiringMicros
                r: 636226
                w: 0
        deadline6db
            timeLockedMicros
                r: 111932145062
                w: 99069634562
            timeAcquiringMicros
                r: 11100874948009
                w: 6370670356853
    network
        bytesIn: 226960925491
        bytesOut: 22366616265080
        numRequests: 1639625547
    opcounters
        insert: 1096180
        query: 996848343
        update: 328895057
        delete: 17597083
        getmore: 23655268
        command: 565811037
    opcountersRepl
        insert: 0
        query: 0
        update: 0
        delete: 0
        getmore: 0
        command: 0
    recordStats
        accessesNotInMemory: 3465
        pageFaultExceptionsThrown: 1056
        deadline6db
            accessesNotInMemory: 3465
            pageFaultExceptionsThrown: 1056
        local
            accessesNotInMemory: 0
            pageFaultExceptionsThrown: 0
    writeBacksQueued: false
    mem
        bits: 64
        resident: 5036
        virtual: 613635
        supported: true
        mapped: 8221
        mappedWithJournal: 16442
    metrics
        document
            deleted: 555400
            inserted: 1096180
            returned: 7180542832
            updated: 169358252
        getLastError
            wtime
                num: 241029187
                totalMillis: 127
            wtimeouts: 0
        operation
            fastmod: 7204361
            idhack: 659916481
            scanAndOrder: 0
        queryExecutor
            scanned: 6165940166
        record
            moves: 425027
        repl
            apply
                batches
                    num: 0
                    totalMillis: 0
                ops: 0
            buffer
                count: 0
                maxSizeBytes: 268435456
                sizeBytes: 0
            network
                bytes: 0
                getmores
                    num: 0
                    totalMillis: 0
                ops: 0
                readersCreated: 0
            oplog
                insert
                    num: 0
                    totalMillis: 0
                insertBytes: 0
            preload
                docs
                    num: 0
                    totalMillis: 0
                indexes
                    num: 0
                    totalMillis: 0
        ttl
            deletedDocuments: 0
            passes: 4369
    ok: 1

I had to reinstall pulse, i could not start it in any other way (autoupdate did not seem to work).

Once pulse was up, the farm seems to have come back to life. It appears that without pulse running, the farm basically halts.

Mongo is now a bit down from the previous cpu usage, fluctuates between 150-350% cpus usage (instead of 5-600). On beta9, mongo usually did not go over 100%.

Pulse seems to crash quite often, taking down the farm with it (well, halting everything)

I noticed these in /var/log/messages:

Nov 10 15:33:19 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 15:33:19 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 15:33:20 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 15:33:20 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 15:33:28 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:18:53 deadline abrt[4373]: Saved core dump of pid 8761 (/opt/mono-2.10.9/bin/mono) to /var/spool/abrt/ccpp-2013-11-10-17:15:01-8761 (3642036224 bytes)
Nov 10 17:18:53 deadline abrtd: Directory 'ccpp-2013-11-10-17:15:01-8761' creation detected
Nov 10 17:18:54 deadline abrtd: Executable '/opt/mono-2.10.9/bin/mono' doesn't belong to any package
Nov 10 17:18:54 deadline abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-11-10-17:15:01-8761' exited with 1
Nov 10 17:18:54 deadline abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2013-11-10-17:15:01-8761, deleting
Nov 10 17:21:36 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:36 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:37 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:37 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:37 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:37 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:39 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:40 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:40 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:40 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:41 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:41 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:43 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:46 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:46 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:47 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:47 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:21:54 deadline avahi-daemon[1864]: Invalid query packet.
Nov 10 17:45:20 deadline abrtd: Directory 'ccpp-2013-11-10-17:41:24-24027' creation detected
Nov 10 17:45:20 deadline abrt[23987]: Saved core dump of pid 24027 (/opt/mono-2.10.9/bin/mono) to /var/spool/abrt/ccpp-2013-11-10-17:41:24-24027 (4405407744 bytes)
Nov 10 17:45:20 deadline abrtd: Executable '/opt/mono-2.10.9/bin/mono' doesn't belong to any package
Nov 10 17:45:20 deadline abrtd: 'post-create' on '/var/spool/abrt/ccpp-2013-11-10-17:41:24-24027' exited with 1
Nov 10 17:45:20 deadline abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2013-11-10-17:41:24-24027, deleting

Each of those abrt commands happens when pulse crashes

Pulse in beta10 is so unstable that we cant keep it running for more than 10-15 mins. And then the whole farm stops :-\

Will try to turn off auto updating and roll pulse back to beta9

Ran beta10 pulse in command line, waited 5 mins, got this crash:

#17 0x00007fef8c000cb0 in ?? ()
#18 0x00007fef83ffaad0 in ?? ()
#19 0x000000004152f0bc in ?? ()
#20 0x00000000021da0c0 in ?? ()
#21 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7fef985b1700 (LWP 16801)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=250) at threads.c:1320
#3  0x0000000040e6f09a in ?? ()
#4  0x00007fef70000c20 in ?? ()
#5  0x08d0ac56b36b7314 in ?? ()
#6  0x08d0ac99c18eb314 in ?? ()
#7  0x08d0ac99c18eb314 in ?? ()
#8  0x00007fef8c000cb0 in ?? ()
#9  0x00007fef985b0ad0 in ?? ()
#10 0x00007fef985b00b0 in ?? ()
#11 0x00007fef9a5facc0 in ?? ()
#12 0x00007fef987e2958 in ?? ()
#13 0x00007fef987e2958 in ?? ()
#14 0x00000000000000fa in ?? ()
#15 0x0000000040e6effc in ?? ()
#16 0x00007fef987e2958 in ?? ()
#17 0x88d0ac56b36b7314 in ?? ()
#18 0x08d0ac99c18eb314 in ?? ()
#19 0x000000004153b3c4 in ?? ()
#20 0x0000000000000fff in ?? ()
#21 0x00007fef2b88e990 in ?? ()
#22 0x00007fef2d25d900 in ?? ()
#23 0x00007feeeb0e71c0 in ?? ()
#24 0x00007fef99eb49f0 in ?? ()
#25 0x00007fef99eb49f0 in ?? ()
#26 0x00007fef98fa92e0 in ?? ()
#27 0x00007fef98fa92e0 in ?? ()
#28 0x00007fef98d34ea0 in ?? ()
#29 0x0000007000000077 in ?? ()
#30 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7fef983ac700 (LWP 16802)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=250) at threads.c:1320
#3  0x0000000040e6f09a in ?? ()
#4  0x00007fef74000c20 in ?? ()
#5  0x08d0ac56b35d1bde in ?? ()
#6  0x08d0ac99c1805bde in ?? ()
#7  0x08d0ac99c1805bde in ?? ()
#8  0x00007fef8c000cb0 in ?? ()
#9  0x00007fef983abad0 in ?? ()
#10 0x00007fef983ab7a0 in ?? ()
#11 0x00007fef9a5facc0 in ?? ()
#12 0x00007fef987e2888 in ?? ()
#13 0x00007fef987e2888 in ?? ()
#14 0x00000000000000fa in ?? ()
#15 0x0000000040e6effc in ?? ()
#16 0x00007fef987e2888 in ?? ()
#17 0x88d0ac56b35d1bde in ?? ()
#18 0x08d0ac99c1805bde in ?? ()
#19 0x0000000041530a8c in ?? ()
#20 0x00007fef983abad0 in ?? ()
#21 0x00007fef99eb4900 in ?? ()
#22 0x00007fef99eb4900 in ?? ()
#23 0x0000012c0000012c in ?? ()
#24 0x00007fef996c0660 in ?? ()
#25 0x00007fef9a31d820 in ?? ()
#26 0x00007fef9a31d820 in ?? ()
#27 0x00007fef9a31d820 in ?? ()
#28 0x00007fef994a6c00 in ?? ()
#29 0x00007fef9a31d820 in ?? ()
#30 0x00007fef98fa92e0 in ?? ()
#31 0x00007fef98fa92e0 in ?? ()
#32 0x00007fef98fa92e0 in ?? ()
#33 0x00007fef98d34ea0 in ?? ()
#34 0x0000000000000001 in ?? ()
#35 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7fef83dfa700 (LWP 16803)):
#0  0x000000338d20f09d in waitpid () from /lib64/libpthread.so.0
#1  0x000000000048cd5d in mono_handle_native_sigsegv (signal=<value optimized out>, ctx=<value optimized out>) at mini-exceptions.c:2192
#2  <signal handler called>
#3  0x000000338ca328a5 in raise () from /lib64/libc.so.6
#4  0x000000338ca34085 in abort () from /lib64/libc.so.6
#5  0x00000000005def1b in GC_abort (msg=0x6a78b0 "Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS") at misc.c:1099
#6  0x00000000005d51ef in GC_add_to_heap (p=0x7fee5d00c000, bytes=8388608) at alloc.c:840
#7  0x00000000005d5678 in GC_expand_hp_inner (n=2048) at alloc.c:994
#8  0x00000000005d5863 in GC_collect_or_expand (needed_blocks=1, ignore_off_page=0) at alloc.c:1070
#9  0x00000000005d5a20 in GC_allocobj (sz=10, kind=0) at alloc.c:1125
#10 0x00000000005d8fe5 in GC_generic_malloc_inner (lb=80, k=0) at malloc.c:136
#11 0x00000000005da1c6 in GC_generic_malloc_many (lb=80, k=0, result=0x7fef986b3090) at mallocx.c:513
#12 0x00000000005e1445 in GC_local_malloc_atomic (bytes=76) at pthread_support.c:395
#13 0x000000000055e242 in mono_object_allocate_ptrfree (vtable=0x21ceb20, n=22) at object.c:4220
#14 mono_array_new_specific (vtable=0x21ceb20, n=22) at object.c:4779
#15 0x00000000403ee59d in ?? ()
#16 0x00007fef68000c20 in ?? ()
#17 0x00007fef83df8960 in ?? ()
#18 0x0000000000000001 in ?? ()
#19 0x0000000000000002 in ?? ()
#20 0x000000000000095f in ?? ()
#21 0x00000000000036b7 in ?? ()
#22 0x00007fef83df8920 in ?? ()
#23 0x00007fef99205840 in ?? ()
#24 0x00007fef54536000 in ?? ()
#25 0x00000000000036a1 in ?? ()
#26 0x0000000000000016 in ?? ()
#27 0x000000004069c73c in ?? ()
#28 0x00007fef99205840 in ?? ()
#29 0x00007fef54536000 in ?? ()
#30 0x00000000000036a1 in ?? ()
#31 0x0000000000000016 in ?? ()
#32 0x000000000000095f in ?? ()
#33 0x000000004069c6e4 in ?? ()
#34 0x00007fee5e827e00 in ?? ()
#35 0x00007fef54536000 in ?? ()
#36 0x00000000000036a1 in ?? ()
#37 0x0000000000000016 in ?? ()
#38 0x00000000000036a1 in ?? ()
#39 0x000000004069c6ac in ?? ()
#40 0x00007fef99205840 in ?? ()
#41 0x00007fef54536000 in ?? ()
#42 0x00000000000036a1 in ?? ()
#43 0x0000000000000016 in ?? ()
#44 0x000000000000095f in ?? ()
#45 0x000000004069c63b in ?? ()
#46 0x00007fee5e827e00 in ?? ()
#47 0x00007fef9a600920 in ?? ()
#48 0x00007fef83df8b58 in ?? ()
#49 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7fef83bf9700 (LWP 16804)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=100) at threads.c:1320
#3  0x0000000040e6f09a in ?? ()
#4  0x00007fef6c000c20 in ?? ()
#5  0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7fef839f8700 (LWP 16805)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=100) at threads.c:1320
#3  0x0000000040e6f09a in ?? ()
#4  0x00007fef60000c20 in ?? ()
#5  0x0000000000002710 in ?? ()
#6  0x00007fef00000019 in ?? ()
#7  0x0000000041e38538 in ?? ()
#8  0x00007fef8c000cb0 in ?? ()
#9  0x00007fef839f7ad0 in ?? ()
#10 0x00007fef839f7710 in ?? ()
#11 0x00007fef9a5facc0 in ?? ()
#12 0x00007fef987e2618 in ?? ()
#13 0x00007fef987e2618 in ?? ()
#14 0x0000000000000064 in ?? ()
#15 0x0000000040e6effc in ?? ()
#16 0x00007fef987e2618 in ?? ()
#17 0x0000000041535bac in ?? ()
#18 0x0000000041535b77 in ?? ()
#19 0x00000000415349bc in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7fef83668700 (LWP 16855)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=250) at threads.c:1320
#3  0x0000000040e6f09a in ?? ()
#4  0x00007fef64000c20 in ?? ()
#5  0x00007fef9a5facc8 in ?? ()
#6  0x00000000000041d7 in ?? ()
#7  0x000000007fffffff in ?? ()
#8  0x00007fef8c000cb0 in ?? ()
#9  0x00007fef83667b50 in ?? ()
#10 0x00007fef83667a50 in ?? ()
#11 0x00007fef9a5facc0 in ?? ()
#12 0x00007fef987ffbc8 in ?? ()
#13 0x00007fef987ffbc8 in ?? ()
#14 0x00000000000000fa in ?? ()
#15 0x0000000040e6effc in ?? ()
#16 0x00007fef9a323ae0 in ?? ()
#17 0x00007fef9a323ae0 in ?? ()
#18 0x00007fef83667b50 in ?? ()
#19 0x00000000407d2c38 in ?? ()
#20 0x00007fef987ffbc8 in ?? ()
#21 0x00000000413c9aa2 in ?? ()
#22 0x0000000000000000 in ?? ()

Thread 2 (Thread 0x7fef5cf4f700 (LWP 32629)):
#0  0x000000338d20e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000338d209388 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x000000338d209257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000005e2929 in GC_lock () at pthread_support.c:1757
#4  0x00000000005d6911 in GC_register_finalizer_inner (obj=0x7fee5e4cdcf8, fn=0, cd=0x0, ofn=0x0, ocd=0x0, mp=0x5d68d1 <GC_null_finalize_mark_proc>) at finalize.c:345
#5  0x00000000005d6dd3 in GC_register_finalizer_no_order (obj=0x7fee5e4cdcf8, fn=0, cd=0x0, ofn=0x0, ocd=0x0) at finalize.c:510
#6  0x0000000041b7b92a in ?? ()
#7  0x00007fef58000c20 in ?? ()
#8  0x0000000040e76eb0 in ?? ()
#9  0x00007fef58000c20 in ?? ()
#10 0x00007fef987778f0 in ?? ()
#11 0x00007fef84006000 in ?? ()
#12 0x00007fef5cf4ea50 in ?? ()
#13 0x00007fef5cf4e6a0 in ?? ()
#14 0x00007fef9a5facc0 in ?? ()
#15 0x00007fee5d90e618 in ?? ()
#16 0x00007fee5e4cdcf8 in ?? ()
#17 0x00007fee5e4cdcf8 in ?? ()
#18 0x0000000040647a40 in ?? ()
#19 0x00007fee5d90e618 in ?? ()
#20 0x00007fef5cf4ecc0 in ?? ()
#21 0x0000000000000001 in ?? ()
#22 0x0000000041b7b619 in ?? ()
#23 0x00007fee5ead7ce0 in ?? ()
#24 0x00000000415344dc in ?? ()
#25 0x0000000041534497 in ?? ()
#26 0x0000000041533ea0 in ?? ()
#27 0x00007fef9a3237e0 in ?? ()
#28 0x01007fef5cf4e870 in ?? ()
#29 0x00007fef98fa1a20 in ?? ()
#30 0x00007fef987778f0 in ?? ()
#31 0x00007fef98fa1a20 in ?? ()
#32 0x00007fee5df4eba0 in ?? ()
#33 0x00007fee5df4eba0 in ?? ()
#34 0x00007fee5e4cdbe0 in ?? ()
#35 0x00007fef994aad00 in ?? ()
#36 0x00007feef4f06f00 in ?? ()
#37 0x00007fef5cf4e870 in ?? ()
#38 0x0000000000590bba in mono_thread_interruption_checkpoint_request (bypass_abort_protection=1559554640) at threads.c:4180
#39 0x00007fee5df4eba0 in ?? ()
#40 0x00007fee5df4eba0 in ?? ()
#41 0x00007fee5e4cdbe0 in ?? ()
#42 0x000001f4000001f4 in ?? ()
#43 0x00007fef996c0660 in ?? ()
#44 0x00007fef9a31d820 in ?? ()
#45 0x00007fef9a31d820 in ?? ()
#46 0x00007fef9a31d820 in ?? ()
#47 0x00007fef994a6c00 in ?? ()
#48 0x00007fef9a31d820 in ?? ()
#49 0x00007fef98fa92e0 in ?? ()
#50 0x00007fef98fa92e0 in ?? ()
#51 0x00007fef98fa92e0 in ?? ()
#52 0x00007fef9a3237e0 in ?? ()
#53 0x88d0ac56b19de8d2 in ?? ()
#54 0x00007fef9a3237e0 in ?? ()
#55 0x00007fef5cf4e9a0 in ?? ()
#56 0x0000000000000000 in ?? ()

Thread 1 (Thread 0x7fefa4abe740 (LWP 13220)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005a98dc in _wapi_handle_timedwait_signal_handle (handle=0xfa24, timeout=0x0, alertable=1, poll=<value optimized out>) at handles.c:1655
#2  0x00000000005b9e1d in WaitForSingleObjectEx (handle=0xfa24, timeout=4294967295, alertable=1) at wait.c:203
#3  0x000000000058ed61 in ves_icall_System_Threading_Thread_Join_internal (this=0x7fef986b8d68, ms=-1, thread=0xfa24) at threads.c:1522
#4  0x00000000407d2b80 in ?? ()
#5  0x00000000020fe700 in ?? ()
#6  0x00007fef9a5facc0 in ?? ()
#7  0x0000000000000000 in ?? ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Ill try to rerun it, as i didnt have enough rollback for the whole crash…

The last lines of the log are:

2013-11-10 18:05:43:      Pending Job Scan - Released pending task 1 with frame(s) 986-990 for job "[BURN] LEM_006_1210_v0042_mmu_voxel04_uvON_images_render3d_FL-VTOL-Wash_0" because the frames it depends on have finished and/or its required assets are available.
2013-11-10 18:05:43:      Pending Job Scan - Done.
2013-11-10 18:05:44:  Clean Up Thread - Performing house cleaning
2013-11-10 18:05:45:  Process exit code: 0
2013-11-10 18:05:45:  Performing Job Cleanup Scan...
2013-11-10 18:05:45:      Job Cleanup Scan - Loading jobs
2013-11-10 18:05:59:      Job Cleanup Scan - Cleaning up jobs
2013-11-10 18:05:59:      Job Cleanup Scan - Cleaning up orphaned tasks
2013-11-10 18:06:01:      Job Cleanup Scan - Done.
2013-11-10 18:06:11:  Power Management - Thermal Shutdown: Skipping zone "Test" because it is disabled
2013-11-10 18:06:11:  Power Management - Thermal Shutdown: Skipping zone "Slaves" because it is disabled
2013-11-10 18:06:11:  Power Management - Thermal Shutdown: Skipping zone "Laszlo" because it is disabled
2013-11-10 18:06:11:  Power Management - Thermal Shutdown: Skipping zone "AnimatorWorkstations" because it is disabled
2013-11-10 18:06:11:  Power Management - Idle Shutdown: Skipping idle shutdown group "Test" because it is disabled
2013-11-10 18:06:11:  Power Management - Idle Shutdown: Skipping idle shutdown group "Slaves" because it is disabled
2013-11-10 18:06:11:  Power Management - Idle Shutdown: Skipping idle shutdown group "Laszlo" because it is disabled
2013-11-10 18:06:11:  Power Management - Idle Shutdown: Skipping idle shutdown group "AnimatorWorkstations" because it is disabled
2013-11-10 18:06:11:  Power Management - Machine Startup: There are no slaves that need to be woken up at this time
2013-11-10 18:06:11:  Power Management - Machine Restart: Skipping machine group "Test" because it is disabled
2013-11-10 18:06:11:  Power Management - Machine Restart: Skipping machine group "Slaves" because it is disabled
2013-11-10 18:06:11:  Power Management - Machine Restart: Skipping machine group "Laszlo" because it is disabled
2013-11-10 18:06:11:  Power Management - Machine Restart: Skipping machine group "AnimatorWorkstations" because it is disabled
2013-11-10 18:06:11:  Power Management - Slave Scheduling: Skipping scheduling group "Test" because it is disabled
2013-11-10 18:06:11:  Power Management - Slave Scheduling: Skipping scheduling group "Slaves" because it is disabled
2013-11-10 18:06:11:  Power Management - Slave Scheduling: Skipping scheduling group "Laszlo" because it is disabled
2013-11-10 18:06:11:  Power Management - Slave Scheduling: Skipping scheduling group "AnimatorWorkstations" because it is disabled

AAAnnnnd 5 mins later, on queue:

Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Stacktrace:

  at (wrapper managed-to-native) object.__icall_wrapper_mono_array_new_specific (intptr,int) <0xffffffff>
  at ICSharpCode.SharpZipLib.Zip.Compression.DeflaterEngine..ctor (ICSharpCode.SharpZipLib.Zip.Compression.DeflaterPending) <0x0006f>
  at ICSharpCode.SharpZipLib.Zip.Compression.Deflater..ctor (int,bool) <0x000bb>
  at ICSharpCode.SharpZipLib.Zip.ZipOutputStream..ctor (System.IO.Stream) <0x000d3>
  at (wrapper remoting-invoke-with-check) ICSharpCode.SharpZipLib.Zip.ZipOutputStream..ctor (System.IO.Stream) <0xffffffff>
  at FranticX.IO.Compression.Zip.CompressBytesZip (byte[],FranticX.IO.Compression.CompressionLevel) <0x0008f>
  at Deadline.Net.DeadlineMessageUtils.SendDeadlineMessage (System.Net.Sockets.NetworkStream,Deadline.Net.DeadlineMessage) <0x000d3>
  at Deadline.Pulses.PulseServerThread.OnConnect (System.IAsyncResult) <0x004af>
  at (wrapper runtime-invoke) <Module>.runtime_invoke_void__this___object (object,intptr,intptr,intptr) <0xffffffff>

Native stacktrace:

        mono() [0x48cc26]
        /lib64/libpthread.so.0() [0x338d20f500]
        /lib64/libc.so.6(gsignal+0x35) [0x338ca328a5]
        /lib64/libc.so.6(abort+0x175) [0x338ca34085]
        mono() [0x5def1b]
        mono() [0x5d51ef]
        mono() [0x5d5678]
        mono() [0x5d5863]
        mono() [0x5d8dd5]
        mono() [0x5d919c]
        mono() [0x5d9363]
        mono() [0x5e136d]
        mono(mono_array_new_specific+0xb2) [0x55e242]
        [0x4180e59d]

Debug info from gdb:

Mono support loaded.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f18b026f700 (LWP 7994)]
[New Thread 0x7f17f081f700 (LWP 30661)]
[New Thread 0x7f18dccb0700 (LWP 3332)]
[New Thread 0x7f18dfb64700 (LWP 31568)]
[New Thread 0x7f18ec9a5700 (LWP 31509)]
[New Thread 0x7f18ecbaa700 (LWP 31507)]
[New Thread 0x7f18ecdaf700 (LWP 31505)]
[New Thread 0x7f18ecfb4700 (LWP 31504)]
[New Thread 0x7f18ed1b9700 (LWP 31503)]
[New Thread 0x7f18ed3be700 (LWP 31502)]
[New Thread 0x7f18ed407700 (LWP 31501)]
[New Thread 0x7f18ed448700 (LWP 31500)]
[New Thread 0x7f18ed793700 (LWP 13389)]
[New Thread 0x7f18ed7d4700 (LWP 13388)]
[New Thread 0x7f18ee3e0700 (LWP 13177)]
[New Thread 0x7f18eede2700 (LWP 13168)]
[New Thread 0x7f18efd4f700 (LWP 13166)]
[New Thread 0x7f18f0750700 (LWP 13165)]
[New Thread 0x7f18f1151700 (LWP 13164)]
[New Thread 0x7f18f1b52700 (LWP 13163)]
[New Thread 0x7f18f2553700 (LWP 13162)]
[New Thread 0x7f18f2f54700 (LWP 13161)]
[New Thread 0x7f18f3955700 (LWP 13160)]
0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  24 Thread 0x7f18f3955700 (LWP 13160)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  23 Thread 0x7f18f2f54700 (LWP 13161)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  22 Thread 0x7f18f2553700 (LWP 13162)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  21 Thread 0x7f18f1b52700 (LWP 13163)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  20 Thread 0x7f18f1151700 (LWP 13164)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  19 Thread 0x7f18f0750700 (LWP 13165)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  18 Thread 0x7f18efd4f700 (LWP 13166)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  17 Thread 0x7f18eede2700 (LWP 13168)  0x000000338d20d720 in sem_wait () from /lib64/libpthread.so.0
  16 Thread 0x7f18ee3e0700 (LWP 13177)  0x000000338d20b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
  15 Thread 0x7f18ed7d4700 (LWP 13388)  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
  14 Thread 0x7f18ed793700 (LWP 13389)  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
  13 Thread 0x7f18ed448700 (LWP 31500)  0x000000338cae8713 in epoll_wait () from /lib64/libc.so.6
  12 Thread 0x7f18ed407700 (LWP 31501)  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
  11 Thread 0x7f18ed3be700 (LWP 31502)  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
  10 Thread 0x7f18ed1b9700 (LWP 31503)  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
  9 Thread 0x7f18ecfb4700 (LWP 31504)  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
  8 Thread 0x7f18ecdaf700 (LWP 31505)  0x000000338d20e054 in __lll_lock_wait () from /lib64/libpthread.so.0
  7 Thread 0x7f18ecbaa700 (LWP 31507)  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
  6 Thread 0x7f18ec9a5700 (LWP 31509)  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
  5 Thread 0x7f18dfb64700 (LWP 31568)  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
  4 Thread 0x7f18dccb0700 (LWP 3332)  0x000000338d20f09d in waitpid () from /lib64/libpthread.so.0
  3 Thread 0x7f17f081f700 (LWP 30661)  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
  2 Thread 0x7f18b026f700 (LWP 7994)  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
* 1 Thread 0x7f18f97e7740 (LWP 13154)  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0

Thread 24 (Thread 0x7f18f3955700 (LWP 13160)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005e29eb in GC_wait_marker () at pthread_support.c:1863
#2  0x00000000005dc261 in GC_help_marker (my_mark_no=91) at mark.c:1116
#3  0x00000000005e176c in GC_mark_thread (id=0x0) at pthread_support.c:552
#4  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 23 (Thread 0x7f18f2f54700 (LWP 13161)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005e29eb in GC_wait_marker () at pthread_support.c:1863
#2  0x00000000005dc261 in GC_help_marker (my_mark_no=91) at mark.c:1116
#3  0x00000000005e176c in GC_mark_thread (id=0x1) at pthread_support.c:552
#4  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 22 (Thread 0x7f18f2553700 (LWP 13162)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005e29eb in GC_wait_marker () at pthread_support.c:1863
#2  0x00000000005dc261 in GC_help_marker (my_mark_no=91) at mark.c:1116
#3  0x00000000005e176c in GC_mark_thread (id=0x2) at pthread_support.c:552
#4  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 21 (Thread 0x7f18f1b52700 (LWP 13163)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005e29eb in GC_wait_marker () at pthread_support.c:1863
#2  0x00000000005dc261 in GC_help_marker (my_mark_no=91) at mark.c:1116
#3  0x00000000005e176c in GC_mark_thread (id=0x3) at pthread_support.c:552
#4  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 20 (Thread 0x7f18f1151700 (LWP 13164)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005e29eb in GC_wait_marker () at pthread_support.c:1863
#2  0x00000000005dc261 in GC_help_marker (my_mark_no=91) at mark.c:1116
#3  0x00000000005e176c in GC_mark_thread (id=0x4) at pthread_support.c:552
#4  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 19 (Thread 0x7f18f0750700 (LWP 13165)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005e29eb in GC_wait_marker () at pthread_support.c:1863
#2  0x00000000005dc261 in GC_help_marker (my_mark_no=91) at mark.c:1116
#3  0x00000000005e176c in GC_mark_thread (id=0x5) at pthread_support.c:552
#4  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 18 (Thread 0x7f18efd4f700 (LWP 13166)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005e29eb in GC_wait_marker () at pthread_support.c:1863
#2  0x00000000005dc261 in GC_help_marker (my_mark_no=91) at mark.c:1116
#3  0x00000000005e176c in GC_mark_thread (id=0x6) at pthread_support.c:552
#4  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#5  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 17 (Thread 0x7f18eede2700 (LWP 13168)):
#0  0x000000338d20d720 in sem_wait () from /lib64/libpthread.so.0
#1  0x00000000005c1b78 in mono_sem_wait (sem=0x8f09e0, alertable=1) at mono-semaphore.c:113
#2  0x000000000051045b in finalizer_thread (unused=<value optimized out>) at gc.c:1066
#3  0x000000000059054b in start_wrapper_internal (data=0x181d530) at threads.c:784
#4  start_wrapper (data=0x181d530) at threads.c:832
#5  0x00000000005bbb84 in thread_start_routine (args=0x17ffb00) at wthreads.c:287
#6  0x00000000005e264e in GC_start_routine (arg=0x7f18ef052fc0) at pthread_support.c:1468
#7  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#8  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 16 (Thread 0x7f18ee3e0700 (LWP 13177)):
#0  0x000000338d20b7bb in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005a981d in _wapi_handle_timedwait_signal_handle (handle=0xfa08, timeout=0x7f18ee3df810, alertable=1, poll=<value optimized out>) at handles.c:1653
#2  0x00000000005b9c91 in WaitForSingleObjectEx (handle=0xfa08, timeout=996, alertable=1) at wait.c:205
#3  0x000000000058d40b in ves_icall_System_Threading_WaitHandle_WaitOne_internal (this=<value optimized out>, handle=0xfa08, ms=996, exitContext=<value optimized out>) at threads.c:1665
#4  0x0000000040743428 in ?? ()
#5  0x00007f18e0000c20 in ?? ()
#6  0x00007fff621ffa37 in clock_gettime ()
#7  0x000000004074295b in ?? ()
#8  0x00000000000003e4 in ?? ()
#9  0x00000000000003e4 in ?? ()
#10 0x0000000040740c30 in ?? ()
#11 0x0000000040740c6b in ?? ()
#12 0x000000338cd8ee80 in ?? () from /lib64/libc.so.6
#13 0x00007f18ee3eafc8 in ?? ()
#14 0x00007f18ef329020 in ?? ()
#15 0x000014584eb29d55 in ?? ()
#16 0x0000000000000038 in ?? ()
#17 0x00000000018e78e8 in ?? ()
#18 0x000000000185c2f8 in ?? ()
#19 0x00007f18ef04aee0 in ?? ()
#20 0x00007f18ef04aeb0 in ?? ()
#21 0x0000000000000038 in ?? ()
#22 0x7fffffffffffffff in ?? ()
#23 0x00007f18ef04aeb0 in ?? ()
#24 0x00000000018e78e8 in ?? ()
#25 0x000000000185c2f8 in ?? ()
#26 0x00000001e0000cb0 in ?? ()
#27 0x00007f17e6d220b8 in ?? ()
#28 0x00007f17e6d22090 in ?? ()
#29 0x00007f18ef04aeb0 in ?? ()
#30 0x00000000018e78e8 in ?? ()
#31 0x00007f18ef04aee8 in ?? ()
#32 0x0000000000000038 in ?? ()
#33 0x00007f18ef04aeb0 in ?? ()
#34 0x00007f18ef04aeb0 in ?? ()
#35 0x00007f18ef04aee8 in ?? ()
#36 0x0000000000000000 in ?? ()

Thread 15 (Thread 0x7f18ed7d4700 (LWP 13388)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x000000000058961b in monitor_thread (unused=<value optimized out>) at threadpool.c:777
#3  0x000000000059054b in start_wrapper_internal (data=0x7f18e0000e60) at threads.c:784
#4  start_wrapper (data=0x7f18e0000e60) at threads.c:832
#5  0x00000000005bbb84 in thread_start_routine (args=0x17ffee8) at wthreads.c:287
#6  0x00000000005e264e in GC_start_routine (arg=0x7f18ef052f18) at pthread_support.c:1468
#7  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#8  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 14 (Thread 0x7f18ed793700 (LWP 13389)):
#0  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
#1  0x00000000005c1c83 in mono_sem_timedwait (sem=0x8f1568, timeout_ms=<value optimized out>, alertable=1) at mono-semaphore.c:76
#2  0x000000000058aeef in async_invoke_thread (data=0x0) at threadpool.c:1485
#3  0x000000000059054b in start_wrapper_internal (data=0x7f18e00053a0) at threads.c:784
#4  start_wrapper (data=0x7f18e00053a0) at threads.c:832
#5  0x00000000005bbb84 in thread_start_routine (args=0x17fffb0) at wthreads.c:287
#6  0x00000000005e264e in GC_start_routine (arg=0x7f18ef052f18) at pthread_support.c:1468
#7  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#8  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 13 (Thread 0x7f18ed448700 (LWP 31500)):
#0  0x000000338cae8713 in epoll_wait () from /lib64/libc.so.6
#1  0x0000000000589fd8 in tp_epoll_wait (p=0x8f14c0) at ../../mono/metadata/tpool-epoll.c:101
#2  0x000000000059054b in start_wrapper_internal (data=0x226fb10) at threads.c:784
#3  start_wrapper (data=0x226fb10) at threads.c:832
#4  0x00000000005bbb84 in thread_start_routine (args=0x1800398) at wthreads.c:287
#5  0x00000000005e264e in GC_start_routine (arg=0x7f18ef052ee0) at pthread_support.c:1468
#6  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#7  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 12 (Thread 0x7f18ed407700 (LWP 31501)):
#0  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
#1  0x00000000005c1c83 in mono_sem_timedwait (sem=0x8f1628, timeout_ms=<value optimized out>, alertable=1) at mono-semaphore.c:76
#2  0x000000000058aeef in async_invoke_thread (data=0x0) at threadpool.c:1485
#3  0x000000000059054b in start_wrapper_internal (data=0x2270f80) at threads.c:784
#4  start_wrapper (data=0x2270f80) at threads.c:832
#5  0x00000000005bbb84 in thread_start_routine (args=0x1800460) at wthreads.c:287
#6  0x00000000005e264e in GC_start_routine (arg=0x7f18ef052ee0) at pthread_support.c:1468
#7  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#8  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 11 (Thread 0x7f18ed3be700 (LWP 31502)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=100) at threads.c:1320
#3  0x000000004138d0ea in ?? ()
#4  0x00007f18d0000c20 in ?? ()
#5  0x0000000000000014 in ?? ()
#6  0x88d0ac5800000024 in ?? ()
#7  0x0000000040fab538 in ?? ()
#8  0x00007f18e0000cb0 in ?? ()
#9  0x00007f18ed3bdad0 in ?? ()
#10 0x00007f18ed3bd9b0 in ?? ()
#11 0x00007f18ef323cc0 in ?? ()
#12 0x00007f18edd02bc8 in ?? ()
#13 0x00007f18edd02bc8 in ?? ()
#14 0x0000000000000064 in ?? ()
#15 0x000000004138d04c in ?? ()
#16 0x00007f18edd02bc8 in ?? ()
#17 0x00007f18e0000cb0 in ?? ()
#18 0x00007f18ed3bdad0 in ?? ()
#19 0x0000000041c910bc in ?? ()
#20 0x00000000018740d0 in ?? ()
#21 0x0000000000000000 in ?? ()

Thread 10 (Thread 0x7f18ed1b9700 (LWP 31503)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=250) at threads.c:1320
#3  0x000000004138d0ea in ?? ()
#4  0x00007f18c4000c20 in ?? ()
#5  0x08d0ac581fbdf0e0 in ?? ()
#6  0x08d0ac9b2de130e0 in ?? ()
#7  0x08d0ac9b2de130e0 in ?? ()
#8  0x00007f18e0000cb0 in ?? ()
#9  0x00007f18ed1b8ad0 in ?? ()
#10 0x00007f18ed1b80b0 in ?? ()
#11 0x00007f18ef323cc0 in ?? ()
#12 0x00007f18edd028f0 in ?? ()
#13 0x00007f18edd028f0 in ?? ()
#14 0x00000000000000fa in ?? ()
#15 0x000000004138d04c in ?? ()
#16 0x00007f18edd028f0 in ?? ()
#17 0x88d0ac581fbdf0e0 in ?? ()
#18 0x08d0ac9b2de130e0 in ?? ()
#19 0x0000000041c9a804 in ?? ()
#20 0x0000000000000fff in ?? ()
#21 0x00007f181139c198 in ?? ()
#22 0x00007f1812f9aa00 in ?? ()
#23 0x00007f1888cf8060 in ?? ()
#24 0x00007f18dff64ea0 in ?? ()
#25 0x00007f18dff64ea0 in ?? ()
#26 0x00007f18edcd32e0 in ?? ()
#27 0x00007f18edcd32e0 in ?? ()
#28 0x00007f18dff3dea0 in ?? ()
#29 0x0000007000000077 in ?? ()
#30 0x0000000000000001 in ?? ()
#31 0x00007f18c4000cb0 in ?? ()
#32 0x00000000fffffbe4 in ?? ()
#33 0x00007f18c4000020 in ?? ()
#34 0x00007f18c4000cb0 in ?? ()
#35 0x00007f18c40b5c50 in ?? ()
#36 0x00000000fffffd20 in ?? ()
#37 0x0000000000000005 in ?? ()
#38 0x00007f18c40b5cc0 in ?? ()
#39 0x00000000004153e5 in g_slist_append_mempool (cfg=0x7f18e0000cb0, when=<value optimized out>, tag=-281920320, reg=-305125136, val=250) at ../../mono/metadata/mempool-internals.h:45
#40 mono_emit_unwind_op (cfg=0x7f18e0000cb0, when=<value optimized out>, tag=-281920320, reg=-305125136, val=250) at mini.c:561
#41 0x00007f18edaa96e0 in ?? ()
#42 0x00007f18edaa96e0 in ?? ()
#43 0x00007f18ee407c00 in ?? ()
#44 0x00007f18edaa96e0 in ?? ()
#45 0x00007f18edcd32e0 in ?? ()
#46 0x00007f18edcd32e0 in ?? ()
#47 0x00007f18edcd32e0 in ?? ()
#48 0x00007f18dff3dea0 in ?? ()
#49 0x00007f18edaa96e0 in ?? ()
#50 0x00007f18edaa96e0 in ?? ()
#51 0x00007f18edaa96e0 in ?? ()
#52 0x00007f18ee407c00 in ?? ()
#53 0x00007f18edaa96e0 in ?? ()
#54 0x00007f18edcd32e0 in ?? ()
#55 0x00007f18edcd32e0 in ?? ()
#56 0x00007f18edcd32e0 in ?? ()
#57 0x00007f18dff3dea0 in ?? ()
#58 0x00007f18aaf67000 in ?? ()
#59 0x00007f18edac8268 in ?? ()
#60 0x00007f18edac8268 in ?? ()
#61 0x00007f18edac8268 in ?? ()
#62 0x00007f18edac8268 in ?? ()
#63 0x00007f18ee407c00 in ?? ()
#64 0x00007f18edac8268 in ?? ()
#65 0x00007f18edcd32e0 in ?? ()
#66 0x00007f18edcd32e0 in ?? ()
#67 0x00007f18edcd32e0 in ?? ()
#68 0x00007f18dff3dea0 in ?? ()
#69 0x00007f18113a9be0 in ?? ()
#70 0x00007f18ed1b87f8 in ?? ()
#71 0x00007f1813f48bb0 in ?? ()
#72 0x00007f1813f48bb0 in ?? ()
#73 0x00007f18dff3dea0 in ?? ()
#74 0x00007f181139c198 in ?? ()
#75 0x00007f18ed1b87f8 in ?? ()
#76 0x00007f18ed1b8810 in ?? ()
#77 0x00007f18ed1b8810 in ?? ()
#78 0x00007f1813f48c08 in ?? ()
#79 0x00007f1813f48c08 in ?? ()
#80 0x00007f18dff3dea0 in ?? ()
#81 0x00007f1813f48bb0 in ?? ()
#82 0x00007f1813f48bb0 in ?? ()
#83 0x00007f1813f48bb0 in ?? ()
#84 0x00007f18dff3dea0 in ?? ()
#85 0x00007f1813f48c08 in ?? ()
#86 0x00007f1813f48c08 in ?? ()
#87 0x00007f1813f48c08 in ?? ()
#88 0x00007f18dff3dea0 in ?? ()
#89 0x00007f18c40b64c0 in ?? ()
#90 0x0000000a0000000a in ?? ()
#91 0x00007f18ef323990 in ?? ()
#92 0x00007f18edaa96e0 in ?? ()
#93 0x00007f18edaa96e0 in ?? ()
#94 0x00007f18edaa96e0 in ?? ()
#95 0x00007f18ee407c00 in ?? ()
#96 0x00007f18edaa96e0 in ?? ()
#97 0x00007f18edcd32e0 in ?? ()
#98 0x00007f18edcd32e0 in ?? ()
#99 0x00007f18edcd32e0 in ?? ()
#100 0x00007f18dff3dea0 in ?? ()
#101 0x00007f18edaa96e0 in ?? ()
#102 0x00007f18edaa96e0 in ?? ()
#103 0x00007f18edaa96e0 in ?? ()
#104 0x00007f18ee407c00 in ?? ()
#105 0x00007f18edaa96e0 in ?? ()
#106 0x00007f18edcd32e0 in ?? ()
#107 0x00007f18edcd32e0 in ?? ()
#108 0x00007f18edcd32e0 in ?? ()
#109 0x00007f18dff3dea0 in ?? ()
#110 0x00007f18aa8a8000 in ?? ()
#111 0x00007f18de7c1000 in ?? ()
#112 0x00007f18edac8268 in ?? ()
#113 0x00007f18edac8268 in ?? ()
#114 0x00007f18edac8268 in ?? ()
#115 0x00007f18edac8268 in ?? ()
#116 0x00007f18ee407c00 in ?? ()
#117 0x00007f18edac8268 in ?? ()
#118 0x00007f18edcd32e0 in ?? ()
#119 0x00007f18edcd32e0 in ?? ()
#120 0x00007f18edcd32e0 in ?? ()
#121 0x00007f18dff3dea0 in ?? ()
#122 0x00007f1814a45130 in ?? ()
#123 0x00007f1814a45130 in ?? ()
#124 0x00007f18ed1b8840 in ?? ()
#125 0x00007f18ed1b8830 in ?? ()
#126 0x00007f18ed1b8830 in ?? ()
#127 0x00007f18ed1b8830 in ?? ()
#128 0x00007f1813f48e70 in ?? ()
#129 0x00007f1813f48e70 in ?? ()
#130 0x00007f18dff3dea0 in ?? ()
#131 0x00007f1813f48e70 in ?? ()
#132 0x00007f1813f48e70 in ?? ()
#133 0x00007f1813f48e70 in ?? ()
#134 0x00007f18dff3dea0 in ?? ()
#135 0x00007f18113a9be0 in ?? ()
#136 0x00007f18ed1b87f8 in ?? ()
#137 0x00007f1813f48bb0 in ?? ()
#138 0x00007f1813f48bb0 in ?? ()
#139 0x00007f18dff3dea0 in ?? ()
#140 0x00007f181139c198 in ?? ()
#141 0x00007f18ed1b87f8 in ?? ()
#142 0x00007f18ed1b8810 in ?? ()
#143 0x00007f18ed1b8810 in ?? ()
#144 0x00007f1813f48c08 in ?? ()
#145 0x00007f1813f48c08 in ?? ()
#146 0x00007f18dff3dea0 in ?? ()
#147 0x00007f1813f48bb0 in ?? ()
#148 0x00007f1813f48bb0 in ?? ()
#149 0x00007f1813f48bb0 in ?? ()
#150 0x00007f18dff3dea0 in ?? ()
#151 0x00007f1813f48c08 in ?? ()
#152 0x00007f1813f48c08 in ?? ()
#153 0x00007f1813f48c08 in ?? ()
#154 0x00007f18dff3dea0 in ?? ()
#155 0x0101000000000000 in ?? ()
#156 0x00007f18ef323990 in ?? ()
#157 0x00007f18edaa96e0 in ?? ()
#158 0x00007f18edaa96e0 in ?? ()
#159 0x00007f18edaa96e0 in ?? ()
#160 0x00007f18ee407c00 in ?? ()
#161 0x00007f18edaa96e0 in ?? ()
#162 0x00007f18edcd32e0 in ?? ()
#163 0x00007f18edcd32e0 in ?? ()
#164 0x00007f18edcd32e0 in ?? ()
#165 0x00007f18dff3dea0 in ?? ()
#166 0x01017f18ed1b8630 in ?? ()
#167 0x00007f18ef323990 in ?? ()
#168 0x00007f18edaa96e0 in ?? ()
#169 0x00007f18edaa96e0 in ?? ()
#170 0x00007f18edaa96e0 in ?? ()
#171 0x00007f18ee407c00 in ?? ()
#172 0x00007f18edaa96e0 in ?? ()
#173 0x00007f18edcd32e0 in ?? ()
#174 0x00007f18edcd32e0 in ?? ()
#175 0x00007f18edcd32e0 in ?? ()
#176 0x00007f18dff3dea0 in ?? ()
#177 0x010100000224f200 in ?? ()
#178 0x00007f18ef323990 in ?? ()
#179 0x00007f18edaa96e0 in ?? ()
#180 0x00007f18edaa96e0 in ?? ()
#181 0x00007f18edaa96e0 in ?? ()
#182 0x00007f18ee407c00 in ?? ()
#183 0x00007f18edaa96e0 in ?? ()
#184 0x00007f18edcd32e0 in ?? ()
#185 0x00007f18edcd32e0 in ?? ()
#186 0x00007f18edcd32e0 in ?? ()
#187 0x00007f18dff3dea0 in ?? ()
#188 0x403edd4738a3b57c in ?? ()
#189 0x00007f18ed1b8808 in ?? ()
#190 0x00000001edd028f0 in ?? ()
#191 0x404e000000000000 in ?? ()
#192 0x00007f18ed1b8808 in ?? ()
#193 0x0000003c0000003c in ?? ()
#194 0x00007f18ef323990 in ?? ()
#195 0x00007f18edaa96e0 in ?? ()
#196 0x00007f18edaa96e0 in ?? ()
#197 0x00007f18edaa96e0 in ?? ()
#198 0x00007f18ee407c00 in ?? ()
#199 0x00007f18edaa96e0 in ?? ()
#200 0x00007f18edcd32e0 in ?? ()
#201 0x00007f18edcd32e0 in ?? ()
#202 0x00007f18edcd32e0 in ?? ()
#203 0x00000001018b44a8 in ?? ()
#204 0x00007f1800000000 in ?? ()
#205 0x0000000000491495 in mono_delegate_trampoline (regs=<value optimized out>, code=0x1e0000001e <Address 0x1e0000001e out of bounds>, tramp_data=<value optimized out>, tramp=<value optimized out>) at mini-trampolines.c:972
#206 0x0000000023c34600 in ?? ()
#207 0x0000000012658760 in ?? ()
#208 0x88d0ac581fbdf0e0 in ?? ()
#209 0x88d0ac580e3a094e in ?? ()
#210 0x000000008820c14a in ?? ()
#211 0x0000000430e23400 in ?? ()
#212 0x000000008820c14a in ?? ()
#213 0x0000000430e23400 in ?? ()
#214 0x00007f1813f48c08 in ?? ()
#215 0x0000053d00000000 in ?? ()
#216 0x0000000000000000 in ?? ()

Thread 9 (Thread 0x7f18ecfb4700 (LWP 31504)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=250) at threads.c:1320
#3  0x000000004138d0ea in ?? ()
#4  0x00007f18c8000c20 in ?? ()
#5  0x08d0ac581fdecc66 in ?? ()
#6  0x08d0ac9b2e020c66 in ?? ()
#7  0x08d0ac9b2e020c66 in ?? ()
#8  0x00007f18e0000cb0 in ?? ()
#9  0x00007f18ecfb3ad0 in ?? ()
#10 0x00007f18ecfb37a0 in ?? ()
#11 0x00007f18ef323cc0 in ?? ()
#12 0x00007f18edd027b8 in ?? ()
#13 0x00007f18edd027b8 in ?? ()
#14 0x00000000000000fa in ?? ()
#15 0x000000004138d04c in ?? ()
#16 0x00007f18edd027b8 in ?? ()
#17 0x88d0ac581fdecc66 in ?? ()
#18 0x08d0ac9b2e020c66 in ?? ()
#19 0x0000000041c91c8c in ?? ()
#20 0x00007f18ecfb3ad0 in ?? ()
#21 0x00007f18dff64e40 in ?? ()
#22 0x00007f18dff64e40 in ?? ()
#23 0x0000012c0000012c in ?? ()
#24 0x00007f18ef323990 in ?? ()
#25 0x00007f18edaa96e0 in ?? ()
#26 0x00007f18edaa96e0 in ?? ()
#27 0x00007f18edaa96e0 in ?? ()
#28 0x00007f18ee407c00 in ?? ()
#29 0x00007f18edaa96e0 in ?? ()
#30 0x00007f18edcd32e0 in ?? ()
#31 0x00007f18edcd32e0 in ?? ()
#32 0x00007f18edcd32e0 in ?? ()
#33 0x00007f18dff3dea0 in ?? ()
#34 0x0000000000000001 in ?? ()
#35 0x0000000000000000 in ?? ()

Thread 8 (Thread 0x7f18ecdaf700 (LWP 31505)):
#0  0x000000338d20e054 in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x000000338d209388 in _L_lock_854 () from /lib64/libpthread.so.0
#2  0x000000338d209257 in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x00000000005e2902 in GC_generic_lock (lock=0x8f2720) at pthread_support.c:1676
#4  0x00000000005e2935 in GC_lock () at pthread_support.c:1759
#5  0x00000000005d9dea in GC_generic_malloc_many (lb=48, k=0, result=0x7f18eda74870) at mallocx.c:375
#6  0x00000000005e1445 in GC_local_malloc_atomic (bytes=46) at pthread_support.c:395
#7  0x000000000055e242 in mono_object_allocate_ptrfree (vtable=0x1868b30, n=7) at object.c:4220
#8  mono_array_new_specific (vtable=0x1868b30, n=7) at object.c:4779
#9  0x000000004180e59d in ?? ()
#10 0x00007f18bc000c20 in ?? ()
#11 0x00007f18ecdad960 in ?? ()
#12 0x0000000000000001 in ?? ()
#13 0x0000000000000002 in ?? ()
#14 0x0000000000003064 in ?? ()
#15 0x0000000000000fa3 in ?? ()
#16 0x00007f18ecdad920 in ?? ()
#17 0x00007f18edf38480 in ?? ()
#18 0x00007f17afc8c000 in ?? ()
#19 0x0000000000000f9c in ?? ()
#20 0x0000000000000007 in ?? ()
#21 0x00000000419ee77c in ?? ()
#22 0x00007f18edf38480 in ?? ()
#23 0x00007f17afc8c000 in ?? ()
#24 0x0000000000000f9c in ?? ()
#25 0x0000000000000007 in ?? ()
#26 0x0000000000003064 in ?? ()
#27 0x00000000419ee724 in ?? ()
#28 0x00007f17ae8a6c40 in ?? ()
#29 0x00007f17afc8c000 in ?? ()
#30 0x0000000000000f9c in ?? ()
#31 0x0000000000000007 in ?? ()
#32 0x0000000000000f9c in ?? ()
#33 0x00000000419ee6ec in ?? ()
#34 0x00007f18edf38480 in ?? ()
#35 0x00007f17afc8c000 in ?? ()
#36 0x0000000000000f9c in ?? ()
#37 0x0000000000000007 in ?? ()
#38 0x0000000000003064 in ?? ()
#39 0x00000000419ee67b in ?? ()
#40 0x00007f17ae8a6c40 in ?? ()
#41 0x00007f18ef329920 in ?? ()
#42 0x00007f18ecdadb58 in ?? ()
#43 0x0000000000000000 in ?? ()

Thread 7 (Thread 0x7f18ecbaa700 (LWP 31507)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=100) at threads.c:1320
#3  0x000000004138d0ea in ?? ()
#4  0x00007f18c0000c20 in ?? ()
#5  0x0000000000004e20 in ?? ()
#6  0x00007f17e99c8fa8 in ?? ()
#7  0x00007f18dffc8030 in ?? ()
#8  0x00007f18e0000cb0 in ?? ()
#9  0x00007f18ecba9ad0 in ?? ()
#10 0x00007f18ecba9980 in ?? ()
#11 0x00007f18ef323cc0 in ?? ()
#12 0x00007f18edd023a8 in ?? ()
#13 0x00007f18edd023a8 in ?? ()
#14 0x0000000000000064 in ?? ()
#15 0x000000004138d04c in ?? ()
#16 0x00007f18edd023a8 in ?? ()
#17 0x0000000041c92918 in ?? ()
#18 0x0000000041c928d9 in ?? ()
#19 0x0000000041c9294c in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 6 (Thread 0x7f18ec9a5700 (LWP 31509)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=100) at threads.c:1320
#3  0x000000004138d0ea in ?? ()
#4  0x00007f18b4000c20 in ?? ()
#5  0x0000000000002710 in ?? ()
#6  0x00007f1800000023 in ?? ()
#7  0x0000000040fab538 in ?? ()
#8  0x00007f18e0000cb0 in ?? ()
#9  0x00007f18ec9a4ad0 in ?? ()
#10 0x00007f18ec9a4710 in ?? ()
#11 0x00007f18ef323cc0 in ?? ()
#12 0x00007f18edd02138 in ?? ()
#13 0x00007f18edd02138 in ?? ()
#14 0x0000000000000064 in ?? ()
#15 0x000000004138d04c in ?? ()
#16 0x00007f18edd02138 in ?? ()
#17 0x0000000041c97afc in ?? ()
#18 0x0000000041c97ac7 in ?? ()
#19 0x0000000041c9690c in ?? ()
#20 0x0000000000000000 in ?? ()

Thread 5 (Thread 0x7f18dfb64700 (LWP 31568)):
#0  0x000000338d20ed2d in nanosleep () from /lib64/libpthread.so.0
#1  0x00000000005bab34 in SleepEx (ms=<value optimized out>, alertable=1) at wthreads.c:1027
#2  0x0000000000591eb9 in ves_icall_System_Threading_Thread_Sleep_internal (ms=250) at threads.c:1320
#3  0x000000004138d0ea in ?? ()
#4  0x00007f18b8000c20 in ?? ()
#5  0x00007f18ef323cc8 in ?? ()
#6  0x0000000000007b50 in ?? ()
#7  0x000000007fffffff in ?? ()
#8  0x00007f18e0000cb0 in ?? ()
#9  0x00007f18dfb63b50 in ?? ()
#10 0x00007f18dfb63a50 in ?? ()
#11 0x00007f18ef323cc0 in ?? ()
#12 0x00007f18edccad00 in ?? ()
#13 0x00007f18edccad00 in ?? ()
#14 0x00000000000000fa in ?? ()
#15 0x000000004138d04c in ?? ()
#16 0x00007f18ef04cae0 in ?? ()
#17 0x00007f18ef04cae0 in ?? ()
#18 0x00007f18dfb63b50 in ?? ()
#19 0x00000000401fe038 in ?? ()
#20 0x00007f18edccad00 in ?? ()
#21 0x000000004073faa2 in ?? ()
#22 0x0000000000000000 in ?? ()

Thread 4 (Thread 0x7f18dccb0700 (LWP 3332)):
#0  0x000000338d20f09d in waitpid () from /lib64/libpthread.so.0
#1  0x000000000048cd5d in mono_handle_native_sigsegv (signal=<value optimized out>, ctx=<value optimized out>) at mini-exceptions.c:2192
#2  <signal handler called>
#3  0x000000338ca328a5 in raise () from /lib64/libc.so.6
#4  0x000000338ca34085 in abort () from /lib64/libc.so.6
#5  0x00000000005def1b in GC_abort (msg=0x6a78b0 "Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS") at misc.c:1099
#6  0x00000000005d51ef in GC_add_to_heap (p=0x7f17ad990000, bytes=8388608) at alloc.c:840
#7  0x00000000005d5678 in GC_expand_hp_inner (n=2048) at alloc.c:994
#8  0x00000000005d5863 in GC_collect_or_expand (needed_blocks=17, ignore_off_page=0) at alloc.c:1070
#9  0x00000000005d8dd5 in GC_alloc_large (lw=8196, k=0, flags=0) at malloc.c:60
#10 0x00000000005d919c in GC_generic_malloc (lb=65568, k=0) at malloc.c:204
#11 0x00000000005d9363 in GC_malloc_atomic (lb=65568) at malloc.c:270
#12 0x00000000005e136d in GC_local_malloc_atomic (bytes=65568) at pthread_support.c:380
#13 0x000000000055e242 in mono_object_allocate_ptrfree (vtable=0x179ac70, n=65536) at object.c:4220
#14 mono_array_new_specific (vtable=0x179ac70, n=65536) at object.c:4779
#15 0x000000004180e59d in ?? ()
#16 0x00007f18ac002ff0 in ?? ()
#17 0x00000000417fdc34 in ?? ()
#18 0x00007f17b3545aa0 in ?? ()
#19 0x00007f17c6390578 in ?? ()
#20 0x00007f18d8008a00 in ?? ()
#21 0x00007f18dccaf6e0 in ?? ()
#22 0x00007f18dccaf530 in ?? ()
#23 0x00007f18ef323cc0 in ?? ()
#24 0x00007f17df928150 in ?? ()
#25 0x00007f17b3545aa0 in ?? ()
#26 0x00007f17c6390578 in ?? ()
#27 0x00000000417fd8e0 in ?? ()
#28 0x0000000000000006 in ?? ()
#29 0x00007f17bab54e00 in ?? ()
#30 0x00007f17b35589d8 in ?? ()
#31 0x00000000417fd7cc in ?? ()
#32 0x00007f17b60d22d8 in ?? ()
#33 0x00007f17b3545b28 in ?? ()
#34 0x0000000000000001 in ?? ()
#35 0x00007f17b3545aa0 in ?? ()
#36 0x00007f17c6390578 in ?? ()
#37 0x00007f17c6390578 in ?? ()
#38 0x00007f17bab54e00 in ?? ()
#39 0x00000000417fd6d4 in ?? ()
#40 0x00007f17b3545b28 in ?? ()
#41 0x00007f17bab54e00 in ?? ()
#42 0x00007f17df928150 in ?? ()
#43 0xffffffffffffffff in ?? ()
#44 0x00007f18dccaf5e0 in ?? ()
#45 0x00000000417fd4cc in ?? ()
#46 0x00007f17b0cb0bb0 in ?? ()
#47 0x00007f17b0cb0bb0 in ?? ()
#48 0x00007f17b0cb0bb0 in ?? ()
#49 0x00000000417fd200 in ?? ()
#50 0x0000000000000000 in ?? ()

Thread 3 (Thread 0x7f17f081f700 (LWP 30661)):
#0  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
#1  0x00000000005c1c83 in mono_sem_timedwait (sem=0x8f1628, timeout_ms=<value optimized out>, alertable=1) at mono-semaphore.c:76
#2  0x000000000058aeef in async_invoke_thread (data=0x0) at threadpool.c:1485
#3  0x000000000059054b in start_wrapper_internal (data=0x7f18e4000cf0) at threads.c:784
#4  start_wrapper (data=0x7f18e4000cf0) at threads.c:832
#5  0x00000000005bbb84 in thread_start_routine (args=0x1802210) at wthreads.c:287
#6  0x00000000005e264e in GC_start_routine (arg=0x7f18ed48b5b0) at pthread_support.c:1468
#7  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#8  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 2 (Thread 0x7f18b026f700 (LWP 7994)):
#0  0x000000338d20d811 in sem_timedwait () from /lib64/libpthread.so.0
#1  0x00000000005c1c83 in mono_sem_timedwait (sem=0x8f1628, timeout_ms=<value optimized out>, alertable=1) at mono-semaphore.c:76
#2  0x000000000058aeef in async_invoke_thread (data=0x0) at threadpool.c:1485
#3  0x000000000059054b in start_wrapper_internal (data=0x7f18e4000cf0) at threads.c:784
#4  start_wrapper (data=0x7f18e4000cf0) at threads.c:832
#5  0x00000000005bbb84 in thread_start_routine (args=0x1803d68) at wthreads.c:287
#6  0x00000000005e264e in GC_start_routine (arg=0x7f18ed48b3f0) at pthread_support.c:1468
#7  0x000000338d207851 in start_thread () from /lib64/libpthread.so.0
#8  0x000000338cae811d in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f18f97e7740 (LWP 13154)):
#0  0x000000338d20b43c in pthread_cond_wait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00000000005a98dc in _wapi_handle_timedwait_signal_handle (handle=0xfa21, timeout=0x0, alertable=1, poll=<value optimized out>) at handles.c:1655
#2  0x00000000005b9e1d in WaitForSingleObjectEx (handle=0xfa21, timeout=4294967295, alertable=1) at wait.c:203
#3  0x000000000058ed61 in ves_icall_System_Threading_Thread_Join_internal (this=0x7f18eda7fd68, ms=-1, thread=0xfa21) at threads.c:1522
#4  0x00000000401fe0c0 in ?? ()
#5  0x0000000001798700 in ?? ()
#6  0x00007f18ef323cc0 in ?? ()
#7  0x0000000000000000 in ?? ()

=================================================================
Got a SIGABRT while executing native code. This usually indicates
a fatal error in the mono runtime or one of the native libraries
used by your application.
=================================================================

Rolling back to beta9

Can I ask what OS you are running?

Its centOS 6.3

viewtopic.php?f=86&t=10669#p46289