Having an odd occasional error on our farm. Am running python jobs with high concurrency (have two workers on each system, running 15 tasks each from same job).
Very rarely we’ll see tasks fail with the error log below. Other tasks queued to same worker will continue to run fine, and requeuing this specific failed task works, even when picked up on system that initially failed it.
Does this indicate an issue with our DB server that is lurking?
=======================================================
Error
=======================================================
An item with the same key has already been added. Key: FileMetadataType
=======================================================
Type
=======================================================
ArgumentException
=======================================================
Stack Trace
=======================================================
at System.Collections.Generic.Dictionary`2.TryInsert(TKey key, TValue value, InsertionBehavior behavior)
at System.Collections.Generic.Dictionary`2.Add(TKey key, TValue value)
at Deadline.StorageDB.Proxy.Utils.DictionaryHelper`2.KeepBuilding(T1 key, T2 value)
at Deadline.StorageDB.Proxy.ProxyFileStorage.GetNetworkFileList(String subDirectory, FileMetadataType metadataType)