Database corruption

Alcium · January 4, 2013, 3:09pm

It sometimes randomly happens that jobs are not removed correctly
as follows, jobs starting with _ID as 1st information are not correct:

{ “Arch” : false, “Aux” : [ ], “Bad” : [ ], “CompletedChunks” : 1, “Date” : ISODate(“2013-01-04T11:53:54.816Z”), “DateComp” : ISODate(“2013-01-04T12:51:39Z”), “DateStart” : ISODate(“2013-01-04T11:53:55.191Z”), “Errs” : 0, “FailedChunks” : 0, “IsSub” : true, “LastWriteTime” : ISODate(“2013-01-04T12:51:39.019Z”), “Mach” : “WALK004”, “OutDir” : [ ], “OutFile” : [ ], “PendingChunks” : 0, “Plug” : “Modo”, “PlugInfo” : { “Version” : “6xx”, “Build” : “None”, “Threads” : “0”, “PassGroup” : “RN_Passes”, “SceneFile” : “M:\Film\q020\s022\Scenes\Lighting\q020_s022_lgt_CHAR_v002.lxo” }, “Props” : { “Name” : “q020_s022_lgt_CHAR_v002.lxo”, “User” : “dtassenoy”, “Cmmt” : “”, “CmmtTag” : “”, “Dept” : “relink mdd with old tool”, “Frames” : “162”, “Chunk” : 1, “Tasks” : 1, “Grp” : “none”, “Pool” : “modo_601”, “Pri” : 50, “Conc” : 1, “ConcLimt” : true, “AuxSync” : false, “Int” : false, “Seq” : false, “Reload” : false, “NoEvnt” : false, “OnComp” : 2, “AutoTime” : false, “TimeScrpt” : false, “MinTime” : 0, “MaxTime” : 0, “Timeout” : 1, “Dep” : [ ], “DepFrame” : false, “DepComp” : true, “DepDel” : false, “DepFail” : false, “DepPer” : -1, “NoBad” : false, “JobFailOvr” : false, “JobFailErr” : 0, “TskFailOvr” : false, “TskFailErr” : 0, “SndWarn” : true, “NotOvr” : false, “SndEmail” : false, “NotEmail” : [ ], “NotUser” : [ “dtassenoy” ], “NotNote” : “”, “Limits” : [ ], “ListedSlaves” : [ ], “White” : false, “MachLmt” : 0, “MachLmtProg” : -1, “PrJobScrp” : “”, “PoJobScrp” : “”, “PrTskScrp” : “”, “PoTskScrp” : “”, “Schd” : 0, “SchdDays” : 1, “SchdDate” : ISODate(“0001-01-01T00:00:00Z”), “SchdDateRan” : ISODate(“0001-01-01T00:00:00Z”), “Ex0” : “”, “Ex1” : “”, “Ex2” : “”, “Ex3” : “”, “Ex4” : “”, “Ex5” : “”, “Ex6” : “”, “Ex7” : “”, “Ex8” : “”, “Ex9” : “”, “ExDic” : { } }, “QueuedChunks” : 0, “RenderingChunks” : 0, “Stat” : 3, “SuspendedChunks” : 0, “Tile” : false, “TileFrame” : 0, “TileX” : 0, “TileY” : 0, “_id” : “50e6c2d2d18e9045300f7c96” }
{ “_id” : “50e6caa0b6587218f04a9142”, “Props” : { “Name” : “Shot Builder: 028-001”, “User” : “oantignac”, “Cmmt” : “”, “CmmtTag” : “”, “Dept” : “”, “Frames” : “0”, “Chunk” : 1, “Tasks” : 2, “Grp” : “none”, “Pool” : “maya_2012”, “Pri” : 51, “Conc” : 1, “ConcLimt” : true, “AuxSync” : false, “Int” : false, “Seq” : false, “Reload” : false, “NoEvnt” : false, “OnComp” : 1, “AutoTime” : false, “TimeScrpt” : false, “MinTime” : 0, “MaxTime” : 0, “Timeout” : 1, “Dep” : [ ], “DepFrame” : false, “DepComp” : true, “DepDel” : false, “DepFail” : false, “DepPer” : -1, “NoBad” : false, “JobFailOvr” : false, “JobFailErr” : 0, “TskFailOvr” : false, “TskFailErr” : 0, “SndWarn” : true, “NotOvr” : false, “SndEmail” : false, “NotEmail” : [ ], “NotUser” : [ “oantignac” ], “NotNote” : “”, “Limits” : [ ], “ListedSlaves” : [ ], “White” : false, “MachLmt” : 0, “MachLmtProg” : -1, “PrJobScrp” : “”, “PoJobScrp” : “Z:\WG_Code\mdc\maya\scripts\deadline_shotbuilder_AddSound.py”, “PrTskScrp” : “”, “PoTskScrp” : “”, “Schd” : 0, “SchdDays” : 1, “SchdDate” : ISODate(“0001-01-01T00:00:00Z”), “SchdDateRan” : ISODate(“0001-01-01T00:00:00Z”), “Ex0” : “”, “Ex1” : “”, “Ex2” : “”, “Ex3” : “”, “Ex4” : “”, “Ex5” : “”, “Ex6” : “”, “Ex7” : “”, “Ex8” : “”, “Ex9” : “”, “ExDic” : { } }, “IsSub” : false, “Mach” : “WALK026”, “Date” : ISODate(“2013-01-04T12:27:12.529Z”), “DateStart” : ISODate(“0001-01-01T00:00:00Z”), “DateComp” : ISODate(“0001-01-01T00:00:00Z”), “Arch” : false, “Plug” : “CommandScript”, “OutDir” : [ ], “OutFile” : [ ], “Tile” : false, “TileFrame” : 0, “TileX” : 0, “TileY” : 0, “Stat” : 1, “Aux” : [ “CommandScript_commandscriptBatch.txt” ], “Bad” : [ ], “PlugInfo” : { “StartupDirectory” : “C:\Program Files\Thinkbox\Deadline6\bin\” }, “CompletedChunks” : 0, “QueuedChunks” : 2, “SuspendedChunks” : 0, “RenderingChunks” : 0, “FailedChunks” : 0, “PendingChunks” : 0, “Errs” : 0 }
{ “Arch” : false, “Aux” : [ ], “Bad” : [ ], “CompletedChunks” : 0, “Date” : ISODate(“2013-01-04T13:42:03.324Z”), “DateComp” : ISODate(“0001-01-01T00:00:00Z”), “DateStart” : ISODate(“2013-01-04T13:42:03.745Z”), “Errs” : 10, “FailedChunks” : 44, “IsSub” : true, “LastWriteTime” : ISODate(“2013-01-04T14:09:46.582Z”), “Mach” : “WALK004”, “OutDir” : [ ], “OutFile” : [ ], “PendingChunks” : 0, “Plug” : “Modo”, “PlugInfo” : { “Version” : “6xx”, “Build” : “None”, “Threads” : “0”, “PassGroup” : “RN_Passes”, “SceneFile” : “M:\Film\q020\s022\Scenes\Lighting\q020_s022_lgt_CHAR_v002.lxo” }, “Props” : { “Name” : “q020_s022_lgt_CHAR_v002.lxo”, “User” : “dtassenoy”, “Cmmt” : “”, “CmmtTag” : “”, “Dept” : “relink mdd with old tool”, “Frames” : “109,112,114,117-118,125,127,130,133,138,140-141,143-145,147-148,153-179”, “Chunk” : 1, “Tasks” : 44, “Grp” : “none”, “Pool” : “modo_601”, “Pri” : 50, “Conc” : 1, “ConcLimt” : true, “AuxSync” : false, “Int” : false, “Seq” : false, “Reload” : false, “NoEvnt” : false, “OnComp” : 2, “AutoTime” : false, “TimeScrpt” : false, “MinTime” : 0, “MaxTime” : 0, “Timeout” : 1, “Dep” : [ ], “DepFrame” : false, “DepComp” : true, “DepDel” : false, “DepFail” : false, “DepPer” : -1, “NoBad” : false, “JobFailOvr” : false, “JobFailErr” : 0, “TskFailOvr” : false, “TskFailErr” : 0, “SndWarn” : true, “NotOvr” : false, “SndEmail” : false, “NotEmail” : [ ], “NotUser” : [ “dtassenoy” ], “NotNote” : “”, “Limits” : [ ], “ListedSlaves” : [ ], “White” : false, “MachLmt” : 0, “MachLmtProg” : -1, “PrJobScrp” : “”, “PoJobScrp” : “”, “PrTskScrp” : “”, “PoTskScrp” : “”, “Schd” : 0, “SchdDays” : 1, “SchdDate” : ISODate(“0001-01-01T00:00:00Z”), “SchdDateRan” : ISODate(“0001-01-01T00:00:00Z”), “Ex0” : “”, “Ex1” : “”, “Ex2” : “”, “Ex3” : “”, “Ex4” : “”, “Ex5” : “”, “Ex6” : “”, “Ex7” : “”, “Ex8” : “”, “Ex9” : “”, “ExDic” : { } }, “QueuedChunks” : 0, “RenderingChunks” : 0, “Stat” : 4, “SuspendedChunks” : 0, “Tile” : false, “TileFrame” : 0, “TileX” : 0, “TileY” : 0, “_id” : “50e6dc2bd18e90cd94485998” }
{ “Arch” : false, “Aux” : [ ], “Bad” : [ ], “CompletedChunks” : 0, “Date” : ISODate(“2013-01-04T13:59:12.320Z”), “DateComp” : ISODate(“0001-01-01T00:00:00Z”), “DateStart” : ISODate(“2013-01-04T13:59:13.428Z”), “Errs” : 0, “FailedChunks” : 0, “IsSub” : true, “LastWriteTime” : ISODate(“2013-01-04T14:12:52.319Z”), “Mach” : “WALK028”, “OutDir” : [ ], “OutFile” : [ ], “PendingChunks” : 0, “Plug” : “MayaCmd”, “PlugInfo” : { “Version” : “2012”, “Build” : “None”, “ProjectPath” : “M:\temp\”, “StrictErrorChecking” : “True”, “LocalRendering” : “False”, “MaxProcessors” : “0”, “UseOnlyCommandLineOptions” : “0”, “IgnoreError211” : “False”, “SceneFile” : “M:/Film/q026/s022/Scenes/Layout/q026_s022_lay_v005.ma”, “OutputFilePath” : “M:/Film/q026/s022/Renders/Layout/v005”, “Renderer” : “MayaSoftware”, “CommandLineOptions” : "-preRender "python \“execfile(\\\“Z:/WG_Code/mdc/maya/scripts/renderpreset_layoutPreview.py\\\”)\” “” }, “Props” : { “Name” : “026-022 - Preview Layout (v005)”, “User” : “sbrys”, “Cmmt” : “”, “CmmtTag” : “”, “Dept” : “”, “Frames” : “101-304”, “Chunk” : 204, “Tasks” : 1, “Grp” : “none”, “Pool” : “small_tasks”, “Pri” : 55, “Conc” : 1, “ConcLimt” : true, “AuxSync” : false, “Int” : false, “Seq” : false, “Reload” : false, “NoEvnt” : false, “OnComp” : 1, “AutoTime” : false, “TimeScrpt” : false, “MinTime” : 0, “MaxTime” : 0, “Timeout” : 1, “Dep” : [ ], “DepFrame” : false, “DepComp” : true, “DepDel” : false, “DepFail” : false, “DepPer” : -1, “NoBad” : false, “JobFailOvr” : false, “JobFailErr” : 0, “TskFailOvr” : false, “TskFailErr” : 0, “SndWarn” : true, “NotOvr” : false, “SndEmail” : false, “NotEmail” : [ ], “NotUser” : [ “sbrys” ], “NotNote” : “”, “Limits” : [ ], “ListedSlaves” : [ ], “White” : false, “MachLmt” : 1, “MachLmtProg” : -1, “PrJobScrp” : “”, “PoJobScrp” : “”, “PrTskScrp” : “”, “PoTskScrp” : “”, “Schd” : 0, “SchdDays” : 1, “SchdDate” : ISODate(“0001-01-01T00:00:00Z”), “SchdDateRan” : ISODate(“0001-01-01T00:00:00Z”), “Ex0” : “”, “Ex1” : “”, “Ex2” : “”, “Ex3” : “”, “Ex4” : “”, “Ex5” : “”, “Ex6” : “”, “Ex7” : “”, “Ex8” : “”, “Ex9” : “”, “ExDic” : { } }, “QueuedChunks” : 0, “RenderingChunks” : 1, “Stat” : 1, “SuspendedChunks” : 0, “Tile” : false, “TileFrame” : 0, “TileX” : 0, “TileY” : 0, “_id” : “50e6e0305168270b3080d60d” }
{ “Arch” : false, “Aux” : [ ], “Bad” : [ ], “CompletedChunks” : 0, “Date” : ISODate(“2013-01-04T14:31:38.926Z”), “DateComp” : ISODate(“0001-01-01T00:00:00Z”), “DateStart” : ISODate(“2013-01-04T14:41:20.690Z”), “Errs” : 0, “FailedChunks” : 0, “IsSub” : true, “LastWriteTime” : ISODate(“2013-01-04T14:41:28.133Z”), “Mach” : “WALK014”, “OutDir” : [ ], “OutFile” : [ ], “PendingChunks” : 0, “Plug” : “MayaCmd”, “PlugInfo” : { “Version” : “2012”, “Build” : “None”, “ProjectPath” : “M:\temp\”, “StrictErrorChecking” : “True”, “LocalRendering” : “False”, “MaxProcessors” : “0”, “UseOnlyCommandLineOptions” : “0”, “IgnoreError211” : “False”, “SceneFile” : “M:/Film/q020/s006/Scenes/Layout/q020_s006_lay_build.ma”, “OutputFilePath” : “M:/Film/q020/s006/Renders/Layout/Build”, “Renderer” : “MayaSoftware”, “CommandLineOptions” : "-preRender "python \“execfile(\\\“Z:/WG_Code/mdc/maya/scripts/renderpreset_layoutPreview.py\\\”)\” “” }, “Props” : { “Name” : “020-006 - Preview Layout (build)”, “User” : “pculas”, “Cmmt” : “”, “CmmtTag” : “”, “Dept” : “”, “Frames” : “101-385”, “Chunk” : 285, “Tasks” : 1, “Grp” : “none”, “Pool” : “maya_2012”, “Pri” : 55, “Conc” : 1, “ConcLimt” : true, “AuxSync” : false, “Int” : false, “Seq” : false, “Reload” : false, “NoEvnt” : false, “OnComp” : 1, “AutoTime” : false, “TimeScrpt” : false, “MinTime” : 0, “MaxTime” : 0, “Timeout” : 1, “Dep” : [ “50e6e7c7b9d23d0fbce4bb6e” ], “DepFrame” : false, “DepComp” : true, “DepDel” : true, “DepFail” : true, “DepPer” : -1, “NoBad” : false, “JobFailOvr” : false, “JobFailErr” : 0, “TskFailOvr” : false, “TskFailErr” : 0, “SndWarn” : true, “NotOvr” : false, “SndEmail” : false, “NotEmail” : [ ], “NotUser” : [ “pculas” ], “NotNote” : “”, “Limits” : [ ], “ListedSlaves” : [ ], “White” : false, “MachLmt” : 1, “MachLmtProg” : -1, “PrJobScrp” : “”, “PoJobScrp” : “”, “PrTskScrp” : “”, “PoTskScrp” : “”, “Schd” : 0, “SchdDays” : 1, “SchdDate” : ISODate(“0001-01-01T00:00:00Z”), “SchdDateRan” : ISODate(“0001-01-01T00:00:00Z”), “Ex0” : “”, “Ex1” : “”, “Ex2” : “”, “Ex3” : “”, “Ex4” : “”, “Ex5” : “”, “Ex6” : “”, “Ex7” : “”, “Ex8” : “”, “Ex9” : “”, “ExDic” : { } }, “QueuedChunks” : 0, “RenderingChunks” : 1, “Stat” : 1, “SuspendedChunks” : 0, “Tile” : false, “TileFrame” : 0, “TileX” : 0, “TileY” : 0, “_id” : “50e6e7cbb9d23d0a9827ade5” }
{ “_id” : “50e6e94151682704a45359ae”, “Props” : { “Name” : “Shot Builder: 026-036”, “User” : “sbrys”, “Cmmt” : “”, “CmmtTag” : “”, “Dept” : “”, “Frames” : “0”, “Chunk” : 1, “Tasks” : 2, “Grp” : “none”, “Pool” : “maya_2012”, “Pri” : 51, “Conc” : 1, “ConcLimt” : true, “AuxSync” : false, “Int” : false, “Seq” : false, “Reload” : false, “NoEvnt” : false, “OnComp” : 1, “AutoTime” : false, “TimeScrpt” : false, “MinTime” : 0, “MaxTime” : 0, “Timeout” : 1, “Dep” : [ ], “DepFrame” : false, “DepComp” : true, “DepDel” : false, “DepFail” : false, “DepPer” : -1, “NoBad” : false, “JobFailOvr” : false, “JobFailErr” : 0, “TskFailOvr” : false, “TskFailErr” : 0, “SndWarn” : true, “NotOvr” : false, “SndEmail” : false, “NotEmail” : [ ], “NotUser” : [ “sbrys” ], “NotNote” : “”, “Limits” : [ ], “ListedSlaves” : [ ], “White” : false, “MachLmt” : 0, “MachLmtProg” : -1, “PrJobScrp” : “”, “PoJobScrp” : “Z:\WG_Code\mdc\maya\scripts\deadline_shotbuilder_AddSound.py”, “PrTskScrp” : “”, “PoTskScrp” : “”, “Schd” : 0, “SchdDays” : 1, “SchdDate” : ISODate(“0001-01-01T00:00:00Z”), “SchdDateRan” : ISODate(“0001-01-01T00:00:00Z”), “Ex0” : “”, “Ex1” : “”, “Ex2” : “”, “Ex3” : “”, “Ex4” : “”, “Ex5” : “”, “Ex6” : “”, “Ex7” : “”, “Ex8” : “”, “Ex9” : “”, “ExDic” : { } }, “IsSub” : false, “Mach” : “WALK028”, “Date” : ISODate(“2013-01-04T14:37:53.629Z”), “DateStart” : ISODate(“0001-01-01T00:00:00Z”), “DateComp” : ISODate(“0001-01-01T00:00:00Z”), “Arch” : false, “Plug” : “CommandScript”, “OutDir” : [ ], “OutFile” : [ ], “Tile” : false, “TileFrame” : 0, “TileX” : 0, “TileY” : 0, “Stat” : 1, “Aux” : [ “CommandScript_commandscriptBatch.txt” ], “Bad” : [ ], “PlugInfo” : { “StartupDirectory” : “C:\Program Files\Thinkbox\Deadline6\bin\” }, “CompletedChunks” : 0, “QueuedChunks” : 2, “SuspendedChunks” : 0, “RenderingChunks” : 0, “FailedChunks” : 0, “PendingChunks” : 0, “Errs” : 0 }
{ “Arch” : false, “Aux” : [ ], “Bad” : [ ], “CompletedChunks” : 0, “Date” : ISODate(“2013-01-04T14:44:26.148Z”), “DateComp” : ISODate(“0001-01-01T00:00:00Z”), “DateStart” : ISODate(“2013-01-04T14:48:36.583Z”), “Errs” : 0, “FailedChunks” : 0, “IsSub” : true, “LastWriteTime” : ISODate(“2013-01-04T14:48:36.627Z”), “Mach” : “WALK018”, “OutDir” : [ ], “OutFile” : [ ], “PendingChunks” : 0, “Plug” : “MayaCmd”, “PlugInfo” : { “Version” : “2012”, “Build” : “None”, “ProjectPath” : “M:\temp\”, “StrictErrorChecking” : “True”, “LocalRendering” : “False”, “MaxProcessors” : “0”, “UseOnlyCommandLineOptions” : “0”, “IgnoreError211” : “False”, “SceneFile” : “M:/Film/q023/s014/Scenes/Layout/q023_s014_lay_build.ma”, “OutputFilePath” : “M:/Film/q023/s014/Renders/Layout/Build”, “Renderer” : “MayaSoftware”, “CommandLineOptions” : "-preRender "python \“execfile(\\\“Z:/WG_Code/mdc/maya/scripts/renderpreset_layoutPreview.py\\\”)\” “” }, “Props” : { “Name” : “023-014 - Preview Layout (build)”, “User” : “kdebien”, “Cmmt” : “”, “CmmtTag” : “”, “Dept” : “”, “Frames” : “101-442”, “Chunk” : 342, “Tasks” : 1, “Grp” : “none”, “Pool” : “maya_2012”, “Pri” : 55, “Conc” : 1, “ConcLimt” : true, “AuxSync” : false, “Int” : false, “Seq” : false, “Reload” : false, “NoEvnt” : false, “OnComp” : 1, “AutoTime” : false, “TimeScrpt” : false, “MinTime” : 0, “MaxTime” : 0, “Timeout” : 1, “Dep” : [ “50e6eac6cfba120cfc9642fa” ], “DepFrame” : false, “DepComp” : true, “DepDel” : true, “DepFail” : true, “DepPer” : -1, “NoBad” : false, “JobFailOvr” : false, “JobFailErr” : 0, “TskFailOvr” : false, “TskFailErr” : 0, “SndWarn” : true, “NotOvr” : false, “SndEmail” : false, “NotEmail” : [ ], “NotUser” : [ “kdebien” ], “NotNote” : “”, “Limits” : [ ], “ListedSlaves” : [ ], “White” : false, “MachLmt” : 1, “MachLmtProg” : -1, “PrJobScrp” : “”, “PoJobScrp” : “”, “PrTskScrp” : “”, “PoTskScrp” : “”, “Schd” : 0, “SchdDays” : 1, “SchdDate” : ISODate(“0001-01-01T00:00:00Z”), “SchdDateRan” : ISODate(“0001-01-01T00:00:00Z”), “Ex0” : “”, “Ex1” : “”, “Ex2” : “”, “Ex3” : “”, “Ex4” : “”, “Ex5” : “”, “Ex6” : “”, “Ex7” : “”, “Ex8” : “”, “Ex9” : “”, “ExDic” : { } }, “QueuedChunks” : 0, “RenderingChunks” : 1, “Stat” : 1, “SuspendedChunks” : 0, “Tile” : false, “TileFrame” : 0, “TileX” : 0, “TileY” : 0, “_id” : “50e6eacacfba121364c25696” }

Resulting in an endless loop within pulse and jobs update rusty

Pulse constantly reporting :

once these jobs are removed from db using [code]> db.Jobs.remove({"_id" : “50e6caa0b6587218f04a9142”})

db.Jobs.remove({"_id" : “50e6e94151682704a45359ae”})
[/code]

pulse returns to normal

anon19176295 · January 5, 2013, 2:46pm

Does this happen when removing a group of jobs or a single job? Please let me know if there are any more details you could provide.

Alcium · January 7, 2013, 10:52am

This happens randomly but :

seems to happen mostly with packs of small jobs with auto-delete on complete
these are really lots of small jobs that do not stay long in queue.

sometimes on of them does not succeed in deletion and this ends up with database having a “corrupted” job,
but this job does not show up in monitor, nore does it retry deletion

so this ends up with pulse in infinite loop because it does not know what to do with this job

once job is manually removed from db, pulse immediatly returns to normal

anon19176295 · January 8, 2013, 1:38pm

Ok, that information helps.

I’m still testing to reproduce the error on my end.