Monitor memory usage info, statistics

nrusch · June 19, 2015, 6:22pm

I don’t know if this is a “bug report” per se, but I wanted to mention that the Monitor’s memory footprint seems to A) get quite high, and B) continue to grow over time.

I’ve had a monitor running for 21 hours, and the memory usage was over 3 GB when I came in this morning. I figured I would dump the object snapshots before and after a forced GC and post that information here.

The repository contains 7,087 jobs, 219 total slaves, 1 Pulse, and 13 Limit Groups. My monitor layout contains two Jobs panes with different filter sets, as well as Tasks, Remote Commands, Console, Slaves, Pulses, Limits, and Job Reports panes. This is using 7.0.2.3 on Fedora 19.

Here are the results:

Initial object dump output (Real private memory usage: 3.19 GB)

2015-06-19 10:57:59: Python Object Counts [>= 10]: 2015-06-19 10:57:59: 50396 <type 'dict'> 2015-06-19 10:57:59: 19446 <type 'list'> 2015-06-19 10:57:59: 12925 <type 'wrapper_descriptor'> 2015-06-19 10:57:59: 8887 <class 'ThinkboxUI.Models.GroupByUpdateModel.UpdateTreeItem'> 2015-06-19 10:57:59: 7086 <class 'Deadline.Jobs.Job'> 2015-06-19 10:57:59: 6516 <type 'function'> 2015-06-19 10:57:59: 5341 <type 'tuple'> 2015-06-19 10:57:59: 4520 <type 'sip.methoddescriptor'> 2015-06-19 10:57:59: 3779 <type 'weakref'> 2015-06-19 10:57:59: 3441 <class 'PyQt5.QtGui.QFont'> 2015-06-19 10:57:59: 1863 <type 'builtin_function_or_method'> 2015-06-19 10:57:59: 1818 <type 'getset_descriptor'> 2015-06-19 10:57:59: 1670 <class 'DeadlineUI.Models.JobListModel.JobBatchData'> 2015-06-19 10:57:59: 1208 <type 'PyQt5.QtCore.pyqtWrapperType'> 2015-06-19 10:57:59: 1006 <type 'method_descriptor'> 2015-06-19 10:57:59: 983 <class 'CLR.CLR Metatype'> 2015-06-19 10:57:59: 552 <type 'module'> 2015-06-19 10:57:59: 536 <type 'sip.enumtype'> 2015-06-19 10:57:59: 438 <class 'System.String[]'> 2015-06-19 10:57:59: 432 <class 'PyQt5.QtCore.Key'> 2015-06-19 10:57:59: 408 <type 'set'> 2015-06-19 10:57:59: 364 <class 'PyQt5.QtWidgets.QCheckBox'> 2015-06-19 10:57:59: 306 <type 'property'> 2015-06-19 10:57:59: 245 <type 'type'> 2015-06-19 10:57:59: 241 <class 'ThinkboxUI.Models.UpdateModel.UpdateItem'> 2015-06-19 10:57:59: 234 <class 'PyQt5.QtGui.QIcon'> 2015-06-19 10:57:59: 226 <class 'PyQt5.QtGui.QPixmap'> 2015-06-19 10:57:59: 219 <class 'Deadline.Slaves.SlaveInfoSettings'> 2015-06-19 10:57:59: 211 <type 'instancemethod'> 2015-06-19 10:57:59: 204 <type 'member_descriptor'> 2015-06-19 10:57:59: 198 <class '_weakrefset.WeakSet'> 2015-06-19 10:57:59: 178 <class 'Deadline.Permissions.Feature'> 2015-06-19 10:57:59: 138 <class 'PyQt5.QtCore.Type'> 2015-06-19 10:57:59: 127 <class 'PyQt5.QtWidgets.QAction'> 2015-06-19 10:57:59: 117 <class 'DeadlineMonitor.Forms.MonitorDockWidget.MonitorMorphWidget'> 2015-06-19 10:57:59: 105 <class 'PyQt5.QtWidgets.StyleHint'> 2015-06-19 10:57:59: 104 <class 'PyQt5.QtChart.QPieSlice'> 2015-06-19 10:57:59: 103 <class 'Deadline.Importers.JobDetailsEntry'> 2015-06-19 10:57:59: 99 <class 'PyQt5.QtCore.WidgetAttribute'> 2015-06-19 10:57:59: 94 <class 'PyQt5.QtGui.QColor'> 2015-06-19 10:57:59: 94 <class 'PyQt5.QtWidgets.PixelMetric'> 2015-06-19 10:57:59: 72 <class 'PyQt5.QtWidgets.StandardPixmap'> 2015-06-19 10:57:59: 69 <class 'PyQt5.QtGui.StandardKey'> 2015-06-19 10:57:59: 57 <type 'sip.variabledescriptor'> 2015-06-19 10:57:59: 56 <class 'PyQt5.QtWidgets.SubElement'> 2015-06-19 10:57:59: 55 <class 'PyQt5.QtWidgets.QVBoxLayout'> 2015-06-19 10:57:59: 52 <class 'PyQt5.QtWidgets.PrimitiveElement'> 2015-06-19 10:57:59: 48 <class 'PyQt5.QtWidgets.ControlElement'> 2015-06-19 10:57:59: 47 <type 'classobj'> 2015-06-19 10:57:59: 46 <class 'PyQt5.QtWidgets.StandardButton'> 2015-06-19 10:57:59: 43 <class 'PyQt5.QtWidgets.SubControl'> 2015-06-19 10:57:59: 39 <class 'PyQt5.QtCore.WindowType'> 2015-06-19 10:57:59: 38 <class 'PyQt5.QtGui.CompositionMode'> 2015-06-19 10:57:59: 35 <class 'PyQt5.QtCore.MouseButton'> 2015-06-19 10:57:59: 34 <class 'PyQt5.QtWidgets.GraphicsItemChange'> 2015-06-19 10:57:59: 31 <class 'PyQt5.QtChart.QPieSeries'> 2015-06-19 10:57:59: 30 <class 'ThinkboxUI.Commands.QMenuPlusFiles.QMenuPlus'> 2015-06-19 10:57:59: 29 <class 'PyQt5.QtWidgets.QToolButton'> 2015-06-19 10:57:59: 28 <class 'PyQt5.QtWidgets.StateFlag'> 2015-06-19 10:57:59: 27 <class 'PyQt5.QtCore.QModelIndex'> 2015-06-19 10:57:59: 27 <class 'PyQt5.QtWidgets.QWidget'> 2015-06-19 10:57:59: 26 <class 'PyQt5.QtWidgets.OptionType'> 2015-06-19 10:57:59: 25 <class 'PyQt5.QtCore.CursorShape'> 2015-06-19 10:57:59: 25 <class 'PyQt5.QtGui.MoveOperation'> 2015-06-19 10:57:59: 24 <class 'PyQt5.QtWidgets.ContentsType'> 2015-06-19 10:57:59: 24 <type 'operator.itemgetter'> 2015-06-19 10:57:59: 23 <class 'PyQt5.QtGui.ColorRole'> 2015-06-19 10:57:59: 23 <class 'PyQt5.QtWidgets.QLabel'> 2015-06-19 10:57:59: 22 <type 'classmethod'> 2015-06-19 10:57:59: 21 <class 'PyQt5.QtCore.InputMethodHint'> 2015-06-19 10:57:59: 20 <class 'PyQt5.QtCore.Filter'> 2015-06-19 10:57:59: 20 <class 'PyQt5.QtCore.GlobalColor'> 2015-06-19 10:57:59: 20 <class 'PyQt5.QtCore.QTimer'> 2015-06-19 10:57:59: 20 <class 'PyQt5.QtGui.QKeySequence'> 2015-06-19 10:57:59: 20 <class 'PyQt5.QtWidgets.ButtonRole'> 2015-06-19 10:57:59: 19 <class 'PyQt5.QtCore.BrushStyle'> 2015-06-19 10:57:59: 19 <class 'PyQt5.QtWidgets.QHBoxLayout'> 2015-06-19 10:57:59: 18 <class 'PyQt5.QtCore.ItemDataRole'> 2015-06-19 10:57:59: 18 <class 'abc.ABCMeta'> 2015-06-19 10:57:59: 18 <type 'staticmethod'> 2015-06-19 10:57:59: 17 <class 'PyQt5.QtCore.InputMethodQuery'> 2015-06-19 10:57:59: 17 <type 'classmethod_descriptor'> 2015-06-19 10:57:59: 17 <type 'frozenset'> 2015-06-19 10:57:59: 16 <class 'PyQt5.QtCore.ApplicationAttribute'> 2015-06-19 10:57:59: 16 <class 'PyQt5.QtWidgets.GraphicsItemFlag'> 2015-06-19 10:57:59: 16 <type 'collections.deque'> 2015-06-19 10:57:59: 15 <class 'PyQt5.QtWidgets.ControlType'> 2015-06-19 10:57:59: 15 <type 'instance'> 2015-06-19 10:57:59: 14 <class 'PyQt5.QtCore.AlignmentFlag'> 2015-06-19 10:57:59: 13 <class 'Deadline.LimitGroups.LimitGroup'> 2015-06-19 10:57:59: 13 <class 'PyQt5.QtCore.ItemFlags'> 2015-06-19 10:57:59: 13 <class 'PyQt5.QtCore.QByteArray'> 2015-06-19 10:57:59: 13 <class 'PyQt5.QtGui.StyleHint'> 2015-06-19 10:57:59: 13 <class 'abc.abstractproperty'> 2015-06-19 10:57:59: 12 <class 'PyQt5.QtChart.SeriesType'> 2015-06-19 10:57:59: 12 <class 'PyQt5.QtCore.ImageConversionFlag'> 2015-06-19 10:57:59: 12 <class 'PyQt5.QtCore.QSettings'> 2015-06-19 10:57:59: 12 <class 'PyQt5.QtCore.SortFlag'> 2015-06-19 10:57:59: 12 <class 'PyQt5.QtGui.StyleStrategy'> 2015-06-19 10:57:59: 12 <class 'ThinkboxUI.Controls.CollapsibleGroupBox.CollapsibleGroupBox'> 2015-06-19 10:57:59: 11 <class 'DeadlineMonitor.DisplayWidgets.FilterListWidget.FilterListWidget_META'> 2015-06-19 10:57:59: 11 <class 'PyQt5.QtCore.SelectionFlag'> 2015-06-19 10:57:59: 11 <class 'PyQt5.QtGui.PaintDeviceMetric'> 2015-06-19 10:57:59: 11 <class 'PyQt5.QtWidgets.Section'> 2015-06-19 10:57:59: 10 <class 'FranticX.Diagnostics.Profiling.ProfilingSection'> 2015-06-19 10:57:59: 10 <class 'PyQt5.QtCore.DateFormat'> 2015-06-19 10:57:59: 10 <class 'PyQt5.QtCore.MatchFlag'> 2015-06-19 10:57:59: 10 <class 'PyQt5.QtCore.TextFlag'> 2015-06-19 10:57:59: 10 <class 'PyQt5.QtCore.WindowFrameSection'> 2015-06-19 10:57:59: 10 <class 'PyQt5.QtWidgets.ComplexControl'> 2015-06-19 10:57:59: 10 <class 'PyQt5.QtWidgets.CursorAction'> 2015-06-19 10:57:59: 10 <class 'PyQt5.QtWidgets.QSplitter'> 2015-06-19 10:57:59: 10 <class 'ThinkboxUI.Controls.DownArrowToolButton.DownArrowToolButton'> 2015-06-19 10:57:59: 10 <class 'threading._RLock'> 2015-06-19 10:57:59: [Ommitted 376 other object types] 2015-06-19 10:57:59: Python Object Deltas (since last dump): 2015-06-19 10:57:59: +45840 <type 'dict'> 2015-06-19 10:57:59: +17974 <type 'list'> 2015-06-19 10:57:59: +8885 <class 'ThinkboxUI.Models.GroupByUpdateModel.UpdateTreeItem'> 2015-06-19 10:57:59: +7086 <class 'Deadline.Jobs.Job'> 2015-06-19 10:57:59: +3437 <class 'PyQt5.QtGui.QFont'> 2015-06-19 10:57:59: +1797 <type 'sip.methoddescriptor'> 2015-06-19 10:57:59: +1670 <class 'DeadlineUI.Models.JobListModel.JobBatchData'> 2015-06-19 10:57:59: +438 <class 'System.String[]'> 2015-06-19 10:57:59: +364 <class 'PyQt5.QtWidgets.QCheckBox'> 2015-06-19 10:57:59: +287 <type 'weakref'> 2015-06-19 10:57:59: +241 <class 'ThinkboxUI.Models.UpdateModel.UpdateItem'> 2015-06-19 10:57:59: +231 <class 'PyQt5.QtGui.QIcon'> 2015-06-19 10:57:59: +219 <class 'Deadline.Slaves.SlaveInfoSettings'> 2015-06-19 10:57:59: +214 <class 'PyQt5.QtGui.QPixmap'> 2015-06-19 10:57:59: +170 <type 'instancemethod'> 2015-06-19 10:57:59: +145 <type 'wrapper_descriptor'> 2015-06-19 10:57:59: +138 <class 'PyQt5.QtCore.Type'> 2015-06-19 10:57:59: +136 <class 'Deadline.Permissions.Feature'> 2015-06-19 10:57:59: +121 <type 'builtin_function_or_method'> 2015-06-19 10:57:59: +117 <class 'DeadlineMonitor.Forms.MonitorDockWidget.MonitorMorphWidget'> 2015-06-19 10:57:59: +104 <class 'PyQt5.QtChart.QPieSlice'> 2015-06-19 10:57:59: +75 <class 'PyQt5.QtWidgets.QAction'> 2015-06-19 10:57:59: +56 <type 'sip.variabledescriptor'> 2015-06-19 10:57:59: +55 <class 'PyQt5.QtWidgets.QVBoxLayout'> 2015-06-19 10:57:59: +38 <class 'PyQt5.QtGui.CompositionMode'> 2015-06-19 10:57:59: +34 <class 'PyQt5.QtWidgets.GraphicsItemChange'> 2015-06-19 10:57:59: +31 <class 'PyQt5.QtChart.QPieSeries'> 2015-06-19 10:57:59: +29 <class 'PyQt5.QtWidgets.QToolButton'> 2015-06-19 10:57:59: +26 <class 'PyQt5.QtWidgets.OptionType'> 2015-06-19 10:57:59: +26 <class 'PyQt5.QtWidgets.QWidget'> 2015-06-19 10:57:59: +25 <class 'PyQt5.QtGui.MoveOperation'> 2015-06-19 10:57:59: +23 <class 'PyQt5.QtWidgets.QLabel'> 2015-06-19 10:57:59: +20 <class 'PyQt5.QtCore.Filter'> 2015-06-19 10:57:59: +20 <class 'ThinkboxUI.Commands.QMenuPlusFiles.QMenuPlus'> 2015-06-19 10:57:59: +19 <class 'PyQt5.QtWidgets.QHBoxLayout'> 2015-06-19 10:57:59: +19 <class 'PyQt5.QtWidgets.StandardButton'> 2015-06-19 10:57:59: +16 <class 'CLR.CLR Metatype'> 2015-06-19 10:57:59: +16 <class 'PyQt5.QtWidgets.GraphicsItemFlag'> 2015-06-19 10:57:59: +13 <class 'Deadline.Importers.JobDetailsEntry'> 2015-06-19 10:57:59: +13 <class 'Deadline.LimitGroups.LimitGroup'> 2015-06-19 10:57:59: +13 <class 'PyQt5.QtCore.QByteArray'> 2015-06-19 10:57:59: +13 <type 'instance'> 2015-06-19 10:57:59: +12 <class 'PyQt5.QtChart.SeriesType'> 2015-06-19 10:57:59: +12 <class 'PyQt5.QtCore.SortFlag'> 2015-06-19 10:57:59: +12 <class 'ThinkboxUI.Controls.CollapsibleGroupBox.CollapsibleGroupBox'> 2015-06-19 10:57:59: +11 <class 'PyQt5.QtCore.SelectionFlag'> 2015-06-19 10:57:59: +11 <class 'PyQt5.QtWidgets.Section'> 2015-06-19 10:57:59: +10 <class 'PyQt5.QtCore.QSettings'> 2015-06-19 10:57:59: +10 <class 'PyQt5.QtWidgets.ButtonRole'> 2015-06-19 10:57:59: +10 <class 'PyQt5.QtWidgets.QSplitter'> 2015-06-19 10:57:59: +10 <class 'ThinkboxUI.Controls.DownArrowToolButton.DownArrowToolButton'> 2015-06-19 10:57:59: +9 <class 'DeadlineMonitor.Forms.MonitorDockWidget.MonitorDockWidget'> 2015-06-19 10:57:59: +9 <class 'PyQt5.QtCore.QTimer'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtChart.ChartTheme'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtChart.QChart'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtChart.QChartView'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtWidgets.MenuItemType'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtWidgets.QScrollArea'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtWidgets.QSpacerItem'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtWidgets.QStackedWidget'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtWidgets.QTreeView'> 2015-06-19 10:57:59: +8 <class 'PyQt5.QtWidgets.SliderAction'> 2015-06-19 10:57:59: +8 <class 'ThinkboxUI.Controls.ElidedLabel.ElidedLabel'> 2015-06-19 10:57:59: +8 <class 'ThinkboxUI.Controls.IconComboBox.IconComboBox'> 2015-06-19 10:57:59: +8 <class 'ThinkboxUI.Controls.IconLineEdit.IconButton'> 2015-06-19 10:57:59: +8 <class 'ThinkboxUI.Controls.IconLineEdit.IconLineEdit'> 2015-06-19 10:57:59: +8 <class 'ThinkboxUI.Controls.KeyPressTreeView.KeyPressTreeView'> 2015-06-19 10:57:59: +7 <class 'Deadline.Jobs.Task'> 2015-06-19 10:57:59: +7 <class 'PyQt5.QtWidgets.QFrame'> 2015-06-19 10:57:59: +7 <class 'PyQt5.QtWidgets.StyleOptionType'> 2015-06-19 10:57:59: +7 <class 'PyQt5.QtWidgets.StyleOptionVersion'> 2015-06-19 10:57:59: +6 <class 'PyQt5.QtCore.PatternSyntax'> 2015-06-19 10:57:59: +6 <class 'PyQt5.QtGui.RenderHint'> 2015-06-19 10:57:59: +6 <class 'PyQt5.QtWidgets.Direction'> 2015-06-19 10:57:59: +6 <class 'PyQt5.QtWidgets.SizeConstraint'> 2015-06-19 10:57:59: +6 <class 'PyQt5.QtWidgets.ToolButtonFeature'> 2015-06-19 10:57:59: +6 <class 'PyQt5.QtWidgets.ViewItemFeature'> 2015-06-19 10:57:59: +6 <type 'classobj'> 2015-06-19 10:57:59: +6 <type 'function'> 2015-06-19 10:57:59: +5 <class 'PyQt5.QtGui.Flag'> 2015-06-19 10:57:59: +5 <class 'PyQt5.QtGui.WrapMode'> 2015-06-19 10:57:59: +5 <class 'PyQt5.QtWidgets.ResizeMode'> 2015-06-19 10:57:59: +5 <class 'PyQt5.QtWidgets.ViewItemPosition'> 2015-06-19 10:57:59: +5 <type 'collections.deque'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtChart.AnimationOption'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtChart.LabelPosition'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtChart.RubberBand'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtCore.Base64Option'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtGui.ResourceType'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtGui.SelectionType'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtGui.TabType'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtWidgets.ButtonLayout'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtWidgets.Position'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtWidgets.SceneLayer'> 2015-06-19 10:57:59: +4 <class 'PyQt5.QtWidgets.SliderChange'> 2015-06-19 10:57:59: +4 <type 'type'> 2015-06-19 10:57:59: +3 <class 'DeadlineMonitor.Controls.JobListControl.JobListDelegate'> 2015-06-19 10:57:59: +3 <class 'DeadlineUI.Commands.JobCommands.JobModifyProperties'> 2015-06-19 10:57:59: +3 <class 'DeadlineUI.Commands.MonitorCommands.MonitorLoadPinnedLayout'> 2015-06-19 10:57:59: +3 <class 'DeadlineUI.Commands.MonitorCommands.MonitorManageLayouts'> 2015-06-19 10:57:59: +3 <class 'DeadlineUI.Commands.MonitorCommands.MonitorPinLayout'> 2015-06-19 10:57:59: +3 <class 'DeadlineUI.Models.JobListModel.JobListModel'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtChart.ChartType'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtCore.CaretMode'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtCore.ItemFlags'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtGui.FindFlag'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtGui.Stacks'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtWidgets.CacheMode'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtWidgets.CheckType'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtWidgets.FrameFeature'> 2015-06-19 10:57:59: +3 <class 'PyQt5.QtWidgets.PanelModality'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.GroupGraph'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.JobListControl'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.JobNameDelegate'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.JobProgressDelegate'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.PluginGraph'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.PoolGraph'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.SecondaryPoolGraph'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.StateGraph'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.Controls.JobListControl.UserGraph'> 2015-06-19 10:57:59: +2 <class 'DeadlineMonitor.DisplayWidgets.JobListWidget.JobListWidget'> 2015-06-19 10:57:59: +2 <class 'DeadlineUI.Models.JobListProxyModel.JobListProxyModel'> 2015-06-19 10:57:59: +2 <class 'PyQt5.QtCore.MonthNameType'> 2015-06-19 10:57:59: +2 <class 'PyQt5.QtGui.ItemType'> 2015-06-19 10:57:59: +2 <class 'PyQt5.QtGui.MetaInformation'> 2015-06-19 10:57:59: +2 <class 'PyQt5.QtGui.MoveMode'> 2015-06-19 10:57:59: +2 <class 'PyQt5.QtGui.RegionType'> 2015-06-19 10:57:59: +2 <class 'PyQt5.QtWidgets.ItemIndexMethod'> 2015-06-19 10:57:59: +2 <class 'PyQt5.QtWidgets.QWidgetAction'> 2015-06-19 10:57:59: +2 <class 'ThinkboxUI.Controls.LineNumberTextEdit.LineNumberTextEdit'> 2015-06-19 10:57:59: +2 <class 'ThinkboxUI.Controls.LineNumberTextEdit.NumberBar'> 2015-06-19 10:57:59: +2 <class 'ThinkboxUI.Controls.LineNumberTextEdit.PlainTextEdit'> 2015-06-19 10:57:59: +2 <class 'codecs.CodecInfo'> 2015-06-19 10:57:59: +2 <type 'module'> 2015-06-19 10:57:59: +1 <class 'Deadline.Balancer.BalancerInfoSettings'> 2015-06-19 10:57:59: +1 <class 'Deadline.Monitor.MonitorLayout'> 2015-06-19 10:57:59: +1 <class 'Deadline.Pulses.PulseInfoSettings'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.JobReportListControl.ErrorGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.JobReportListControl.JobReportListControl'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.JobReportListControl.StateGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.LimitListControl.LimitListControl'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.LimitListControl.StateGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.PulseListControl.PulseListControl'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.PulseListControl.StateGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.RemoteCommandStatusControl.RemoteCommandStatusControl'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.RemoteCommandStatusControl.StateGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.SlaveListControl.GroupGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.SlaveListControl.PoolGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.SlaveListControl.SlaveListControl'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.SlaveListControl.StateGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.TaskListControl.CPUGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.TaskListControl.ImageSizeGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.TaskListControl.RAMGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.TaskListControl.StateGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.TaskListControl.TaskDelegate'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.TaskListControl.TaskListControl'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Controls.TaskListControl.TaskTimeGraph'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.DisplayWidgets.ConsoleWidget.ConsoleWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.DisplayWidgets.JobReportListWidget.JobReportListWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.DisplayWidgets.LimitsListWidget.LimitsListWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.DisplayWidgets.PulseListWidget.PulseListWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.DisplayWidgets.RemoteCommandStatusWidget.RemoteCommandStatusWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.DisplayWidgets.SlaveListWidget.SlaveListWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.DisplayWidgets.TasksListWidget.TasksListWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineMonitor.Forms.StatusBarWidget.StatusBarWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.DeadlineCommands.DeadlineExploreLogFolder'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.DeadlineCommands.DeadlineManualLink'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.DeadlineCommands.DeadlineSupportLink'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.DeadlineCommands.MonitorViewDatabaseStats'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.EditMonitorOptions'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorChangeRepository'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorChangeUser'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorConfigLocalSlave'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorLockWidgets'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorRefresh'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewBalancerList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewCloudViewer'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewConsoleWidget'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewJobDependencyNodeViewer'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewJobInfo'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewJobList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewJobReportList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewLimitList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewPulseList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewRemoteCommandStatus'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewSlaveList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewSlaveReportList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Commands.MonitorCommands.MonitorSpawnNewTaskList'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Models.JobReportListProxyModel.JobReportListProxyModel'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Models.LimitListProxyModel.LimitListProxyModel'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Models.PulseListProxyModel.PulseListProxyModel'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Models.RemoteCommandStatusProxyModel.RemoteCommandStatusProxyModel'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Models.SlaveListProxyModel.SlaveListProxyModel'> 2015-06-19 10:57:59: +1 <class 'DeadlineUI.Models.TaskListProxyModel.TaskListProxyModel'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtCore.QDate'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtCore.QSortFilterProxyModel'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtGui.PixmapFragmentHint'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtGui.QColor'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtGui.QPalette'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtWidgets.QComboBox'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtWidgets.QListView'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtWidgets.QScrollBar'> 2015-06-19 10:57:59: +1 <class 'PyQt5.QtWidgets.QShortcut'> 2015-06-19 10:57:59: +1 <class 'ThinkboxUI.Commands.ApplicationCommands.ApplicationAbout'> 2015-06-19 10:57:59: +1 <class 'ThinkboxUI.Controls.FilterLogBoxView.FilterLogBoxModel'> 2015-06-19 10:57:59: +1 <class 'ThinkboxUI.Controls.FilterLogBoxView.FilterLogBoxView'> 2015-06-19 10:57:59: -1 <class 'Deadline.Monitor.MonitorManager'> 2015-06-19 10:57:59: -1 <type 'frame'> 2015-06-19 10:57:59: -143 <type 'tuple'>

Forced GC Output

2015-06-19 11:00:10: Before Collection: 2015-06-19 11:00:10: [memory usage] 2.477 GB / 0.000 Bytes 2015-06-19 11:00:10: After Collection: 2015-06-19 11:00:10: [memory usage] 1.819 GB / 0.000 Bytes

Post-GC object dump output (Real private memory usage: 2.99 GB)

2015-06-19 11:01:31: Python Object Counts [>= 10]: 2015-06-19 11:01:31: 50415 <type 'dict'> 2015-06-19 11:01:31: 19448 <type 'list'> 2015-06-19 11:01:31: 13002 <type 'wrapper_descriptor'> 2015-06-19 11:01:31: 8887 <class 'ThinkboxUI.Models.GroupByUpdateModel.UpdateTreeItem'> 2015-06-19 11:01:31: 7086 <class 'Deadline.Jobs.Job'> 2015-06-19 11:01:31: 6516 <type 'function'> 2015-06-19 11:01:31: 5357 <type 'tuple'> 2015-06-19 11:01:31: 4690 <type 'sip.methoddescriptor'> 2015-06-19 11:01:31: 3794 <type 'weakref'> 2015-06-19 11:01:31: 3441 <class 'PyQt5.QtGui.QFont'> 2015-06-19 11:01:31: 1871 <type 'builtin_function_or_method'> 2015-06-19 11:01:31: 1818 <type 'getset_descriptor'> 2015-06-19 11:01:31: 1670 <class 'DeadlineUI.Models.JobListModel.JobBatchData'> 2015-06-19 11:01:31: 1208 <type 'PyQt5.QtCore.pyqtWrapperType'> 2015-06-19 11:01:31: 1006 <type 'method_descriptor'> 2015-06-19 11:01:31: 991 <class 'CLR.CLR Metatype'> 2015-06-19 11:01:31: 552 <type 'module'> 2015-06-19 11:01:31: 536 <type 'sip.enumtype'> 2015-06-19 11:01:31: 438 <class 'System.String[]'> 2015-06-19 11:01:31: 432 <class 'PyQt5.QtCore.Key'> 2015-06-19 11:01:31: 408 <type 'set'> 2015-06-19 11:01:31: 364 <class 'PyQt5.QtWidgets.QCheckBox'> 2015-06-19 11:01:31: 306 <type 'property'> 2015-06-19 11:01:31: 245 <type 'type'> 2015-06-19 11:01:31: 241 <class 'ThinkboxUI.Models.UpdateModel.UpdateItem'> 2015-06-19 11:01:31: 234 <class 'PyQt5.QtGui.QIcon'> 2015-06-19 11:01:31: 226 <class 'PyQt5.QtGui.QPixmap'> 2015-06-19 11:01:31: 219 <class 'Deadline.Slaves.SlaveInfoSettings'> 2015-06-19 11:01:31: 211 <type 'instancemethod'> 2015-06-19 11:01:31: 204 <type 'member_descriptor'> 2015-06-19 11:01:31: 198 <class '_weakrefset.WeakSet'> 2015-06-19 11:01:31: 178 <class 'Deadline.Permissions.Feature'> 2015-06-19 11:01:31: 138 <class 'PyQt5.QtCore.Type'> 2015-06-19 11:01:31: 127 <class 'PyQt5.QtWidgets.QAction'> 2015-06-19 11:01:31: 117 <class 'DeadlineMonitor.Forms.MonitorDockWidget.MonitorMorphWidget'> 2015-06-19 11:01:31: 105 <class 'PyQt5.QtWidgets.StyleHint'> 2015-06-19 11:01:31: 104 <class 'PyQt5.QtChart.QPieSlice'> 2015-06-19 11:01:31: 103 <class 'Deadline.Importers.JobDetailsEntry'> 2015-06-19 11:01:31: 99 <class 'PyQt5.QtCore.WidgetAttribute'> 2015-06-19 11:01:31: 94 <class 'PyQt5.QtGui.QColor'> 2015-06-19 11:01:31: 94 <class 'PyQt5.QtWidgets.PixelMetric'> 2015-06-19 11:01:31: 72 <class 'PyQt5.QtWidgets.StandardPixmap'> 2015-06-19 11:01:31: 69 <class 'PyQt5.QtGui.StandardKey'> 2015-06-19 11:01:31: 61 <type 'sip.variabledescriptor'> 2015-06-19 11:01:31: 56 <class 'PyQt5.QtWidgets.SubElement'> 2015-06-19 11:01:31: 55 <class 'PyQt5.QtWidgets.QVBoxLayout'> 2015-06-19 11:01:31: 52 <class 'PyQt5.QtWidgets.PrimitiveElement'> 2015-06-19 11:01:31: 48 <class 'PyQt5.QtWidgets.ControlElement'> 2015-06-19 11:01:31: 47 <type 'classobj'> 2015-06-19 11:01:31: 46 <class 'PyQt5.QtWidgets.StandardButton'> 2015-06-19 11:01:31: 43 <class 'PyQt5.QtWidgets.SubControl'> 2015-06-19 11:01:31: 39 <class 'PyQt5.QtCore.WindowType'> 2015-06-19 11:01:31: 38 <class 'PyQt5.QtGui.CompositionMode'> 2015-06-19 11:01:31: 35 <class 'PyQt5.QtCore.MouseButton'> 2015-06-19 11:01:31: 34 <class 'PyQt5.QtWidgets.GraphicsItemChange'> 2015-06-19 11:01:31: 31 <class 'PyQt5.QtChart.QPieSeries'> 2015-06-19 11:01:31: 30 <class 'ThinkboxUI.Commands.QMenuPlusFiles.QMenuPlus'> 2015-06-19 11:01:31: 29 <class 'PyQt5.QtWidgets.QToolButton'> 2015-06-19 11:01:31: 28 <class 'PyQt5.QtWidgets.StateFlag'> 2015-06-19 11:01:31: 27 <class 'PyQt5.QtCore.QModelIndex'> 2015-06-19 11:01:31: 27 <class 'PyQt5.QtWidgets.QWidget'> 2015-06-19 11:01:31: 26 <class 'PyQt5.QtWidgets.OptionType'> 2015-06-19 11:01:31: 25 <class 'PyQt5.QtCore.CursorShape'> 2015-06-19 11:01:31: 25 <class 'PyQt5.QtGui.MoveOperation'> 2015-06-19 11:01:31: 24 <class 'PyQt5.QtWidgets.ContentsType'> 2015-06-19 11:01:31: 24 <type 'operator.itemgetter'> 2015-06-19 11:01:31: 23 <class 'PyQt5.QtGui.ColorRole'> 2015-06-19 11:01:31: 23 <class 'PyQt5.QtWidgets.QLabel'> 2015-06-19 11:01:31: 22 <type 'classmethod'> 2015-06-19 11:01:31: 21 <class 'PyQt5.QtCore.InputMethodHint'> 2015-06-19 11:01:31: 20 <class 'PyQt5.QtCore.Filter'> 2015-06-19 11:01:31: 20 <class 'PyQt5.QtCore.GlobalColor'> 2015-06-19 11:01:31: 20 <class 'PyQt5.QtCore.QTimer'> 2015-06-19 11:01:31: 20 <class 'PyQt5.QtGui.QKeySequence'> 2015-06-19 11:01:31: 20 <class 'PyQt5.QtWidgets.ButtonRole'> 2015-06-19 11:01:31: 19 <class 'PyQt5.QtCore.BrushStyle'> 2015-06-19 11:01:31: 19 <class 'PyQt5.QtWidgets.QHBoxLayout'> 2015-06-19 11:01:31: 18 <class 'PyQt5.QtCore.ItemDataRole'> 2015-06-19 11:01:31: 18 <class 'abc.ABCMeta'> 2015-06-19 11:01:31: 18 <type 'staticmethod'> 2015-06-19 11:01:31: 17 <class 'PyQt5.QtCore.InputMethodQuery'> 2015-06-19 11:01:31: 17 <type 'classmethod_descriptor'> 2015-06-19 11:01:31: 17 <type 'frozenset'> 2015-06-19 11:01:31: 16 <class 'PyQt5.QtCore.ApplicationAttribute'> 2015-06-19 11:01:31: 16 <class 'PyQt5.QtWidgets.GraphicsItemFlag'> 2015-06-19 11:01:31: 16 <type 'collections.deque'> 2015-06-19 11:01:31: 15 <class 'PyQt5.QtWidgets.ControlType'> 2015-06-19 11:01:31: 15 <type 'instance'> 2015-06-19 11:01:31: 14 <class 'PyQt5.QtCore.AlignmentFlag'> 2015-06-19 11:01:31: 13 <class 'Deadline.LimitGroups.LimitGroup'> 2015-06-19 11:01:31: 13 <class 'PyQt5.QtCore.ItemFlags'> 2015-06-19 11:01:31: 13 <class 'PyQt5.QtCore.QByteArray'> 2015-06-19 11:01:31: 13 <class 'PyQt5.QtCore.QSettings'> 2015-06-19 11:01:31: 13 <class 'PyQt5.QtGui.StyleHint'> 2015-06-19 11:01:31: 13 <class 'abc.abstractproperty'> 2015-06-19 11:01:31: 12 <class 'PyQt5.QtChart.SeriesType'> 2015-06-19 11:01:31: 12 <class 'PyQt5.QtCore.ImageConversionFlag'> 2015-06-19 11:01:31: 12 <class 'PyQt5.QtCore.SortFlag'> 2015-06-19 11:01:31: 12 <class 'PyQt5.QtGui.StyleStrategy'> 2015-06-19 11:01:31: 12 <class 'ThinkboxUI.Controls.CollapsibleGroupBox.CollapsibleGroupBox'> 2015-06-19 11:01:31: 11 <class 'DeadlineMonitor.DisplayWidgets.FilterListWidget.FilterListWidget_META'> 2015-06-19 11:01:31: 11 <class 'PyQt5.QtCore.SelectionFlag'> 2015-06-19 11:01:31: 11 <class 'PyQt5.QtGui.PaintDeviceMetric'> 2015-06-19 11:01:31: 11 <class 'PyQt5.QtWidgets.Section'> 2015-06-19 11:01:31: 10 <class 'FranticX.Diagnostics.Profiling.ProfilingSection'> 2015-06-19 11:01:31: 10 <class 'PyQt5.QtCore.DateFormat'> 2015-06-19 11:01:31: 10 <class 'PyQt5.QtCore.MatchFlag'> 2015-06-19 11:01:31: 10 <class 'PyQt5.QtCore.TextFlag'> 2015-06-19 11:01:31: 10 <class 'PyQt5.QtCore.WindowFrameSection'> 2015-06-19 11:01:31: 10 <class 'PyQt5.QtWidgets.ComplexControl'> 2015-06-19 11:01:31: 10 <class 'PyQt5.QtWidgets.CursorAction'> 2015-06-19 11:01:31: 10 <class 'PyQt5.QtWidgets.QSplitter'> 2015-06-19 11:01:31: 10 <class 'ThinkboxUI.Controls.DownArrowToolButton.DownArrowToolButton'> 2015-06-19 11:01:31: 10 <class 'threading._RLock'> 2015-06-19 11:01:31: [Ommitted 381 other object types] 2015-06-19 11:01:31: Python Object Deltas (since last dump): 2015-06-19 11:01:31: +170 <type 'sip.methoddescriptor'> 2015-06-19 11:01:31: +77 <type 'wrapper_descriptor'> 2015-06-19 11:01:31: +19 <type 'dict'> 2015-06-19 11:01:31: +16 <type 'tuple'> 2015-06-19 11:01:31: +15 <type 'weakref'> 2015-06-19 11:01:31: +8 <class 'CLR.CLR Metatype'> 2015-06-19 11:01:31: +8 <class 'PyQt5.QtCore.OpenModeFlag'> 2015-06-19 11:01:31: +8 <type 'builtin_function_or_method'> 2015-06-19 11:01:31: +6 <class 'PyQt5.QtWidgets.ButtonFeature'> 2015-06-19 11:01:31: +4 <type 'sip.variabledescriptor'> 2015-06-19 11:01:31: +3 <class 'DeadlineUI.Commands.MonitorCommands.MonitorLoadPinnedLayout'> 2015-06-19 11:01:31: +3 <class 'PyQt5.QtGui.Mode'> 2015-06-19 11:01:31: +3 <class 'PyQt5.QtWidgets.FieldGrowthPolicy'> 2015-06-19 11:01:31: +3 <class 'PyQt5.QtWidgets.ItemRole'> 2015-06-19 11:01:31: +3 <class 'PyQt5.QtWidgets.RowWrapPolicy'> 2015-06-19 11:01:31: +2 <class 'DeadlineUI.Commands.MonitorCommands.MonitorManageLayouts'> 2015-06-19 11:01:31: +2 <class 'DeadlineUI.Commands.MonitorCommands.MonitorPinLayout'> 2015-06-19 11:01:31: +2 <type 'list'> 2015-06-19 11:01:31: +1 <class 'PyQt5.QtCore.QSettings'> 2015-06-19 11:01:31: +1 <class 'PyQt5.QtWidgets.QShortcut'> 2015-06-19 11:01:31: +1 <class 'PyQt5.QtWidgets.StyleOptionType'> 2015-06-19 11:01:31: +1 <class 'PyQt5.QtWidgets.StyleOptionVersion'>

jgaudet · June 19, 2015, 8:37pm

Hey Nathan,

Weird, all those object counts seem more or less in line with what I’d expect (based on the number of Jobs/Slaves/etc you mentioned), at least from a List View / Data Model perspective.

Do you switch between Layouts a lot? I noticed the deltas in the second dump mention a bunch of Layout switch-related stuff, though this is obviously a bit of a shot in the dark. I’m sure the objects in that delta wouldn’t be responsible for this growth on their own, but I wonder if there’s a bunch of the underlying C++ Qt controls not getting cleaned up during switches, due to a broken parent chain or something.

What’s the memory usage like on a freshly opened monitor for you with a similar job count? Have you noticed if it grows faster over the day when things are active or at night when things are maybe more quiet?

Are you the only one experiencing this leak, or is this a more widespread problem?

Also, and I should probably know this by now, but which version are you guys on exactly again?

nrusch · June 19, 2015, 10:17pm

I never switch layouts.

The queue in question is on a really high-latency connection for us (because it’s in Australia), so it takes a reeeaally long time to load the jobs, and I usually get a lot of connection errors that freeze the whole thing up and stop it from loading further jobs for a while. I’ll restart it and see if I can get it to load the whole queue eventually.

I haven’t really paid attention to this, but I’ll see if I can spot any patterns.

Haven’t checked, but I would bet money it affects everyone who leaves their monitor open for extended periods of time.

Currently 7.0.2.3, but planning to update to 7.1 as soon as our current major show wraps.

nrusch · June 19, 2015, 10:49pm

OK, it eventually loaded just about all of the existing jobs (now 7,101), and the private memory usage is at 2.49 GB.

nrusch · June 20, 2015, 1:36am

OK, in the last 2.5 to 3 hours, the job count has been reduced to 6,179, but the memory usage of the monitor has gone up to 3.14 GB.

nrusch · June 20, 2015, 2:25am

And now it has dropped back to 2.6 GB, so I guess some kind of cleanup actually happened. We’ll see what it does over the weekend.

dwallbridge · June 22, 2015, 9:11pm

Hello Nathan,

I’ve asked Jon to look over your responses, but he is out today. I’ll have him reply asap.

jgaudet · June 23, 2015, 11:02pm

Hey Nathan,

How did the Monitor fare over the weekend? Either way, even if it didn’t climb much, 2.7GB is rather high for a Monitor with ~7k jobs to begin with. Since the Python stuff all looked in line (based on those object counts), I wonder if it’s happening in the managed C# code…

Can you do a run through of the Monitor using Mono Profiler (Edwin has a good post about it here)? That should give us some more insight into what’s causing this.

Cheers,
Jon

nrusch · June 24, 2015, 12:06am

Memory usage climbed to around 3.4 GB over the weekend, even as the job count dropped some more. However, it then seemed to stabilize to ~3.1 GB. I don’t remember what the final job count was, but I’ll try a profiled run and post the results.

nrusch · June 30, 2015, 4:40pm

So I finally got around to doing a profiled run of the Monitor, since I realized the build of Mono that ships with Deadline already has the debugging tools bundled. I started it at about 6 PM last night pointed at the high-latency remote repository, which had about 6,800 jobs at the time, but it crashed at 11:40 PM. Thus, I don’t know how helpful this log will be, but I’m attaching it anyway, because it does appear there may be some object counts that are growing constantly over time. I’m going to do some more of these runs and see if I can get a good long-term report.
monitor-mprof-report-1.txt (93.4 KB)

eamsler · July 1, 2015, 8:28pm

It looks like the garbage collector is a bit lazy with these strings and buffers.

	2239604616     718550     3116 System.String (bytes: +158733664, count: +109005)
	 232278488        556   417767 System.Char[] (bytes: +149026080, count: +229)
	 133216624       3693    36072 System.Byte[] (bytes: +74513920, count: +1198)

What I find interesting is this guy:

	  10472496      31450      332 System.Collections.Generic.Link[] (bytes: +347840, count: +1284)

That’s not part of the Microsoft documentation. I wonder if it’s part of Mono’s Linked List implementation…

It’s going to be hard to track these down. I need to learn more about the profiler toolset to see if we can figure out who created these, and if the counts for the Link class are normal (we make a whole bunch of List instances in the codebase).

jgaudet · July 6, 2015, 5:48pm

There’s definitely something out of whack with those strings. 2GB of string objects is insane, especially at an instance count of only ~700-900k.

Here’s what I’m seeing on windows, for 4.5k jobs:

445166 (instance count)     24103034 (bytes)   System.String

So about 24MB for 445k instances, for an average of 54 bytes per string (~27 chars), which seems pretty normal to me. Obviously this will go up the more data you have in jobs, and mine are probably on the low side, but it shouldn’t be several orders of magnitudes higher.

From what I’m seeing in your data, strings are averaging 3KB (~1500 characters). I know you guys were talking about storing extra data in Jobs, but are you storing THAT much data in the job/slaves? I’m guessing the answer is ‘no’. You mentioned connecting to a remote and getting lots of errors – I wonder if those are somehow not getting cleaned up by Mono, and all the stack traces and stuff are inflating these numbers…

What kind of errors are you getting? DB Timeouts and stuff I presume? Or is there more to them?

nrusch · July 7, 2015, 7:07pm

The answer is actually “yes”. We need to include a fairly large amount of data with the jobs, including some large-ish binary objects. Because there is no way to store said data as anything other than flat strings (i.e. no binary types, document storage, etc.), we end up having to serialize and compress them, and then convert them to ASCII.

Yeah, connection timeouts.

nrusch · July 8, 2015, 7:28pm

Why would the monitor need to load custom job data for normal operation?

jgaudet · July 8, 2015, 8:27pm

For the Monitor we load the entire Job object for use by our data model, and typically remains in our cache layer. If you’re using the ExtraInfoKVPs for your custom binary data, that will be part of it.

It’s normally not a huge deal, because the large majority of the data in the Job object needs to be displayed in the Monitor anyways, but I suppose your case is a bit special. Since the objects you guys are storing are rather large, it might be worth exploring alternatives to storing them serialized in the Job KVPs.

As per our previous discussion, it should be fairly safe to add a new JSON property to Jobs instead, and store it in there. The Monitor would then throw that away when de-serializing the Job, so it wouldn’t be kept in memory, but you’d run the risk of clobbering your data when we do a full Job save (only Undelete and (Re)submission do this, afaik). You’d also still be sending all that data over the wire from the DB to the Monitor machines, so you’d still be using a lot of bandwidth, though… I suppose you also then wouldn’t really be able to access it from scripts and stuff either, so I’m not sure that would have any benefit over storing it entirely separate, aside from the lifetime of the data being directly tied to the Job.

Would it be feasible to store your custom info (at least the bigger binary data) in a completely separate collection, and tie it to the Job ID?

nrusch · July 17, 2015, 9:06pm

We aren’t using the ExtraInfo. We use the pluginInfo exclusively for our render pipeline data.

I think we’re going to try going this route in the near future. Losing the data encapsulation within the job document is unfortunate, since it makes things like post-deletion data cleanup and undeleting more involved, but on the plus side, we’ll be able to store actual binary blobs, so the data size should be reduced significantly.

However, I still think it would be good for the job loading to be done more intelligently by the client applications, particularly with regards to the pluginInfo data.

jgaudet · July 20, 2015, 5:14pm

Gotcha, they’re both handled as opaque dictionaries that are part of the Job though, so the same thing applies.

Normalizing the Job data in the DB is probably a good idea; we’ll have to revisit how we have it structured at some point. Even without custom data in there, the Plugin Info can get rather large – 3dsMax jobs being the biggest offenders.