Issue

• The Iguana service is leaking memory overtime

• This can be seen in Windows task manager - Iguana memory increasing

• This leads to a crash, the crashdumps often have 0 bytes, crash-failure.txt, or failed to create (ran out of memory while creating the file), nothing meaningful in the ServiceErrorLog usually

• In Windows Event Viewer, an error such as “low virtual memory condition”

  

Cause

• Most often, there is a memory leak in a channel

• There could be another process leaking memory and making it look like Iguana is leaking memory

• RAM too small

Possible Solutions

• https://help.interfaceware.com/v6/how-to-troubleshoot-iguana-memory-leak

• Find the channel leaking memory then do a code review

• You can use the checkMemory.lua script developed by Paul Le to find a channel leaking memory over time.

  • The client would need to add this module in the shared folder, add a require statement in their main.lua, and call checkMemory() in the main function.

  • All of their channels need this script and the client needs to run the channels as normal to simulate the traffic when the memory leak occurred

  • An error should show in the Iguana logs for the leaking channel

  • If no leak is found, try decreasing the variable memoryBuffer (This is the case if the leak is small)

    -- Keep track of last memory usage and how many times memory has increased
    memoryUsage = {
       ['lastMemoryUsage'] = 0,
       ['increasedMemoryCount'] = 0
    }
    
    isTriggered = false
    
    local memoryLeakThreshold = 20 -- How many times memory increases to trigger an alert
    local memoryBuffer        = 10 -- Buffer for what is inconsidered a memory increase
    
    -- Useful information on how Lua handles memory: 
    --   https://www.lua.org/pil/17.html
    
    function checkMemory()
    
       -- Trigger garbage collection to clear unreferenced memory
       collectgarbage('collect')
    
       -- Get referenced memory usage
       local memoryUsageCount = collectgarbage('count')
       iguana.logDebug('Total memory in use by Lua (in Kbytes): ' .. memoryUsageCount)
    
       -- Check if the memory is increasing even after garbage is collected
       if memoryUsageCount - memoryBuffer > memoryUsage.lastMemoryUsage then
          iguana.logDebug('Memory in use by Lua increased (in Kbytes) by: ' .. memoryUsageCount - memoryUsage.lastMemoryUsage)
          iguana.logDebug('Total memory in use by Lua (in Kbytes): ' .. memoryUsageCount)
          -- Increase count for increasing memory usage
          memoryUsage.increasedMemoryCount = memoryUsage.increasedMemoryCount + 1
          -- Set last memory usage to the current memory usage count
          memoryUsage.lastMemoryUsage = memoryUsageCount
       elseif  memoryUsageCount < memoryUsage.lastMemoryUsage then
          -- Reset memory increase count if memory decreases
          memoryUsage.increasedMemoryCount = 0
       end
    
       -- If memory is increasing consistently then flag potential memory leak
       if memoryUsage.increasedMemoryCount > memoryLeakThreshold and not isTriggered then 
          iguana.logError('There may be a memory leak!')
          iguana.logError('Total memory in use by Lua (in Kbytes): ' .. memoryUsageCount)
          isTriggered = true
       end
    
    end

Case Studies

• Sometimes, external programs will make it look like Iguana is leaking memory.

  • 35814 - They had a monitoring program outside of Iguana that uses the Log API. It queried all the logs on an Iguana server, which cause memory to not get released properly. They now only query the last 12 hours.

  • 32422