Jan Kubr

Posts Tagged ‘ruby’

Memory leaks in Ruby 1.8

In Uncategorized on January 4, 2008 at 15:10

I’ve found two Ruby methods that seem to leak memory. At least according to dike.

[Update: Might be I just can’t use dike or there’s some problem with it. Make sure you read the comments for more information.]

[Update2:

On Jan 9, 2008 4:12 AM, ara howard <ara.t.howard..> wrote:

so i do think that dike is causing the issue, but i do not think it’s a bug ;-) 

More info coming, too early now.. ]

First example:

require 'rubygems'
require 'dike'
Dike.logfactory './log/'

class Leak

 def http_call
	puts 'making http call'
	url = URI.parse('http://www.google.com')
	Net::HTTP.start(url.host) do |http|
		puts http.get('/').code
	end
	''
 end
end

5.times {
	leak = Leak.new
	leak.http_call
	GC.start
	Dike.finger
}

The get method leaves the Net::HTTPFound (HTTPResponse) objects in memory. Replace it with request_get and the leak is gone.

Second example:

require 'rubygems'
require 'dike'
Dike.logfactory './log/'

class Leak

  def http_call
	url = URI.parse('http://www.google.com')
	puts 'making the call'
	Net::HTTP.start(url.host) {|http|
		http.request_get('/') do |response|
			response.read_body do |segment|
			end
		end
	}
	''
  end
end

5.times {
	leak = Leak.new
	leak.http_call
	GC.start
	Dike.finger
}

The read_body methods leaves Net::ReadAdapter objects in memory and because of them other objects remain in memory, too. This happens only when you pass a block to read_body, without it the leak does not happen. This is not that easy to fix because without the block the whole body of the response is read into memory and if are downloading a large file, that is not what you want to do. My solution was write my own read_body method. I looked at the net/http.rb file for inspiration:

def read_chunked(dest)
	len = nil
	total = 0
	while true
		line = @socket.readline
		hexlen = line.slice(/[0-9a-fA-F]+/) or
			raise HTTPBadResponse, "wrong chunk size line: #{line}"
		len = hexlen.hex
		break if len == 0
		@socket.read len, dest; total += len
		@socket.read 2   # \r\n
	end
	until @socket.readline.empty?
		# none
	end
end

Hmm I must say I didn’t really get the code much, so I just wrote this (which might not work in all the cases, I warned you!):

def read_body(response)
	socket = response.instance_variable_get(:@socket)
	bytes_left = response.send(:content_length)
	read_size = calculate_read_size(bytes_left)
	while bytes_left > 0
		line = ''
		socket.send(:read, read_size, line)
		yield line
		bytes_left -= read_size
		read_size = calculate_read_size(bytes_left)
	end
end

def calculate_read_size(bytes_left)
	bytes_left > BUFFER_SIZE ? BUFFER_SIZE : bytes_left
end

To try the examples, paste the code in a file (named leak.rb in my case) and run it with a following command:

ruby leak.rb && dike log

I don’t see anything wrong neither with the examples nor in the code of the library, if anyone does, please let me know.

I tried to run the examples with Ruby 1.9 and there are “only” some String objects leaking (I can show you what changes I had to do in dike to make it work). So there might be only something minor with Strings in the new (development) version of Ruby. I’ll need to look at this soon again. Oh and by the way, your tests are leaking memory, too.

Advertisements