Retrying Ruby Geocoder Requests after OVER_QUERY_LIMIT errors
QuotaGuard is designed to give you access to the Google Maps Geocoding API on cloud platforms like Heroku or CloudControl. Our proxy network spreads out your API requests across a number of machines, adding more machines as necessary to handle everybody’s requests. This service means you should not see any OVER_QUERY_LIMIT errors until you reach your daily limit (2500 as of November 2013).
Unfortunately Google also limits requests on a per second basis so if you submit more than 4 requests a second you will temporarily receive an OVER_QUERY_LIMIT error. We mitigate this to a large extent by having a buffer of spare capacity but if you attempt to make hundreds of calls a second you may still see this error.
So what is the best way to handle this temporary error?
Solution
Try, try and try again! Unless you are a website with high loads doing geocoding based on incoming web requests you will most commonly hit this issue when doing some bulk geocoding. In Ruby this is commonly done in a rake task. This gist shows you how to use the Ruby Geocoder gem to bulk geocode a collection of objects, retrying if you temporarily hit the OVER_QUERY_LIMIT error.
namespace :geocoder do desc "Geocode all objects without coordinates." task :all_with_retry => :environment do class_name = ENV['CLASS'] || ENV['class'] sleep_timer = ENV['SLEEP'] || ENV['sleep'] raise "Please specify a CLASS (model)" unless class_name klass = class_from_string(class_name) klass.not_geocoded.each do |obj| geocode_with_retry obj sleep(sleep_timer.to_f) unless sleep_timer.nil? end end end def geocode_with_retry(obj) tries ||= 3 puts "Trying again, #{tries} tries left..." if tries < 3 puts Time.now.strftime("%Y%m%d%H%M%S")+": Geocoding " + obj.to_s obj.geocode if obj.geocoded? obj.save else sleep(3.0/tries) #Back off in 1s, 2s, 3s intervals raise Geocoder::OverQueryLimitError end rescue Geocoder::OverQueryLimitError => e retry unless (tries -= 1).zero? end ## # Get a class object from the string given in the shell environment. # Similar to ActiveSupport's +constantize+ method. # def class_from_string(class_name) parts = class_name.split("::") constant = Object parts.each do |part| constant = constant.const_get(part) end constant end
Caveats
- The Ruby Geocoder doesn’t raise exceptions when calls to geocode fail so to determine if a retry if necessary you have to check whether it has been geocoded. This could result in false retries where you are trying to geocode invalid data.