ruby - Efficient way to render ton of JSON on Heroku -

ruby - Efficient way to render ton of JSON on Heroku -

i built simple api 1 endpoint. scrapes files , has around 30,000 records. ideally able fetch records in json 1 http call.

here sinatra view code:

require 'sinatra' require 'json' require 'mongoid' mongoid.identity_map_enabled = false '/' content_type :json book.all end

i've tried following: using multi_json

require './require.rb' require 'sinatra' require 'multi_json' multijson.engine = :yajl mongoid.identity_map_enabled = false '/' content_type :json multijson.encode(book.all) end

the problem approach error r14 (memory quota exceeded). same error when seek utilize 'oj' gem.

i concatinate 1 long redis string, heroku's redis service $30 per month instance size need (> 10mb).

my current solution utilize background task creates objects , stuffs them total of jsonified objects @ near mongoid object size limit (16mb). problems approach: still takes 30 seconds render, , have run post-processing on receiving app extract json objects.

does have improve thought how can render json 30k records in 1 phone call without switching away heroku?

sounds want stream json straight client instead of building in memory. it's best way cutting downwards memory usage. illustration utilize yajl encode json straight stream.

edit: rewrote entire code yajl, because api much more compelling , allows much cleaner code. included illustration reading response in chunks. here's streamed json array helper wrote:

require 'yajl' module jsonarray class streamwriter def initialize(out) super() @out = out @encoder = yajl::encoder.new @first = true end def <<(object) @out << ',' unless @first @out << @encoder.encode(object) @out << "\n" @first = false end end def self.write_stream(app, &block) app.stream |out| out << '[' block.call streamwriter.new(out) out << ']' end end end

usage:

require 'sinatra' require 'mongoid' mongoid.identity_map_enabled = false # utilize server supports streaming set :server, :thin '/' content_type :json jsonarray.write_stream(self) |json| book.all.each |book| json << book.attributes end end end

to decode on client side can read , parse response in chunks, illustration em-http. note solution requires clients memory big plenty store entire objects array. here's corresponding streamed parser helper:

require 'yajl' module jsonarray class streamparser def initialize(&callback) @parser = yajl::parser.new @parser.on_parse_complete = callback end def <<(str) @parser << str end end def self.parse_stream(&callback) streamparser.new(&callback) end end

usage:

require 'em-http' parser = jsonarray.parse_stream |object| # block called when done parsing # entire array; can handle info p object end eventmachine.run http = eventmachine::httprequest.new('http://localhost:4567').get http.stream |chunk| parser << chunk end http.callback eventmachine.stop end end

alternative solution

you simplify whole thing lot when give need generating "proper" json array. above solution generates json in form:

[{ ... book_1 ... } ,{ ... book_2 ... } ,{ ... book_3 ... } ... ,{ ... book_n ... } ]

we stream each book separate json , cut down format following:

{ ... book_1 ... } { ... book_2 ... } { ... book_3 ... } ... { ... book_n ... }

the code on server much simpler:

require 'sinatra' require 'mongoid' require 'yajl' mongoid.identity_map_enabled = false set :server, :thin '/' content_type :json encoder = yajl::encoder.new stream |out| book.all.each |book| out << encoder.encode(book.attributes) << "\n" end end end

as client:

require 'em-http' require 'yajl' parser = yajl::parser.new parser.on_parse_complete = proc.new |book| # called separately every book p book end eventmachine.run http = eventmachine::httprequest.new('http://localhost:4567').get http.stream |chunk| parser << chunk end http.callback eventmachine.stop end end

the great thing client not have wait entire response, instead parses every book separately. however, not work if 1 of clients expects 1 single big json array.

ruby json heroku sinatra mongoid