Crawler in Groovy (JSoup VS Crawler4j) -



Crawler in Groovy (JSoup VS Crawler4j) -

i wish develop web crawler in groovy(using grails framework , mongodb database) has ability crawl website, creating list of site urls , resource types, content, response times , number of redirects involved.

i debating on jsoup vs crawler4j. have read cannot understand difference between two. can suggest improve 1 above functionality? or totally wrong compare two?

thanks.

crawler4j crawler, jsoup parser. could/should utilize both. crawler4j easy-multithreaded interface urls , pages(content) of site want. after can utilize jsoup in order parse data, amazing (jquery-like) css selectors , it. of course of study have consider dynamic (javascript generated) content. if want content too, have utilize else includes javascript engine (headless browser + parser) htmlunit or webdriver (selenium), execute javascript before parsing content.

jsoup web-crawler crawler4j

Comments

Post a Comment

Popular posts from this blog

model view controller - MVC Rails Planning -

model view controller - MVC Rails Planning - i'm starting rails , i'm building app stores films info, user can watch film , give grade on web application. i have coded cinema model, controller , views. now, have 2 questions: should need create model , controller grading? in case, should create or should wait , create first users model , controller? yes, on model , controller grades. that'd feasible way maintain track of who's graded what. it's create first, i'd start users, since grades depend on both films , users. ruby-on-rails model-view-controller
Read more

ruby on rails - Devise Logout Error in RoR -

ruby on rails - Devise Logout Error in RoR - while using devise in ror, encountering unusual error when trying log out url different root url. displays me : actioncontroller::invalidauthenticitytoken in devise::sessionscontroller#destroy" for illustration if rooting path after log in '/structure', , seek log out '/general' error, while logging out '/structure' fine. any help highly appreciated. ruby-on-rails ruby devise
Read more

html - Submenu setup with jquery and effect 'fold' -

html - Submenu setup with jquery and effect 'fold' - i want open submenu jquery effect fold, problem if user "hover effect" fast menu remain open, how can avoid this, jquery code is: $('ul.mainmenu li').hover( function() { $(this).children('ul').show('fold', 570); }, function() { $(this).children('ul').hide('fold', 500); } ); my jsfiddle link is: http://jsfiddle.net/9wkbf/ updated fiddle the reason behind problem is, first event $(this).children('ul').show('fold', 570); queued , until completes, sec animation not start. the next snippet can workaround $('ul.mainmenu > li').hover( function() { $(this).children('ul').show('fold', 570); }, function() { $('ul:not(.mainmenu)').hide('fold', 500); } ); *important note : work current scenario. jquery html css
Read more