python - Scrape page with generator -



python - Scrape page with generator -

i scraping site beautiful soup. problem have parts of site paginated js, unknown (varying) number of pages scrape. i'm trying around generator, it's first time writing 1 , i'm having hard time wrapping head around , figuring out if i'm doing makes sense.

code:

from bs4 import beautifulsoup import urllib import urllib2 import jabba_webkit jw import csv import string import re import time tlds = csv.reader(open("top_level_domains.csv", 'r'), delimiter=';') sites = csv.writer(open("websites_to_scrape.csv", "w"), delimiter=',') tld = "uz" has_next = true page = 0 def create_link(tld, page): if page == 0: link = "https://domaintyper.com/top-websites/most-popular-websites-with-" + tld + "-domain" else: link = "https://domaintyper.com/top-websites/most-popular-websites-with-" + tld + "-domain/page/" + repr(page) homecoming link def check_for_next(soup): disabled_nav = soup.find(class_="pagingdivdisabled") if disabled_nav: if "next" in disabled_nav: homecoming false else: homecoming true else: homecoming true def make_soup(link): html = jw.get_page(link) soup = beautifulsoup(html, "lxml") homecoming soup def all_the_pages(counter): while true: link = create_link(tld, counter) soup = make_soup(link) if check_for_next(soup) == true: yield counter else: break counter += 1 def scrape_page(soup): table = soup.find('table', {'class': 'ranktable'}) th = table.find('tbody') test = th.find_all("td") correct_cells = range(1,len(test),3) cell in correct_cells: #print test[cell] url = repr(test[cell]) content = re.sub("<[^>]*>", "", url) sites.writerow([tld]+[content]) def main(): page in all_the_pages(0): print page link = create_link(tld, page) print link soup = make_soup(link) scrape_page(soup) main()

my thinking behind code: scraper should page, determine if there page follows, scrape current page , move next one, repreating process. if there no next page, should stop. create sense how i'm going here?

as told you, utilize selenium programmatically clicking on next button, since not alternative you, can think of next method number of pages using pure bs4:

import requests bs4 import beautifulsoup def page_count(): pages = 1 url = "https://domaintyper.com/top-websites/most-popular-websites-with-uz-domain/page/{}" while true: html = requests.get(url.format(pages)).content soup = beautifulsoup(html) table = soup.find('table', {'class': 'ranktable'}) if len(table.find_all('tr')) <= 1: homecoming pages pages += 1

python web-scraping beautifulsoup generator

Comments

Post a Comment

Popular posts from this blog

model view controller - MVC Rails Planning -

model view controller - MVC Rails Planning - i'm starting rails , i'm building app stores films info, user can watch film , give grade on web application. i have coded cinema model, controller , views. now, have 2 questions: should need create model , controller grading? in case, should create or should wait , create first users model , controller? yes, on model , controller grades. that'd feasible way maintain track of who's graded what. it's create first, i'd start users, since grades depend on both films , users. ruby-on-rails model-view-controller
Read more

ruby on rails - Devise Logout Error in RoR -

ruby on rails - Devise Logout Error in RoR - while using devise in ror, encountering unusual error when trying log out url different root url. displays me : actioncontroller::invalidauthenticitytoken in devise::sessionscontroller#destroy" for illustration if rooting path after log in '/structure', , seek log out '/general' error, while logging out '/structure' fine. any help highly appreciated. ruby-on-rails ruby devise
Read more

html - Submenu setup with jquery and effect 'fold' -

html - Submenu setup with jquery and effect 'fold' - i want open submenu jquery effect fold, problem if user "hover effect" fast menu remain open, how can avoid this, jquery code is: $('ul.mainmenu li').hover( function() { $(this).children('ul').show('fold', 570); }, function() { $(this).children('ul').hide('fold', 500); } ); my jsfiddle link is: http://jsfiddle.net/9wkbf/ updated fiddle the reason behind problem is, first event $(this).children('ul').show('fold', 570); queued , until completes, sec animation not start. the next snippet can workaround $('ul.mainmenu > li').hover( function() { $(this).children('ul').show('fold', 570); }, function() { $('ul:not(.mainmenu)').hide('fold', 500); } ); *important note : work current scenario. jquery html css
Read more