python - Crawl multiple domains with Scrapy without criss-cross -



python - Crawl multiple domains with Scrapy without criss-cross -

i have set crawlspider aggregating outbound links (crawling start_urls depth via e.g. depth_limit = 2).

class linknetworkspider(crawlspider): name = "network" allowed_domains = ["examplea.com"] start_urls = ["http://www.examplea.com"] rules = (rule(sgmllinkextractor(allow=()), callback='parse_item', follow=true),) def parse_start_url(self, response): homecoming self.parse_item(response) def parse_item(self, response): hxs = htmlxpathselector(response) links = hxs.select('//a/@href').extract() outgoing_links = [] link in links: if ("http://" in link): base_url = urlparse(link).hostname base_url = base_url.split(':')[0] # drop ports base_url = '.'.join(base_url.split('.')[-2:]) # remove subdomains url_hit = sum(1 in self.allowed_domains if base_url not in i) if url_hit != 0: outgoing_links.append(link) if outgoing_links: item = linknetworkitem() item['internal_site'] = response.url item['out_links'] = outgoing_links homecoming [item] else: homecoming none

i want extend multiple domains (examplea.com, exampleb.com, examplec.com ...). @ first, thought can add together list start_urls allowed_domains in sentiment causes next problems:

will settings depth_limit applied each start_urls/allowed_domain? more important: if sites connected spider jump examplea.com exampleb.com because both in allowed_domains? need avoid criss-cross later on want count outbound links each site gain info relationship between websites!

so how can scale more spider without running problem of criss-crossing , using settings per website?

additional image showing realize:

you need maintain info construction (ex hashmap) of urls crawler has visited. it's matter of adding urls hashmap visit them , not visiting urls if they're in hashmap (as means have visited them). there more complicated ways of doing give greater performace, these harder implement.

python scrapy

Comments

Post a Comment

Popular posts from this blog

php - Android app custom user registration and login with cookie using facebook sdk -

php - Android app custom user registration and login with cookie using facebook sdk - i developing android application in users need register , log in using facebook. approximately, flow. open app first time, authenticate facebook, data, send info server create them business relationship in application, start session new business relationship in server, cookie perform later interactions. so far good. what need know wich info have send server, can able identify users when log in seccond time. because need start session in server , send cookie android app farther interactions. i'm using php server side language. i thinking getting getaccesstoken() in android facebook sdk, pass server on business relationship creation , storing in android app. but can't manage how login users on sec time utilize app. any suggestions ? in advance ! to uniquely identify user within app, should create request /me , user's id. guaranteed unique , remain con...
Read more

ruby on rails - Devise Logout Error in RoR -

ruby on rails - Devise Logout Error in RoR - while using devise in ror, encountering unusual error when trying log out url different root url. displays me : actioncontroller::invalidauthenticitytoken in devise::sessionscontroller#destroy" for illustration if rooting path after log in '/structure', , seek log out '/general' error, while logging out '/structure' fine. any help highly appreciated. ruby-on-rails ruby devise
Read more

c# - Create a Notification Object (Email or Page) At Run Time -- Dependency Injection or Factory -

c# - Create a Notification Object (Email or Page) At Run Time -- Dependency Injection or Factory - i have code emails me notification when happens. trying add together page , also remove dependency on email. for reference; have name of notification want send stored string in database. @ runtime have list of people , string representation of want receive. i thought dependency injection ninject, kernel bindings got big , seemed hacking together. then thought simple factory, activator.createinstance(email/page) inotifcation, don't know how handle differing constructor parameters (unless utilize common, complex property, employee, has info need). so far have this: public interface inotification { void sendmessage(); } public class email: inotification { private readonly string _toemailaddress; private string body { { homecoming "email"; } } private static string smtphost { ...
Read more