python - scrappy log message in log file -

python - scrappy log message in log file -

here setting scrappy

log_enabled = true stats_enabled = true log_file = 'crawl.log'

and spider is..

class abcspider(xmlfeedspider): handle_httpstatus_list = [404, 500] name = 'abctv' allowed_domains = ['abctvnepal.com.np'] start_urls = [ 'http://www.abctvnepal.com.np', ] def parse(self, response): mesg = "spider {} not working".format(name) if response.status in self.handle_httpstatus_list: homecoming log.msg(mesg, level=log.error) hxs = htmlxpathselector(response) # xpath selector sites = hxs.select('//div[@class="marlr respo-left"]/div/div/h3') items = [] site in sites: item = newsitem() item['title'] = escape(''.join(site.select('a/text()').extract())).strip() item['link'] = escape(''.join(site.select('a/@href').extract())).strip() item['description'] = escape(''.join(site.select('p/text()').extract())) item = request(item['link'],meta={'item': item},callback=self.parse_detail) items.append(item) homecoming items def parse_detail(self, response): item = response.meta['item'] sel = htmlxpathselector(response) details = sel.select('//div[@class="entry"]/p/text()').extract() detail = '' piece in details: detail = detail + piece item['details'] = detail item['location'] = detail.split(",",1)[0] item['published_date'] = (detail.split(" ",1)[1]).split(" ",1)[0]+' '+((detail.split(" ",1)[1]).split(" ",1)[1]).split(" ",1)[0] homecoming item

here want send log message if response code in handle_httpstatus_list = [404, 500]. can give me illustration how it? helpful.

the scrapy documentation written , contains lot of illustration code. if you're working on first scrapy project, worthwhile having browse there. :)

for example, quick scan of logging documentation turns next sample code:

from scrapy import log log.msg("this warning", level=log.warning)

so adding import , removing return should prepare code

also, should mesg line utilize self.name?

mesg = "spider {} not working".format(self.name)

python django web-scraping scrapy