Scrapy - Using an Item


Item objects are the regular dicts of Python. We can use the following syntax to access the attributes of the class −

>>> item = DmozItem()
>>> item['title'] = 'sample title'
>>> item['title']
'sample title'

Add the above code to the following example −

import scrapy

from tutorial.items import DmozItem

class MyprojectSpider(scrapy.Spider):
   name = "project"
   allowed_domains = [""]
   start_urls = [
   def parse(self, response):
      for sel in response.xpath('//ul/li'):
         item = DmozItem()
         item['title'] = sel.xpath('a/text()').extract()
         item['link'] = sel.xpath('a/@href').extract()
         item['desc'] = sel.xpath('text()').extract()
         yield item

The output of the above spider will be −

[scrapy] DEBUG: Scraped from <200>
   {'desc': [u' - By David Mertz; Addison Wesley. Book in progress, full text, 
      ASCII format. Asks for feedback. [author website, Gnosis Software, Inc.\n],
   'link': [u''],
   'title': [u'Text Processing in Python']}
[scrapy] DEBUG: Scraped from <200>
   {'desc': [u' - By Sean McGrath; Prentice Hall PTR, 2000, ISBN 0130211192, 
      has CD-ROM. Methods to build XML applications fast, Python tutorial, DOM and 
      SAX, new Pyxie open source XML processing library. [Prentice Hall PTR]\n'],
   'link': [u''],
   'title': [u'XML Processing with Python']}
Kickstart Your Career

Get certified by completing the course

Get Started