How to Scrap Amazon Products with Python Scrapy

In this article we want to learn How to Scrap Amazon Products with Python Scrapy , so as we know that Amazon is one the most popular and big marketplace. it has millions of products, and finding specific items can be a difficult task, also Scrapy is an open source web scraping framework written in Python. It provides an easy way to extract data from websites by defining spiders, which are Python classes that define how to navigate web pages, extract data from them and store the scraped data.

First of all we need to install Scrapy and you can use pip for that.

pip install scrapy

1	pip install scrapy

After that open a command prompt and navigate to the directory where you want to create your project.
Run the following command to create a new Scrapy project named amazon_scraper, this will create a directory structure for your project with different files and folders.

scrapy startproject amazon_scraper

1	scrapy startproject amazon_scraper

Now Open amazon_scraper/items.py file and define the structure of the item you want to scrape. for example, if you want to scrape product titles, prices, and ratings, you can define an item like this, especially in here we are not going to scrap all products from Amazon, we want to just scrap kitchen products.

import scrapy


class KitchenProductItem(scrapy.Item):
    title = scrapy.Field()
    price = scrapy.Field()
    rating = scrapy.Field()

import scrapy

class KitchenProductItem(scrapy.Item):

title = scrapy.Field()

price = scrapy.Field()

rating = scrapy.Field()

In Scrapy a spider is responsible for defining how to crawl a website and extract data. Create a new Python file named amazon_spider.py in the amazon_scraper/amazon_scraper/spiders directory. open amazon_scraper/amazon_scraper/spiders/amazon_spider.py file and replace its content with the following code:

import scrapy
from ..items import KitchenProductItem

class AmazonSpider(scrapy.Spider):
    name = 'amazon'
    start_urls = ['https://www.amazon.com/s?k=kitchen']

    def start_requests(self):
        headers = {
            'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',
        }
        for url in self.start_urls:
            yield scrapy.Request(url, headers=headers)

    def parse(self, response):
        products = response.xpath('//div[@data-asin]')
        for product in products:
            item = KitchenProductItem()
            item['title'] = product.xpath('.//h2/a/span/text()').get()
            item['price'] = product.xpath('.//span[@class="a-offscreen"]/text()').get()
            item['rating'] = product.xpath('.//span[@class="a-icon-alt"]/text()').get()
            yield item

        next_page_url = response.css('div.a-section.a-text-center.s-pagination-container.s-pagination-dynamic-container a.s-pagination-item.s-pagination-next::attr(href)').get()
        if next_page_url:
            yield response.follow(next_page_url, self.parse)

import scrapy

from ..items import KitchenProductItem

class AmazonSpider(scrapy.Spider):

name = 'amazon'

start_urls = ['https://www.amazon.com/s?k=kitchen']

def start_requests(self):

headers = {

'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/93.0.4577.82 Safari/537.36',

}

for url in self.start_urls:

yield scrapy.Request(url, headers=headers)

def parse(self, response):

products = response.xpath('//div[@data-asin]')

for product in products:

item = KitchenProductItem()

item['title'] = product.xpath('.//h2/a/span/text()').get()

item['price'] = product.xpath('.//span[@class="a-offscreen"]/text()').get()

item['rating'] = product.xpath('.//span[@class="a-icon-alt"]/text()').get()

yield item

next_page_url = response.css('div.a-section.a-text-center.s-pagination-container.s-pagination-dynamic-container a.s-pagination-item.s-pagination-next::attr(href)').get()

if next_page_url:

yield response.follow(next_page_url, self.parse)

And now Open a terminal or command prompt and navigate to amazon_scraper directory. Run the following command to start the spider: