Scrapy - Command Line Tools

Advertisements

Previous Page
Next Page  

Description

The Scrapy command line tool is used for controlling Scrapy, which is often referred to as 'Scrapy tool'. It includes the commands for various objects with a group of arguments and options.

Configuration Settings

Scrapy will find configuration settings in the scrapy.cfg file. Following are a few locations −

  • C:\scrapy(project folder)\scrapy.cfg in the system

  • ~/.config/scrapy.cfg ($XDG_CONFIG_HOME) and ~/.scrapy.cfg ($HOME) for global settings

  • You can find the scrapy.cfg inside the root of the project.

Scrapy can also be configured using the following environment variables −

  • SCRAPY_SETTINGS_MODULE
  • SCRAPY_PROJECT
  • SCRAPY_PYTHON_SHELL

Default Structure Scrapy Project

The following structure shows the default file structure of the Scrapy project.

scrapy.cfg                - Deploy the configuration file
project_name/             - Name of the project
   _init_.py
   items.py               - It is project's items file
   pipelines.py           - It is project's pipelines file
   settings.py            - It is project's settings file
   spiders                - It is the spiders directory
      _init_.py
      spider_name.py
      . . .

The scrapy.cfg file is a project root directory, which includes the project name with the project settings. For instance −

[settings] 
default = [name of the project].settings  

[deploy] 
#url = http://localhost:6800/ 
project = [name of the project]

Using Scrapy Tool

Scrapy tool provides some usage and available commands as follows −

Scrapy X.Y  - no active project 
Usage: 
   scrapy  [options] [arguments] 
Available commands: 
   crawl      It puts spider (handle the URL) to work for crawling data 
   fetch      It fetches the response from the given URL

Creating a Project

You can use the following command to create the project in Scrapy −

scrapy startproject project_name

This will create the project called project_name directory. Next, go to the newly created project, using the following command −

cd  project_name

Controlling Projects

You can control the project and manage them using the Scrapy tool and also create the new spider, using the following command −

scrapy genspider mydomain mydomain.com

The commands such as crawl, etc. must be used inside the Scrapy project. You will come to know which commands must run inside the Scrapy project in the coming section.

Scrapy contains some built-in commands, which can be used for your project. To see the list of available commands, use the following command −

scrapy -h

When you run the following command, Scrapy will display the list of available commands as listed −

  • fetch − It fetches the URL using Scrapy downloader.

  • runspider − It is used to run self-contained spider without creating a project.

  • settings − It specifies the project setting value.

  • shell − It is an interactive scraping module for the given URL.

  • startproject − It creates a new Scrapy project.

  • version − It displays the Scrapy version.

  • view − It fetches the URL using Scrapy downloader and show the contents in a browser.

You can have some project related commands as listed −

  • crawl − It is used to crawl data using the spider.

  • check − It checks the items returned by the crawled command.

  • list − It displays the list of available spiders present in the project.

  • edit − You can edit the spiders by using the editor.

  • parse − It parses the given URL with the spider.

  • bench − It is used to run quick benchmark test (Benchmark tells how many number of pages can be crawled per minute by Scrapy).

Custom Project Commands

You can build a custom project command with COMMANDS_MODULE setting in Scrapy project. It includes a default empty string in the setting. You can add the following custom command −

COMMANDS_MODULE = 'mycmd.commands'

Scrapy commands can be added using the scrapy.commands section in the setup.py file shown as follows −

from setuptools import setup, find_packages  

setup(name = 'scrapy-module_demo', 
   entry_points = { 
      'scrapy.commands': [ 
         'cmd_demo = my_module.commands:CmdDemo', 
      ], 
   }, 
)

The above code adds cmd_demo command in the setup.py file.

Advertisements