Warning: preg_replace(): The /e modifier is no longer supported, use preg_replace_callback instead in [path]/includes/class_bootstrap.php(430) : eval()'d code on line 110
What is the limit of a robot.txt file?

Results 1 to 6 of 6

Thread: What is the limit of a robot.txt file?

  1. #1
    Member
    Join Date
    Mar 2018
    Posts
    54
    Thanks
    0
    Thanked 0 Times in 0 Posts

    What is the limit of a robot.txt file?

    Hello friends,

    What is the limit of a robot.txt file?

  2. #2
    Senior Member charujewels's Avatar
    Join Date
    Jan 2018
    Location
    Surat,India
    Posts
    147
    Thanks
    7
    Thanked 4 Times in 4 Posts
    Your robots.txt file must be smaller than 500KB.

  3. #3
    Senior Member
    Join Date
    Apr 2013
    Posts
    779
    Thanks
    0
    Thanked 5 Times in 5 Posts
    A maximum file size may be enforced per crawler. Content which is after the maximum file size may be ignored. Google currently enforces a size limit of 500kb.

  4. #4

  5. #5

  6. #6
    Senior Member RH-Calvin's Avatar
    Join Date
    Apr 2015
    Location
    Forum
    Posts
    1,025
    Thanks
    0
    Thanked 4 Times in 4 Posts
    Google crawlers crawl only 500KB of the text file. Hence it is beneficial to have only 500KB text file.
    Cheap VPS | $1 VPS Hosting
    Cheap Windows VPS | Remote Desktop Access
    Cheap Dedicated Server | Free Setup and IPMI

Similar Threads

  1. Is it necessary to use Robots.txt file?
    By johnexplo in forum Off Page / On Page SEO
    Replies: 9
    Last Post: 05-03-2018, 05:31 AM
  2. What is the character limit for AdWords ad?
    By ninadordev in forum Off Page / On Page SEO
    Replies: 4
    Last Post: 03-13-2018, 07:34 AM
  3. What is the limit of a robot.txt file?
    By ninadordev in forum Off Page / On Page SEO
    Replies: 6
    Last Post: 03-05-2018, 06:58 AM
  4. My Robots.txt file got changed??
    By Jasmine in forum Web Programming
    Replies: 1
    Last Post: 01-09-2013, 11:06 PM
  5. signature limit
    By kevinvam in forum Suggestions for SamuraisSEO forum
    Replies: 0
    Last Post: 09-13-2012, 12:25 PM

Tags for this Thread

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •