Ticket #6891 (closed defect: fixed)

Opened 7 months ago

Last modified 7 months ago

Do not add trailing slash to the robots.txt URL

Reported by: thenlich Assigned to: anonymous
Priority: normal Milestone: 2.6
Component: General Version: 2.5.1
Severity: normal Keywords: has-patch
Cc:

Description

The /robots.txt is redirected to /robots.txt/ if configured. This is in violation of the robot exclusion standard, which specifies

This file must be accessible via HTTP on the local URL "/robots.txt".

It is also not network nice because it generates extra traffic for the additional request after redirection.

Attachments

6891.diff (0.7 kB) - added by DD32 on 05/01/08 10:25:02.

Change History

05/01/08 10:24:54 changed by DD32

  • keywords set to has-patch.
  • milestone set to 2.6.

I spotted this the other day, Didnt get around to submitting a trac ticket though, Thanks for bringing it up and reminding me :)

05/01/08 10:25:02 changed by DD32

  • attachment 6891.diff added.

05/01/08 21:38:41 changed by ryan

  • status changed from new to closed.
  • resolution set to fixed.

(In [7871]) Don't redirect robots. Props thenlich and DD32. fixes #6891