skip to primary navigationskip to content
 

Stopping some or all of your site from being indexed by search engines (by editing your robots.txt file)

While your site is under development (before it has a 'proper' name) it is blocked from indexers. When it has its 'proper' name it will be open to indexers if they find it.

Falcon adds a file at the top level of your site to stop some or all of it from being indexed by search engines. By default this allows everything to be indexed - if you have restricted a folder to being viewed by a group of people this will stop Google from indexing this content.

To alter the robots.txt file, go to Site Setup and then Zope Management Interface. From the list available, select portal_skins (Controls skin behaviour (search order etc)) and then select plone_templates. You need then to select robots.txt and click on the button that says 'Customize'. This gives you a screen similar to this:

Robots.txt editing screen

You can directly edit this file in the window, and then save the changes. There is full information on how to construct your page at http://help.uis.cam.ac.uk/website-resources/website-components/web-search/robot-text (including how to allow the University search authenticated access which blocking other search engines) - to exclude all indexing of the whole of your site the page needs the following:

User-agent: *
Disallow: /

To prevent just the folder 'new' and all within it being indexed, add the following:

User-agent: *
Disallow: /new/

Remember to periodically review this file when additional information or functionality is added to your site or when you decide to make the site public.