[MF3D.FORUM:1411] Re: MF 3D Expo 2000 update

[Brian Reynolds <reynolds@xxxxxxxxx>]

Paul Talbot wrote:
> I won't post the location as a URL because I don't the document to
> get into the search engines (Marco's archive is crawled by the
> search engines).  Remove the embedded spaces from the following line
> to convert it to the URL for the page: http://www.rmm3d.com/
> funlearn/ expo2000entries .html
> 

You can prevent this by having a robots.txt file at the top directory
of your web page.  I do this to keep the search engines from indexing
my JPEG files.  I've included the robots.txt file I use for my
personal site below.  The "User-agent:" line specifies what User Agent
(program) these rules apply to (everything in my case).  The
"Disallow:" lines specify what directories (or files) I don't want
searched.  Nothing in my "images" directories (or anything below them)
should be searched.

Paul, if you created a directory funlearn/expo2000, put the Expo files
in there and then put the line "Disallow: /funlearn/expo2000/" on your
server the Expo stuff wouldn't be indexed.  If Marco put an
appropriate robots.txt at the archive site the archives wouldn't be
indexed by the search engines.

Of course this assumes that the spiders (the programs that crawl
through the web for the search engines) are well behaved.  People that
run spiders that ignore robots.txt aren't going to be fooled by most
attempts to thwart them.

-- 
Brian Reynolds                  | "Dee Dee!  Don't touch that button!"
reynolds@xxxxxxxxx              | "Oooh!"
http://www.panix.com/~reynolds  |    -- Dexter and Dee Dee
NAR# 54438                      |       "Dexter's Laboratory"

----- cut here ----- robots.txt ----- cut here -----
User-agent: *
Disallow: /images/
Disallow: /photography/lf/rhinebeck/images/
Disallow: /photography/lf/firehouse/images/
Disallow: /photography/lf/nyc/images/
Disallow: /photography/pinhole/images/
Disallow: /photography/family/images/
Disallow: /photography/stereo/images/
Disallow: /photography/35mm/images/
----- cut here ----- robots.txt ----- cut here -----