Title: Theory / Unimportant / CS / Descriptive / Web / Robots
Tags: hasNotNewEffectReturnsMeaning FromInstruction computer_to_instruction FromVariables comment examplesSwitchExample1

Name: WWW robots
Definition: programs that traverse many pages in the World Wide Web by recursively retrieving linked pages
Page: http://www.robotstxt.org/orig.html
Name2: wanderers
Name3: Spiders
Name4: web crawlers

Question: Reason not to want a www robot
Answer: cgi script with side effect
Page: http://www.robotstxt.org/orig.html
Answer2: temporary information
Answer3: very deep virtual trees
Answer4: duplicated informations

Variable: /robots.txt
Meaning: Telling www robots what they have the right to check on this url
Page: http://www.robotstxt.org/orig.html
Typ: file

Instruction: <field>:<optionalspace><value><optionalspace>(CR|CR/NL|NL)
Page: http://www.robotstxt.org/orig.html
Effect: An entry of robots.txt file

Instruction: #foo
Other fields (not shown on cards):
Page: http://www.robotstxt.org/orig.html
Context: robots.txt

Variable: User-agent
Meaning: the name of the robot the record is describing access policy for
Page: http://www.robotstxt.org/orig.html
Context: robots.txt

Variable: Disallow
Meaning: a partial URL that is not to be visited
Page: http://www.robotstxt.org/orig.html
Context: robots.txt

Question: Empty robots.txt means
Answer: the same thing as absence of robots.txt
Page: http://www.robotstxt.org/orig.html
Context: robots.txt

Name: Autonomous agents
Definition: programs that do travel between sites, deciding themselves when to move and what to do
Page: http://www.robotstxt.org/faq/agent.html
Context: robots.txt

Name: Intelligent agents
Definition: programs that help users with things, such as choosing a product, or guiding a user through form filling, or even helping users find things. These have generally little to do with networking.
Page: http://www.robotstxt.org/faq/agent.html
Context: robots.txt

Name: User-agent
Definition: technical name for programs that perform networking tasks for a user
Page: http://www.robotstxt.org/faq/agent.html
Context: robots.txt
Example: IE, outlook

Name: WebAnts
Definition: distributed cooperating robots.
Page: http://www.robotstxt.org/faq/agent.html
Context: robots.txt

Name: rapid-fire
Definition: Traversing quickly an entire website
Page: http://www.robotstxt.org/faq/agent.html
Context: robots.txt

Instruction: <meta NAME="ROBOTS" CONTENT="(no)index"/>
Page: http://www.robotstxt.org/meta.html
Effect: State to robots whether they can(not) index this page
Context: HTML
Where to use it: head
Name: rapid-fire

Instruction: <meta NAME="ROBOTS" CONTENT="(no)follow"/>
Page: http://www.robotstxt.org/meta.html
Effect: State to robots whether they can(not) follow link from this page
Context: HTML
Where to use it: head

Instruction: rel="nofollow"
Page: http://www.robotstxt.org/meta.html
Effect: Don't use this link to give credit to improve the pagerank of the page linked here
Context: HTML
Where to use it: <a