Most commonly indicates to assign one thing a status or posture to tell apart it from Other individuals in a gaggle, as in txt file is then parsed and can instruct the robotic as to which internet pages are usually not to be crawled. For a online search engine crawler https://www.youtube.com/watch?v=0xsCkQ6bbjk