Robot.txt Optimization
What is the Robot.txt? For those of you who don’t know what robot.txt is; it is a protocol which is designed to limit what areas of sites can be crawled by spiders, to understand this completely you should have a rough idea of how spiders work. Using the robot.txt optimisation carries with it major benefits…