إن أنت تتلقّى [ولّ دسند] وجيّدا يجعل موقعة مع الكلمة المفتاح محتوى غنيّة أن يجذب زائرات و [سرش نجن] أنّ يكون حقّا عظيمة غير أنّ أنت مفقودة شيء أنّ يكون جدّا مهمّة. أنت تعرف ماذا أنّ يكون? أنّ Robots.txt مبرد.
Robots.txt يتلقّى مبرد حصص الأهمية بما أنّ هو يسمح عناكب أو زحّافات أن يسمح أو ينقض أن يزحف كلّ صفحات من موقعة أو [وببج] خاصّة. أحيانا يتلقّى الناس بعض معطيات سرّيّة على موقعتهم وب يستعمل robots.txt مبرد هم يستطيع قيّدت زحّافات أو عنكبوت أن لا زحفت أو فهرست أنّ صفحة خاصّة لذلك لا أحد يستطيع بلغت على أنّ صفحة وفي هذا طريق معطيات سرّيّة على أنّ صفحة كنت يؤمّن.
فتّشت بينما يذهب أن يزحف موقعة أو [وببج] [سرش نجن] عناكب أو زحّافات غالبا [سبسل فيل] أنّ يكون دعات Robots.txt مبرد لأنّ من خلال robots.txt مبرد [سرش نجن] عناكب أو زحّافات يأتون أن يعرف حول الذي [وب بج] من أنّ موقعة يضطرّ زحفت أو فهرست والذي [وب بج] يضطرّ تجاهلت.
Robots.txt file is a simple text file that must be placed in root directory of a website. For Example:
Robots.txt file must be like as-
http://www.abc.com/robots.txt
Creating Robots.txt File:
As mentioned above, robots.txt file is a simple text file and you can create it by opening a simple text editor like notepad. The data or a command mentioned in robots.txt file is called as “records”.
A record includes the information of a particular search engine and each record have two fields- User agent where you mention the robots or spider name and other field is disallow lines that may be one or more where you have to mention that which pages or files have to be ignored. For Example:
User-agent: googlebot
Disallow: /cgi-bin/
In above example robots.txt file allows “googlebot” that is the search engine spider of major search engine Google to crawl each and every page of the website except of files from “cgi-bin” directory. Means googlebot have to ignore all files from “cgi-bin”directory.
And if you enter like below:
User-agent: googlebot
Disallow: /support
Googlebot will not crawl any file from support directory as robots.txt file has instructions to googlebot to not crawl any file from support directory.
In case you leave disallow field blank then it will indicate to googlebot to crawl all files of the website. But in any case you must have a disallow field for every user agent.
The all above example were only for googlebot but if you want to give same rights to all other search engine’s spiders then use asterisk (*) instead of googlebot in user agent field. For example:
User-agent: *
Disallow: /cgi-bin/
In above example * represents all search engine spiders and robots.txt file above allows all search engine spiders to crawl each and every page of the website except of files from “cgi-bin” directory. Means all spiders from different search engines have to ignore all files from “cgi-bin”directory.
If you want to know user agent names for other search engines then you can find it in your log files by checking for requests to robots.txt. Most often, all search engine spiders should be given the same rights. in that case, use User-agent: * as mentioned above.
Webexcel Solutions (ISO 9001:2000 Certified) A Software Development Company and an SEO Company
Keyword Tags: robots, Search Engine Marketing, SEO
No Responses to “Importance Of Using Robots.txt File”
Please Wait
Leave a Reply