Macam macam robot dan cara kerja

Macam macam robot dan cara kerja 

http://www.seohocasi.com/wp-content/uploads/robots_txt.gif 
 
 
 
 
# robots.txt for http://www.blogger.com

User-agent: *
Disallow: /blog_this.pyra
Disallow: /comment.g
Disallow: /comment-iframe.g
Disallow: /create-blog.g
Disallow: /delete-backlink.g
Disallow: /delete-comment.g
Disallow: /email-post.g
Disallow: /post-edit.g
Disallow: /profile-find.g
Disallow: /rearrange
Disallow: /share-post.g
Disallow: /share-post-menu.g
 
 
 
User-agent: *
Disallow: /p/
Disallow: /r/
Disallow: /bin/
Disallow: /includes/
Disallow: /blank.html

Sitemap: https://www.yahoo.com/food/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/tech/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/travel/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/movies/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/beauty/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/health/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/style/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/diy/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/parenting/sitemaps/sitemap_index_us_en-US.xml.gz
Sitemap: https://www.yahoo.com/music/sitemaps/sitemap_index_us_en-US.xml.gz 
 
 
User-agent: *
Disallow: /account/
Disallow: /bfp/search
Disallow: /blogs/search/
Disallow: /entities/search
Disallow: /fd/
Disallow: /history
Disallow: /hotels/search
Disallow: /images/search?
Disallow: /images?
Disallow: /local
Disallow: /maps/clicks.ashx?
Disallow: /maps/GeoCommunity.aspx
Disallow: /news/apiclick.aspx
Disallow: /news/search?
Disallow: /notifications/
Disallow: /offers/proxy/dealsserver/api/log
Disallow: /offers/proxy/dealsserver/buy
Disallow: /ping
Disallow: /profile/history?
Disallow: /proFile/history?
Disallow: /Proxy.ashx
Disallow: /results
Disallow: /rewardsapp/
Disallow: /search
Disallow: /Search
Disallow: /settings
Disallow: /shenghuo
Disallow: /shopping/
Allow: /shopping/$
Allow: /shopping$
Disallow: /social/search?
Disallow: /spbasic
Disallow: /spresults
Disallow: /static/
Disallow: /th?
Disallow: /th$
Disallow: /translator/?
Disallow: /translator?
Disallow: /travel/css
Disallow: /travel/flight/flightSearch
Disallow: /travel/flight/flightSearchAction
Disallow: /travel/flight/search?
Disallow: /travel/flight/search/?
Disallow: /travel/hotel/hotelMiniSearchRequest
Disallow: /travel/hotel/hotelSearch
Disallow: /travel/hotels/search?
Disallow: /travel/hotels/search/?
Disallow: /travel/scripts
Disallow: /travel/secure
Disallow: /url
Disallow: /videos?
Disallow: /videos/?
Disallow: /videos/search?
Disallow: /videos/search/?
Disallow: /widget/cr
Disallow: /widget/entity/search/?
Disallow: /widget/render
Disallow: /widget/snapshot


Sitemap: http://cn.bing.com/dict/sitemap-index.xml
Sitemap: http://www.bing.com/offers/sitemap.xml 






User-agent: *
Disallow: /search
Disallow: /sdch
Disallow: /groups
Disallow: /images
Disallow: /catalogs
Allow: /catalogs/about
Allow: /catalogs/p?
Disallow: /catalogues
Allow: /newsalerts
Disallow: /news
Allow: /news/directory
Disallow: /nwshp
Disallow: /setnewsprefs?
Disallow: /index.html?
Disallow: /?
Allow: /?hl=
Disallow: /?hl=*&
Allow: /?hl=*&gws_rd=ssl$
Disallow: /?hl=*&*&gws_rd=ssl
Allow: /?gws_rd=ssl$
Allow: /?pt1=true$
Disallow: /addurl/image?
Allow:    /mail/help/
Disallow: /mail/
Disallow: /pagead/
Disallow: /relpage/
Disallow: /relcontent
Disallow: /imgres
Disallow: /imglanding
Disallow: /sbd
Disallow: /keyword/
Disallow: /u/
Disallow: /univ/
Disallow: /cobrand
Disallow: /custom
Disallow: /advanced_group_search
Disallow: /googlesite
Disallow: /preferences
Disallow: /setprefs
Disallow: /swr
Disallow: /url
Disallow: /default
Disallow: /m?
Disallow: /m/
Allow:    /m/finance
Disallow: /wml?
Disallow: /wml/?
Disallow: /wml/search?
Disallow: /xhtml?
Disallow: /xhtml/?
Disallow: /xhtml/search?
Disallow: /xml?
Disallow: /imode?
Disallow: /imode/?
Disallow: /imode/search?
Disallow: /jsky?
Disallow: /jsky/?
Disallow: /jsky/search?
Disallow: /pda?
Disallow: /pda/?
Disallow: /pda/search?
Disallow: /sprint_xhtml
Disallow: /sprint_wml
Disallow: /pqa
Disallow: /palm
Disallow: /gwt/
Disallow: /purchases
Disallow: /hws
Disallow: /bsd?
Disallow: /linux?
Disallow: /mac?
Disallow: /microsoft?
Disallow: /unclesam?
Disallow: /answers/search?q=
Disallow: /local?
Disallow: /local_url
Disallow: /shihui?
Disallow: /shihui/
Disallow: /froogle?
Disallow: /products?
Disallow: /froogle_
Disallow: /product_
Disallow: /products_
Disallow: /products;
Disallow: /print
Disallow: /books/
Disallow: /bkshp?*q=*
Disallow: /books?*q=*
Disallow: /books?*output=*
Disallow: /books?*pg=*
Disallow: /books?*jtp=*
Disallow: /books?*jscmd=*
Disallow: /books?*buy=*
Disallow: /books?*zoom=*
Allow: /books?*q=related:*
Allow: /books?*q=editions:*
Allow: /books?*q=subject:*
Allow: /books/about
Allow: /booksrightsholders
Allow: /books?*zoom=1*
Allow: /books?*zoom=5*
Disallow: /ebooks/
Disallow: /ebooks?*q=*
Disallow: /ebooks?*output=*
Disallow: /ebooks?*pg=*
Disallow: /ebooks?*jscmd=*
Disallow: /ebooks?*buy=*
Disallow: /ebooks?*zoom=*
Allow: /ebooks?*q=related:*
Allow: /ebooks?*q=editions:*
Allow: /ebooks?*q=subject:*
Allow: /ebooks?*zoom=1*
Allow: /ebooks?*zoom=5*
Disallow: /patents?
Disallow: /patents/download/
Disallow: /patents/pdf/
Disallow: /patents/related/
Disallow: /scholar
Disallow: /citations?
Allow: /citations?user=
Disallow: /citations?*cstart=
Allow: /citations?view_op=new_profile
Allow: /citations?view_op=top_venues
Disallow: /complete
Disallow: /s?
Disallow: /sponsoredlinks
Disallow: /videosearch?
Disallow: /videopreview?
Disallow: /videoprograminfo?
Allow: /maps?*output=classic*
Allow: /maps/api/js?
Allow: /maps/d/
Disallow: /maps?
Disallow: /mapstt?
Disallow: /mapslt?
Disallow: /maps/stk/
Disallow: /maps/br?
Disallow: /mapabcpoi?
Disallow: /maphp?
Disallow: /mapprint?
Disallow: /maps/api/js/
Disallow: /maps/api/staticmap?
Disallow: /mld?
Disallow: /staticmap?
Disallow: /places/
Allow: /places/$
Disallow: /maps/preview
Disallow: /maps/place
Disallow: /help/maps/streetview/partners/welcome/
Disallow: /help/maps/indoormaps/partners/
Disallow: /lochp?
Disallow: /center
Disallow: /ie?
Disallow: /sms/demo?
Disallow: /katrina?
Disallow: /blogsearch?
Disallow: /blogsearch/
Disallow: /blogsearch_feeds
Disallow: /advanced_blog_search
Disallow: /uds/
Disallow: /chart?
Disallow: /transit?
Disallow: /mbd?
Disallow: /extern_js/
Disallow: /xjs/
Disallow: /calendar/feeds/
Disallow: /calendar/ical/
Disallow: /cl2/feeds/
Disallow: /cl2/ical/
Disallow: /coop/directory
Disallow: /coop/manage
Disallow: /trends?
Disallow: /trends/music?
Disallow: /trends/hottrends?
Disallow: /trends/viz?
Disallow: /trends/embed.js?
Disallow: /trends/fetchComponent?
Disallow: /notebook/search?
Disallow: /musica
Disallow: /musicad
Disallow: /musicas
Disallow: /musicl
Disallow: /musics
Disallow: /musicsearch
Disallow: /musicsp
Disallow: /musiclp
Disallow: /browsersync
Disallow: /call
Disallow: /archivesearch?
Disallow: /archivesearch/url
Disallow: /archivesearch/advanced_search
Disallow: /base/reportbadoffer
Disallow: /urchin_test/
Disallow: /movies?
Disallow: /codesearch?
Disallow: /codesearch/feeds/search?
Disallow: /wapsearch?
Disallow: /safebrowsing
Allow: /safebrowsing/diagnostic
Allow: /safebrowsing/report_badware/
Allow: /safebrowsing/report_error/
Allow: /safebrowsing/report_phish/
Disallow: /reviews/search?
Disallow: /orkut/albums
Allow: /jsapi
Disallow: /views?
Disallow: /c/
Disallow: /cbk
Allow: /cbk?output=tile&cb_client=maps_sv
Disallow: /recharge/dashboard/car
Disallow: /recharge/dashboard/static/
Disallow: /translate_a/
Disallow: /translate_c
Disallow: /translate_f
Disallow: /translate_static/
Disallow: /translate_suggestion
Disallow: /profiles/me
Allow: /profiles
Disallow: /s2/profiles/me
Allow: /s2/profiles
Allow: /s2/oz
Allow: /s2/photos
Allow: /s2/search/social
Allow: /s2/static
Disallow: /s2
Disallow: /transconsole/portal/
Disallow: /gcc/
Disallow: /aclk
Disallow: /cse?
Disallow: /cse/home
Disallow: /cse/panel
Disallow: /cse/manage
Disallow: /tbproxy/
Disallow: /imesync/
Disallow: /shenghuo/search?
Disallow: /support/forum/search?
Disallow: /reviews/polls/
Disallow: /hosted/images/
Disallow: /ppob/?
Disallow: /ppob?
Disallow: /adwordsresellers
Disallow: /accounts/ClientLogin
Disallow: /accounts/ClientAuth
Disallow: /accounts/o8
Allow: /accounts/o8/id
Disallow: /topicsearch?q=
Disallow: /xfx7/
Disallow: /squared/api
Disallow: /squared/search
Disallow: /squared/table
Disallow: /toolkit/
Allow: /toolkit/*.html
Disallow: /globalmarketfinder/
Allow: /globalmarketfinder/*.html
Disallow: /qnasearch?
Disallow: /app/updates
Disallow: /sidewiki/entry/
Disallow: /quality_form?
Disallow: /labs/popgadget/search
Disallow: /buzz/post
Disallow: /compressiontest/
Disallow: /analytics/reporting/
Disallow: /analytics/admin/
Disallow: /analytics/web/
Disallow: /analytics/feeds/
Disallow: /analytics/settings/
Allow: /alerts/manage
Allow: /alerts/remove
Disallow: /alerts/
Allow: /alerts/$
Disallow: /ads/search?
Disallow: /ads/plan/action_plan?
Disallow: /ads/plan/api/
Disallow: /phone/compare/?
Disallow: /travel/clk
Disallow: /hotelfinder/rpc
Disallow: /hotels/rpc
Disallow: /flights/rpc
Disallow: /commercesearch/services/
Disallow: /evaluation/
Disallow: /chrome/browser/mobile/tour
Disallow: /compare/*/apply*
Disallow: /forms/perks/
Disallow: /baraza/*/search
Disallow: /baraza/*/report
Disallow: /shopping/suppliers/search
Disallow: /ct/
Disallow: /edu/cs4hs/
Disallow: /trustedstores/s/
Disallow: /trustedstores/tm2
Disallow: /trustedstores/verify
Disallow: /adwords/proposal
Disallow: /shopping/product/
Disallow: /shopping/seller
Disallow: /shopping/reviewer
Disallow: /about/careers/apply/
Disallow: /about/careers/applications/
Disallow: /landing/signout.html
Allow: /gb/images
Allow: /gb/js
Disallow: /gallery/
Sitemap: http://www.gstatic.com/culturalinstitute/sitemaps/www_google_com_culturalinstitute/sitemap-index.xml
Sitemap: https://www.google.com/edu/sitemap.xml
Sitemap: https://www.google.com/work/sitemap.xml
Sitemap: https://www.google.com/intx/sitemap.xml
Sitemap: http://www.google.com/hostednews/sitemap_index.xml
Sitemap: http://www.google.com/maps/views/sitemap.xml
Sitemap: http://www.google.com/sitemaps_webmasters.xml
Sitemap: http://www.google.com/ventures/sitemap_ventures.xml
Sitemap: http://www.gstatic.com/dictionary/static/sitemaps/sitemap_index.xml
Sitemap: http://www.gstatic.com/earth/gallery/sitemaps/sitemap.xml
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.gstatic.com/trends/websites/sitemaps/sitemapindex.xml
Sitemap: http://www.google.com/adwords/sitemap.xml
Sitemap: http://www.google.com/drive/sitemap.xml 
 
 
 
 
# robots.txt file for YouTube
# Created in the distant future (the year 2000) after
# the robotic uprising of the mid 90's which wiped out all humans.

User-agent: Mediapartners-Google*
Disallow:

User-agent: *
Disallow: /comment
Disallow: /get_video
Disallow: /get_video_info
Disallow: /login
Disallow: /results
Disallow: /signup
Disallow: /t/terms
Disallow: /verify_age
Disallow: /watch_ajax
Disallow: /watch_popup
Disallow: /watch_queue_ajax 
 
 
 
 
# Robots.txt file for http://www.microsoft.com
#

User-agent: *
Disallow: /*/security/search-results.aspx?
Disallow: /*/search/
Disallow: /*/newsearch/
Disallow: *action=catalogsearch&
Allow: *action=catalogsearch&catalog_mode=grid&page=2$
Allow: *action=catalogsearch&catalog_mode=grid&page=3$
Allow: *action=catalogsearch&catalog_mode=grid&page=4$
Allow: *action=catalogsearch&catalog_mode=grid&page=5$
Allow: *action=catalogsearch&catalog_mode=grid&page=6$
Allow: *action=catalogsearch&catalog_mode=grid&page=7$
Allow: *action=catalogsearch&catalog_mode=grid&page=8$
Allow: *action=catalogsearch&catalog_mode=list&page=2$
Allow: *action=catalogsearch&catalog_mode=list&page=3$
Allow: *action=catalogsearch&catalog_mode=list&page=4$
Allow: *action=catalogsearch&catalog_mode=list&page=5$
Allow: *action=catalogsearch&catalog_mode=list&page=6$
Allow: *action=catalogsearch&catalog_mode=list&page=7$
Allow: *action=catalogsearch&catalog_mode=list&page=8$
Disallow: *action=accessorysearch&product=*&*
Allow: *action=accessorysearch&product=*$
Disallow: *action=accessorysearch&
Allow: *action=accessorysearch&page=2$
Allow: *action=accessorysearch&page=3$
Allow: *action=accessorysearch&page=4$
Allow: *action=accessorysearch&page=5$
Allow: *action=accessorysearch&page=6$
Allow: *action=accessorysearch&page=7$
Allow: *action=accessorysearch&page=8$
Disallow: *action=productcompareaction&
Disallow: *action=productLinkAction&
Disallow: *action=overlay&
Disallow: *action=quickSearch&
Disallow: *action=writeReview
Disallow: *rep=hc
Disallow: *fe=true
Disallow: *?intc=
Disallow: *&solved=
Disallow: */wal/
Disallow: */layout/
Disallow: */base-en/
Disallow: *action=siteSearch
Disallow: *action=productSupportSearch
Disallow: *?cid=
Disallow: *=imgmanager
Disallow: */unsubscribe/
Disallow: *ActivityUID=
Disallow: /feeds/TechNet/fr-fr/screenshot/screenshot%20surface.jpg
Disallow: /imaginecup/*
Disallow: /*/download/confirmation.aspx?
Disallow: /*navV3Index=0$
Disallow: /*navV3Index=1$
Disallow: /*navV3Index=2$
Disallow: /*navV3Index=3$
Disallow: /*navV3Index=4$
Disallow: /*mnui=-1$
Disallow: /*mnui=0$
Disallow: /*mnui=1$
Disallow: /*mnui=2$
Disallow: /*mnui=3$
Disallow: /*mnui=4$
Disallow: /*mnui=5$
Disallow: /*acci=0$
Disallow: /*acci=1$
Disallow: /*acci=2$
Disallow: /*acci=3$
Disallow: /*acci=4$
Disallow: /*acci=5$
Disallow: /*acci=6$
Disallow: /*crsci=-1$
Disallow: /*crsci=0$
Disallow: /*crsci=1$
Disallow: /*crsci=2$
Disallow: /*crsci=3$
Disallow: /*crsci=4$
Disallow: /*crsci=5$
Disallow: /*crsci=6$
Disallow: /*crsci=7$
Disallow: /*crsci=8$
Disallow: /*hdrFo=mthdr01$
Disallow: /*hdrFo=mthdr02$
Disallow: /*hdrFo=mthdr03$
Disallow: /*hdrFo=mthdr04$
Disallow: /*hdrFo=mthdr05$
Disallow: /*hdrFo=mthdr06$
Disallow: /*hdrFo=mthdr07$
Disallow: /*hdrFo=mthdr08$
Disallow: /*hroi=-1$
Disallow: /*hroi=0$
Disallow: /*hroi=1$
Disallow: /*hroi=2$
Disallow: /*hroi=3$
Disallow: /*hroi=4$
Disallow: /*hroi=5$
Disallow: /*hroi=6$
Disallow: /*pvti=-1$
Disallow: /*pvti=0$
Disallow: /*pvti=1$
Disallow: /*pvti=2$
Disallow: /*pvti=3$
Disallow: /*pvti=4$
Disallow: /*pvti=5$
Disallow: /*pvti=6$
Disallow: /*pvtsi=-1$
Disallow: /*pvtsi=0$
Disallow: /*pvtsi=1$
Disallow: /*pvtsi=2$
Disallow: /*pvtsi=3$
Disallow: /*pvtsi=4$
Disallow: /*pvtsi=5$
Disallow: /*pvtsi=6$
Disallow: /*HpOptOut=true$
Disallow: /*TOCLinksForCrawlers*
Disallow: /*mac/help.mspx?
Disallow: /*mactopia/help.mspx?
Disallow: /blacklisted*
Disallow: /canada/Library/mnp/2/aspx/
Disallow: /communities/bin.aspx?
Disallow: /communities/blogs/PortalResults.mspx?
Disallow: /communities/eventdetails.mspx?
Disallow: /communities/rss.aspx*
Disallow: /*/download/confirmation.aspx?
Disallow: /*/download/registration-suggested.aspx?
Disallow: /*/download/results.aspx?
Disallow: /*/download/Browse.aspx?
Disallow: /*/download/browse.aspx?
Disallow: /*/download/info.aspx?
Disallow: /*/download/thankyou.aspx
Disallow: /*/download/thankyou.aspx?
Disallow: /france/formation/centres/planning.asp?
Disallow: /france/ie/default.asp?
Disallow: /france/mnp_utility.mspx?
Disallow: /genuine/
Disallow: /Germany/kleinunternehmen/euga/detail.mspx?
Disallow: /Germany/kleinunternehmen/euga/results.mspx?
Disallow: /germany/library/images/mnp/
Disallow: /germany/video/de/de/related*
Disallow: /hpc/*/supported-applications.aspx?
Disallow: /ie/ie40/
Disallow: /info/customerror.htm*
Disallow: /info/smart404.asp*
Disallow: /intlkb/
Disallow: /isapi/
Disallow: /Japan/DirectX/default.asp?
Disallow: /japan/directx/default.asp?
Disallow: /japan/enable/textview.asp?
Disallow: /japan/products/library/search.asp?
Disallow: /japan/showcase/print/default.aspx?
Disallow: /japan/terminology/query.asp?
Disallow: /*mnp_utility.mspx?
Disallow: /rus/licensing/Unilateral.aspx/*
Disallow: /spain/empresas/
Disallow: /spain/medianaempresa/
Disallow: /windows/compatibility/windows-vista/
Disallow: /windows/compatibility/windows-7/*/search.aspx?
Disallow: /windows/compatibility/windows-7/*/Search.aspx?
Disallow: /windows/compatibility/windows-7/*/browse.aspx?
Disallow: /windows/compatibility/windows-7/*/Browse.aspx?
Disallow: /windows/compatibility/windows-7/*/details.aspx?
Disallow: /windows/compatibility/windows-7/*/Details.aspx?
Disallow: /windows/404.aspx?*
Disallow: /windows/campaign/meet-start.aspx
Disallow: /windows/campaign/meet-apps.aspx
Disallow: /windows/campaign/features-built-in-apps.aspx
Disallow: /ru-ru/events/platforma/materials/default.aspx?speaker*
Disallow: /de-de/corporate/rechtliche-hinweise/impressum_de.aspx

Sitemap: http://www.microsoft.com/en-us/explore/msft_sitemap_index.xml 
 
 
 
 
 
User-agent: ia_archiver
Disallow: /

User-agent: *
Disallow: /dynamic/
Disallow: /dropbox/
Disallow: /templates/
Disallow: /myaccount/
Disallow: /api/
Disallow: /basicapi/
Disallow: /filedrop/ 
 
 
 
 
User-agent: *
Disallow: /radar
Disallow: /audio_file
Disallow: /dashboard
Disallow: /x
Disallow: /svc/account
Disallow: /dashboard/notes
Disallow: /customize
Disallow: /impixu
Disallow: /liked
Disallow: /search/*?before=*
Disallow: /tagged/*?before=*
Disallow: /search/*?language=*
Disallow: /tagged/*?language=* 
 
 
 
 
User-agent: *
Disallow: /web
Disallow: /webans?
Disallow: /maps?
Disallow: /pictures?
Disallow: /allabout?
Disallow: /shopping?
Disallow: /touchWeb?
Disallow: /ar?
Disallow: /news?
Disallow: /youtube
Disallow: /touch/
Disallow: /wiki
Disallow: /web-explore
Disallow: /answers
Disallow: /web-answers
Disallow: /web-question
Disallow: /ans
Disallow: /settings
Disallow: /ref

User-agent: Mediapartners-Google
Disallow:
User-agent: Teoma 
Disallow:
User-agent: netseer
Disallow: 
User-agent: AdsBot-Google
Disallow:

Sitemap: http://www.ask.com/sitemap.xml
Sitemap: http://www.ask.com/question/sitemap_index.xml
 
 
 
 
#Google Search Engine Robot
User-agent: Googlebot
Allow: /?_escaped_fragment_

Allow: /?lang=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/followers
Disallow: /*/following

Disallow: /account/not_my_account

#Yahoo! Search Engine Robot
User-Agent: Slurp
Allow: /?_escaped_fragment_

Allow: /?lang=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/followers
Disallow: /*/following

Disallow: /account/not_my_account

#Yandex Search Engine Robot
User-agent: Yandex
Allow: /?_escaped_fragment_

Allow: /?lang=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/followers
Disallow: /*/following

Disallow: /account/not_my_account

#Microsoft Search Engine Robot
User-Agent: msnbot
Allow: /?_escaped_fragment_

Allow: /?lang=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/followers
Disallow: /*/following

Disallow: /account/not_my_account

# Every bot that might possibly read and respect this file.
User-agent: *
Allow: /?lang=
Allow: /hashtag/*?src=
Allow: /search?q=%23
Disallow: /search/realtime
Disallow: /search/users
Disallow: /search/*/grid

Disallow: /*?
Disallow: /*/followers
Disallow: /*/following

Disallow: /account/not_my_account

Disallow: /oauth
Disallow: /1/oauth

# Wait 1 second between successive requests. See ONBOARD-2698 for details.
Crawl-delay: 1

# Independent of user agent. Links in the sitemap are full URLs using https:// and need to match
# the protocol of the sitemap.
Sitemap: https://twitter.com/sitemap.xml
 
 
 
 
 
User-agent: *
Allow: /_/scs/
Allow: /_/apps-static/
Allow: /_/explore
Allow: /_/initialdata
Allow: /_/socialgraph/lookup/hovercards
Disallow: /_/
Disallow: /s/
Sitemap: http://www.gstatic.com/s2/sitemaps/profiles-sitemap.xml
Sitemap: http://www.gstatic.com/communities/sitemap/communities-sitemap.xml 
 
 
File robots.txtadalah file teks yang menghentikan perangkat lunak perayap web seperti Googlebot agar tidak merayapi laman tertentu di situs Anda. File ini pada dasarnya merupakan daftar perintah, seperti Allow dan Disallow, yang memberi tahu perayap web tentang URL yang dapat atau tidak dapat diambil. Jadi, jika URL tidak diizinkan dalam robots.txt, URL tersebut dan kontennya tidak akan muncul di hasil Google Penelusuran.
Anda hanya memerlukan file robots.txt jika situs menyertakan konten yang tidak ingin disertakan dalam pengindeksan Google atau mesin telusur lain. Untuk memungkinkan Google mengindeks seluruh situs, jangan buat file robots.txt (bahkan yang kosong sekalipun).
Untuk menguji URL yang dapat dan tidak dapat diakses Google di situs web, coba gunakan Penguji robots.txt.

Memahami batasan robots.txt

Sebelum membuat robots.txt, Anda harus mengerti risiko penggunaan metode pemblokiran URL ini. Terkadang, Anda dapat mempertimbangkan mekanisme lain guna memastikan URL tidak dapat ditemukan di web.

  • Pastikan informasi pribadi Anda aman

    Perintah dalam file robots.txt bukanlah peraturan yang harus dipatuhi semua perayap; sebagai gantinya, lebih baik menganggap perintah ini sebagai pedoman. Googlebot dan perayap web tepercaya lainnya mematuhi petunjuk yang ada di file robots.txt , namun perayap lainnya belum tentu. Oleh karena itu, sangat penting untuk mengetahui konsekuensi berbagi informasi yang Anda blokir dengan cara ini. Untuk menjaga keamanan informasi pribadi, sebaiknya gunakan metode pemblokiran lain seperti file pribadi yang dilindungi sandi pada server Anda.

  • Gunakan sintaksis yang benar untuk setiap perayap

    Meskipun perayap web tepercaya mengikuti petunjuk dalam file robots.txt, beberapa perayap dapat mengartikannya secara berbeda. Anda perlu mengetahui sintaksis yang sesuai untuk mengatasi perayap web yang berbeda karena beberapa di antaranya mungkin tidak memahami perintah tertentu.

  • Blokir perayap agar tidak merujuk ke URL Anda di situs lain

    Meskipun Google tidak akan merayapi atau mengindeks konten yang diblokir dengan robots.txt, kami mungkin tetap menemukan dan mengindeks informasi tentang URL yang tidak diizinkan dari tempat lain di web. Akibatnya, alamat URL dan, kemungkinan, informasi lain yang tersedia secara publik seperti teks tautan dalam tautan ke situs masih dapat muncul di hasil penelusuran Google. Anda dapat menghentikan URL agar tidak muncul di hasil Penelusuran sepenuhnya dengan menggunakan robots.txt yang dikombinasikan dengan metode pemblokiran URL lain seperti file yang dilindungi sandi di server, atau menyisipkan tag meta ke dalam HTML. 
 
 

Share this:

ABOUT THE AUTHOR

Ceyron Louis

Hey , Silahkan mampir ya! Jangan ragu kalo mau ngasih komentar.. . . . .. .

    Blogger Comment
    Facebook Comment