# Robots.txt for www.nla.gov.au # # Disallow gathering of /apps and /servlet where java apps live. # # Tomasz Ciolek 29/05/2000 # Lesley Bezear 2/7/2004 added from Ultraseek filter and sorted second section alphabetically User-agent: * Allow: /apps/oaicat Disallow: /apps # Site java apps servlets Disallow: /servlet # BAT java servlets Disallow: /admin # added for Lesley Bezear Disallow: /cgi-bin Disallow: /abn # inactive site Disallow: /aclis # inactive site Disallow: /apais/thesaurus/html Disallow: /apais/thesaurus/htmla Disallow: /apais/thesaurus/htmltop # These 3 to disallow indexing of individual thesaurus terms Disallow: /css Disallow: /download Disallow: /ecommerce Disallow: /ferguson Disallow: /images Disallow: /initiatives/sg/metadata.xls Disallow: /jslib # Javascript files Disallow: /kinetica/rap # Excluded as only retained for 6 months Disallow: /linkscan Disallow: /list-archives Disallow: /manual Disallow: /nacq # Exclude new acquisitions lists Disallow: /news Disallow: /nla/pandora Disallow: /nla/staffpaper/archive Disallow: /nla/tutorial Disallow: /nph-arch # For PANDORA Disallow: /parchive # For PANDORA Disallow: /pathways/jnls/austjnls # Duplicates entries Disallow: /piview Disallow: /policy/annrep01/textonly # Duplicates content Disallow: /publictest Disallow: /software Disallow: /structuralmap Disallow: /stylesheets Disallow: /sys Disallow: /temp Disallow: /templates Disallow: /test Disallow: /unavail Disallow: /users Disallow: /wgroups User-agent: AGLS Meta: /harvest/aglsharvest.txt