Learn More





DNN Community Blog

The Community Blog is a personal opinion of community members and by no means the official standpoint of DNN Corp or DNN Platform. This is a place to express personal thoughts about DNNPlatform, the community and its ecosystem. Do you have useful information that you would like to share with the DNN Community in a featured article or blog? If so, please contact .

The use of the Community Blog is covered by our Community Blog Guidelines - please read before commenting or posting.

A simple SEO tip - create a robots.txt


I discovered something interesting this week using the Google webmaster tools & thought I's share it.  I had a look at the diagnostic tools (many of these are only available if you "verify" your site), and Google listed several hundred Urls (mostly forum posts) that it had not indexed, with an explanation that it did not index the page because it did not find a robots.txt file for my site.

A robots.txt file is used to provide instructions to web "robots" like the one that Google uses to index web sites.  The absence of a robots.txt is generally meant to tell robots to go ahead and index everything, but from the results I saw in Google diagnostics, it appears that you will get a better indexing result from Google by explicitly including a robots.txt file in your site.

Here's my robots.txt:  I ended up adding all the sub-folders in DNN except for /Portals, even though many of them would never get linked to anyway.  (For example, including /bin is a bit of overkill, since no page would ever have a link to its contents). 

User-agent: *
Disallow: /Admin/
Disallow: /App_Browser/
Disallow: /App_Code/
Disallow: /App_Data/
Disallow: /App_GlobalResources/
Disallow: /bin/
Disallow: /Components/
Disallow: /Config/
Disallow: /Controls/
Disallow: /DesktopModules/
Disallow: /Documentation/
Disallow: /Install/
Disallow: /js/
Disallow: /Providers/
Disallow: /Resources/

A simple version of the above would be:

User-agent: *

I couldn't find any documentation to back this observation up, but after I added the robots.txt file above and waited a couple of days, the errors disapeared from my Google diagnostics page, and the pages were indexed.  If anyone else out there is using the Google webmaster tools & can check to see if their results match mine, I'd appreciate it if you'd let me know by posting a blog comment.


There are currently no comments, be the first to post one.

Comment Form

Only registered users may post comments.


2sic Daniel Mettler (125)
Aderson Oliveira (15)
Alec Whittington (11)
Alex Shirley (10)
Andrew Nurse (30)
Anthony Glenwright (5)
Antonio Chagoury (28)
Ash Prasad (22)
Ben Schmidt (1)
Benjamin Hermann (25)
Benoit Sarton (9)
Beth Firebaugh (12)
Bill Walker (36)
Bob Kruger (5)
Brian Dukes (2)
Brice Snow (1)
Bruce Chapman (20)
Bryan Andrews (1)
cathal connolly (55)
Charles Nurse (163)
Chris Hammond (203)
Chris Paterra (55)
Clinton Patterson (28)
Cuong Dang (21)
Daniel Bartholomew (2)
Dave Buckner (2)
David Poindexter (3)
David Rodriguez (2)
Doug Howell (11)
Erik van Ballegoij (30)
Ernst Peter Tamminga (74)
Geoff Barlow (6)
Gifford Watkins (3)
Gilles Le Pigocher (3)
Ian Robinson (7)
Israel Martinez (17)
Jan Blomquist (2)
Jan Jonas (3)
Jaspreet Bhatia (1)
Jenni Merrifield (6)
Joe Brinkman (270)
John Mitchell (1)
Jon Henning (14)
Jonathan Sheely (4)
Jordan Coopersmith (1)
Joseph Craig (2)
Kan Ma (1)
Keivan Beigi (3)
Ken Grierson (10)
Kevin Schreiner (6)
Leigh Pointer (31)
Lorraine Young (60)
Malik Khan (1)
Matthias Schlomann (15)
Mauricio Márquez (5)
Michael Doxsey (7)
Michael Tobisch (3)
Michael Washington (202)
Mike Horton (19)
Mitchel Sellers (28)
Nathan Rover (3)
Navin V Nagiah (14)
Néstor Sánchez (31)
Nik Kalyani (14)
Peter Donker (52)
Philip Beadle (135)
Philipp Becker (4)
Richard Dumas (22)
Robert J Collins (5)
Roger Selwyn (8)
Ruben Lopez (1)
Ryan Martinez (1)
Salar Golestanian (4)
Sanjay Mehrotra (9)
Scott McCulloch (1)
Scott S (11)
Scott Wilkinson (3)
Scott Willhite (97)
Sebastian Leupold (80)
Shaun Walker (237)
Shawn Mehaffie (17)
Stefan Cullmann (12)
Stefan Kamphuis (12)
Steve Fabian (31)
Timo Breumelhof (24)
Tony Henrich (3)
Torsten Weggen (2)
Vicenç Masanas (27)
Vincent Nguyen (3)
Vitaly Kozadayev (6)
Will Morgenweck (37)
Will Strohl (163)
William Severance (5)
Try Evoq
For Free
Start Free Trial
a Demo
See Evoq Live
Need More Information?