Is it possible to exclude certain URLs from being indexed?
Yes. If you are trying to exclude a page, simply untick the checkbox “Include in Search Engine” on the top of the page in edit mode. However, in some rare cases this will not work, for example when your page outputs CSV data instead of HTML. In such case, you will have to configure the WebManager Search Engine to ignore the content of the offending URL. The configuration file “parser.txt” determines which parser will be used for what URL-pattern. By sending the content for any URL through a parser, the WebManager Search Engine will get readable content that can be indexed. The special notation “-” in this case means “no parser”, which in turn means “discard the content, do not index”. For example, let us assume we want to exclude the following URL:http://www.gxdeveloperweb.com/Excluded/Do-Not-Index.htm We now tell the WebManager Search Engine for this URL, for all content types (“.*”), use no parser (“-“). This can be done by adding the URL on the top of the “parser.
