notexOPENOverview
Notex/Systems/Detect and avoid problematic content
09.11LESSON 11 OF 12 · 1 MIN READ

Detect and avoid problematic content

Always free to read
SECTION VIDEODetect and avoid problematic content

Video coming soon. Start with the reading below.

COMING SOON

This section discusses the detection and prevention of redundant, meaningless, or harmful content.

  1. 1. Redundant content As discussed previously, nearly 30% of the web pages are duplicates. Hashes or checksums help to detect duplication [11].
  1. 2. Spider traps A spider trap is a web page that causes a crawler in an infinite loop. For instance, an infinite deep directory structure is listed as follows: www.spidertrapexample.com/foo/bar/foo/bar/foo/bar/...

Such spider traps can be avoided by setting a maximal length for URLs. However, no onesize-fits-all solution exists to detect spider traps. Websites containing spider traps are easy to identify due to an unusually large number of web pages discovered on such websites. It is hard to develop automatic algorithms to avoid spider traps; however, a user can manually verify and identify a spider trap, and either exclude those websites from the crawler or apply some customized URL filters.

  1. 3. Data noise Some of the contents have little or no value, such as advertisements, code snippets, spam URLs, etc. Those contents are not useful for crawlers and should be excluded if possible.
Progress is saved on this device.
KEEP EXPLORING

One note leads to another.

Back to the library →
⌕
Search across the entire library ⌘ K