Capt. Horatio T.P. Webb

Crawling the Web in ASP (spiders and bots)

Parks -- SPRING 2009
Last Updated 6 PM 5/18/2009

The most common activity on the web is searching the web using the large variety of search engines. Companies like Google who do this for a living spend a tremendous amount of effort in wandering around the web, finding and storing the search results in an easily accessible fashion. This processes is generally a trade secret of the firms that perform this service. However, it is NOT rocket science.

We can employ ASP to do this task in VERY limited way using server side AJAX. Suppose we wish to use the MIS 4372 syllabus as a starting point and develop a spider that crawls the homepage, looks around and then starts searching all the pages that appear as links on the page. It is just a matter of storing the results of the searches and their URLs.

The code uses the IE version of AJAX to read the page's content:

set xmlhttp = CreateObject("MSXML2.ServerXMLHTTP")
xmlhttp.open "GET", full_url, false
xmlhttp.send ""
page_content=xmlhttp.responseText

It sends no data to the page -- it just retrieves the page and stores it in the variable "page_content". Now the entire page is just a string. The code (as shown below) will search the course syllabus page at http://www.bauer.uh.edu/parks/disc4372.htm (the string "page_content") and retrieve all the links. It does this by:

finding each occurrence of "<a href="
stripping out the address between the quotes
storing the URL if either the href:
1. has an HTTP prefix and the URL equal to "http://www.bauer.uh.edu/parks/"
2. if no HTTP/URL prefix exists, then it must be a local file in the same directory as the syllabus.
  So it adds "http://www.bauer.uh.edu/parks/" to the local filename

Once the list of local URL's has been made, it uses the same AJAX to retrieve each page that is linked, stores the page content as a string then tries to find the desired search term (in the code this stored in a variable named "search_for").

<html> <body> Mikey's Spidey Search: <% t1=Timer() '*** store the start time dim u(2000) server_path = "http://www.bauer.uh.edu/parks/" file_name="disc4372.htm" search_for="Snidley Whiplash" response.write "Searching for all local pages (links) that appear on the page: " response.write server_path+file_name response.write " (local pages means all page that are at: " response.write server_path ' '*** ucase the names ' server_path=ucase(server_path) file_name=ucase(file_name) full_url=server_path+file_name ' '*** set up the IE ajax ' set xmlhttp = CreateObject("MSXML2.ServerXMLHTTP") xmlhttp.open "GET", full_url, false xmlhttp.send "" page_content=xmlhttp.responseText len_page=len(page_content) page_content=ucase(page_content) response.write "Length of page="+formatnumber(len_page,0)+" bytes" response.write "Searching for the string: "+search_for search_for=ucase(search_for) response.write "Found the following links on the page (all links shown): " start_of_url="<a href="+chr(34) start_of_url=ucase(start_of_url) cur_char=1 urls_found=0 urls_stored=0 non_local_urls=0 ' '***find and extract all the links ' while cur_char > 0 url_loc=Instr(cur_char,page_content,start_of_url) if (url_loc > 1 ) then urls_found=urls_found+1 first_quote=Instr( url_loc , page_content , chr(34) ) cur_char=first_quote + 1 '*** skip past the first quote end_quote=Instr(cur_char , page_content , chr(34) ) num_chars = end_quote - first_quote -1 temp_url=mid( page_content , (first_quote+1) , num_chars ) response.write " "+cstr(urls_found)+". "+ temp_url if Instr(1,temp_url,"HTTP") then ' '*** link has an HTTP prefix ' if Instr(1, temp_url,server_path) then ' '*** store the local HTTP page ' urls_stored=urls_stored+1 u(urls_stored)= temp_url response.write " stored as local with no revision" else ' '*** not local ' non_local_urls=non_local_urls+1 response.write " NOT LOCAL" end if else ' '*** no HTTP prefix, must be local, so add the server and path and store it ' temp_url=server_path+temp_url urls_stored=urls_stored+1 u(urls_stored)= temp_url response.write " revised to: "+temp_url+"" end if cur_char=end_quote+1 else cur_char=0 end if wend response.write "Total URLs (links) found on "+server_path+file_name+" = "+cstr(urls_found) response.write " Local URLs stored = "+cstr(urls_stored) response.write " Non-local URLS = "+cstr(non_local_urls) response.write "Searching the "+cstr(urls_stored)+" local pages for "+search_for+" " find_count=0 pages_searched=0 for i=1 to urls_stored pages_searched = pages_searched + 1 xmlhttp.open "GET", u(i), false xmlhttp.send "" page_content=xmlhttp.responseText len_page=formatnumber(len(page_content),0) page_content=ucase(page_content) if Instr(1, page_content, search_for) > 0 then find_count=find_count+1 response.write " "+cstr(pages_searched)+". found on "+u(i)+"" response.write " (byte count = "+cstr(len_page)+")" else response.write " "+cstr(pages_searched)+". NOT found on "+u(i) response.write " (byte count = "+cstr(len_page)+")" end if next set xmlhttp = nothing t2=Timer() '*** store end time search_time=cdbl(t2)-cdbl(t1) search_time=formatnumber(search_time,4) response.write "Found '"+search_for+"' "+ cstr(find_count) +" times on "+cstr(pages_searched)+" pages" response.write " Search Time ="+ cstr(search_time) +" seconds" %> </body> </html>

Execute this search here (i.e., find "Snidely Whiplash" by crawling my MIS 4372 syllabus).

Because this page is on the syllabus, this page is searched (along with lotsa others). Because this page contains the value of the "search_for" variable in the code shown above (search_for="Snidely Whiplash"), this page is the only one identified as a successful search.

While you might assume that you could write this code and execute it in a browser -- You CANNOT do this. This is because many "cross-site scripting" features have been disallowed by the browser makers. As this example demonstrates, it is possible to do some things inside your own sandbox. Further, without this server-side capability, search engines would be unable to automate the web crawling ability that allows them to search and index the web.

Be careful what you ask for...