Saving Files from a Website using VB.Net

    Date: 01/28/09 (Asp Dot Net)    Keywords: html, database, web

    I have a site that we are moving from Flat HTML into a Database using in house
    written VB.Net applications. The website is roughly 10,000 pages. The volume of
    pages however is not the problem. The Page HTML content is all now held in a
    database, Now I am looping through the pages and checking each page using regular
    expressions for either a href or img src tag so I can collect the URL for all
    images and attachments. Once I have these, i pass the URL (currently) to the
    HttpWebRequest.DownloadData which makes a request to the website for the url
    provided and then the attachment is downloaded and eventually written out using
    the WriteAllBytes() function. This works for one iterration and then times out.
    What i need to know is why does it time out and is there a better method of doing
    this? the files can be rather large that are getting moved the site is about 5.7gb
    and most of this will be down to diffrent file formats.

    Source: http://aspdotnet.livejournal.com/102801.html

« Using DataRelation in DataSet || Postback question »


antivirus | apache | asp | blogging | browser | bugtracking | cms | crm | css | database | ebay | ecommerce | google | hosting | html | java | jsp | linux | microsoft | mysql | offshore | offshoring | oscommerce | php | postgresql | programming | rss | security | seo | shopping | software | spam | spyware | sql | technology | templates | tracker | virus | web | xml | yahoo | home