Custom Search

You are hereBlog / idea for extracting and archiving online content

idea for extracting and archiving online content


Posted on July 18, 2008 by Denis

Printer-friendly versionSend to friend

This morning a neato, million dollar idea popped into my head right after I woke up:

There's so much online content available these days but no efficient, user-friendly way to guarantee the availability of such content in the future. There needs to be more tools for us to extract and archive all forms of online content.

In other words, we're putting faith in the content providers and assuming that they won't remove the content we want from their servers. That in a nutshell is a lot of blind faith!

Going along this chain of thought, allow me to provide context and elaborate...

With content storage solutions decreasing in cost over time (i.e. hard drives are increasing in size but cost per byte ratio is getting cheaper), content storage is less of an issue to many online content providers as opposed to other costs like bandwidth. Consider the following 3 examples to prove my point:

  1. Google, Yahoo!, and MSN offer gigabytes++ of email storage these days as opposed to the measly < 100 MB or so a decade or so ago.
  2. Popular online social networking and media sites such as Facebook, Flickr, and YouTube offer essentially unlimited media content storage space (i.e. photos and videos).
  3. Web hosting businesses such as Bluehost.com (which this site uses) are beginning to offer unlimited content storage (while bandwidth and memory usage may still be limited).

However, consider the following examples to illustrate hypothetical problems but common scenarios:

  • An online news article can no longer be found after a certain amount of time.
  • Online media is taken down due to potential copyright violations.
  • Content cannot be found after a site goes out of business or its domain expires.
  • Content-rich forums close down due to lack of funding or managing.

In short, the old saying "all good things must come to an end" is pretty much still true in modern times.

On top of my head, I can think of the following hodgepodge of non-perfect methods to tackle this problem:

Method
Issues
Manually download content and store
  • Slow, inefficient
Convert static content into PDF and store*
  • Slow, inefficient
  • Expensive: Adobe Acrobat or other PDF creation software required
Online "Archivers": Archive.org, Google News, etc.
  • Still dependent on 3rd party; content extraction/storage not user controlled
  • Content not available on-demand (need to search)
Online content syndication services
  • Original content is usually still linked to 3rd parties
  • Content delivery oriented; not designed for storage

* A new feature idea for Adobe: Embedding and viewing non-static media content (videos and audio) in PDFs.

The interesting thing is that we have a ton of consumer-friendly content distribution and syndication services available (just click on the ShareThis link or look at the RSS Feeds block on the left), but no content extraction/archive solutions (at least that I know of) right now.

Any one that knows of a solution - please inform me. Otherwise, these ideas would sound nice to propose to some angel investors...

recent comments

Anonymous's picture

Great blog. Thanks.

Anonymous's picture

It is better to take sure shot stock tips or advices from someone before investing in Stocks and Commodities market instead of losing money and then regretting.
We at Capitalstars Financial Research Pvt. Ltd. Do this for you and help you make profit in stocks and commodities market. Our technical analysis team is an experienced one and provides sure shot stock recommendation with 90-95% accuracy rate. Avail our free trial service to know more.
Stock Tips

Anonymous's picture

It is better to take sure shot stock tips or advices from someone before investing in Stocks and Commodities market instead of losing money and then regretting.
We at Capitalstars Financial Research Pvt. Ltd. Do this for you and help you make profit in stocks and commodities market. Our technical analysis team is an experienced one and provides sure shot stock recommendation with 90-95% accuracy rate. Avail our free trial service to know more.

Foxkeh

View Denis Lam's profile on LinkedIn