Relevant Search on the Web with a Personal Crawler

I thought (and I strongly believe that somebody already thought about the same before me :D) that it would be great to have a search that's relevant only to you. With Google you can kind of personalize your searching experience, but I mean another thing.

Let's say:

• I've got hundreds of followers on Twitter and I follow a hundred people


• I'm on Facebook where I've got many friends


• I'm using Delicious for bookmarks


• I use Google Reader for keeping myself up to date


• I blog myself, and I receive comments on my posts


• etc.

In all the listed services/activities there is a substantial set of links present or produced either by me or by people in my network. These links form a slice of Internet that is relevant to me.

Intuitively, the part of Internet that is relevant to me is growing:

1. as I'm expanding my social graph


2. as I'm sharing links with others


3. as I'm actively participating on the Web, e.g. commenting

however, a big part of my subInternet is actually hidden (not public), e.g. links shared by friends on Facebook.

Now, imagine that you have your personal Web crawler (maybe plugged into Google). This way you could search in 3 modes:

1. The Web - as we are used today


2. The relevant part of the Web only - available via OAuth, Facebook Connect, etc.


3. Both

As an example consider the keyword "birthday" in all the three cases. It's pretty clear that the results will be quite different.