I like Gnutella, but I think Freenet has the potential to be even better. The only problem is, Freenet wants me to push all my files out on the network before people can get them rather than letting people request whatever files I have that they want (like Gnutella). Here I present ideas on how to do Gnutella-like activities over Freenet. This is not completely fleshed-out, but it should give a good idea what I'm up to.
The need for a Freenet/Gnutella hybrid
I want to share my files using Freenet, but I don't want to insert them all. I have two reasons:
What I want to do is provide to Freenet a list of files that I'm offering, and insert them on request. My plan is to do this with a special client application that connects to my local Freenet server node, and interacts with the network through it. The client, which I'm calling a "chicken", will provide info about what I'm sharing and answer requests from people who want it.
How Chicken works
In general, talking chickens post messages in predictable places on Freenet. Listening chickens poll those places to see if anyone has spoken. We use Freenet to cache things such as search results, file meta data, and the actual files requested.
Messages
First, let me define a few variables.
Messages generally contain URL encoded strings like 'q=britney&max=256'. They're nice and easy to parse.
Requesting a file - chicken/request/date/sha1
What's in this key is unimportant. It's a flag. Each chicken has computed a sha1 hash for each file it is sharing. It polls for requests of those hashes and answers them when it sees one.
Announcing a file - chicken/info/sha1/index
This will contain a URL encoded string with meta info about a file inserted. When a client successfully inserts a file, it will also insert an info file about it. That file must contain the Freenet key for the file inserted, and may contain any other meta data the client wants to disclose.
Clients who have requested a file should poll for these info items to see when the file has been inserted and how to retrieve it.
Searching - chicken/query/time/index
What's in this key is the thing the user is searching for. It should contain at least a simple keyword query and a maximum number of results, but it can also have other fields to match.
Search results - chicken/found/keyword/index
In this case, the index is a little special. To avoid getting a huge list of duplicate results, the index is the sha1 of the result file mod the max results. So, if you have a file whose sha1 is x, and the search requested at most y results, the index should be (x mod y) + 1. The client searching for these results has to poll the whole results space (1 .. max) to get all the results.
On the plus side, results are cached in Freenet. They're still there the next time someone else wants to search for the same thing. It's a good idea to look for previous search results for your keyword before posting a new search.
Each result is a URL encoded line with info for one file. At a minimum it should contain the file's sha1 hash, name, and size. It can also contain any other file meta data.
File announcements - chicken/list/id/date/index
The id part is a unique ID for a client. You get this ID from a client's announcement. This key contains a list of all the files the client is sharing. Each line is a URL encoded string with the same information that would be found in a search result.
Other clients can look through client announcements and use them to fetch the lists of files those clients are sharing. By caching those lists, clients can do searches locally instead of over the network.
Clients might also insert another key which contains not only their own files, but all the other files they've heard about. The down side, is that could be a very large list for a client that's been in operation a long time.
Client announcements - chicken/announce/time/index
I haven't completely decided what this message contains. Each client should give an ID or a key where another client can find its file announcement. It could also contain statistics about the client, such as how many files it's sharing, but pointing to a file announcement is the only real reason to do a client announcement.
A little implementation
If I ever get time to implement this, I'll do it in Perl. I plan to have a script that forks to do several jobs:
I'd have the master run a HTTP::Daemon loop and present the UI to the user via a browser that talks to that daemon as if it were some CGI scripts.
I'd communicate with the Freenet server with RPC::XML.
I've already written some code to index files and make the indices persist across executions (so I'm not blowing a lot of time computing sha1 hashes over and over).
Generally good things
I get to cache shared files and search results in Freenet. Every user is anonymous because of Freenet. Also, since we're polling for everything, there's no chance of getting flooded off the network. If you can't keep up, the messages you can't get to are effectively dropped, but your connection is never saturated. Also, it's easy to do things as quickly or as slowly as you want; nothing is being forced down your pipe.
Generally bad things
I like Freenet, but it's still alpha/beta/whatever. I've found it hard to set up. Adding another layer on top of that will be a barrier to users.
Last word
I don't really have a lot of time to implement this, but I welcome suggestions and discussion. I'm not convinced that I haven't reinvented the wheel, and I'm not sure that all this stuff will work well in a real network. Any gaps you can fill in would be appreciated.
Votes: 10