Log in

No account? Create an account
getting great big lists of friends - LiveJournal Client Discussions [entries|archive|friends|userinfo]
LiveJournal Client Discussions

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

getting great big lists of friends [Oct. 11th, 2001|02:39 pm]
LiveJournal Client Discussions


i've been toying around with the idea of writing an lj app, not a client for posting and editing friends and all that, but something completely different. i've been fascinated by the interconnected aspect of LJ, and i think it would be great to have an app that mapped it out for you in a number of ways. such as:

1) showing the number of degrees of seperation between you and any other user, and what that path is
2) showing you statistical info such as the average number of hops it takes you to get to every other user, and showing the top ranked people - the "center" of the LJ universe
3) other silly things, like a graphical map of your friends and their locations... the ability to map your friends on it, your friends' friends... etc

the hardest part of writing this app would be getting an entire database of lj users and the stuff listed in their user profile. that could be kind of hairy. i can think of two ways of doing this:

1) asking the lj developers to provide an image of all the user profiles. but i don't think this would really be appropriate. after all, what i'd be writing would essentially be a toy.

this leaves me with:

2) write a mini app that slowly spiders across all the user info's it can find, using "friends" and "friends of" listings to make its own image. i see two problems with this.
- it would be an incomplete listing. it wouldn't find journals of people who aren't friends of anyone, because it would have no way of getting to them.
- in order to be polite, it would probably take a month or so for the spider to get user info on some 130,000 active users.

i think people would have a lot of fun playing around with something like this, and i know i'd have a lot of fun writing it. i'd appreciate any suggestions or comments as to how i might work through this hurdle.

[User Picture]From: suppafly
2001-10-11 01:49 pm (UTC)
propigate your own database with the information using some kind of webspider or something..
(Reply) (Thread)
[User Picture]From: xb95
2001-10-11 02:29 pm (UTC)
I have considered this idea and even included the bare thoughts for it in LochJournal back when it was still LJU, but I canned the idea when I realized how hard of a hit it is on the DBs to do that, and how fast Brad would sitelock me. ;) But, if you're going to do this, why not just use userinfo.bml to convert their userid to names and stuff? I think userinfo.bml has a method to convert userids to usernames. I think. Hmm. Either way, good luck. You will probably want to ask Brad first. :P
(Reply) (Thread)
From: evan
2001-10-11 02:43 pm (UTC)
We've done a lot of graphical / path-finding analyses, and we've found that they're basically unfeasable.

Graphing: Even graphing the relationships between just the list of all of the people on my friends list was a 2000x2000 pixel image littered with lines, and it brought my 800mhz Athlon to its knees. (I was using graphviz, which is pretty state-of-the-art in terms of graphing software.)

Shortest path: Unless you can propose a previously-unheard of algorithm, anything you do will be a horrible hit on the servers.
There are almost 400,000 users, and the hardcore users have over 100 friends each; that's a possible 40,000,000 links.

Most importantly (and also a response to your earlier lj_dev post): some people hide their friends list, for privacy purposes. (Like me.)

Finally, if you spider the database, I think dormando will murder you in your sleep.
We used to export friends list data for interested parties like you, but some people started abusing it.

Mail me privately and I'll give you some data to play with.
(Reply) (Thread)