?

Log in

No account? Create an account
USENET-style LJ client? - LiveJournal Client Discussions [entries|archive|friends|userinfo]
LiveJournal Client Discussions

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

USENET-style LJ client? [Aug. 23rd, 2004|12:48 am]
LiveJournal Client Discussions

lj_clients

[pyesetz]
I've been looking around at the available LJ clients and not finding one that does what I want.  Do I have to write my own?  Most clients seem to think that their job begins and ends with helping me write WYSIWYG journal entries, but I find the LJ web interface is fine for that.  What I want a client for is
  • Tell me when new comments have been attached to my friends' journal entries.
  • Tell me which comments are ones I haven't read yet.  (Currently I have to squint at the dates and usually just end up rereading comments I've already seen.)
It seems to me that this is most easily accomplished by adapting a USENET reader, such as Pan, using the following analogy:
LiveJournal USENET
User login  → Server
Friend  → Newsgroup
Journal entry  → Thread-starting post
Comment  → Reply post
And so forth.  Of course, there are some differences (deleting a comment has different semantics from cancelling a reply, thread-starting posts can be edited if yours, etc.) but I suspect a decent job could be achieved by "just" replacing Pan's backend and leaving the UI pretty much the same.

And I completely out to lunch on this?
linkReply

Comments:
[User Picture]From: vanbeast
2004-08-22 06:48 pm (UTC)
The big problem is that none of that data is available in a machine-friendly format. You'd have to scrape all sorts of HTML pages, which is a big no-no.

I suppose you could do syncitems for your own journal, but that seems over the top and only works for your personal journal.
(Reply) (Thread)
[User Picture]From: pyesetz
2004-08-23 08:15 am (UTC)
Where is it stated that mechanical screen-scaping is bad? In the basic Bots policy page it just says "don't make repeated requests for the same URL over a short time". Currently I'm reloading my "friends" page and manually scraping the screen for color-changes in the "nn comments" links. A bot that automates doing the same thing at the same rate is somehow unethical???
(Reply) (Parent) (Thread)
[User Picture]From: bleything
2004-08-23 08:21 am (UTC)
The Bot Policy page says You are encouraged to use these resources instead of "screen-scraping" user pages.

If you have a look at the post in lj_dev located here, you'll find more info.
(Reply) (Parent) (Thread)
[User Picture]From: xb95
2004-08-22 06:49 pm (UTC)
There is no way to determine if new comments have been posted to entries on your friends page or to determine what the posts on your friends page are without screen scraping, which is expressly disallowed at this point.

As long as you follow the guidelines previously set out in lj_dev about client policy (go back a month or two), you'll be fine.

It'd be nice and cool, but our systme really just doesn't support it right now.
(Reply) (Thread)
[User Picture]From: lordindra
2004-08-22 06:51 pm (UTC)
Thats actually an interesting idea.

Hmm. You might want to consider basing it off of Mozilla. You can take Deepest Sender, and hack that to fit the Mozilla Mail/Thunderbird UI. Most of the heavy lifting code has already been written, you just have to figure out how to weld the pieces together. The ForumZilla extension might also have useful code you can use... check on the licenses though, I'm not up on the licensing practices for extensions, and any conditions the main mozilla license imposes on licenses of extensions.
(Reply) (Thread)
(Deleted comment)
[User Picture]From: pyesetz
2004-08-23 04:33 am (UTC)

Re: NNTB, Zilla #1576

Great!  Glad to hear that other people think this is needed.  But I'm finding it hard to follow the politics implied by Zilla #1576: did bradfitz reject your patch, despite his comments there that he seemed to like the getcomments approach?  It's interesting that there are so many undocumented extensions to the client protocol, which seems like an odd situation for an open-source project.

I note that the export_comments.bml method does not take a username argument, implying that it works for one's personal journal only.  And why are users only "allowed" to download for journals they administrate, yet via the friends page they can manually download comments for non-administered journals?  If this is a server-load issue, a rate-limit on comment downloads would seem a lot friendlier than an outright ban on using the protocol to automate what I end up doing by hand multiple times a day (reload friends page, look for "nn comments" links in black indicating that the number has changed, click on link, scan page looking for comments I haven't read yet -- waste, waste, waste!).
(Reply) (Parent) (Thread)
[User Picture]From: vanbeast
2004-08-23 05:16 am (UTC)

Re: NNTB, Zilla #1576

At the risk of getting into another argument, generally speaking users are only allowed to download content they own. Sure, anybody can go and save an HTML page, but our policy has been to only allow programmatic access to those posts and comments that belong to the user.

Basically, we don't want to provide a way for someone to take a copy of another user's journal.
(Reply) (Parent) (Thread)
[User Picture]From: pyesetz
2004-08-23 08:25 am (UTC)

Re: NNTB, Zilla #1576

I believe ownership of comments stays with the commenter, not the owner of the journal it's posted in--at least in terms of who would be asked for permission before comments can be republished.

My journal is marked "Do not block spiders".  Does this have any relevance?

I really like the rate-limiter idea, although I don't know how practical it might be to implement.  A per-IP limit of 4 comments/sec, 60/min, 1000/hr would be plenty for legitimate users, yet annoyingly inadequate to the bad guys.

I originally came to LJ because the USENET groups I used to read have become ghost towns as everyone migrates over here.  But, because of this comment situation, LJ does not offer a service equivalent to that of the competitor it has killed off.
(Reply) (Parent) (Thread)
[User Picture]From: vanbeast
2004-08-23 08:44 am (UTC)

Re: NNTB, Zilla #1576

I believe ownership of comments stays with the commenter, not the owner of the journal it's posted in--at least in terms of who would be asked for permission before comments can be republished.

That's a bit of a grey area. My understanding has been that comments attached to an entry are derivative work, so therefore ownership remains with the poster of the original entry. Regardless of where the truth lays on the matter, it's something we have to at least give consideration.

Our spider blocking is meant for off-site spiders, so wouldn't have relevance (as it stands now) to the proposed feature. That doesn't mean that it couldn't!

At one point there was discussion about a flag you could set on your journal that would allow others to download it... for instance, I hoenstly don't care if people want a local backup of my journal. That just means people like what I write! So I could set that flag, and people could go to town.
(Reply) (Parent) (Thread)
[User Picture]From: novalis
2004-11-20 05:02 pm (UTC)

Re: NNTB, Zilla #1576

My understanding has been that comments attached to an entry are derivative work, so therefore ownership remains with the poster of the original entry.

Rarely. A "criticism" or "comment" to something is usually not a dervative work (a work "based upon" another work). When it is, it constitutes fair use in nearly all cases.
(Reply) (Parent) (Thread)
From: emarkienna
2004-08-23 01:28 pm (UTC)

Re: NNTB, Zilla #1576

I originally came to LJ because the USENET groups I used to read have become ghost towns as everyone migrates over here. But, because of this comment situation, LJ does not offer a service equivalent to that of the competitor it has killed off.

Just to say that I share these feelings..

I've often thought that NNTP is the best system for dealing with online discussions. It allows customised clients which can access the information in a machine readable format. Unlike email clients and mailing lists, you get things like threading.

But then, Usenet gets full of spam, and few people go there. There's no reason why you couldn't have NNTP servers that weren't part of Usenet, and required sign ups and had moderators in the same way that LJ and other web forums do - but sadly that hasn't happened (at least, not on a wide scale).

I think the web is just about the worst medium for conducting online discussion - you're stuck with online tools which are often slow, basic and limited compared to what offline ones could be, and you're dependant on the central server being up all the time. *Hits select-all/copy in case my web browser crashes*
(Reply) (Parent) (Thread)
From: emarkienna
2004-08-23 01:21 pm (UTC)

Re: NNTB, Zilla #1576

At the risk of getting into another argument,

Okay;) I'll accept that RSS is allowed by the TOS, but that new custom download methods wouldn't be.

So would it be possible to use the same workaround for comments, as RSS is used for accessing entries? That is, by accessing a particular URL, the web server returns the comments not as HTML web pages, but in a machine readable format (which could be something like RSS, or an entirely custom format). This would satisfy the TOS restrictions of being over "the web" just as much as RSS, surely.

As to your later comment about copyright belonging to the journal owner.. well, IANAL etc, but I thought it was pretty standard that copyright for anything by default belongs with the person who writes it. The only way that can be transferred is if permission is given in any way. As I type this, I don't see any messages telling me that the copyright will be owned by the owner of this journal entry..

I can't believe that something is a derivative work merely because it responds to that (and if that were true, I would be breaking copyright law by making a comment in the first place).
(Reply) (Parent) (Thread)
[User Picture]From: bleything
2004-08-23 01:28 pm (UTC)

Re: NNTB, Zilla #1576

Regarding the "workaround" you mentioned, I think you're completely missing the point of what I've been saying all along. I'm simply suggesting that we need to be careful what we do with other people's data, particularly when there are binding agreements in place. Lame "workarounds" to satisfy these agreements are just that: lame. The correct solution is to make a new agreement.

...I thought it was pretty standard that copyright for anything by default belongs with the person who writes it.

There are a number of circumstances under which this isn't true. One of the grey areas is email, because it's difficult to assign ownership to a conversation under which people made equal contribution. LiveJournal posts are the same way. There isn't a lot of legal precedent for it right now, so it's hard to say exactly what the case is. Because of that, it's best to play things conservatively.

and if that were true, I would be breaking copyright law by making a comment in the first place

How so? There's nothing that says you can't make a derivative work, just that you don't own 100% of the rights to it.
(Reply) (Parent) (Thread)
From: emarkienna
2004-08-23 01:35 pm (UTC)

Re: NNTB, Zilla #1576

Regarding the "workaround" you mentioned, I think you're completely missing the point of what I've been saying all along. I'm simply suggesting that we need to be careful what we do with other people's data, particularly when there are binding agreements in place. Lame "workarounds" to satisfy these agreements are just that: lame. The correct solution is to make a new agreement.

Well I don't think RSS is a workaround in the bad sense, just that it was another way of achieving the same thing, whilst (supposedly) not causing the problems with the TOS.

Either RSS fits with the TOS, in which case fine, but if not, surely these problems already exist with the RSS that is being broadcast on everyone's LJ? (Was there discussion of these issues when RSS was implemented?)

I'd agree with making a new agreement for all new entries/comments, it seems this would be the best way to avoid any potential problems..

How so? There's nothing that says you can't make a derivative work, just that you don't own 100% of the rights to it.

Well I meant in the sense of creating a derivative work and then publishing it (which surely isn't allowed), although I see that posting on the original journal entry could be seen as not counting as distributing it.
(Reply) (Parent) (Thread)
[User Picture]From: bleything
2004-08-23 01:44 pm (UTC)

Re: NNTB, Zilla #1576

My position on all of this is in the thread in lj_dev, so I won't repeat it here. I will say this: it is wrong to look for ways to comply with the letter of the TOS (and the law) while evading the spirit thereof. Not that I'm suggesting that's what you were doing, but that's what the term "workaround" implies in this sense.

In this case, we're talking about a thing that would be useful to many people and is difficult to find objectionable. Again, let me be clear: I don't oppose the functionality requested here, I'm just trying to make sure people consider all the ramifications of what they're talking about. I'm trying to foster reasoned discussion about more than simply the technical aspect of things.
(Reply) (Parent) (Thread)
From: emarkienna
2004-08-25 11:44 am (UTC)

Re: NNTB, Zilla #1576

Well, my viewpoint is that it doesn't violate even the spirit of the TOS (so it's not like I'm trying to evade it), but I realise that others may think it does.

I guess my curiosity is then, why didn't these issues get discussed when RSS was introduced - or if they were, what was decided.
(Reply) (Parent) (Thread)
From: snej
2004-08-23 03:41 pm (UTC)

Re: NNTB, Zilla #1576

I've never felt that argument held much water. The distinction between HTTP fetches of regular HTML pages by an accredited web browser, vs any other form of downloading, makes little sense.

Worse, the cows have already left the barn. It is already possible to take a copy of someone's journal. My newsreader app is doing it every half hour as it fetches my friends' Atom feeds and appends the posts to its database.

It is public content. Redistribution of that content is legally iffy, but I can store that content for my own personal use, via my web browser or a newsreader or any other mechanism. Any LJ user who thinks that's wrong is laboring under a common but regrettable misconception about what "public" means.
(Reply) (Parent) (Thread)
[User Picture]From: cryo
2004-08-22 08:42 pm (UTC)
The only solution for this is to have the LJ client<->server protocol support it. I'd be happy with having article reference #'s returned in a metadata for the entry. As for maintaining a .newsrc for read 'entries/community entries', you're looking at basically eliminating the web interface to LJ, which does two things:

1) pretty much eliminates the LiveJournal look-n-feel(tm).
2) moves the onus of responsibility more to the client [where I feel more needs to be done on the server side to keep track of things like this (since the server requires you to login and keeps track of your other preferences anyway (the scope of LJ changed when it went from individual journals (email-like paradigm) to having communities (nntp-like paradigm))].

brainsplode.
--クライオ

(Reply) (Thread)
[User Picture]From: hythloday
2004-08-22 09:34 pm (UTC)
I actually have an LJ->NNTP server written in Python, which I can't release because the method it uses to get comments is to screen scrape an XML s2-styled comments page, which is verboten. Something I have on the back burner is to convert the backend from going straight to LJ into a module which reads pages from the mozilla or konqueror cache, which have an embedded XML section. (So the user can use a browser normally and also the newsreader.)

That said, generating s2 code that would do this for each user is probably a monstrous hack, and I don't know anything about the mozilla cache. (And working on something that only worked on konqueror is a bit silly, imo. :-)
(Reply) (Thread)
[User Picture]From: mattp
2005-11-10 11:07 pm (UTC)
I've written a basic engine for this, based on papercut (which is a NNTP presenation suite for a filestore) and my own fetcher program. It's in Python and in alpha stage right now. Would you be interested?
(Reply) (Thread)