?

Log in

No account? Create an account
LJSM v.2.7 - LiveJournal Client Discussions [entries|archive|friends|userinfo]
LiveJournal Client Discussions

[ userinfo | livejournal userinfo ]
[ archive | journal archive ]

LJSM v.2.7 [Jan. 8th, 2006|01:46 am]
LiveJournal Client Discussions

lj_clients

[ati]
Hi,

There's a new version of LJSM, Livejournal backup tool.
Now it uses 'sessiongenerate' call of LJ client protocol for login instead of /login.bml screen scraping.
See http://www.offtopia.net/ljsm/index_en.html for download links, my previous post in this community for some discussion.
linkReply

Comments:
[User Picture]From: marksmith
2006-01-09 04:27 pm (UTC)
Downloading HTML files programmatically is not a good idea. Please reference the bot policy.

Since you are getting entries and comments, you should easily be able to use our client protocol which lets you download the data without having to screen scrape. Then your client will work for S1, S2, and everything else we might add in the future.
(Reply) (Thread)
[User Picture]From: ati
2006-01-09 05:36 pm (UTC)
LJSM tries to be as polite to LJ server as possible.
No multithreading, caching of downloaded data, uses as much of a client protocol as possible. (BTW, there's still no support for memories, as far as I know)
But sometimes I really want to save someone else's journal.
(Reply) (Parent) (Thread)
[User Picture]From: marksmith
2006-01-09 06:00 pm (UTC)
Saving someone else's journal that you do not own is not allowed.

You are allowed access to your own content for backup purposes as well as content you control by being a maintainer of a community/shared account. We do not allow access to accounts you do not control except in web browsers for the purpose of human viewing.

This has been our policy for ages now. I would try to dig up old discussions here, but I don't have time right now. There should be discussions about it in either lj_dev or lj_clients sometime in the past year.
(Reply) (Parent) (Thread)
[User Picture]From: ati
2006-01-09 06:31 pm (UTC)
My tool brings the possibility to save any journal in bandwidth effective (and in accordance with the bot policy) way, but of course it shoud be used in compliance with LJ's TOS.
(Reply) (Parent) (Thread)
From: (Anonymous)
2006-01-09 06:14 pm (UTC)
I have an idea for a feature addition. You should add the ability to generate a hybrid index page for your journal as well as your friend' journals. It would look the same, except have a username next to each entry. That way, you could download the most recent updates to your friend' page and view them by subject line.
(Reply) (Thread)
[User Picture]From: ati
2006-01-10 04:56 am (UTC)
My motivation to create LJSM was to have an ability to save the best journals of the best people (a kind of 'offline memories'), not to use it as a better interface to LJ.

(Reply) (Parent) (Thread)
[User Picture]From: mrda
2006-01-21 06:46 am (UTC)
This thing simply does not collect my posts in its current incarnation....
(Reply) (Thread)
[User Picture]From: ati
2006-01-21 06:30 pm (UTC)
I'd rather say 'in current incarnation of LJ's link system'.
Since it was changed to username.livejournal.com/postid.html, it will take some time to teach ljsm to follow such links.
(Reply) (Parent) (Thread)
[User Picture]From: davidlevine
2006-01-24 02:17 am (UTC)
Here's (at least one part of) the fix:
        # process links.
        foreach $link (reverse sort $extor->a) {
             if ($link =~ m#/$user.livejournal.com/(\d+)\.html#) {
                 push @posts, {
This does what I need on my Mac. Unfortunately, I have to use ljsm.exe on my PCs. How can I find out when a new .exe is available?

Thanks for providing this tool!
(Reply) (Parent) (Thread)
(Deleted comment)
[User Picture]From: mrda
2006-02-01 02:30 am (UTC)
This gives me trouble loading it - my download manager continuously crashes and restarts, when I try to retrieve it.
(Reply) (Parent) (Thread)
(Deleted comment)
[User Picture]From: mrda
2006-02-01 11:15 am (UTC)
Still no joy, alas...
(Reply) (Parent) (Thread)
From: phenolog
2006-02-05 08:14 pm (UTC)
thanks for the fix, but still some problems arouse:

1) Friends-only posts were not saved (although logged in correctly)

2) The posts is the index file are not sorted by date (within every month), and no day mentioned near the post (like it was in previous version)
(Reply) (Parent) (Thread)
[User Picture]From: marrch_caat
2006-02-06 01:56 pm (UTC)
A bit more should be fixed. For now, it realy doesn't work if username contains an underscore (_). Here's a fix for this problem I suppose:

[...]
sub get_posts {
my ($user) = @_;
# 06.02.2006 atmc: in the domain name underscore should be replaced with dash:
$userpatched = $user;
$userpatched =~ s/_/-/gs;
[...]
# 06.02.2006 atmc: in the domain name underscore should be replaced with dash:
# if ($link =~ m#/$user.livejournal.com/(\d+)\.html#) {
if ($link =~ m#/$userpacthed.livejournal.com/(\d+)\.html#) {
$itemid = $1;
push @posts, {
'type' => 'post',
'status' => 0,
'year' => $year,
'month' => $month,
'link' => $link,
'itemid' => $itemid,
'keyword' => '',
'comments' => ($content =~ m#$itemid\.html\D+(\d+)\s+repl#)? $1 : 0
};
}
# 06.02.2006 atmc:one more case exists (when username starts with underscore):
if ($link =~ m#/users.livejournal.com\/$user/(\d+)\.html#) {
$itemid = $1;
push @posts, {
'type' => 'post',
'status' => 0,
'year' => $year,
'month' => $month,
'link' => $link,
'itemid' => $itemid,
'keyword' => '',
'comments' => ($content =~ m#$itemid\.html\D+(\d+)\s+repl#)? $1 : 0
};
}
[...]

I haven't written a line of Perl code for years, so I mya be wrong in syntax - I'm not realy sure if it possible to merge two lines with #userpatched into one and if "_" symbol should be escaped with backslash in the replace pattern. But I hope you got the point...

If you find this fix usable, couldn't you please provide me with Win32 executable compiled with - I couldn't make it myself using Perl2Exe. Here is the output I got trying to make the file:

Perl2Exe V8.70 Copyright (c) 1997-2005 IndigoSTAR Software
Registered to Alex the Marrch Caat:AlexMcCensy:20060106, ENT version
Converting 'D:/Programs/INetTools/LJSM/ljsm_272.pl' to ljsm_272.exe
ERROR: Can't open c:\Perl\p2x\p2x582exe.lib
Warning: Can't locate Digest/Perl/MD5.pm
at C:\Perl\lib\Digest\MD5.pm line 22
@INC = C:\Perl\lib, C:\Perl\site\lib, .

ERROR: Can't open c:\Perl\p2x\p2x582.dll

If you share any ideas how could I solve the problem with perl2exe, it would be wonderful too ;)
(Reply) (Parent) (Thread)
[User Picture]From: ati
2006-02-07 09:52 am (UTC)
perl2exe version should match version of perl installed on your computer.

I'll compile .exe version tommorow, in the meantime you can use perl version, that fixes issues with new LJ's address scheme.
(Reply) (Parent) (Thread)
[User Picture]From: marrch_caat
2006-02-07 11:20 am (UTC)
Thank you!!!
(Reply) (Parent) (Thread)
[User Picture]From: mrda
2006-01-25 01:06 pm (UTC)
Will you create a new upgrade that gets around this problem?

(Despite the discrepancy on this update, LJSM proves an asset for cataloguing the journals of myself and my LJ friends - thanks for making it!)
(Reply) (Parent) (Thread)
[User Picture]From: ati
2006-02-07 09:54 am (UTC)
there's new perl version of LJSM, that fixes authentication problems and knows how to deal with the new address scheme.
I'll compile .exe version of this script tomorrow
(Reply) (Parent) (Thread)