[JSch-users] Transfering large number of small files

[Luís Lobo]

Hi all.

I'm trying to transfer a large number of small files to a remote server (with reasonable latency) using JSch.

The code I'm using creates a Session and a ChannelSftp and then iterates over the local files (some times thousands) to transfer them to the server, with something like (I omitted exception handling for simplicity):

Session session=...;
ChannelSftp sftp=(ChannelSftp)session.openChannel("sftp");
for(Path path:...){
sftp.put(path.toString(), String.format("/destination/%s",path.getFileName().toString()));
}
sftp.disconnect();
session.disconnect();

The problem is this is very slow (sometimes it takes hours) compared with the something like:

rsync -avu /local/<some glob> server:/destination/ # takes seconds

Is there any faster way to transfer large number of small files with JSch?

Regards,
LL

[Leonardo K. Shikida]

you can't compare sftp with rsync

rsync calculates the incremental changes it must send over the wire, sftp
will always send the whole file, no matter it's overwriting the same file
on the other side

you can compare jsch with native sftp, but I believe sftp will still be
faster for obvious reasons

[]

Leo

Post by Luís Lobo
Hi all.
I'm trying to transfer a large number of small files to a remote server
(with reasonable latency) using JSch.
The code I'm using creates a Session and a ChannelSftp and then iterates
over the local files (some times thousands) to transfer them to the server,
Session session=...;
ChannelSftp sftp=(ChannelSftp)session.openChannel("sftp");
for(Path path:...){
sftp.put(path.toString(), String.format("/destination/%s",path.getFileName().toString()));
}
sftp.disconnect();
session.disconnect();
The problem is this is very slow (sometimes *it takes hours*) compared
rsync -avu /local/<some glob> server:/destination/ # takes seconds
Is there any faster way to transfer large number of small files with JSch?
Regards,
LL
------------------------------------------------------------------------------
What NetFlow Analyzer can do for you? Monitors network bandwidth and
traffic
patterns at an interface-level. Reveals which users, apps, and protocols
are
consuming the most bandwidth. Provides multi-vendor support for NetFlow,
J-Flow, sFlow and other flows. Make informed decisions using capacity
planning
reports.http://sdm.link/zohodev2dev
_______________________________________________
JSch-users mailing list
https://lists.sourceforge.net/lists/listinfo/jsch-users