[JSch-users] SFTP slow file transfer speed with Titan FTP

[Thomas Küstermann]

Hi!


We're using Camel<http://camel.apache.org/> to read files from a SFTP server. Camel is using JSch internally which used to work quite good for us. I recently had to upgrade JSch in our product from 0.1.38 to 0.1.53. Although the upgrade works perfectly with the majority of our customers I am currently faced with a weird issue with Titan FTP (SSH-2.0-srtSSHServer_11.00). The transfer speed by using Channel#get(String) is awfully slow (~ 10 KB/s). I tried several JSch versions and observed that the issue had been introduced in the transition of 0.1.47 to 0.1.48 whose changelog states:


- change: the file transfer speed with ChannelSftp#get(String src) has been

improved; sending multiple requests at any one time.
- change: by the default, at most, 16 requests will be sent at any one time
in ChannelSftp.


Before this change the download rate was at least at several 100 KB/s. So, I'm particularly interested in "sending multiple requests at any one time" in ChannelSftp. What does it mean and how can this actually result in a download slow down?

Current work around is to make use of ChannelSftp#get(String, OutputStream) which is still satisfactorily fast. For testing purposes I set the number of bulk requests to 1 which resulted in a known good performance.

I already tried to use ChannelSftp#setBulkRequests(int) but with no luck. Current work around is to make use of ChannelSftp#get(String, OutputStream) which is still satisfactorily fast.

Any hints are appreciated!

Regards,

-- Thomas

[Thomas Küstermann]

I've managed to overcome this issue myself. First, I tried to find out where the blocking actually occurs. After a number of sent bulk requests the ChannelSftp.header(Buffer, Header) method gets blocked for exactly 60 seconds, that's the stack trace:

Daemon Thread [Camel (camel-1) thread #0 - sftp://secret/archive] (Suspended)
waiting for: Channel$MyPipedInputStream (id=26)
Object.wait(long) line: not available [native method]
Channel$MyPipedInputStream(PipedInputStream).read() line: 326
Channel$MyPipedInputStream(PipedInputStream).read(byte[], int, int) line: 377
ChannelSftp.fill(byte[], int, int) line: 2605
ChannelSftp.header(Buffer, ChannelSftp$Header) line: 2631
ChannelSftp.access$11(ChannelSftp, Buffer, ChannelSftp$Header) line: 2629
ChannelSftp$2.read(byte[], int, int) line: 1282
ChannelSftp$2.read(byte[]) line: 1232
FileOperations.writeFileByStream(InputStream, File) line: 382

I don't know what is causing this. After the "timeout" occurred processing continued until the next outage. I tried to reproduce the "bug" with Openssh's sftp command and the -R (num_requests) argument which I think is meant by "sending multiple requests at any one time". As Openssh's sftp works well I compared the ChannelSftp implementation with what is in sftp-client.c. The only obvious difference I found is the "rq.count()==0" condition in "ChannelSftp.get(String, SftpProgressMonitor, long).new InputStream() {...}.read(byte[], int, int)". Removing the condition fixes the issue for me.

Can this be considered as a bug? Is this something eligible for a next version of JSCH?

The following patch is based upon jsch-0.1.53:

--- src/com/jcraft/jsch/ChannelSftp.java
+++ src/com/jcraft/jsch/ChannelSftp.java
@@ -1374,17 +1374,16 @@
len=1024;
}

- if(rq.count()==0) {
- int request_len = buf.buffer.length-13;
- if(server_version==0){ request_len=1024; }

- while(rq.count() < request_max){
- try{
- sendREAD(handle, request_offset, request_len, rq);
- }
- catch(Exception e){ throw new IOException("error"); }
- request_offset += request_len;
+ int request_len = buf.buffer.length-13;
+ if(server_version==0){ request_len=1024; }
+
+ while(rq.count() < request_max){
+ try{
+ sendREAD(handle, request_offset, request_len, rq);
}
+ catch(Exception e){ throw new IOException("error"); }
+ request_offset += request_len;
}

header=header(buf, header);

Thanks for considering this.

Regards,
Thomas

[Atsuhiko Yamanaka]

Hi,

Sorry for my long long delay.

On Tue, Mar 29, 2016 at 9:17 PM, Thomas Küstermann

Post by Thomas Küstermann
I don't know what is causing this. After the "timeout" occurred processing continued until the next outage. I tried to reproduce the "bug" with Openssh's sftp command and the -R (num_requests) argument which I think is meant by "sending multiple requests at any one time". As Openssh's sftp works well I compared the ChannelSftp implementation with what is in sftp-client.c. The only obvious difference I found is the "rq.count()==0" condition in "ChannelSftp.get(String, SftpProgressMonitor, long).new InputStream() {...}.read(byte[], int, int)". Removing the condition fixes the issue for me.
Can this be considered as a bug? Is this something eligible for a next version of JSCH?
--- src/com/jcraft/jsch/ChannelSftp.java
+++ src/com/jcraft/jsch/ChannelSftp.java
@@ -1374,17 +1374,16 @@
len=1024;
}
- if(rq.count()==0) {

...

As you have suggested, we will remove "if(rq.count()==0)" check in the
next release.


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk