[JSch-users] Issues uploading to ProFTPd with mod_sftp

[Erick Lichtas]

The latest versions of JSCH 0.1.49 and 0.1.50 are having issues transferring larger files (I'm testing with ~400MB) to ProFTPD server, however, the older version 0.1.44 that I've tested works (and has been working) just fine for quite some time. I've included the configuration information of ProFTPd below. The transfer of the file usually hangs anywhere between 10% and 20% of the transfer and is reproducible 100% of the time (for me) using the Sftp.java example class to upload the file using jsch version 1.49 and 1.50. Jsch version 1.44, it works every time.

I have the following setting configured in my proftpd config as recommended by the link below.

SFTPClientMatch ".*JSCH.*" channelWindowSize 1GB

https://forums.proftpd.org/smf/index.php?topic=4616.0

Can you please explain what change would have affected this and what adjustments I can make in the short term to get this transfer working?

Waiting Thread
"main" prio=6 tid=0x000000000062c000 nid=0x1ff8 runnable [0x000000000256f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:92)
at java.net.SocketOutputStream.write(SocketOutputStream.java:136)
at com.jcraft.jsch.IO.put(IO.java:77)
at com.jcraft.jsch.Session._write(Session.java:1461)
- locked <0x00000000ecd951f0> (a java.lang.Object)
at com.jcraft.jsch.Session.write(Session.java:1424)
at com.jcraft.jsch.ChannelSftp.sendWRITE(ChannelSftp.java:3329)
at com.jcraft.jsch.ChannelSftp._put(ChannelSftp.java:795)
at com.jcraft.jsch.ChannelSftp.put(ChannelSftp.java:623)
at JSchTest.main(JSchTest.java:281)


ProFTPd configuration

Compile-time Settings:
Version: 1.3.4d (maint)
Platform: LINUX [Linux 2.6.32-38-generic i686]
Built: Thu Oct 3 2013 11:07:09 CDT
Built With:
configure '--prefix=/usr' '--with-includes=/usr/include/postgresql:/usr/include/mysql:/usr/openssl/include/' '--mandir=/usr/share/man' '--sysconfdir=/etc/proftpd' '--localstatedir=/var/run' '--libexecdir=/usr/lib/proftpd' '--enable-sendfile' '--enable-facl' '--enable-dso' '--enable-autoshadow' '--enable-ctrls' '--with-modules=mod_readme:mod_sftp' '--enable-ipv6' '--enable-nls' '--build' 'i486-linux-gnu' 'build_alias=i486-linux-gnu' 'CFLAGS=-O2 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -DHAVE_OPENSSL -DUSE_LDAP_TLS ' 'LDFLAGS=-Wl,-Bsymbolic-functions' 'CPPFLAGS=' 'CXXFLAGS=-g -O2' 'FFLAGS=-g -O2' '--enable-openssl' '--with-libraries=/usr/openssl/lib' '--with-shared=mod_ctrls_admin:mod_ban'

CFLAGS: -O2 -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64 -DHAVE_OPENSSL -DUSE_LDAP_TLS -Wall
LDFLAGS: -L$(top_srcdir)/lib -Wl,-Bsymbolic-functions -L/usr/openssl/lib
LIBS: -lssl -lcrypto -lssl -lcrypto -L$(top_srcdir)/lib/libcap -lcap -lcrypto -lsupp -lcrypt -ldl -ldl -lz

Files:
Configuration File:
/etc/proftpd/proftpd.conf
Pid File:
/var/run/proftpd.pid
Scoreboard File:
/var/run/proftpd.scoreboard
Header Directory:
/usr/include/proftpd
Shared Module Directory:
/usr/lib/proftpd

Features:
+ Autoshadow support
+ Controls support
- curses support
- Developer support
+ DSO support
+ IPv6 support
+ Largefile support
- Lastlog support
- Memcache support
- ncurses support
+ NLS support
+ OpenSSL support
- PCRE support
+ POSIX ACL support
+ Shadow file support
+ Sendfile support
+ Trace support

Tunable Options:
PR_TUNABLE_BUFFER_SIZE = 1024
PR_TUNABLE_DEFAULT_RCVBUFSZ = 8192
PR_TUNABLE_DEFAULT_SNDBUFSZ = 8192
PR_TUNABLE_GLOBBING_MAX_MATCHES = 100000
PR_TUNABLE_GLOBBING_MAX_RECURSION = 8
PR_TUNABLE_HASH_TABLE_SIZE = 40
PR_TUNABLE_NEW_POOL_SIZE = 512
PR_TUNABLE_SCOREBOARD_BUFFER_SIZE = 80
PR_TUNABLE_SCOREBOARD_SCRUB_TIMER = 30
PR_TUNABLE_SELECT_TIMEOUT = 30
PR_TUNABLE_TIMEOUTIDENT = 10
PR_TUNABLE_TIMEOUTIDLE = 600
PR_TUNABLE_TIMEOUTLINGER = 30
PR_TUNABLE_TIMEOUTLOGIN = 300
PR_TUNABLE_TIMEOUTNOXFER = 300
PR_TUNABLE_TIMEOUTSTALLED = 3600
PR_TUNABLE_XFER_SCOREBOARD_UPDATES = 10

Thanks in advance!

Erick Lichtas

[Atsuhiko Yamanaka]

Hi,

+-From: Erick Lichtas <***@linoma.com> --
|_Date: Thu, 3 Oct 2013 21:20:10 +0000 _______
|

|The latest versions of JSCH 0.1.49 and 0.1.50 are having issues
|transferring larger files (I'm testing with ~400MB) to ProFTPD server,
|however, the older version 0.1.44 that I've tested works (and has been
|working just fine for quite some time.

How about version 0.1.47 and 0.1.48?
If 0.1.47 works fine, but 0.1.48 does not, try
((ChannelSftp)c).setBulkRequests(1)
before invoking Channel#connect().


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk

[Erick Lichtas]

Hi Atsuhiko,

Thanks again for your reply. I've verified every version after 0.1.44 and they all fail (lock up) after a substantial number of bytes. The bulk requests setting does not resolve the issue.

Channel c = session.openChannel("sftp");
((ChannelSftp)c).setBulkRequests(1);
c.connect();

So going back to the differences between 0.1.44 and 0.1.49, the block of code that is causing the failure with ProFTPd is the following section from the _put(...) method in ChannelSftp:

int foo = count;
while (foo > 0) {
if ((seq - 1) == startid
|| ((seq - startid) - ackcount) >= bulk_requests) {
while (((seq - startid) - ackcount) >= bulk_requests) {
if (this.rwsize >= foo)
break;
if (checkStatus(ackid, header)) {
int _ackid = ackid[0];
if (startid > _ackid || _ackid > seq - 1) {
if (_ackid == seq) {
System.err.println("ack error: startid="
+ startid
+ " seq="
+ seq
+ " _ackid=" + _ackid);
}
else {
throw new SftpException(SSH_FX_FAILURE,
"ack error: startid=" + startid
+ " seq=" + seq + " _ackid="
+ _ackid);
}
}
ackcount++;
}
else {
break;
}
}
}
foo -= sendWRITE(handle, offset, data, 0, foo);
}

If I change this block of code back to what it was in 0.1.44 (below), then the transfers complete successfully.

int _i = count;
while (_i > 0) {
_i -= sendWRITE(handle, offset, data, 0, _i);
if ((seq - 1) == startid || io_in.available() >= 1024) {
while (io_in.available() > 0) {
if (checkStatus(ackid, header)) {
_ackid = ackid[0];
if (startid > _ackid || _ackid > seq - 1) {
if (_ackid == seq) {
System.err.println("ack error: startid="
+ startid
+ " seq="
+ seq
+ " _ackid=" + _ackid);
}
else {
throw new SftpException(SSH_FX_FAILURE,
"ack error: startid=" + startid
+ " seq=" + seq + " _ackid="
+ _ackid);
}
}
ackcount++;
}
else {
break;
}
}
}
}

What are your thoughts on reverting this back? And in doing so, what negative impacts would this present?

Regards,
Erick

-----Original Message-----
From: Atsuhiko Yamanaka [mailto:***@jcraft.com]
Sent: Thursday, October 10, 2013 1:59 AM
To: Erick Lichtas
Cc: jsch-***@lists.sourceforge.net
Subject: Re: [JSch-users] Issues uploading to ProFTPd with mod_sftp

Hi,

+-From: Erick Lichtas <***@linoma.com> --
|_Date: Thu, 3 Oct 2013 21:20:10 +0000 _______
|

|The latest versions of JSCH 0.1.49 and 0.1.50 are having issues
|transferring larger files (I'm testing with ~400MB) to ProFTPD server,
|however, the older version 0.1.44 that I've tested works (and has been
|working just fine for quite some time.

How about version 0.1.47 and 0.1.48?
If 0.1.47 works fine, but 0.1.48 does not, try
((ChannelSftp)c).setBulkRequests(1)
before invoking Channel#connect().


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk

[Atsuhiko Yamanaka]

Hi,

+-From: Erick Lichtas <***@linoma.com> --
|_Date: Thu, 10 Oct 2013 19:21:51 +0000 ______
|

|So going back to the differences between 0.1.44 and 0.1.49,
|the block of code that is causing the failure with ProFTPd is
|the following section from the _put(...) method in ChannelSftp:

Could you try to comment two lines out as follows?


| int foo = count;
| while (foo > 0) {
| if ((seq - 1) == startid
| || ((seq - startid) - ackcount) >= bulk_requests) {
| while (((seq - startid) - ackcount) >= bulk_requests) {
|
// if (this.rwsize >= foo) // !!
// break; // !!
|
| if (checkStatus(ackid, header)) {
| int _ackid = ackid[0];
...

By the way, does OpenSSH's sftp command have a similar problem with
the following configuration?
SFTPClientMatch ".*OpenSSH.*" channelWindowSize 1GB


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk

[Erick Lichtas]

Hi Atsuhiko,

You are correct, removing those lines does resolve the transfer issues with ProFTPd.

Without those lines, the uploads are successful. The bulk request default setting seems to be working fine (no change from default required) and the SFTPClientMatch setting in ProFTPD doesn't seem to have any affect, at least on the files I'm testing with.

What are the implications of removing this conditional break statement?

// if (this.rwsize >= foo)
// break;

Does this affect the efficiency or are there other compatibility concerns with other servers by not having this setup?

Regards,
Erick Lichtas

-----Original Message-----
From: Atsuhiko Yamanaka [mailto:***@jcraft.com]
Sent: Friday, October 11, 2013 2:49 AM
To: Erick Lichtas
Cc: jsch-***@lists.sourceforge.net
Subject: Re: [JSch-users] Issues uploading to ProFTPd with mod_sftp

Hi,

+-From: Erick Lichtas <***@linoma.com> --
|_Date: Thu, 10 Oct 2013 19:21:51 +0000 ______
|

|So going back to the differences between 0.1.44 and 0.1.49,
|the block of code that is causing the failure with ProFTPd is
|the following section from the _put(...) method in ChannelSftp:

Could you try to comment two lines out as follows?


| int foo = count;
| while (foo > 0) {
| if ((seq - 1) == startid
| || ((seq - startid) - ackcount) >= bulk_requests) {
| while (((seq - startid) - ackcount) >= bulk_requests) {
|
// if (this.rwsize >= foo) // !!
// break; // !!
|
| if (checkStatus(ackid, header)) {
| int _ackid = ackid[0];
...

By the way, does OpenSSH's sftp command have a similar problem with the following configuration?
SFTPClientMatch ".*OpenSSH.*" channelWindowSize 1GB


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk

[Atsuhiko Yamanaka]

Hi,

+-From: Erick Lichtas <***@linoma.com> --
|_Date: Thu, 3 Oct 2013 21:20:10 +0000 _______
...
|I have the following setting configured in my proftpd config as
|recommended by the link below.
|SFTPClientMatch ".*JSCH.*" channelWindowSize 1GB
|https://forums.proftpd.org/smf/index.php?topic=3D4616.0

This is an off-topic, but we have found the reason why that config
is required for ProFTPd. It seems, by the default, ProFTPd will
choose 4GB for its channelWindowSize, and JSch's sftp channel will be stopped
in uploading huge data to ProFTPd. We hope the following version will
work well without such a config for ProFTPd.
http://www.jcraft.com/jsch/jsch-0.1.51-rc7.zip

We have planned to release that version as the next formal one.
It has following changes,
Changes since version 0.1.50:
- bugfix: reproducibility of "verify: false". FIXED.
Hundreds of thousands connections had caused that exception.
- bugfix: resource leaks at the failure of making local port forwarding. FIXED.
- bugfix: NPE in connecting to the non-standard TCP port. FIXED.
This problem had appeared if a host-key does not exist in
"known_host" file.
- bugfix: TCP connection may not be dropped if error messages from
the remote are too long. FIXED.
- bugfix: SftpATTRS#getAtimeString() returns the wrong string. FIXED.
- bugfix: bytes to be added by SSH_MSG_CHANNEL_WINDOW_ADJUST must be in
unsigned integer. FIXED.
- bugfix: Util.checkTilde() should not convert a tilde in
"C:\PROGRA~1\". FIXED.
- bugfix: A long long command for ChannelExec will cause
an ArrayIndexOutOfBoundsException. FIXED.
- bugfix: ChannelSftp should not send bulk request greedily even if the remote
window has the enough space. FIXED.
- bugfix: Util.createSocket() should throw an exception with 'cause'. FIXED.
- bugfix: failed to parse .ssh/config in the EBCDIC environment. FIXED.
- bugfix: com.jcraft.jsch.jcraft.HMACSHA1(used only for MacOSX) is not
reusable. FIXED.
- bugfix: NPE caused by the delayed response for channels. FIXED.
- bugfix: hung-up in uploading huge data to ProFTPd without the config
'SFTPClientMatch "JSCH.*" channelWindowSize 1GB' FIXED.
- change: com.jcraft.jsch.jcraft.HMAC* will not be used.
It seems Java6 on Mac OS X has fixed some memory leak bug in JCE,
so there is no reason to use c.j.j.jt.HMAC* introduced at 0.1.30.
- change: updating copyright messages; 2013 -> 2014
- feature: added the support for private keys in PKCS#8 format.


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk

[Erick Lichtas]

Hi,

Thanks for the reply. We've confirmed the issue also exists against the SFTP server in the Cerebus FTP Server software (http://www.cerberusftp.com/) and that the latest 0.1.51.rc7 version does fix this issue which is great. I have a couple questions because I want to implement this fix in our environment soon.

Would it be safe for me to patch in the change (the commented out line of code) from ChannelSftp.put(..) into 0.1.49? or are there additional changes directly related to this that need to be updated elsewhere?

What are your expectations for a release date of the stable 0.1.51?

Regards,
Erick

-----Original Message-----
From: Atsuhiko Yamanaka [mailto:***@jcraft.com]
Sent: Wednesday, February 12, 2014 12:54 AM
To: Erick Lichtas
Cc: jsch-***@lists.sourceforge.net
Subject: Re: [JSch-users] Issues uploading to ProFTPd with mod_sftp

Hi,

+-From: Erick Lichtas <***@linoma.com> --
|_Date: Thu, 3 Oct 2013 21:20:10 +0000 _______
...
|I have the following setting configured in my proftpd config as
|recommended by the link below.
|SFTPClientMatch ".*JSCH.*" channelWindowSize 1GB
|https://forums.proftpd.org/smf/index.php?topic=3D4616.0

This is an off-topic, but we have found the reason why that config is required for ProFTPd. It seems, by the default, ProFTPd will choose 4GB for its channelWindowSize, and JSch's sftp channel will be stopped in uploading huge data to ProFTPd. We hope the following version will work well without such a config for ProFTPd.
http://www.jcraft.com/jsch/jsch-0.1.51-rc7.zip

We have planned to release that version as the next formal one.
It has following changes,
Changes since version 0.1.50:
- bugfix: reproducibility of "verify: false". FIXED.
Hundreds of thousands connections had caused that exception.
- bugfix: resource leaks at the failure of making local port forwarding. FIXED.
- bugfix: NPE in connecting to the non-standard TCP port. FIXED.
This problem had appeared if a host-key does not exist in
"known_host" file.
- bugfix: TCP connection may not be dropped if error messages from
the remote are too long. FIXED.
- bugfix: SftpATTRS#getAtimeString() returns the wrong string. FIXED.
- bugfix: bytes to be added by SSH_MSG_CHANNEL_WINDOW_ADJUST must be in
unsigned integer. FIXED.
- bugfix: Util.checkTilde() should not convert a tilde in
"C:\PROGRA~1\". FIXED.
- bugfix: A long long command for ChannelExec will cause
an ArrayIndexOutOfBoundsException. FIXED.
- bugfix: ChannelSftp should not send bulk request greedily even if the remote
window has the enough space. FIXED.
- bugfix: Util.createSocket() should throw an exception with 'cause'. FIXED.
- bugfix: failed to parse .ssh/config in the EBCDIC environment. FIXED.
- bugfix: com.jcraft.jsch.jcraft.HMACSHA1(used only for MacOSX) is not
reusable. FIXED.
- bugfix: NPE caused by the delayed response for channels. FIXED.
- bugfix: hung-up in uploading huge data to ProFTPd without the config
'SFTPClientMatch "JSCH.*" channelWindowSize 1GB' FIXED.
- change: com.jcraft.jsch.jcraft.HMAC* will not be used.
It seems Java6 on Mac OS X has fixed some memory leak bug in JCE,
so there is no reason to use c.j.j.jt.HMAC* introduced at 0.1.30.
- change: updating copyright messages; 2013 -> 2014
- feature: added the support for private keys in PKCS#8 format.


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk