[JSch-users] jsch sftp put hangs on large file

[Doug Kent]

We are attempting to put files from Solaris 10 server to a Novell sftp server running NetWare OpneSSH sftp-server(NICI) 3.7.1p6 build 80). Using jsch 1.44. We are going throught a firewall. What happens is that when putting the file, it will just stop and not exchange any data with the server. We have looked at the connection using network sniffers and see no traffic at all. Just stops. If the connection is broken by the network people, the code throws an exception as it should. Has anyone seen this behavior? Any solution? This happens sometimes, maybe one out of 5 tries, and it's usually with larger files 300MB and up, although it has happened on files as little as 9MB. If it stops, the part of the file transferred is still there on the target server. In my code, each file has a fresh connection created for that transmission. I've tried setting a timeout, calling setTimeout() on the session - no difference. Also tried calling session.setServerAliveInterval() - nothing. Maybe the answer lies in starting a separate thread and calling session.sendKeepAliveMsg()? Following is the code:


JSch jsch=new JSch();
Session session=null;
Channel channel=null;
ChannelSftp c=null;

Hashtable hashtable = new Hashtable();

hashtable.put("StrictHostKeyChecking","no");

//SFTP default port is 22

try {
session=jsch.getSession(User, Host, Port);
} catch (JSchException ex) {
throw new Exception("getSession Exception: " + ex.getMessage());
}
session.setConfig(hashtable);
session.setPassword(Password);
try {
session.connect();
} catch (JSchException ex) {
throw new Exception("session.connect Exception: " + ex.getMessage());
}
try {
channel=session.openChannel("sftp");
} catch (JSchException ex) {
throw new Exception("session.openChannel Exception: " + ex.getMessage());
}
try {
channel.connect();
} catch (JSchException ex) {
throw new Exception("channel.connect Exception: " + ex.getMessage());
}

try {
c=(ChannelSftp)channel;
} catch (Exception ex) {
throw new Exception("(ChannelSftp)channel Exception: " + ex.getMessage());
}
//check if file exists
SftpATTRS sftpATTRS = null;
Boolean fileExists = true;
try {
sftpATTRS = c.lstat(RemoteTargetFolder + Filename);
} catch (Exception ex) {
fileExists = false;
}

if (Overlay == true & fileExists) { //allowed to overlay and file exists
try {
c.rm(RemoteTargetFolder + Filename);
} catch (Exception ex) {
c.quit();
session.disconnect();
throw new Exception("rm Exception on existing file: " + RemoteTargetFolder + Filename + " " + ex.getMessage());
}
}
if (Overlay == false & fileExists) { //not allowed to overlay and file exists
c.quit();
session.disconnect();
throw new Exception("File " + Filename + " exists in directory " + RemoteTargetFolder);
}
// check if compression
if (Compression == true) {
session.setConfig("compression.s2c", "***@openssh.com,zlib,none");
session.setConfig("compression.c2s", "***@openssh.com,zlib,none");
}

//check if using temp file
String filename = null;
if (UseTempFile == true) {
filename = TempFileName;
}
else {
filename = Filename;
}

try {
c.put(LocalFullPathFileName, RemoteTargetFolder + filename);
} catch (Exception ex) {
try { //clean up file
c.rm(RemoteTargetFolder + RemoteTargetFolder + filename);
} catch (Exception ex2) {
}
c.quit();
session.disconnect();
throw new Exception("put Exception: " + ex.getMessage());
}
if (UseTempFile == true) {
try {
c.rename(RemoteTargetFolder + TempFileName, RemoteTargetFolder + Filename);
} catch (Exception ex) {
try {
c.rm(RemoteTargetFolder + TempFileName);
} catch (Exception ex2) {
}
c.quit();
session.disconnect();
throw new Exception("rename Exception: " + ex.getMessage());
}
try {
c.chmod(0664, RemoteTargetFolder + Filename);
} catch (Exception ex) {
}
}
c.quit();
session.disconnect();

[Doug Kent]

What facilities does jsch sftp have for logging? How does one set the level? (examples?) And what is the default name of the log file?

[Viet H. Phan]

Hi,

You need to implement your own Logger that implements com.jcraft.jsch.Logger interface.
Then call com.jcraft.jsch.JSch.setLogger(...) to activate the Logger.

Regards,
Viet


________________________________
From: Doug Kent <***@aol.com>
To: jsch-***@lists.sourceforge.net
Sent: Saturday, August 13, 2011 7:35 AM
Subject: [JSch-users] Setting up Logging


What facilities does jsch sftp have for logging?  How does one set the level? (examples?)  And what is the default name of the log file?

[Doug Kent]

I was able to find out exactly where the code is hanging; It's right after the sendWRITE - it must be hanging on the io_in.available() call. What could cause this?

while(_i>0){
_i-=sendWRITE(handle, offset, data, 0, _i);
System.out.println("sent data"+data.length);
if((seq-1)==startid ||
io_in.available()>=1024){
System.out.println("just after available check");
while(io_in.available()>0){
System.out.println ("just after io_in.available");
if(checkStatus(ackid, header)){
System.out.println ("just after checkstatus");
_ackid=ackid[0];
if(startid>_ackid || _ackid>seq-1){
if(_ackid==seq){






-----Original Message-----
From: Doug Kent <***@aol.com>
To: jsch-users <jsch-***@lists.sourceforge.net>
Sent: Fri, Aug 12, 2011 4:18 pm
Subject: jsch sftp put hangs on large file


We are attempting to put files from Solaris 10 server to a Novell sftp server running NetWare OpneSSH sftp-server(NICI) 3.7.1p6 build 80). Using jsch 1.44. We are going throught a firewall. What happens is that when putting the file, it will just stop and not exchange any data with the server. We have looked at the connection using network sniffers and see no traffic at all. Just stops. If the connection is broken by the network people, the code throws an exception as it should. Has anyone seen this behavior? Any solution? This happens sometimes, maybe one out of 5 tries, and it's usually with larger files 300MB and up, although it has happened on files as little as 9MB. If it stops, the part of the file transferred is still there on the target server. In my code, each file has a fresh connection created for that transmission. I've tried setting a timeout, calling setTimeout() on the session - no difference. Also tried calling session.setServerAliveInterval() - nothing. Maybe the answer lies in starting a separate thread and calling session.sendKeepAliveMsg()? Following is the code:


JSch jsch=new JSch();
Session session=null;
Channel channel=null;
ChannelSftp c=null;

Hashtable hashtable = new Hashtable();

hashtable.put("StrictHostKeyChecking","no");

//SFTP default port is 22

try {
session=jsch.getSession(User, Host, Port);
} catch (JSchException ex) {
throw new Exception("getSession Exception: " + ex.getMessage());
}
session.setConfig(hashtable);
session.setPassword(Password);
try {
session.connect();
} catch (JSchException ex) {
throw new Exception("session.connect Exception: " + ex.getMessage());
}
try {
channel=session.openChannel("sftp");
} catch (JSchException ex) {
throw new Exception("session.openChannel Exception: " + ex.getMessage());
}
try {
channel.connect();
} catch (JSchException ex) {
throw new Exception("channel.connect Exception: " + ex.getMessage());
}

try {
c=(ChannelSftp)channel;
} catch (Exception ex) {
throw new Exception("(ChannelSftp)channel Exception: " + ex.getMessage());
}
//check if file exists
SftpATTRS sftpATTRS = null;
Boolean fileExists = true;
try {
sftpATTRS = c.lstat(RemoteTargetFolder + Filename);
} catch (Exception ex) {
fileExists = false;
}

if (Overlay == true & fileExists) { //allowed to overlay and file exists
try {
c.rm(RemoteTargetFolder + Filename);
} catch (Exception ex) {
c.quit();
session.disconnect();
throw new Exception("rm Exception on existing file: " + RemoteTargetFolder + Filename + " " + ex.getMessage());
}
}
if (Overlay == false & fileExists) { //not allowed to overlay and file exists
c.quit();
session.disconnect();
throw new Exception("File " + Filename + " exists in directory " + RemoteTargetFolder);
}
// check if compression
if (Compression == true) {
session.setConfig("compression.s2c", "***@openssh.com,zlib,none");
session.setConfig("compression.c2s", "***@openssh.com,zlib,none");
}

//check if using temp file
String filename = null;
if (UseTempFile == true) {
filename = TempFileName;
}
else {
filename = Filename;
}

try {
c.put(LocalFullPathFileName, RemoteTargetFolder + filename);
} catch (Exception ex) {
try { //clean up file
c.rm(RemoteTargetFolder + RemoteTargetFolder + filename);
} catch (Exception ex2) {
}
c.quit();
session.disconnect();
throw new Exception("put Exception: " + ex.getMessage());
}
if (UseTempFile == true) {
try {
c.rename(RemoteTargetFolder + TempFileName, RemoteTargetFolder + Filename);
} catch (Exception ex) {
try {
c.rm(RemoteTargetFolder + TempFileName);
} catch (Exception ex2) {
}
c.quit();
session.disconnect();
throw new Exception("rename Exception: " + ex.getMessage());
}
try {
c.chmod(0664, RemoteTargetFolder + Filename);
} catch (Exception ex) {
}
}
c.quit();
session.disconnect();

[Atsuhiko Yamanaka]

Hi,

Thank you for your investigations,

+-From: Doug Kent <***@aol.com> --------------
|_Date: Tue, 16 Aug 2011 17:45:42 -0400 (EDT) __
|
|I was able to find out exactly where the code is hanging; It's right after
| the sendWRITE - it must be hanging on the io_in.available() call.
|What could cause this?

|while(_i>0){
| _i-=sendWRITE(handle, offset, data, 0, _i);
| System.out.println("sent data"+data.length);
| if((seq-1)==startid ||
| io_in.available()>=1024){
...

I had not expected that java.io.InputStream#available() will block.
Could you please try the following version?
http://www.jcraft.com/jsch/jsch-0.1.45-rc6.zip
Its internal has been re-implemented.


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
[phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk

[Doug Kent]

This version looks to be faster with a much bigger buffer being sent with each send, but still hangs up at that same point. We are investigating the server software. Thanks very much.




-----Original Message-----
From: Atsuhiko Yamanaka <***@jcraft.com>
To: dglke <***@aol.com>
Cc: jsch-users <jsch-***@lists.sourceforge.net>
Sent: Tue, Aug 16, 2011 11:09 pm
Subject: Re: [JSch-users] Fwd: jsch sftp put hangs on large file


Hi,

Thank you for your investigations,

+-From: Doug Kent <***@aol.com> --------------
|_Date: Tue, 16 Aug 2011 17:45:42 -0400 (EDT) __
|
|I was able to find out exactly where the code is hanging; It's right after
| the sendWRITE - it must be hanging on the io_in.available() call.
|What could cause this?

|while(_i>0){
| _i-=sendWRITE(handle, offset, data, 0, _i);
| System.out.println("sent data"+data.length);
| if((seq-1)==startid ||
| io_in.available()>=1024){
...

I had not expected that java.io.InputStream#available() will block.
Could you please try the following version?
http://www.jcraft.com/jsch/jsch-0.1.45-rc6.zip
Its internal has been re-implemented.


Sincerely,
--
Atsuhiko Yamanaka
JCraft,Inc.
1-14-20 HONCHO AOBA-KU,
SENDAI, MIYAGI 980-0014 Japan.
Tel [phone]
[phone]
Skype callto://jcraft/
Twitter: http://twitter.com/ymnk
Facebook: http://facebook.com/aymnk