[JSch-users] Jsch issue with compression algorithm

Baweja, Keshav

2014-10-20 04:16:53 UTC

Hello All,

I have been able to narrow down this issue to below -

1) I enabled client to server compression configuration on Jsch by setting compression=2 as an option on camel route.
2) With this, Jsch sets client to server compression configuration as - JSCH -> kex: client: ***@openssh.com, zlib, none
3) When server is configured as below the sftp connection is successful
JSCH -> kex: server: none,***@openssh.com
4) However, if the server is configured as any of the below the sftp connection is not successful
JSCH -> kex: server: none,zlib
JSCH -> kex: server: zlib

So if ***@openssh.com is not configured on server, Jsch fails with the exception below

Caused by: com.jcraft.jsch.JSchException: Algorithm negotiation fail
at com.jcraft.jsch.Session.receive_kexinit(Session.java:583) ~[ftu-server-3.0.3.jar:na]
at com.jcraft.jsch.Session.connect(Session.java:320) ~[ftu-server-3.0.3.jar:na]
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:115) ~[ftu-server-3.0.3.jar:na]

Has someone encountered this? Any ideas to resolve this. Thanks in advance.

Regards
Keshav

-----Original Message-----
From: Baweja, Keshav [mailto:***@jpmorgan.com.INVALID]
Sent: Friday, October 10, 2014 2:56 PM
To: ***@camel.apache.org; jsch-***@lists.sourceforge.net
Subject: RE: JSch connection issue with Maverick SSHD server

Hello

I am using Camel routing library to set up sftp routes to a remote sftp server. Camel user Jsch library (version 0.1.50) for all sftp operations and jsch-zlib (version 1.1.3) for the compression. However I encounter the exception below with this set up. Could someone advise/point to a resolution? Many thanks in advance.

[2014-10-07 10:37:11.937] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using private keyfile: /home/localuser/.ssh/id_rsa
[2014-10-07 10:37:11.937] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using knownhosts file: /home/localuser/.ssh/known_hosts
[2014-10-07 10:37:12.018] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using StrickHostKeyChecking: no
[2014-10-07 10:37:12.018] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using compression: 2
[2014-10-07 10:37:12.018] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Connecting to host port port
[2014-10-07 10:37:12.150] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Connection established
[2014-10-07 10:37:12.279] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Remote version string: SSH-2.0-Maverick_SSHD
[2014-10-07 10:37:12.279] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Local version string: SSH-2.0-JSCH-0.1.50
[2014-10-07 10:37:12.279] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[2014-10-07 10:37:12.284] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes256-ctr is not available.
[2014-10-07 10:37:12.284] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes192-ctr is not available.
[2014-10-07 10:37:12.284] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes256-cbc is not available.
[2014-10-07 10:37:12.285] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes192-cbc is not available.
[2014-10-07 10:37:12.285] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> arcfour256 is not available.
[2014-10-07 10:37:12.285] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> CheckKexes: diffie-hellman-group14-sha1
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> SSH_MSG_KEXINIT sent
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> SSH_MSG_KEXINIT received
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: ssh-rsa
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: aes128-cbc,aes256-cbc,aes192-cbc,3des-cbc,twofish256-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,cast128-cbc
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: aes128-cbc,aes256-cbc,aes192-cbc,3des-cbc,twofish256-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,cast128-cbc
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: hmac-sha1,hmac-sha1-96
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: hmac-sha1,hmac-sha1-96
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: zlib
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: zlib
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server:
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server:
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: ssh-rsa,ssh-dss
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: ***@openssh.com, zlib, none
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: ***@openssh.com, zlib, none
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client:
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client:
[2014-10-07 10:37:12.311] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Disconnecting from host port port
[2014-10-07 10:37:12.316] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpConsumer - Could not connect to:Endpoint[sftp://***@host:port/data?compression=2&delay=70000&disconnect=false&include=filename_2014-10-06.xml&knownHostsFile=%2Fexport%2Fhome%2Flocaluser%2F.ssh%2Fknown_hosts&localWorkDirectory=%2Fexport%2Fhome%2Flocaluser%2FFTU%2Fserver%2Fwork%2F26&noop=true&pollStrategy=%23POLL_STRATEGY_ENDPOINT_16&privateKeyFile=%2Fexport%2Fhome%2Flocaluser%2F.ssh%2Fid_rsa&readLock=changed&readLockCheckInterval=5000&readLockTimeout=20000&reconnectDelay=60000&throwExceptionOnConnectFailed=true]. Will try to recover.
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot connect to sftp://***@host:port
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:143) ~[ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.RemoteFileConsumer.connectIfNecessary(RemoteFileConsumer.java:154) [ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.RemoteFileConsumer.recoverableConnectIfNecessary(RemoteFileConsumer.java:133) [ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.RemoteFileConsumer.prePollCheck(RemoteFileConsumer.java:55) [ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:106) [ftu-server-3.0.2.jar:na]
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) [ftu-server-3.0.2.jar:na]
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114) [ftu-server-3.0.2.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_05]
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_05]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_05]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_05]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_05]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_05]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_05] Caused by: com.jcraft.jsch.JSchException: Algorithm negotiation fail
at com.jcraft.jsch.Session.receive_kexinit(Session.java:582) ~[ftu-server-3.0.2.jar:na]
at com.jcraft.jsch.Session.connect(Session.java:320) ~[ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:115) ~[ftu-server-3.0.2.jar:na]
... 14 common frames omitted
[2014-10-07 10:37:12.316] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpConsumer - Trying to recover connection to: Endpoint[sftp://***@host:port/data?compression=2&delay=70000&disconnect=false&include=filename_2014-10-06.xml&knownHostsFile=%2Fexport%2Fhome%2Flocaluser%2
F.ssh%2Fknown_hosts&localWorkDirectory=%2Fexport%2Fhome%2Flocaluser%2FFTU%2Fserver%2Fwork%2F26&noop=true&pollStrategy=%23POLL_STRATEGY_ENDPOINT_16&privateKeyFile=%2Fexport%2Fhome%2Flocaluser%2F.ssh%2Fid_rsa&readLock=changed&readLockCheckInterval=5000&readLockTimeout=20000&reconnectDelay=60000&throwExceptionOnConnectFailed=true] with a fresh client.

Also debug output from unix sftp command (which connects successfully) is below –

Connecting to host...
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: ssh_connect: needpriv 0
debug1: Connecting to host [host] port port.
debug1: Connection established.
debug1: identity file /export/home/localuser/.ssh/id_rsa type 1
debug1: identity file /export/home/localuser/.ssh/id_dsa type 2
debug1: Logging to host: host
debug1: Local user: localuser Remote user: user
debug1: Remote protocol version 2.0, remote software version Maverick_SSHD
debug1: no match: Maverick_SSHD
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.1.6
debug1: use_engine is 'yes'
debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
debug1: pkcs11 engine initialization complete
debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible Unknown code 0
)
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-sha1 zlib
debug1: kex: client->server aes128-cbc hmac-sha1 zlib
debug1: Peer sent proposed langtags, ctos:
debug1: Peer sent proposed langtags, stoc:
debug1: We proposed langtags, ctos: i-default
debug1: We proposed langtags, stoc: i-default
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: dh_gen_key: priv key bits set: 157/320
debug1: bits set: 523/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'host' is known and matches the RSA host key.
debug1: Found key in /export/home/localuser/.ssh/known_hosts:19
debug1: bits set: 528/1024
debug1: ssh_rsa_verify: signature correct
debug1: newkeys: mode 1
debug1: set_newkeys: setting new keys for 'out' mode
debug1: Enabling compression at level 6.
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: set_newkeys: setting new keys for 'in' mode
debug1: SSH2_MSG_NEWKEYS received
debug1: done: ssh_kex2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug1: got SSH2_MSG_SERVICE_ACCEPT

Regards
Keshav

-----Original Message-----
From: Baweja, Keshav
Sent: Tuesday, September 16, 2014 7:07 PM
To: ***@camel.apache.org
Subject: RE: JSch connection issue with Maverick SSHD server

Hello

JSch has configuration parameters to set compression algorithm for client to server and server to client transport.

compression.c2s
Compression algorithms for client-to-server transport. The default is "none", but this library also supports "zlib" and "***@openssh.com". (Other compression algorithms can't be put in, it seems.)

compression.s2c
Compression algorithms for server-to-client transport. The default is "none", but this library also supports "zlib" and "***@openssh.com". (Other compression algorithms can't be put in, it seems.)

Is it possible to set this configuration parameter in Options part of Camel URI?

The only Option I could see on Camel's FTP2 page is below - compression | Default Value = 0 | SFTP Only: Camel 2.8.3/2.9: To use compression. Specify a level from 1 to 10. Important: You must manually add the needed JSCH zlib JAR to the classpath for compression support.

Does setting this Camel option (with JSch zlib on classpath) automatically sets both these configuration parameters.

Regards
Keshav

-----Original Message-----
From: Baweja, Keshav [mailto:***@jpmorgan.com.INVALID]
Sent: Monday, September 15, 2014 2:30 PM
To: ***@camel.apache.org
Subject: RE: JSch connection issue with Maverick SSHD server

Thanks for your reply Nodet,

The sftp server is hosted externally and is outside our infrastructure. I can't make any changes to the server, and need to make client work with the server configuration. Would you know how I can do it, found the below on Camel's ftp2 page and guess need to look at this -

compression | Default Value = 0 | SFTP Only: Camel 2.8.3/2.9: To use compression. Specify a level from 1 to 10. Important: You must manually add the needed JSCH zlib JAR to the classpath for compression support.

Regards
Keshav

-----Original Message-----
From: Guillaume Nodet [mailto:***@apache.org]
Sent: Monday, September 15, 2014 2:15 PM
To: ***@camel.apache.org
Subject: Re: JSch connection issue with Maverick SSHD server

Your server is configured with "zlib" compression only, while the client only supports compression "none", so there's a mismatch and both can't talk to each other. You need to configure the server with both "zlib" *and* "none" compression factories.

Hello
I am using Camel 2.13.2 in my application to build up routes to
download files from sftp server locations. The codebase has been
tested ok against 5 different sftp servers. However against one
particular sftp server, JSch the sftp implementation used by Camel
seems to disconnect immediately after connecting. Could you please
advise what could be the issue here. The log from application is as
below -
[2014-09-15 05:52:32.236] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - Using private
keyfile: /home/.ssh/id_rsa
[2014-09-15 05:52:32.238] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - Using
knownhosts file: /home/.ssh/known_hosts
[2014-09-15 05:52:32.329] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - Using
StrickHostKeyChecking: no
[2014-09-15 05:52:32.329] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH ->
Connecting to hostname port port
[2014-09-15 05:52:32.462] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH ->
Connection established
[2014-09-15 05:52:32.589] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH -> Remote
version string: SSH-2.0-Maverick_SSHD
[2014-09-15 05:52:32.589] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH -> Local
version string: SSH-2.0-JSCH-0.1.50
[2014-09-15 05:52:32.589] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH ->
aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des
-ctr,arcfour,arcfour128,arcfour256
[2014-09-15 05:52:32.598] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH ->
CheckKexes: diffie-hellman-group14-sha1
[2014-09-15 05:52:32.621] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH ->
SSH_MSG_KEXINIT sent
[2014-09-15 05:52:32.621] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH ->
SSH_MSG_KEXINIT received
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
server: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
server: ssh-rsa
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
aes128-cbc,aes256-cbc,aes192-cbc,3des-cbc,twofish256-cbc,twofish192-cb
c,twofish128-cbc,blowfish-cbc,cast128-cbc
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
aes128-cbc,aes256-cbc,aes192-cbc,3des-cbc,twofish256-cbc,twofish192-cb
c,twofish128-cbc,blowfish-cbc,cast128-cbc
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
server: hmac-sha1,hmac-sha1-96
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
server: hmac-sha1,hmac-sha1-96
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
server: zlib
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
server: zlib
[2014-09-15 05:52:32.622] [Camel (camel-1) thread #0 -
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-
group-exchange-sha1
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
client: ssh-rsa,ssh-dss
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256
-cbc
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc,aes192-cbc,aes256
-cbc
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
client: none
[2014-09-15 05:52:32.623] [Camel (camel-1) thread #0 -
client: none
[2014-09-15 05:52:32.624] [Camel (camel-1) thread #0 -
[2014-09-15 05:52:32.624] [Camel (camel-1) thread #0 -
[2014-09-15 05:52:32.624] [Camel (camel-1) thread #0 -
org.apache.camel.component.file.remote.SftpOperations - JSCH ->
Disconnecting from hostname port port
Regards
Keshav
Regards
Keshav
This email is confidential and subject to important disclaimers and
conditions including on offers for the purchase or sale of securities,
accuracy and completeness of information, viruses, confidentiality,
legal privilege, and legal entity disclaimers, available at
http://www.jpmorgan.com/pages/disclosures/email.

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.

This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.

Baweja, Keshav

2014-10-20 04:33:07 UTC

Permalink

Looking at the code in SftpOperations.java in createSession()

// compression
if (sftpConfig.getCompression() > 0) {
LOG.debug("Using compression: {}", sftpConfig.getCompression());
session.setConfig("compression.s2c", "***@openssh.com, zlib, none");
session.setConfig("compression.c2s", "***@openssh.com, zlib, none");
session.setConfig("compression_level", Integer.toString(sftpConfig.getCompression()));
}

So, the JSch session is set with "***@openssh.com, zlib, none" for "compression.s2c", even though ***@openssh.com might not be configured on sshd for server to client communication. Could this be causing the behavior I encounter below?

Regards
Keshav

-----Original Message-----
From: Baweja, Keshav
Sent: Monday, October 20, 2014 12:17 PM
To: ***@camel.apache.org; jsch-***@lists.sourceforge.net
Subject: Jsch issue with compression algorithm

Hello All,

I have been able to narrow down this issue to below -

1) I enabled client to server compression configuration on Jsch by setting compression=2 as an option on camel route.
2) With this, Jsch sets client to server compression configuration as - JSCH -> kex: client: ***@openssh.com, zlib, none
3) When server is configured as below the sftp connection is successful
JSCH -> kex: server: none,***@openssh.com
4) However, if the server is configured as any of the below the sftp connection is not successful JSCH -> kex: server: none,zlib JSCH -> kex: server: zlib

So if ***@openssh.com is not configured on server, Jsch fails with the exception below

Caused by: com.jcraft.jsch.JSchException: Algorithm negotiation fail
at com.jcraft.jsch.Session.receive_kexinit(Session.java:583) ~[ftu-server-3.0.3.jar:na]
at com.jcraft.jsch.Session.connect(Session.java:320) ~[ftu-server-3.0.3.jar:na]
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:115) ~[ftu-server-3.0.3.jar:na]

Has someone encountered this? Any ideas to resolve this. Thanks in advance.

Regards
Keshav

-----Original Message-----
From: Baweja, Keshav [mailto:***@jpmorgan.com.INVALID]
Sent: Friday, October 10, 2014 2:56 PM
To: ***@camel.apache.org; jsch-***@lists.sourceforge.net
Subject: RE: JSch connection issue with Maverick SSHD server

Hello

I am using Camel routing library to set up sftp routes to a remote sftp server. Camel user Jsch library (version 0.1.50) for all sftp operations and jsch-zlib (version 1.1.3) for the compression. However I encounter the exception below with this set up. Could someone advise/point to a resolution? Many thanks in advance.

[2014-10-07 10:37:11.937] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using private keyfile: /home/localuser/.ssh/id_rsa
[2014-10-07 10:37:11.937] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using knownhosts file: /home/localuser/.ssh/known_hosts
[2014-10-07 10:37:12.018] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using StrickHostKeyChecking: no
[2014-10-07 10:37:12.018] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpOperations - Using compression: 2
[2014-10-07 10:37:12.018] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Connecting to host port port
[2014-10-07 10:37:12.150] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Connection established
[2014-10-07 10:37:12.279] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Remote version string: SSH-2.0-Maverick_SSHD
[2014-10-07 10:37:12.279] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Local version string: SSH-2.0-JSCH-0.1.50
[2014-10-07 10:37:12.279] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> CheckCiphers: aes256-ctr,aes192-ctr,aes128-ctr,aes256-cbc,aes192-cbc,aes128-cbc,3des-ctr,arcfour,arcfour128,arcfour256
[2014-10-07 10:37:12.284] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes256-ctr is not available.
[2014-10-07 10:37:12.284] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes192-ctr is not available.
[2014-10-07 10:37:12.284] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes256-cbc is not available.
[2014-10-07 10:37:12.285] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> aes192-cbc is not available.
[2014-10-07 10:37:12.285] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> arcfour256 is not available.
[2014-10-07 10:37:12.285] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> CheckKexes: diffie-hellman-group14-sha1
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> SSH_MSG_KEXINIT sent
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> SSH_MSG_KEXINIT received
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: diffie-hellman-group1-sha1,diffie-hellman-group-exchange-sha1
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: ssh-rsa
[2014-10-07 10:37:12.308] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: aes128-cbc,aes256-cbc,aes192-cbc,3des-cbc,twofish256-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,cast128-cbc
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: aes128-cbc,aes256-cbc,aes192-cbc,3des-cbc,twofish256-cbc,twofish192-cbc,twofish128-cbc,blowfish-cbc,cast128-cbc
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: hmac-sha1,hmac-sha1-96
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: hmac-sha1,hmac-sha1-96
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: zlib
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server: zlib
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server:
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: server:
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: diffie-hellman-group1-sha1,diffie-hellman-group14-sha1,diffie-hellman-group-exchange-sha1
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: ssh-rsa,ssh-dss
[2014-10-07 10:37:12.309] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: aes128-ctr,aes128-cbc,3des-ctr,3des-cbc,blowfish-cbc
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: hmac-md5,hmac-sha1,hmac-sha2-256,hmac-sha1-96,hmac-md5-96
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: ***@openssh.com, zlib, none
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client: ***@openssh.com, zlib, none
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client:
[2014-10-07 10:37:12.310] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> kex: client:
[2014-10-07 10:37:12.311] [Camel (camel-1) thread #0 - sftp://***@host:port/data] INFO org.apache.camel.component.file.remote.SftpOperations - JSCH -> Disconnecting from host port port
[2014-10-07 10:37:12.316] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpConsumer - Could not connect to:Endpoint[sftp://***@host:port/data?compression=2&delay=70000&disconnect=false&include=filename_2014-10-06.xml&knownHostsFile=%2Fexport%2Fhome%2Flocaluser%2F.ssh%2Fknown_hosts&localWorkDirectory=%2Fexport%2Fhome%2Flocaluser%2FFTU%2Fserver%2Fwork%2F26&noop=true&pollStrategy=%23POLL_STRATEGY_ENDPOINT_16&privateKeyFile=%2Fexport%2Fhome%2Flocaluser%2F.ssh%2Fid_rsa&readLock=changed&readLockCheckInterval=5000&readLockTimeout=20000&reconnectDelay=60000&throwExceptionOnConnectFailed=true]. Will try to recover.
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot connect to sftp://***@host:port
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:143) ~[ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.RemoteFileConsumer.connectIfNecessary(RemoteFileConsumer.java:154) [ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.RemoteFileConsumer.recoverableConnectIfNecessary(RemoteFileConsumer.java:133) [ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.RemoteFileConsumer.prePollCheck(RemoteFileConsumer.java:55) [ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:106) [ftu-server-3.0.2.jar:na]
at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) [ftu-server-3.0.2.jar:na]
at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114) [ftu-server-3.0.2.jar:na]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) [na:1.7.0_05]
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) [na:1.7.0_05]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) [na:1.7.0_05]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) [na:1.7.0_05]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [na:1.7.0_05]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [na:1.7.0_05]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [na:1.7.0_05]
at java.lang.Thread.run(Thread.java:722) [na:1.7.0_05] Caused by: com.jcraft.jsch.JSchException: Algorithm negotiation fail
at com.jcraft.jsch.Session.receive_kexinit(Session.java:582) ~[ftu-server-3.0.2.jar:na]
at com.jcraft.jsch.Session.connect(Session.java:320) ~[ftu-server-3.0.2.jar:na]
at org.apache.camel.component.file.remote.SftpOperations.connect(SftpOperations.java:115) ~[ftu-server-3.0.2.jar:na]
... 14 common frames omitted
[2014-10-07 10:37:12.316] [Camel (camel-1) thread #0 - sftp://***@host:port/data] DEBUG org.apache.camel.component.file.remote.SftpConsumer - Trying to recover connection to: Endpoint[sftp://***@host:port/data?compression=2&delay=70000&disconnect=false&include=filename_2014-10-06.xml&knownHostsFile=%2Fexport%2Fhome%2Flocaluser%2
F.ssh%2Fknown_hosts&localWorkDirectory=%2Fexport%2Fhome%2Flocaluser%2FFTU%2Fserver%2Fwork%2F26&noop=true&pollStrategy=%23POLL_STRATEGY_ENDPOINT_16&privateKeyFile=%2Fexport%2Fhome%2Flocaluser%2F.ssh%2Fid_rsa&readLock=changed&readLockCheckInterval=5000&readLockTimeout=20000&reconnectDelay=60000&throwExceptionOnConnectFailed=true] with a fresh client.

Also debug output from unix sftp command (which connects successfully) is below –

Connecting to host...
debug1: Rhosts Authentication disabled, originating port will not be trusted.
debug1: ssh_connect: needpriv 0
debug1: Connecting to host [host] port port.
debug1: Connection established.
debug1: identity file /export/home/localuser/.ssh/id_rsa type 1
debug1: identity file /export/home/localuser/.ssh/id_dsa type 2
debug1: Logging to host: host
debug1: Local user: localuser Remote user: user
debug1: Remote protocol version 2.0, remote software version Maverick_SSHD
debug1: no match: Maverick_SSHD
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-Sun_SSH_1.1.6
debug1: use_engine is 'yes'
debug1: pkcs11 engine initialized, now setting it as default for RSA, DSA, and symmetric ciphers
debug1: pkcs11 engine initialization complete
debug1: Failed to acquire GSS-API credentials for any mechanisms (No credentials were supplied, or the credentials were unavailable or inaccessible Unknown code 0
)
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-cbc hmac-sha1 zlib
debug1: kex: client->server aes128-cbc hmac-sha1 zlib
debug1: Peer sent proposed langtags, ctos:
debug1: Peer sent proposed langtags, stoc:
debug1: We proposed langtags, ctos: i-default
debug1: We proposed langtags, stoc: i-default
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: dh_gen_key: priv key bits set: 157/320
debug1: bits set: 523/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Host 'host' is known and matches the RSA host key.
debug1: Found key in /export/home/localuser/.ssh/known_hosts:19
debug1: bits set: 528/1024
debug1: ssh_rsa_verify: signature correct
debug1: newkeys: mode 1
debug1: set_newkeys: setting new keys for 'out' mode
debug1: Enabling compression at level 6.
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: newkeys: mode 0
debug1: set_newkeys: setting new keys for 'in' mode
debug1: SSH2_MSG_NEWKEYS received
debug1: done: ssh_kex2.
debug1: send SSH2_MSG_SERVICE_REQUEST
debug1: got SSH2_MSG_SERVICE_ACCEPT

Regards
Keshav

-----Original Message-----
From: Baweja, Keshav
Sent: Tuesday, September 16, 2014 7:07 PM
To: ***@camel.apache.org
Subject: RE: JSch connection issue with Maverick SSHD server

Hello

JSch has configuration parameters to set compression algorithm for client to server and server to client transport.

compression.c2s
Compression algorithms for client-to-server transport. The default is "none", but this library also supports "zlib" and "***@openssh.com". (Other compression algorithms can't be put in, it seems.)

compression.s2c
Compression algorithms for server-to-client transport. The default is "none", but this library also supports "zlib" and "***@openssh.com". (Other compression algorithms can't be put in, it seems.)

Is it possible to set this configuration parameter in Options part of Camel URI?

The only Option I could see on Camel's FTP2 page is below - compression | Default Value = 0 | SFTP Only: Camel 2.8.3/2.9: To use compression. Specify a level from 1 to 10. Important: You must manually add the needed JSCH zlib JAR to the classpath for compression support.

Does setting this Camel option (with JSch zlib on classpath) automatically sets both these configuration parameters.

Regards
Keshav

-----Original Message-----
From: Baweja, Keshav [mailto:***@jpmorgan.com.INVALID]
Sent: Monday, September 15, 2014 2:30 PM
To: ***@camel.apache.org
Subject: RE: JSch connection issue with Maverick SSHD server

Thanks for your reply Nodet,

The sftp server is hosted externally and is outside our infrastructure. I can't make any changes to the server, and need to make client work with the server configuration. Would you know how I can do it, found the below on Camel's ftp2 page and guess need to look at this -

compression | Default Value = 0 | SFTP Only: Camel 2.8.3/2.9: To use compression. Specify a level from 1 to 10. Important: You must manually add the needed JSCH zlib JAR to the classpath for compression support.

Regards
Keshav

-----Original Message-----
From: Guillaume Nodet [mailto:***@apache.org]
Sent: Monday, September 15, 2014 2:15 PM
To: ***@camel.apache.org
Subject: Re: JSch connection issue with Maverick SSHD server

Your server is configured with "zlib" compression only, while the client only supports compression "none", so there's a mismatch and both can't talk to each other. You need to configure the server with both "zlib" *and* "none" compression factories.