sftp clients based on the Mina-sshd and JSCH components upload and download files at very different speeds

[czldb2]

Phenomenon：

Uploading and downloading files with mina-sshd is about twice as slow as JSCH

Code：

Mina-sshd

Dependency

<dependency>
    <groupId>org.apache.sshd</groupId>
    <artifactId>sshd-core</artifactId>
    <version>2.13.0</version>
</dependency>
<dependency>
    <groupId>org.apache.sshd</groupId>
    <artifactId>sshd-common</artifactId>
    <version>2.13.0</version>
</dependency>
<dependency>
    <groupId>org.apache.sshd</groupId>
    <artifactId>sshd-sftp</artifactId>
    <version>2.13.0</version>
</dependency>

demo

import org.apache.sshd.client.SshClient;
import org.apache.sshd.client.future.ConnectFuture;
import org.apache.sshd.client.session.ClientSession;
import org.apache.sshd.sftp.client.SftpClient;
import org.apache.sshd.sftp.client.SftpClientFactory;
import org.apache.sshd.common.util.io.IoUtils;

import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.concurrent.TimeUnit;

public class SftpTestTool {

    public static void main(String[] args) {
        String host = "x.x.x.x";
        int port = 22;
        String user = "root";
        String password = "root123";

        String localFile = "src/main/resources/temp/temp.gz";
        String remoteFile = "/home/user/temp/temp.gz";

        SshClient client = SshClient.setUpDefaultClient();
        client.start();

        long connectStartTime = System.currentTimeMillis();
        try (ClientSession session = createSession(client, host, port, user, password)) {
            SftpClientFactory factory = SftpClientFactory.instance();
            try (SftpClient sftpClient = factory.createSftpClient(session)) {
                long connectEndTime = System.currentTimeMillis();
                System.out.println("Mina SSHD connect Time: " + (connectEndTime - connectStartTime) + " ms");
                long uploadStartTime = System.currentTimeMillis();
                try (InputStream localInputStream = Files.newInputStream(Paths.get(localFile));
                     OutputStream remoteOutputStream = sftpClient.write(remoteFile)) {
                    IoUtils.copy(localInputStream, remoteOutputStream);
                }
                long uploadEndTime = System.currentTimeMillis();
                System.out.println("Mina SSHD Upload Time: " + (uploadEndTime - uploadStartTime) + " ms");

                long downloadStartTime = System.currentTimeMillis();
                try (InputStream remoteInputStream = sftpClient.read(remoteFile);
                     OutputStream localOutputStream = Files.newOutputStream(Paths.get("src/main/resources/temp/temp.gz.bak"))) {
                    IoUtils.copy(remoteInputStream, localOutputStream);
                }
                long downloadEndTime = System.currentTimeMillis();
                System.out.println("Mina SSHD Download Time: " + (downloadEndTime - downloadStartTime) + " ms");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            client.stop();
        }
    }

    private static ClientSession createSession(SshClient client, String host, int port, String user, String password) throws Exception {
        ConnectFuture connectFuture = client.connect(user, host, port);
        connectFuture.await(5, TimeUnit.SECONDS);

        ClientSession session = connectFuture.getSession();
        session.addPasswordIdentity(password);
        session.auth().verify(5, TimeUnit.SECONDS);
        return session;
    }

}

JSCH

Dependency

<dependency>
    <groupId>com.github.mwiede</groupId>
    <artifactId>jsch</artifactId>
    <version>0.2.3</version>
</dependency>

demo

import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.util.Properties;

public class JSCHTestTool {

public static void main(String[] args) {
String host = "x.x.x.x";
 int port = 22;
 String user = "root";
 String password = "root123";

 String localFile = "src/main/resources/temp/temp.gz";
 String remoteFile = "/home/user/temp/temp.gz";

 JSch jsch = new JSch();
 Session session = null;
 ChannelSftp channelSftp = null;

 try {
long connectStartTime = System.currentTimeMillis();
 session = jsch.getSession(user, host, port);
 session.setPassword(password);
 Properties config = new Properties();
 config.put("StrictHostKeyChecking", "no");
 session.setConfig(config);
 session.connect();

 channelSftp = (ChannelSftp) session.openChannel("sftp");
 channelSftp.connect();
 long connectEndTime = System.currentTimeMillis();
 System.out.println("JSch connect Time: " + (connectEndTime - connectStartTime) + " ms");

 long uploadStartTime = System.currentTimeMillis();
 channelSftp.put(new FileInputStream(localFile), remoteFile);
 long uploadEndTime = System.currentTimeMillis();
 System.out.println("JSch Upload Time: " + (uploadEndTime - uploadStartTime) + " ms");

 long downloadStartTime = System.currentTimeMillis();
 channelSftp.get(remoteFile, new FileOutputStream("src/main/resources/temp/temp.gz.bak"));
 long downloadEndTime = System.currentTimeMillis();
 System.out.println("JSch Download Time: " + (downloadEndTime - downloadStartTime) + " ms");

 } catch (Exception e) {
e.printStackTrace();
 } finally {
if (channelSftp != null) {
channelSftp.disconnect();
 }
if (session != null) {
session.disconnect();
 }
}
}

}

Question

Is there something wrong with my coding? Is there any configuration or correct encoding of mina-sshd so that my sftp function can reach the same speed level as jsch?

[tomaswolf]

Cannot reproduce on Mac OS X, running an Apache MINA sshd SFTP client against an OpenSSH SFTP server running in an Alpine container.

IoUtils.copy(InputStream, OutputStream) by default uses a smallish buffer of 8kB. With that, I see performance slightly slower than JSCH. Using a 32kB buffer via IoUtils.copy(InputStream, OutputStream, 32 * 1024) I get slightly better performance with Apache MINA sshd than with JSCH.

[czldb2]

Cannot reproduce on Mac OS X, running an Apache MINA sshd SFTP client against an OpenSSH SFTP server running in an Alpine container.

IoUtils.copy(InputStream, OutputStream) by default uses a smallish buffer of 8kB. With that, I see performance slightly slower than JSCH. Using a 32kB buffer via IoUtils.copy(InputStream, OutputStream, 32 * 1024) I get slightly better performance with Apache MINA sshd than with JSCH.

I modified the buffer size and made some changes in thread switching, but the transfer efficiency is still the same as before

import org.apache.sshd.client.SshClient;
import org.apache.sshd.client.future.ConnectFuture;
import org.apache.sshd.client.session.ClientSession;
import org.apache.sshd.sftp.client.SftpClient;
import org.apache.sshd.sftp.client.SftpClientFactory;

import java.io.InputStream;
import java.io.OutputStream;
import java.nio.file.Files;
import java.nio.file.Paths;
import java.util.EnumSet;
import java.util.concurrent.TimeUnit;

public class SftpTestTool {

    private static final int BUFFER_SIZE = 65536;

    public static void main(String[] args) {
        String host = "x.x.x.x";
        int port = 22;
        String user = "root";
        String password = "rootr123";

        String localFile = "src/main/resources/temp/temp.gz";
        String remoteFile = "/home/user/temp/temp.gz";

        SshClient client = SshClient.setUpDefaultClient();
        client.start();

        long connectStartTime = System.currentTimeMillis();
        try (ClientSession session = createSession(client, host, port, user, password)) {
            SftpClientFactory factory = SftpClientFactory.instance();
            try (SftpClient sftpClient = factory.createSftpClient(session)) {
                long connectEndTime = System.currentTimeMillis();
                System.out.println("Mina SSHD connect Time: " + (connectEndTime - connectStartTime) + " ms");

                long uploadStartTime = System.currentTimeMillis();
                try (InputStream localInputStream = Files.newInputStream(Paths.get(localFile));
                     OutputStream remoteOutputStream = sftpClient.write(remoteFile, EnumSet.of(SftpClient.OpenMode.Create, SftpClient.OpenMode.Write))) {
                    byte[] buffer = new byte[BUFFER_SIZE];
                    int bytesRead;
                    while ((bytesRead = localInputStream.read(buffer)) != -1) {
                        remoteOutputStream.write(buffer, 0, bytesRead);
                    }
                }
                long uploadEndTime = System.currentTimeMillis();
                System.out.println("Mina SSHD Upload Time: " + (uploadEndTime - uploadStartTime) + " ms");

                long downloadStartTime = System.currentTimeMillis();
                try (InputStream remoteInputStream = sftpClient.read(remoteFile);
                     OutputStream localOutputStream = Files.newOutputStream(Paths.get("src/main/resources/temp/temp.gz.bak"))) {
                    byte[] buffer = new byte[BUFFER_SIZE];
                    int bytesRead;
                    while ((bytesRead = remoteInputStream.read(buffer)) != -1) {
                        localOutputStream.write(buffer, 0, bytesRead);
                    }
                }
                long downloadEndTime = System.currentTimeMillis();
                System.out.println("Mina SSHD Download Time: " + (downloadEndTime - downloadStartTime) + " ms");
            }
        } catch (Exception e) {
            e.printStackTrace();
        } finally {
            client.stop();
        }
    }

    private static ClientSession createSession(SshClient client, String host, int port, String user, String password) throws Exception {
        ConnectFuture connectFuture = client.connect(user, host, port);
        connectFuture.await(5, TimeUnit.SECONDS);

        ClientSession session = connectFuture.getSession();
        session.addPasswordIdentity(password);
        session.auth().verify(5, TimeUnit.SECONDS);
        return session;
    }
}

[czldb2]

I found that the write() method of the SftpOutputStreamAsync class seems to check every sftp packet; And then SftpRemotePathChannel has a transferTo() method that looks like it's aggregated internally to check. I am not sure if this is the case, and then I would like to ask the use of transforTo() or transforFrom(). Another scenario is how to handle the sftp ls command in the case of a large number of files or directories like transforTo()?

[tomaswolf]

On Windows, I see inconsistent timings, but indeed most of the time the file transfer with Apache MINA sshd appears to be about 50% slower than with JSch.

[czldb2]

On Windows, I see inconsistent timings, but indeed most of the time the file transfer with Apache MINA sshd appears to be about 50% slower than with JSch.

Do we have plans to optimize this feature?

I found that every time the data is sent, it needs to be confirmed, but jsch seems to confirm the same

[tomaswolf]

This is something else, and indeed I have an idea about providing asynchronous SFTP client interfaces. But this is unrelated to the question of why file transfer on Windows is slower than expected, but apparently not on OS X. Off-hand I don't see where the bottleneck might be; it should not be the ACK handling in SftpOutputStreamAsync: that one only checks ACKs that have already arrived.

This may need extensive profiling or log analysis.

[tomaswolf]

This problem definitely is specific to Windows. It does not occur on OS X. (Didn't test Linux.)

The problem occurs with both the NIO2 and the netty I/O back-end. (Didn't try MINA.) Performance with either back-end is the same. So it's not an I/O back-end problem.

The problem also definitely is not in the SFTP layer. I performed and timed a direct file upload via the equivalent of cat local_file | ssh user@somehost 'cat > remote_file' with Apache MINA sshd:

try (ChannelExec exec = session.createExecChannel("cat > upload/testfile.bin")) {
    exec.setStreaming(Streaming.Async);
    exec.open().verify();
    long start = System.currentTimeMillis();
    try (InputStream localInputStream = Files.newInputStream(localFile)) {
        IoOutputStream out = exec.getAsyncIn();
        byte[] buf = new byte[32 * 1024];
        int n = 0;
        while ((n = localInputStream.read(buf)) >= 0) {
            if (n > 0) {
                out.writeBuffer(new ByteArrayBuffer(buf, 0, n)).verify(1000);
            }
        }
        out.close(false);
    }
    long elapsed = System.currentTimeMillis() - start;
    System.err.println("Apache cat stream upload took " + (elapsed / 1000.0) + "sec");
}

This uploads the file without SFTP, so there is zero overhead for the SFTP protocol and its ACKs. Timing this (repeatedly) showed no significant differences from normal uploads via SFTP. Which means that the bottleneck is not in the SFTP code. (Assuming that on the server side using "cat" is not slower than "internal-sftp". The server again is OpenSSH running in an alpine:latest container.)

As mentioned before, the timings are fairly inconsistent from run to run, too. (For both JSch and Apache MINA sshd.) In some runs JSch is slow, too, and Apache MINA sshd is actually faster. In some runs, both are fast (and about equal). In most runs, though, Apache MINA sshd is noticeably slower.

Might be a threading issue, or a memory issue. Or a channel window issue (though then I'd expect to see the problem also on OS X). But whatever it is, it isn't in the SFTP code.

[czldb2]

This problem definitely is specific to Windows. It does not occur on OS X. (Didn't test Linux.)

The problem occurs with both the NIO2 and the netty I/O back-end. (Didn't try MINA.) Performance with either back-end is the same. So it's not an I/O back-end problem.

The problem also definitely is not in the SFTP layer. I performed and timed a direct file upload via the equivalent of cat local_file | ssh user@somehost 'cat > remote_file' with Apache MINA sshd:

try (ChannelExec exec = session.createExecChannel("cat > upload/testfile.bin")) {
    exec.setStreaming(Streaming.Async);
    exec.open().verify();
    long start = System.currentTimeMillis();
    try (InputStream localInputStream = Files.newInputStream(localFile)) {
        IoOutputStream out = exec.getAsyncIn();
        byte[] buf = new byte[32 * 1024];
        int n = 0;
        while ((n = localInputStream.read(buf)) >= 0) {
            if (n > 0) {
                out.writeBuffer(new ByteArrayBuffer(buf, 0, n)).verify(1000);
            }
        }
        out.close(false);
    }
    long elapsed = System.currentTimeMillis() - start;
    System.err.println("Apache cat stream upload took " + (elapsed / 1000.0) + "sec");
}

This uploads the file without SFTP, so there is zero overhead for the SFTP protocol and its ACKs. Timing this (repeatedly) showed no significant differences from normal uploads via SFTP. Which means that the bottleneck is not in the SFTP code. (Assuming that on the server side using "cat" is not slower than "internal-sftp". The server again is OpenSSH running in an alpine:latest container.)

As mentioned before, the timings are fairly inconsistent from run to run, too. (For both JSch and Apache MINA sshd.) In some runs JSch is slow, too, and Apache MINA sshd is actually faster. In some runs, both are fast (and about equal). In most runs, though, Apache MINA sshd is noticeably slower.

Might be a threading issue, or a memory issue. Or a channel window issue (though then I'd expect to see the problem also on OS X). But whatever it is, it isn't in the SFTP code.

Thank you for your answer, I still need to learn more knowledge and then analyze this problem

[czldb2]

I used the transferFrom() method of SftpRemotePathChannel for file operations in my windows environment twice as fast as the default write() method of the SftpOutputStreamAsync class

[tomaswolf]

I run tests using a wide variety of methods, including transferFrom. I do not get consistently good timings with that either on Windows, but it is generally one of the faster methods. Which is actually a bit surprising, since as you may notice it uses SftpOutputStreamAsync .write(), too.

[ryabovgs]

Hi @tomaswolf. Could you please advise how I can optimise Mina setup (I use default one) to reach same performance as JSCH?
Problem: I got 20% degradation in speed for uploading of 50 files of 40mb
Main concern from profiling point of view(VisualVM) that OutputStream::write looks as 27s total vs 10s of JSCH.
Approach: I use Streams approach as you mentioned as fastest.
Default setup and streams approach details:

• SshClient.setUpDefaultClient()
• DefaultSftpClientFactory.INSTANCE.createSftpClient(clientSession);
• outputStream = sftpClient.write(file) // here during debug session I see that bufferSize 32755 is used
• tried also MinaServiceFactoryFactory/Nio2ServiceFactoryFactory without good speed too via setIoServiceFactoryFactory(sshClient, factory);
  May be the reason could be async nature of Mina and I missed the guide how to use it optimistically - could you plz point out if it exists also.

[tomaswolf]

Could you please advise how I can optimise Mina setup (I use default one) to reach same performance as JSCH?

@ryabovgs : no, I cannot. If you read the comments in this issue, you see that currently I only know that there is a performance problem on Windows (and apparently Windows only), but I don't know yet why. I normally develop on OS X, and I have only temporary and sporadic access to a Windows machine. So it may be a while until I figure this out.