Why is there a limit of 128MB for RDMA transfer?

[bibhas]

I got the following error while running perf_rdma (from the tutorial branch).

$ sudo ./bin/test --ip_addr="192.168.10.3" --runs=10 --operation=1 --max_size=1073741824

-- CLI PARAMETERS:
-----------------------------------------------
Server's TCP address: 192.168.10.3
Benchmark operation: WRITE
Number of test runs: 10
Starting transfer size: 64
Ending transfer size: 1073741824

Queue pair:
Local: : QPN 0x000000, PSN 0xb19b73, VADDR 00007fd2c6800000, SIZE 40000000, IP 0xc0a80960
Remote: : QPN 0x000000, PSN 0x9a7a73, VADDR 000070792d400000, SIZE 40000000, IP 0xc0a80959
Client registered

-- RDMA BENCHMARK: CLIENT
-----------------------------------------------
Size:       64; Average throughput:  112.483 MB/s; Average latency:    3.795 us
Size:      128; Average throughput:  219.898 MB/s; Average latency:  3.85215 us
Size:      256; Average throughput:  447.974 MB/s; Average latency:   3.9043 us
Size:      512; Average throughput:  880.373 MB/s; Average latency:    4.877 us
Size:     1024; Average throughput:  1786.59 MB/s; Average latency:   4.2827 us
Size:     2048; Average throughput:  3546.35 MB/s; Average latency:  4.54125 us
Size:     4096; Average throughput:  6848.15 MB/s; Average latency:  5.08785 us
Size:     8192; Average throughput:  9273.26 MB/s; Average latency:  5.69715 us
Size:    16384; Average throughput:  10856.7 MB/s; Average latency:   6.0724 us
Size:    32768; Average throughput:  11008.5 MB/s; Average latency:     7.89 us
Size:    65536; Average throughput:  10464.5 MB/s; Average latency:  10.0662 us
Size:   131072; Average throughput:  10299.1 MB/s; Average latency:  15.3505 us
Size:   262144; Average throughput:  10212.5 MB/s; Average latency:  28.1202 us
Size:   524288; Average throughput:  10143.9 MB/s; Average latency:  52.2876 us
Size:  1048576; Average throughput:  10129.2 MB/s; Average latency:  101.821 us
Size:  2097152; Average throughput:  10222.6 MB/s; Average latency:  200.815 us
Size:  4194304; Average throughput:    10647 MB/s; Average latency:  394.933 us
Size:  8388608; Average throughput:  10815.1 MB/s; Average latency:  743.399 us
Size: 16777216; Average throughput:  11199.2 MB/s; Average latency:   1444.3 us
Size: 33554432; Average throughput:  11153.9 MB/s; Average latency:  2891.36 us
Size: 67108864; Average throughput:    11227 MB/s; Average latency:  5755.92 us
terminate called after throwing an instance of 'std::runtime_error'
  what():  Transfers over 128MB are currently not supported in Coyote. Exiting...

I can see that bThread.cpp is placing a limit on the transfer size. What is the reason for this? Is this a constant I can change in the HDL code? Or, is this a fundamental limitation of Coyote?

[bo3z]

Currently a limitation - we dedicate 28 bits to the buffer length register in hardware. We may look into changing this in the future; for know, the recommended approach is to partition transfers over 128MB from software. This should have no overhead, as cThread->invoke is asynchronous, so it's possible to fire multiple invoke calls one after another, so:
cThread->invoke(buffer)
cThread->invoke(buffer + 128MB offset)
etc.

[bibhas]

I see. Thank you @bo3z!