Skip to content

Implement core functionalities of the IO component in scaler - YMQ#149

Merged
sharpener6 merged 50 commits intofinos:mainfrom
gxuu:ymq
Jul 22, 2025
Merged

Implement core functionalities of the IO component in scaler - YMQ#149
sharpener6 merged 50 commits intofinos:mainfrom
gxuu:ymq

Conversation

@gxuu
Copy link
Contributor

@gxuu gxuu commented Jun 10, 2025

READY FOR REVIEW.

Implements the C++ core of YMQ, the network part of scaler.

The initial design and implementation differs by about 10%, with majority comes from the idea of RecvMessageCallback should be placed in the IOSocket level.

General execution flow behaves like the following graph.

realFinalYMQ drawio

Limitations

Ability

  • Current implementation does not allow listen on multiple addresses.
  • Current implementation does not allow back-to-back connect to multiple addresses.
  • Current implementation does not allow connect to remote end when an existing connection is retrying.
  • Current implementation does not allow duplicate identity.
  • Current implementation does not support NS lookup.

Performance

  • Current implementation do sendMessage in O(n) time which can be reduced to O(lg n) easily.
  • Current implementation do connection construction in O(n) time which can be reduced to O(lg n) not so easily.

Security

  • Current implementation does not clear empty connection (ones that connect but never send anything).
  • Current implementation does not clear connections with abnormal traffic (for example sending one bytes per 5 seconds).
  • Current implementation does not set limits on package size.
  • Current implementation does not differentiate true close and false close (based on the assumption that our use case don't close until the program finishes).

@gxuu gxuu force-pushed the ymq branch 8 times, most recently from 0e9cd20 to b65c155 Compare June 16, 2025 12:28
@gxuu gxuu changed the title WIP Wire timedQueue up with EventloopThread WIP Implement Exchange Identity etc. Jun 16, 2025
@gxuu gxuu force-pushed the ymq branch 4 times, most recently from b346485 to 90a1744 Compare June 22, 2025 03:15
@linux-foundation-easycla
Copy link

linux-foundation-easycla bot commented Jun 23, 2025

@gxuu gxuu force-pushed the ymq branch 10 times, most recently from 885c371 to b651dd5 Compare June 24, 2025 16:52
gxuu added 16 commits July 15, 2025 08:16
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
@gxuu gxuu changed the title WIP Implement Exchange Identity etc. Implement core functionalities of the IO component in scaler - YMQ Jul 21, 2025
Signed-off-by: gxu <georgexu420@163.com>
Signed-off-by: gxu <georgexu420@163.com>
@sharpener6 sharpener6 dismissed magniloquency’s stale review July 22, 2025 23:45

This is not applicable

@sharpener6 sharpener6 merged commit 4506ba9 into finos:main Jul 22, 2025
5 checks passed
rafa-be pushed a commit to rafa-be/scaler that referenced this pull request Jul 30, 2025
…inos#149)

* Update to use g++-14 to compile instead of the defaulted g++-13 etc.

- update inaccurate wording in the file
- make sure scripts are executable

Signed-off-by: gxu <georgexu420@163.com>

* Use CMAKE_BUILD_TYPE as the source of truth for what build we are on

Signed-off-by: gxu <georgexu420@163.com>

* Update TODOs and DONEs in readme

Signed-off-by: gxu <georgexu420@163.com>

* Implement C++ core of YMQ network library

Signed-off-by: gxu <georgexu420@163.com>

* Update pymod io_context to make sure the code compiles (TMP FIX!)

Signed-off-by: gxu <georgexu420@163.com>

* Provide basic examples on how to use YMQ in C++

Signed-off-by: gxu <georgexu420@163.com>

* Bump version number

Signed-off-by: gxu <georgexu420@163.com>

* Add qualifiers and attributes to functions (noexcept, nodiscard, etc.)

Signed-off-by: gxu <georgexu420@163.com>

* Change calls to move_only_functions so functions are not copied

Signed-off-by: gxu <georgexu420@163.com>

* Extract TCP Operations to their own header

Signed-off-by: gxu <georgexu420@163.com>

* Implement IOSocketType behavior in YMQ

Signed-off-by: gxu <georgexu420@163.com>

* Even Unicast and Multicast needs to handle reconnect

Signed-off-by: gxu <georgexu420@163.com>

* Add pub-sub examples to demonstrate socket types

Signed-off-by: gxu <georgexu420@163.com>

* upd readme (need rebase later)

Signed-off-by: gxu <georgexu420@163.com>

* Improve the examples

Signed-off-by: gxu <georgexu420@163.com>

* Change to new socket types Binder and Connector

Signed-off-by: gxu <georgexu420@163.com>

* Refactor: MessageConnectionTCP is easier to construct correctly

Signed-off-by: gxu <georgexu420@163.com>

* upd readme (need rebase later)

Signed-off-by: gxu <georgexu420@163.com>

* Implement ISO 8601 conforming timestamp output

Signed-off-by: gxu <georgexu420@163.com>

* Reference implementation of the error type

Signed-off-by: gxu <georgexu420@163.com>

* Example usage to the error type

Signed-off-by: gxu <georgexu420@163.com>

* upd readme (need rebase later)

Signed-off-by: gxu <georgexu420@163.com>

* Make comments better looking

Signed-off-by: gxu <georgexu420@163.com>

* fixup! Reference implementation of the error type

Signed-off-by: gxu <georgexu420@163.com>

* fixup! Example usage to the error type

Signed-off-by: gxu <georgexu420@163.com>

* Use switch on enum to handle error->explanation translation

Signed-off-by: gxu <georgexu420@163.com>

* use std::string_view instead of const char*

Signed-off-by: gxu <georgexu420@163.com>

* Fix off by one error when getting format string

Signed-off-by: gxu <georgexu420@163.com>

* Refinements on Error implementation

Signed-off-by: gxu <georgexu420@163.com>

* Refactor: Put network utils in their own file

Signed-off-by: gxu <georgexu420@163.com>

* Refactor: createIOSocket takes a callback and returns nothing

Signed-off-by: gxu <georgexu420@163.com>

* Remove unneeded comments

Signed-off-by: gxu <georgexu420@163.com>

* Fix pymod ymq build due to namespace change

Signed-off-by: gxu <georgexu420@163.com>

* Add namespace scaler::ymq for every file

Signed-off-by: gxu <georgexu420@163.com>

* Refactor: Extract common part of examples to header file

Signed-off-by: gxu <georgexu420@163.com>

* check epoll_wait calls return value

Signed-off-by: gxu <georgexu420@163.com>

* Update automated examples

Signed-off-by: gxu <georgexu420@163.com>

* Fixup typo found in logging.h

Signed-off-by: gxu <georgexu420@163.com>

* Fix: send/recv were implemented incorrectly but not shows up in low tp.

Signed-off-by: gxu <georgexu420@163.com>

* Fix bugs for not adding the correct offset (Thanks magniloquency!)

Signed-off-by: gxu <georgexu420@163.com>

* Update the exmaple to mimic high throughput scenario

Signed-off-by: gxu <georgexu420@163.com>

* Make sure onRead returns early when socket is closed and set to 0

Signed-off-by: gxu <georgexu420@163.com>

* Update automated example to not use hard coded address in syncBindTo

Signed-off-by: gxu <georgexu420@163.com>

* Update error's implementation

Signed-off-by: gxu <georgexu420@163.com>

* Implement error abort for syscalls

Signed-off-by: gxu <georgexu420@163.com>

* Fix: Did not return when receiving EAGAIN

Signed-off-by: gxu <georgexu420@163.com>

* Add notice with respect to thread safety of methods

Signed-off-by: gxu <georgexu420@163.com>

* Remove unneeded IOSocketType(s)

Signed-off-by: gxu <georgexu420@163.com>

---------

Signed-off-by: gxu <georgexu420@163.com>
@gxuu gxuu mentioned this pull request Aug 18, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants