old %ames falls into one-way connectivity

[tlon-gh-bot]

Issue by cgyarvin
~2015.11.03..00.10.57
Originally opened as #496

---

For unknown reasons which are unlikely to be fixed. This can be seen in the ~doznec log:

; ~dolten-hoptes not responding still trying
; ~dolten-hoptes is ok
; ~dolten-hoptes not responding still trying
; ~dolten-hoptes is ok

etc.

[tlon-gh-bot]

Comment by galenwp
~2015.11.03..00.15.21

---

Didn't we track down the root cause here? Like, the bug is actually known in specific?

[tlon-gh-bot]

Comment by cgyarvin
~2015.11.03..00.17.47

---

We did not. We know that if the wrong packet gets lost, we lose
connectivity. But we don't know exactly why, and the codebase is "dontfix."

On Mon, Nov 2, 2015 at 4:15 PM, Galen Wolfe-Pauly notifications@github.com
wrote:

Didn't we track down the root cause here? Like, the bug is actually known in
specific?

—
Reply to this email directly or view it on GitHub
#496 (comment).

[tlon-gh-bot]

Comment by galenwp
~2017.02.08..21.07.08

---

Looking forward to new ames.

[Fang-]

From casual observation while working on jael, it seems old/current ames expects the recipient vane (ie the one whose ++call arm is called) to %give a %mack back. If it doesn't, the message is considered not-acknowledged, which can result in one-way connectivity (when enough non-acked messages build up).

[ohAitch]

that's the %ames semantic yeah, it tries to preserve sequentiality by waiting for the old message to be fully processed first. Is there anything in particular that's failing to give the %mack?

[cgyarvin]

That's definitely correct!

…

On Mon, May 21, 2018 at 1:40 PM, Fang ***@***.***> wrote: From casual observation while working on jael, it seems old/current ames expects the recipient vane (ie the one whose ++call arm is called) to %give a %mack back. If it doesn't, the message is considered not-acknowledged, which can result in one-way connectivity (when enough non-acked messages build up). — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <https://github.com/urbit/arvo/issues/554#issuecomment-390775716>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALyAfrxnpcDEksXjgcgg6VHvGrMkcFGks5t0yY5gaJpZM4RdO8T> .

[Fang-]

My WIP code. Don't know about master arvo, but this seems like a potentially nasty pitfall. And since the OWC state this can get you into lines up with the one we're seeing every now and then on livenet stars, I figured it was worth documenting. Might want to sit down and verify all the vanes are macking when they need to at some point, just in case.

[cgyarvin]

It is a nasty pitfall -- but the definition of a vane, as opposed to a %gall app, is that you're trusted not to blow your own foot off. This actually works pretty well in theory.

…

On Mon, May 21, 2018 at 2:01 PM, Fang ***@***.***> wrote: My WIP code. Don't know about master arvo, but this seems like a potentially nasty pitfall. And since the OWC state this can get you into lines up with the one we're seeing every now and then on livenet stars, I figured it was worth documenting. Might want to sit down and verify all the vanes are macking when they need to at some point, just in case. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/urbit/arvo/issues/554#issuecomment-390781457>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALyAe6iHWnrjSGPpLAOcybXpbI4ngj1ks5t0ysigaJpZM4RdO8T> .

[Fang-]

We should probably start providing gun-safety courses, then. (^:

I'd argue that new ames should send a [%mack ~] by default if the vane doesn't provide an alternative after successfully processing the message, but that might be hairier than I imagine.

[cgyarvin]

Not automatically acknowledging allows you to delay the acknowledgment, which is in many cases what you actually want -- since acknowledgements are supposed to be end-to-end, and indicate that the transaction has completely succeeded. In fact, one feature %gall needs is a way for apps to indicate that message processing will not complete in this event. Of course, since %gall apps are fallible, %gall itself needs to track this and be ready to step in.

…

On Mon, May 21, 2018 at 2:12 PM, Fang ***@***.***> wrote: We should probably start providing gun-safety courses, then. (^: I'd argue that new ames should send a [%mack ~] by default if the vane doesn't provide an alternative after successfully processing the message, but that might be hairier than I imagine. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/urbit/arvo/issues/554#issuecomment-390784505>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALyATistQ_QHbH-GliD1aowr5VOF5Frks5t0y3QgaJpZM4RdO8T> .

[ohAitch]

Definitely *in gall*, if we ever get the e2e poke ack mechanism back, it should work on an immediacy opt-out basis :3 For vanes, code review of every ++take %west is probably sufficient

…

On Mon, May 21, 2018 at 2:12 PM, Fang ***@***.***> wrote: We should probably start providing gun-safety courses, then. (^: I'd argue that new ames should send a [%mack ~] by default if the vane doesn't provide an alternative after successfully processing the message, but that might be hairier than I imagine. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/urbit/arvo/issues/554#issuecomment-390784505>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABxXhrWDIP8in2N9RVYK_scPFCSEYUXQks5t0y3QgaJpZM4RdO8T> .

[cgyarvin]

Yes, opt-out is definitely the right approach to %gall apps (you say "waiting on this one," and if you don't it assumes you succeeded.) On Mon, May 21, 2018 at 2:17 PM, Anton Dyudin <notifications@github.com> wrote:

…

Definitely *in gall*, if we ever get the e2e poke ack mechanism back, it should work on an immediacy opt-out basis :3 For vanes, code review of every ++take %west is probably sufficient On Mon, May 21, 2018 at 2:12 PM, Fang ***@***.***> wrote: > We should probably start providing gun-safety courses, then. (^: > > I'd argue that new ames should send a [%mack ~] by default if the vane > doesn't provide an alternative after successfully processing the message, > but that might be hairier than I imagine. > > — > You are receiving this because you commented. > Reply to this email directly, view it on GitHub > <https://github.com/urbit/arvo/issues/554#issuecomment-390784505>, or mute > the thread > <https://github.com/notifications/unsubscribe-auth/ABxXhrWDIP8in2N9RVYK_ scPFCSEYUXQks5t0y3QgaJpZM4RdO8T> > . > — You are receiving this because you commented. Reply to this email directly, view it on GitHub <https://github.com/urbit/arvo/issues/554#issuecomment-390785707>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AALyAVtSEuT2YA1YxnWrwvY1_TFjf5MKks5t0y7hgaJpZM4RdO8T> .

[ohAitch]

One possible safety layer here is to require a %mack or timeout request, and error the event out if we haven't gotten either by the end of it. The timeout request can then be serviced by ames into an actual %behn timeout for the duct, which when fired again wants a %mack or time bound on checking in with the downstream that the thread isn't dropped; with a reasonable default(%gall arm-not-found) implementation being "convert timeout directly to an error ack".

[ohAitch]

(this ofc screws with CQRS desirables wrt "an ames message either causes an effect or returns an error", but like "an HTTP API request dies after being processed but before being acknowledged" is a real case where we can't just hang forever, the problem is part of the territory)

[cgyarvin]

That’s pretty bureaucratic! But I could see it...

…

Sent from my iPhone

On May 21, 2018, at 2:24 PM, Anton Dyudin ***@***.***> wrote: One possible safety layer here is to require a %mack or timeout request, and error the event out if we haven't gotten either by the end of it. The timeout request can then be serviced by ames into an actual %behn timeout for the duct, which when fired again wants a %mack or time bound on checking in with the downstream that the thread isn't dropped; with a reasonable default(%gall arm-not-found) implementation being "convert timeout directly to an error ack". — You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.