Cypht Webmail (aggregator): Add JMAP support (as an alternative to IMAP + SMTP) and permit unified inbox (and even move or copy emails from one account to another)

[marclaporte]

"JMAP is intended to be a new standard for email clients to connect to mail stores. It therefore intends to primarily replace IMAP + SMTP submission. It is also designed to be more generic such that it can be extended with contacts, calendars in the future (replacing CardDAV/CalDAV). It does not replace MTA-to-MTA SMTP transmission."

http://jmap.io/

[jasonmunro]

Thanks for the feedback. I have heard of jmap before, and briefly reviewed it a while back, but I'm not super familiar with the specification. Could be a cool addition to Cypht as either a stand alone module set or as part of the imap modules.

[marclaporte]

Super!

Someone else asked me "Who uses or supports it?" My answer copied below:

Cyrus IMAP and Apache James are the notable ones here: http://jmap.io/software.html

OpenPaaS is an open source Entreprise Social Platform which is led by LINAGORA (which has more than 150 employees)
https://linagora.com/
http://www.open-paas.org

OpenPaaS is using Apache James. And you can see JMAP used in the code:
https://github.com/linagora/openpaas-esn/search?utf8=%E2%9C%93&q=jmap&type=

In this video: https://www.youtube.com/watch?v=2QZIi3EO6rQ , Benoit Tellier explains why they decided to invest in James and JMAP (among other things). LINAGORA has 5 full time employees which are devoted to the project.

Why LINAGORA picked JMAP:
https://medium.com/linagora-engineering/you-cant-do-imap-in-the-browser-can-you-208838b0ba17

FastMail started this initiative:
"JMAP is FastMail's protocol with the warts removed. We leverage existing standards like HTTP, JSON, and native push channels on platforms which have them - making it easy for developers to work with."
https://blog.fastmail.com/2016/12/22/cyrus-development-and-release-plans/
https://blog.fastmail.com/2016/12/12/why-we-contribute/
https://blog.fastmail.com/2016/12/09/jmap-test-suite/
https://blog.fastmail.com/2015/12/23/the-jmap-momentum-builds/
https://blog.fastmail.com/2015/08/11/an-open-source-jmap-proxy-javascript-library-and-webmail-demo/
https://blog.fastmail.com/2014/12/23/jmap-a-better-way-to-email/

Roundcube Next will support JMAP as well.
https://github.com/search?q=org%3Aroundcube-next+jmap&type=Code

Thanks!

[jasonmunro]

Thanks for that info. I took another peek at the JMAP spec after you submitted this. Looks like a pretty straight forward rest-style api. Cypht was designed to aggregate data from different sources, so a lot of the work of uncoupling from a particular api or protocol is baked in already. Having worked with the IMAP protocol for 15+ years, I understand more than most why something like JMAP is needed :) I think it differs enough from IMAP that it would make sense to create a stand alone module set (also the IMAP module set is already a tremendous amount of code).

I can't promise I will get to this real soon, but I am intrigued. I suspect initial support for this would entail a similar amount of work that the Github module set took - I also suspect we could abstract various bits of the existing IMAP code where the two protocols might overlap.

[marclaporte]

Dear Jason,

Now is a good time to start the implementation:
https://fastmail.blog/2018/12/27/jmap-is-on-the-home-straight/

Thanks!!!

[dumblob]

@marclaporte how I see it, the most important information is, that during the upcoming weeks/months, JMAP will finally become an IETF standard. Which is a huge step from my point of view, but seems to be also a huge step forward form the JMAP developers point of view (the IETF processes increased the JMAP quality significantly).

[jasonmunro]

@dumblob @marclaporte I agree and that was an interesting read, thanks for the link. It's a new year and I'm feeling ambitious so I'm going to bump this up and see what I can get done. The other new feature in the works is cardDav, which has read only support now, and I want to expand that to read/write then move on to calDav support. No reason we can't push both that and JMAP forward at the same time :)

One prerequisite for this is I want to create a "skeleton" module set for new data sources. Sort of a template of all (or at least most) of the entry points required to create a new data source inside Cypht. Then I can use that as the foundation for supporting JMAP, as well as making it easier to build support for any other protocol we want to implement.

[ulfgebhardt]

A slim PHP Library is missing at current time.

There is one for Zend:
https://github.com/WikiSuite/zend-jmap

There is many for JS (which is not compatible with current architecture of cypht?!)
https://github.com/search?l=JavaScript&q=jmap&type=Repositories

Protocol Meta for Client:
https://github.com/jmapio/jmap/blob/master/client-guide/jmap-client-guide.mdown

I think it is possible to extract the Implementation out of the Zend Framework from here:
https://github.com/WikiSuite/zend-jmap/blob/master/src/Jmap.php
(and parent folder respectively)

But the License is quite limiting:
https://github.com/WikiSuite/zend-jmap/blob/master/LICENSE.md

[dumblob]

For the record, there is also a JMAP<->IMAP proxy which could be useful for testing of some corner cases.

[jasonmunro]

@ulfgebhardt and @dumblob thanks for the resources! I suspect we will be writing our own JMAP library in PHP for this. Once we have the "skeleton" module set in place for data sources, most of the boilerplate required to set this up in Cypht is in place. Since JMAP is very RESTful, we can leverage our existing API wrapper code that makes doing RESTy things easy. The final bit is the actual work of deciphering the spec and building out the parsing and logical flows which I suspect won't be nearly as hard as IMAP. We also have a nice set of functions in the core module set for common operations so we can just re-use things we use now (like decoding mime encoded header values for example).

[jasonmunro]

As I dig into this I think I might put JMAP support into the IMAP and SMTP module sets. It will likely be a LOT easier with the only downside being you need those 2 sets enabled to use JMAP.

After much head scratching and trial and error, I have a git checkout of Cyrus IMAP running locally with JMAP support - which should be the most up to date WRT JMAP specs as we can get :)

[jasonmunro]

Update: spent some time on this over the weekend. Some random thoughts:

• It's a pretty cool API. Docs have been spot on so far and the design is consistent
• It's not as "folder" based as IMAP. Messages are still grouped by folders and folders have hierarchy of course, but message uuids are globally unique so there is no need to select a folder before accessing a message
• Performance is great, though it IS Cyrus IMAP running locally so I would expect that :)
• I started hacking around by swapping out the public interface to our IMAP lib with a matching JMAP version. This has been ... interesting. It's a hell of a lot less duplicate code than creating a stand alone module set, and maybe it could be sort of a "drop in" replacement. But It's quickly becoming obvious the protocol differences are going to require some better abstraction at the module layer that interacts with the library for that to work :)
• Pagination support for message lists!
• Our API wrapper for http requests using curl is working out nicely for this. No changes required so far.
• Managed to cobble together folder navigation. Barely tested but it's functional.

After getting folder navigation going I tried to add old school mailbox browsing, but can't seem to get quite what I need yet. Surely I'm doing something wrong but I filed an issue anyway because whatever is not working I just don't see it yet: cyrusimap/cyrus-imapd#2627

[jasonmunro]

As expected the issue I was having with Cyrus JMAP was not a bug, but a configuration issue on my end. With there help we are back in business :)