MORG BOF

1. MORG BOF IETF 72, DublinJuly 30th, 2008 Chairs: Alexey Melnikov <alexey.melnikov@isode.com> Randall Gellens <randy@qualcomm.com> Mailing List: <mailto:morg@ietf.org> Jabber: morg@jabber.ietf.org

2. Note Well Any submission to the IETF intended by the Contributor for publication as all or part of an IETF Internet-Draft or RFC and any statement made within the context of an IETF activity is considered an "IETF Contribution". Such statements include oral statements in IETF sessions, as well as written and electronic communications made at any time or place, which are addressed to: • the IETF plenary session, • any IETF working group or portion thereof, • the IESG or any member thereof on behalf of the IESG, • the IAB or any member thereof on behalf of the IAB, • any IETF mailing list, including the IETF list itself, any working group or design team list, or any other list functioning under IETF auspices, • the RFC Editor or the Internet-Drafts function All IETF Contributions are subject to the rules of BCP 78 and BCP 79.Statements made outside of an IETF session, mailing list or other function, that are clearly not intended to be input to an IETF activity, group or function, are not IETF Contributions in the context of this notice. Please consult BCP 78 for details.

3. Agenda

4. MORG BOF introduction • MORG – Message ORGanization • Goals • improve ability to find messages in an IMAP mailstore • Don't make things worse while doing that (think about extension interaction) • Should work for diverse clients • Mobile devices, web clients, desktop clients

5. MORG BOF history • Various proposals were suggested over the years: • New SEARCH/SORT/THREAD criteria • Extensions to SEARCH: • Multimailbox searches, virtual views • Message grouping and quick ways of returning information about number of messages in such groups • Message contexts (e.g. voice/fax messages), STATUS-in-LIST, per-mailbox annotations

6. draft-gulbrandsen-imap-inthread-02.txt Presented by Timo Sirainen SEARCH=INTHREADandTHREAD=REFS

7. SEARCH=INTHREAD extension • Adds 4 new SEARCH keys: • INTHREAD takes two arguments, a threading algorithm (as defined in [SORT]) and another search key. • The INTHREAD search-key matches a message if its second parameter matches at least one message in the same thread as the message. • [Alexey: Is this the same as: return all messages in threads, which contain at least one message matching the specified search key?]

8. SEARCH=INTHREAD extension(continued) • THREADROOT Search Key • The THREADLEAF Search Key • The MESSAGEID Search Key • Similar to <HEADER “Message-Id” “value”>, but normalizes the value

9. THREAD=REFS extension • Like THREAD=REFERENCES, but: • No merging threads based on subject • References: header field is supported well enough nowadays • False positives – unrelated threads merged • Sort thread roots based on the last received message within the thread, not by the root’s Date: header • New messages are expected to be at the top/bottom of the mailbox

10. THREAD=REFS: Open Issues • Should THREAD=REFS use the In-Reply-To header field value if there is no References header field?

11. draft-karp-morg-sortdisplay-00.txt Presented by Alexey Melnikov New address sort extension to SORT

12. draft-melnikov-imapext-multimailbox-search-03.txt Barry Leiba & Alexey Melnikov IMAP4 Multimailbox SEARCH Extension

13. Recent changes • Clarified that a multimailbox SEARCH/UID SEARCH always returns UIDs in the ESEARCH response • Updated references

14. Open Issues • Replace DEPTH option with something used by IMAP NOTIFY? I.e. • Change “tag1 SEARCH IN (("folder1" "folder2/*") (depth 1)) unseen” • To “tag1 SEARCH IN (Subtree ("folder1" "folder2")) unseen” • Need to define interaction with IMAP CONTEXT (draft-cridland-imap-context) • Might also have to prohibit UPDATE option from draft-ietf-lemonade-imap-notify

15. Multimailbox SEARCH: suggested changes • Timo: ESEARCH replies should include mailbox UIDVALIDITY • Timo: alternative proposal – create virtual view from multiple mailboxes

16. draft-daboo-imap-annotatemore-13.txt Cyrus Daboo IMAP Extension for per-mailbox/per-server attributes

17. draft-melnikov-imapext-status-in-list-00.txt Alexey Melnikov & Timo Sirainen STATUS-in-LIST

18. STATUS-IN-LIST • Most clients display mailbox list with number of (unseen) messages • Requires LIST and STATUS for each listed mailbox • Do it all in one command: LIST "" % RETURN (STATUS (MESSAGES UNSEEN)) • Requires LIST-EXTENDED • Advantages • Avoid one round-trip (or more) • save some bandwidth • allow server to optimize the lookup more easily

19. STATUS-IN-LIST Protocol • Standard LIST and STATUS replies are returned to make it easy for clients to implement support for the extension 1 LIST "" % RETURN (STATUS (MESSAGES UNSEEN)) * LIST () "." "INBOX" * STATUS "INBOX" (MESSAGES 17 UNSEEN 16) * LIST () "." "foo" * STATUS "foo" (MESSAGES 30 UNSEEN 29) 1 OK

20. STATUS-IN-LIST Issues • STATUS replies add more potential failure cases, such as: • ACLs prevent STATUS reply • Internal server failures prevent a successful STATUS lookup (but LIST succeeds) • These are normally handled by a NO reply to STATUS • Possible ways to let clients know about them: • Return \NoSelect and no STATUS (good for ACLs) • STATUS replies with empty parameters • NO reply to LIST if any STATUS replies are missing

21. Alexey Melnikov New collations (comparators) for IMAP and Sieve

22. Suggested comparators • Ned Freed: space ignoring comparator (e.g. for Japanese texts) • Signed numeric comparator (i;ascii-numeric doesn't deal with leading '-', '+' or SP) • Telephone number mapping – ignore punctuation such as SP, '-', '(' and ')' • Case sensitive version of i;unicode-casemap?

23. Sorting by Relevancy • All modern search engines return results sorted by relevancy to the search query. IMAP should allow the same. • How the relevancy is calculated or presented is implementation specific. • The scores may be calculated differently in different installations or even different connections (client location, user’s language).

24. Sorting by Relevancy • Relevancy score is search query-specific, so its value can’t really be FETCHed. • New SCORE SORT key • SORT (SCORE DESC) UTF-8 TEXT “hello world” • Highest score = highest relevancy, so should DESC be used to get highest scores first?

25. Sorting by Relevancy • New SCORE SEARCH key? • SEARCH TEXT “hello world” SCORE 0.5 • Is this useful? It would require some kind of a standardization of score values, which may make it difficult to implement with some search engines. • Would clients still want to know the score and display it to the user in some way (color, percentage, ..)? How would this be possible? • FETCH SCORE would be updated after each SEARCH • SORT/SEARCH returns scores in untagged reply

26. Flag/Keyword Management • Allow explicitly adding and removing keywords • This allows client UI to display a list of existing keywords that can be assigned to messages • Currently keyword with typos or keywords that are no longer relevant can’t be removed • Allow specifying whether flag or keyword should be private or shared in shared mailboxes • Different use cases require different configurations • Notify client about these

27. Flag/Keyword Management • Send shared flag state whenever FLAGS and PERMANENTFLAGS are sent: • * OK [SHAREDFLAGS (\Seen \Answered $hello)] • Add keywords or change private/shared state of an existing flag/keyword: • FLAGS ADD PRIVATE (\Answered $pvt1) SHARED (\Flagged $world) • Remove keywords: • FLAGS REMOVE ($pvt1 $world)

28. Flag/Keyword Management • Backwards compatibility with clients not supporting the new extension: Allow server to drop unused keywords created with STORE • But don’t allow dropping explicitly created keywords?

29. Flag/Keyword Management • Interaction with ACL? • “w” right to create new keywords? • “a” right to modify private/shared state of system flags? What about keywords created by you? Keywords created by someone else? • Allow user to create a private keyword to override an existing shared keyword, effectively hiding the shared keyword from the one user?

30. Proposed WG Charter Discussion

31. Basic questions • Does the list of drafts/ideas presented include documents you will be willing to work on? • Does this look like a WG?

32. Proposed WG description • The IETF Message Organization extensions Working Group is tasked to work on IMAP extensions that improve clients ability to find messages or group of messages in an IMAP mailstore. This includes an extension for multimailbox searches, extensions defining new search and sort criteria, an extension defining client controlled per mailbox attributes. As the secondary goal the WG will try to ensure that designed extensions minimize number of round-trips and and bandwidth overhead.

33. Proposed workitems (1 of 2) [note that not all of them will become WG items] • (a) The IMAP SEARCH=INTHREAD and THREAD=REFS Extensions (draft-gulbrandsen-imap-inthread-02.txt) • (b) New search method using prefix match and a comparator (TBD) ? • (c) New address sort criteria extending the SORT extension (draft-karp-morg-sortdisplay-00.txt) • (d) IMAP4 Multimailbox SEARCH Extension (draft-melnikov-imapext-multimailbox-search-03.txt) • (e) IMAP Extension for per mailbox and per server attributes (draft-daboo-imap-annotatemore-13.txt)

34. Proposed workitems (2 of 2) • (f) STATUS-in-LIST (draft-melnikov-imapext-status-in-list-00.txt) • (g) Formalize a way to return counters by message types (e.g. voice, fax) using STATUS command and ESEARCH extension • (h) New comparators (TBD) ? • (i) Sorting by relevancy (TBD) ? • (j) IMAP keyword management (TBD) ? • (k) Mailbox type management extension (TBD) ?