OMEMO encryption in group chats works basically like it works in 1:1 chats as well. Instead of encrypting using all of your contacts‘ sessions and all of your other devices’ sessions you encrypt using all of the the participants’ devices sesions (plus all of your other devices).
It is recommended to use the very same sessions you’d be using for 1:1 chats as well. So messages going to someone directly are encrypted using the same session as messages going to a group chat with that someone in it.
For this to work we need to be able to discover the real XMPP address of all participants. Therefor it is a fundamental requirement for all OMEMO enabled group chats to be non-anonymous. On top of that Conversations (and probably other clients) limit OMEMO to members-only groups. Encrypting messages in a chat where everyone can join and potentially has the expection to be able to read the backlog does not make much sense. However this is an artificial limitation and not a technical requirement.
Sidenote: For this and other reasons Conversations (ever since the UX Sprint in Brussels) makes a hard distinction between channels (public) and group chats. Group chats are always members only and non-anonymous and can be encrypted. Channels are public (like on IRC) and don’t even display a lock button.
Getting all participants
When sending a message to a group chat you need to compile a list of all participants and their real XMPP addreses. For people currently online in the room that list can be generated from presences. Non-anonymous MUCs carry the real jid in the presence.
However not everyone is online all the time and we want group chats to support offline messages. For members only rooms XEP-0045 has the ability to retrieve the list of all members or more specfically all affilitaions. Namely members, admins and owners. Someone who is an owner of the MUC (usually at least the creator) is not simultaneously on the members list. So you need to query all three lists and merge the results. (In the XEP the syntax on how to do this is devided up in the sections Admin Use Cases and Owner Use Cases. But you can take a look at Example 129 and do the same for
Some older servers don’t allow members to retrieve those lists and you also want to react to live changes. So in practice you would always want to combine the affiliation list with the information you are getting from live presences.
Do not forget to also remove participants from the list when they become
affiliation=outcast (removed or banned from the group chat).
Decrypting a message in a group chat works by first figuring out who is the real sender of that message. The full JID in the from attribute only has the nick name. By looking at the received presences this can be mapped to a real XMPP address.
Special case MAM
When catching up with offline messages one could run into a special case where the original sender is currently not online and therefor not in the
nick->real JID map. Furtunatly MAM (XEP-0313) can transmit the real JID in a special x attribute in the message. (See section 5.1.2.) Since this is a feature old-school MUC history can not provide it is recommended to use MAM over muc-history or offline messages will be unreliable in some corner cases.
Dealing with reflections
Some (older?) XMPP clients display sent messages only by parsing the reflection. (Your sent messages are echoed back to you by the server.) Ignoring the fact that this is generally not nice behaviour because it delays the message display by the round trip time to the server it also breaks OMEMO since you usually do not encrypt to your own device and therefor won’t be able to decrypt the reflected message. So if your client behaves that way it is a good opportunity to refactor your MUC sending behaviour to display the message directly and just change the status from sending to sent (or add a green checkmark or whatever) once you receive the reflection.
To keep track of your reflection you can use message-ids:
<message to="room" id="this-one"/> or origin-ids
<message to="room"><origin-id xmlns='urn:xmpp:sid:0' id='that-one'/></message>. Conversations always sets both to the same value and hopes that at least one of them makes it through some of the MUC services that might change the ID. When parsing a message Conversations first looks at the origin-id and tries to match that. If the origin-id got removed it looks at the message-id. (It is more likely that a server will change the message-id; therefor we prefer to look at the origin-id)