A Week in Matrix

A user accidentally writes in the room of a Matrix space, resulting in every member of that space getting a notification. Matrix spaces are just regular rooms, and the fallback for old clients which don't support spaces is to render them as completely regular chat rooms. On the other side, new clients that do support spaces have very confusing UI when people start chatting in them, because that's not supposed to happen. (Yet, most clients don't adjust the room permissions in spaces to disallow people to chat in there. That would be too easy.)

I figure out how to get Element Web to display the space as a regular room in order to inspect it (it's in the devtools), give myself admin permissions and adjust the room's permissions to disallow chatting in the space room.

A user reports a spurious ban on the forum, saying they got banned from our Matrix space and have no idea why. A quick Draupnir query for the user handle shows nothing of interest. After some tring around, I query Draupnir for the domain of the user handle instead. Indeed, it finds one entry:

Apparently, someone on a moderation list our space is subscribed to banned the user's domain (it looks like it might be a single-user personal instance). I am not in the room !foQLZposNt3iEpkl5O:example.org , which I assume is the policy room from which the ban is coming, and Matrix annoyingly won't tell me its title. But it seems to be public, so I try to join it to find out. My homeserver chokes a little while trying to join me, WARN conduwuit_api::client::sync::v3::joined: timeline for newly joined room is empty , but it then asks another server to backfill the timeline, and then I only have to "clear cache and reload" my client (which will incidentally also mark all my chats as unread, which I'll have to manually clear again afterwards) and I'm in.

The room is one of those community-pooled policy rooms, full of messages of type m.policy.rule.user originating from both mods and moderation bots, which ban people and servers with sometimes very explicit usernames. I search for the domain of the affected user, but Element will only search through messages of type m.messages and thus the search comes in blank. The usual workaround for when the search doesn't work is to export the chat as HTML, then grep through it in the hope that the relevant message is in the exported range. Luckily, the room export indeed includes all message types, and grepping for the domain gives me seven hits.

It turns out, the homeserver of the affected user accidentally had had open registrations at some point in the past, with the expected consequences. Mystery solved. Getting this info took me roughly 40 minutes. It should have been two at most.

Postscriptum: Two weeks later, I get contacted by that user again, they still struggle with being banned. All I can see is that their domain is still on the ban list. I can't find an easy way of making local overrides to the ban list, so there is little we can do but to unsubscribe from the entire ban list, which would potentially expose the entire community to spam and abuse.

While onboarding a new person to our Matrix space, they don't seem to be able to see one of the rooms, despite being "suggested" for that space. After them manually joining that room, it does show up in the space's room list. The issue can be reproduced from various accounts on different homeservers.

Removing and adding the room to the space again does not seem to help. Even tombstoning and recreating the room does not seem to be a solution. However, if one of the space admins joins the room, it suddenly shows up in the space's room list again. When they leave again, the room disappears anew.

We accept the obvious workaround and move on.

After some routine server maintenance, Draupnir stops responding to admin commands. According to the server logs, Draupnir seems to be processing the commands just fine. Some long debugging sessions reveal that this is an outbound federation problem with the Draupnir's homeserver (it is the only account on there). More debugging reveals that this is an issue with the Dendrite default configuration which drops all egress IPv6 traffic for no good reason.

Noticing and debugging the issue took over a month in total, during which the community space was without protection from a moderation bot. I am very glad that we were not hit by a porn&gore spam wave during that time.

Dendrite is kind of deprecated as a homeserver these days, at least it doesn't seem to be the main focus of development and most recent commits are mere dependency bumps. Years ago, when the server was set up, choosing Dendrite seemed like a good decision, because it was easier to deploy and maintain than Synapse and there was a promise for a bright future for the first 2nd generation homeserver implementation. Unfortunately, migrating between Matrix homeserver implementations remains a problem with no satisfying solution, so we're kind of stuck on slowly rotting foundations.

A user accidentally writes in the room of a Matrix space, resulting in every member of that space getting a notification. Again. In the same room as last time, despite me setting the permissions a couple of months ago. I go check, and the permissions have somehow reset themselves. Some testing confirms that the room seems to have desynced, because when I re-apply the permissions change they don't show up for people on other home servers, allowing them to continue chatting in the space room. The workaround is to have someone else on a different homeserver apply the changes and hope that it propagates to sufficiently many other homeservers on the other side of the desync.

A split brained Matrix room like this means that some users might be allowed to chat in that room, and those messages would propagate to other homeservers who haven't received the permissions change. However, homeservers who do would soft-fail those messages, rendering the chat invisible to the other side of the split.

The room is overdue for a room version upgrade anyways, so we end up not investigating this any further for the time being.

My homeserver had degraded federation for a couple of days. The root cause isn't super important, but it was something with DNS (of course) causing server resolution to fail and resulting in the server connecting to its peers on wrong default ports, leading to directional communication breakdown. The issue once identified was quickly fixed, but unfortunately the homeserver did not properly resume operation afterwards: Unfortunately, something desynced along the way, and the server ended up thinking of one account not being in a certain room. Fortunately this was just a bot account, so kicking it from the room and re-inviting it resolved the issue. But still, I wonder: How many others users am I accidentally ghosting because my homeserver just "forgot" about them?

The Matrix Foundation's Mastodon account once again boosts some toot from a person reporting a very happy experience using Matrix. The user looks like they've never joind a group chat larger than a handful of people, mostly use 1:1 DMs and don't know what spaces are. If they self-host, they likely are running a Synapse server with some Element client, but most likely they're just happily chugging along in the matrix.org silo.

I'd like to be happy for them, but I can't help not feeling somewhat gaslit by the stark contrast between the happy stories boosted into my timeline and my personal reality of helping run an actual community in this ecosystem. On a certain level this makes sense: How good a tool is heavily depends on the purpose you use it for, and Matrix undeniably does have its strengths. And the usage patterns for Matrix I find myself in clearly aren't getting as much love and dedication as others. Some are a matter of use case: Focusing on the needs of WhatsApp and Signal users first arguably makes sense. Others are a matter of principle: When the experience suffers from ecosystem software diversity due to interoperability issues, then something fundamental about the project's values is lost. Saying "Matrix is great but only if you use the flagship client on a flagship server made by the flagship company" doesn't have the same ring to it.

Some kind of clash between ideals and financial reality is inevitable to any project with ambition, but expectation management is key here. Because as of right now, the clash between PR touting "we are a serious Discord alternative" on social media all the while the development reality being "spaces are low priority, because they're a feature for more 'advanced' users" is jarring. Maybe I'd be feeling less bitter about all of this if not for all the broken promises over all these years.