Hello users!

I haven’t made a post in a while, so I figured I’d make one about what’s going on with XMPP.is. First of all, Happy New Year! 2018 has been a great year for the server and I’ve received $135 USD in donations via PayPal. Thank you to all the donors, you are what helps keep this server running. Thank you to the few who donated cryptocurrency as well! Since there hasn’t been many posts, here are some things I’ve accomplished last year…

• I made numerous improvements to the server’s configuration, adding support for many new XEPs and becoming 100% compliant according to this compliance tester. I’ve made several improvements to the server’s scripts and even added a few. You can see every change I’ve made on GitHub. Open source is the best 🙂
• I’ve updated a lot of pages on the site, adding more info and descriptions.
• I’ve tried to keep downtime to a bare-minimum. With the new hypervisor things have been more stable than ever. I’ve also improved on scheduling maintenance windows and giving notice beforehand.

Now onto 2019…

Recently I’ve been working on cleaning up registered users. We currently have over 37,000 registered users. Many years ago before I had web registrations and captchas in place, a nefarious individual/group registered over 35,000 accounts en-masse. A lot of the accounts follow a pattern (which I won’t disclose). I am working on cleaning up these users and deleting them, as they’ve never been used (as far as I know). This is a long and drawn out process as I have to make sure the accounts I’m deleting are not legitimate.

I’ve made a couple one-liner scripts in bash to find these accounts based on the pattern in their usernames that I’ve noticed. While it work 95% of the time, it also picks up legitimate users and I have to manually filter them out from the deletion lists. I’ve been going through the lists when I make them and taking out any users that look legitimate. As this is a manual process I might miss a few here and there… If your account gets deleted and you notice you can no longer login, please let me know. I can restore it from encrypted, off-site backups. Keep in mind that these usernames are very uncommon. They are probably taken from a word list and random numbers are added to them.

If you have any questions, feel free to reach out!

~ Lunar

Hello users!

A little late… but Happy New Year!

I wanted to post about some recent changes that XMPP.is underwent this past few months.

• mod_cloud_notify (XEP-0357: Push Notifications) support has been added. twitter.com/xmpp_is/status/946592338607996928
• The security page has been updated with more details about how we care for your data.
• Non-standard ASCII characters in usernames are no longer allowed when registering. People could potentially impersonate other users for malicious purposes. See: twitter.com/xmpp_is/status/954729573173940224
• We now have a fair (in my opinion) TOS/AUP.
• The 2018 donation goal has been created. I lowered it this year to cover at least domain costs, and a little more. Thank you to all that donated USD and crypto-currency!
• The donation page has been updated with our hard-coded crypto-currency wallet addresses. If you’d like to donate with any of those, please opt for the ones we provide (more privacy).

Hello users!

Offline messages are now back after a short maintenance window yesterday. After speaking with a user that thought offline messages are very important, and much needed for instant messaging services.. I fully agreed and decided to enable the module again after the attack we underwent a few months ago. Despite the shortcomings of the offline messages module in Prosody, I believe this to be important. Previously, if you were offline, even for a short period of time, messages could be lost. This would obviously be annoying, as pretty much any modern IM service will just store messages server-side until the user is online. So.. In the meantime hopefully things hold up. In the meantime I will have systems constantly monitor everything server-side to alert me of any potential issues.

In other news.. I’ve patched Tor packages on the server due to the recent CVE‘s. I’m also continuing to research new modules (been busy) that I can add to support more XEPs. The following are on my list for testing:

• XEP-0065: SOCKS5 Bytestreams
• XEP-0313: Message Archive Management
• XEP-0357: Push Notifications

Hello users!

Just wanted to write up a little update, the stability of XMPP.is has been in question recently. 🙁

After a long and strenuous attack it appears that everything is stable now. We’re back on the main dedicated hypervisor. I made a lot of configuration tweaks, and wrote several scripts to make stability a priority and migration easy while the attack was going on. All configuration and scripts are now hosted on GitHub, which I use to pull changes from. If you have a suggestion or would like to contribute feel free to! Everything is open.

In other news:

• The donation goal has been updated to a ‘yearly’ cycle. I’ve also updated the operating costs page which now more accurately reflects what it costs to run the service. Cryptocurrency donations are subtracted from the goal.
• I made a new script that outputs the current certificate fingerprints for all of the domains, which you can find on the transparency page.
• I’m currently testing a password reset module that allows you to reset your password with the email set in your vCard. Source here.

Have questions? Contact me here. Hope everyone has a good weekend!

Hello users!

I just wanted to write up a post on what happened yesterday. If you were online around 3 – 5 PM (CT) you may have noticed the server went down and you were unable to see your contacts online. I first noticed the problem around 3:50 PM (CT) and started investigating. Prosody was using 100% of 1 core on the temporary server. This choked everything up, you could sign in sometimes, other times you could not. If you were online, you may have noticed that all of your contacts were offline. Sadly this was the case until around 5 – 6 PM (CT).

The attack was due to tons and tons of XMPP spam directed towards one user that was offline (DDoS protection won’t protect from this). The way Prosody (the XMPP server) handles offline messages is flawed and has been mentioned here. This attack has occurred before but never seriously interrupted service like it did yesterday. Since the temp server wasn’t as powerful it couldn’t handle the load.

I took some drastic action and enabled debug logs for the time being to see exactly what was going on. There were so many messages coming in, that the log file is huge now (over 20GB) and still climbing. Here’s a snippet from the debug log. I’m posting the full one without censoring anything either. I believe this user may be involved in the attack.

Oct 04 14:20:36 s2sin55dee11bc500 debug Received[s2sin]: <presence type=’subscribe’ to=’xsndr@xmpp.is’ from=’fhr72w1vkcet@keinhorn.de’>
Oct 04 14:20:36 xmpp.is:presence debug inbound presence subscribe from fhr72w1vkcet@keinhorn.de for xsndr@xmpp.is
Oct 04 14:20:36 rostermanager debug load_roster: asked for: xsndr@xmpp.is
Oct 04 14:20:36 rostermanager debug load_roster: loading for offline user: xsndr@xmpp.is
Oct 04 14:20:36 stanzarouter debug Routing to remote…
Oct 04 14:20:36 s2sout55dee4948a30 debug going to send stanza to keinhorn.de from xmpp.is
Oct 04 14:20:36 s2sout55dee4948a30 debug sending: <presence type=’unavailable’ to=’fhr72w1vkcet@keinhorn.de’ from=’xsndr@xmpp.is’>
Oct 04 14:20:36 s2sout55dee4948a30 debug stanza sent over s2sout
Oct 04 14:20:36 rostermanager debug load_roster: asked for: xsndr@xmpp.is
Oct 04 14:20:36 rostermanager debug load_roster: loading for offline user: xsndr@xmpp.is
Oct 04 14:20:36 rostermanager debug load_roster: asked for: xsndr@xmpp.is
Oct 04 14:20:36 rostermanager debug load_roster: loading for offline user: xsndr@xmpp.is
Oct 04 14:20:36 xmpp.is:auth_internal_hashed debug account not found for username ‘xsndr’
Oct 04 14:20:36 rostermanager debug not saving roster for xsndr@xmpp.is: the user doesn’t exist

The temporary fix was to disable offline messages entirely. I also migrated XMPP to a better server (temporarily) and I used my beta deploy script, which in turn didn’t include some important things that I had to fix later on… I haven’t had time to get my dedicated hypervisor reconfigured yet as I’ve been too busy with work.

I’m writings scripts and configs on GitHub to easily push updates and for transparency reasons. I’m going to dedicate more time to XMPP.is in the near future to improve how everything is managed and prevent downtime like this from ever happening again!