TMDA vs. SA: YMMV

With an eye to quantitatively comparing my TMDA install to my old SpamAssassin setup, I had a look at what TMDA's been blocking for me, and how SpamAssassin would have evaluated them. The quick summary is that false positives were slightly better with TMDA, and TMDA blocked a lot of spam that SpamAssassin wanted to let through.

Also, I discovered something interesting about the "spamming third parties with challenges" problem. Basically, normal SMTP servers are causing a much larger problem than all C/R systems combined.

I took the 2000 most recent real mails from my mailbox, and the 2000 most recently blocked mails from the TMDA pending queue. I had SpamAssassin look at them all, and give me their scores. Given this info, I've determined:

• 5 Spams that TMDA allowed and SA would have stopped
  • Four hit my white list (some of which I fixed)
  • One was confirmed by the spammer (which I blacklisted)
• 12 Real mails TMDA allowed that would have been blocked by SA
  • 9 Responses to my complaints to abuse@*
  • 3 Friendly emails
• 463 spams stopped by TMDA that SA would have allowed.
• 109 automatic replies (to spam) stopped by TMDA, which SA would have allowed (in approximate order of prevalence).
  • Poorly formatted bounce messages
  • Anti-spam C-R systems different from TMDA
  • Responses containing only a single line: "This is an autoresponder. I'll never see your message."
  • Responses from mailing lists
  • Vacation messages
• 2 friendly mails stopped by TMDA that SA would have allowed.
  • Envelope sender was wrong, so the challenge didn't get to him.
  • Unknown problem stopped the challenge from reaching him.
• 1 hate mail stopped by TMDA that SA would have allowed.
• 24 messages made it through confirmation.
  • Results of spammers using my domain as their 'from' address
    • 2 "screw you" hate mails
    • 2 "please delete me" from people thinking I was a spammer who'd honor the request.
    • 1 illegible response from another country
  • Friendly mails
    • 3 emails about a project I work on.
    • 1 from an old friend.
    • 3 from friends who changed addresses
    • 4 because of a "big cc" style mailing list I was on.
  • Junk
    • 2 auto-confirmed challenge from another C/R
    • 2 auto-confirmed vacation message
    • 2 spammers who confirmed

A few things stand out:

• Of people writing one-line "screw you" hate mail messages, two out of three confirmed their messages! The investment in a message required to motivate the user to invest an extra response to a challenge must be small indeed.
• The only one real email (a hate mail) may have gotten a challenge and did not respond
• I have a filter that throws away bounce messages that result from spammers spoofing my domain in their headers. In three weeks it tossed over 5000 messages. Compare to the 109 auto-responses from other computers in a month, less than half of which were C/R style responses. C/R systems need to get about 100 times more prevalent before they even equal the volume of spurious responses generated by properly functioning SMTP servers.

The way I look at all this is:

• TMDA
  • Misblocked 2 mails
  • Allowed 5 spams SA wouldn't have
• SA
  • Misblocked 3 mails
  • Allowed 463 spams TMDA blocked

I don't think I'll be going back to SA soon. It's a good program, but it's not meeting my needs. I integrated SA into my mail stream again with the intention of reducing the challenges I send out. I figured I'd deliver or block the mails with scores at far ends of the spectrum and only challenge the middle ground "suspicious" mails. Since doing that two days ago, I've gotten one spam per day that SA thought was very UNspammy and so bypassed TMDA.

I don't think TMDA is for every one. It's definitely the hole hog of anti-spam systems. If you don't have a really bad spam problem, you probably don't need anything so powerful. That having been said, TMDA has worked wonders for me. It's given me my mailbox back. I'm happier with my email than I have been in five years.

Reaction

• 0% I think your analysis is wrong.
• 0% OK, it works great, but it's still wrong to use it.
• 0% I'll be shutting down my SMTP server to stop the flood of spurious bounce messages right NOW
• 25% I see it's good, but I still won't use it.
• 50% It is good!
• 0% It's even worse than this poll
• 0% None of the above
• 25% All of the below.

Votes: 4