Junk Email Filter Server Configuration
From the Department of Redundancy Department
Our infrastructure relies heavily on redundancy. The idea is to have several computers in multiple locations that are capable of doing the job so that if hardware layers go down there is enough spare capacity to get by till the problem is fixed. We do that using redundant hardware and standby system structured in a way that service are automatically rerouted based on either failures or load spikes than bring on additional computing power. The idea of course to be up and fully functional 100% of the time and still provide fast processing so the any delays through the filter aren't noticeable.
One of the technologies we use is OpenVZ which virtualizes the physical computer into several logical virtual servers which serve standard functions. These virtual server can be moved around from box to box as need be and are backed up on our backup server system. So if one box goes down not only are there other computers running the same thing but we can manually move servers to different hardware before we have to go to the data center and do actual work.
OpenVZ is not really true virtualization like VMWare or Xen. It's more like containerization like BSD Jails, but with a lot more power. It runs on the same Linux kernel and is Linux only, but it is very fast (1% overhead) and efficient with resources. We can even do live migrations from one box to another of logical servers. And we can increase the capacity of the system by adding more physical servers and copying standard virtual servers on them. So we don't need to go faster because we can go wider as we grow.
Logical Layers
Junk Email Filter uses a 5 layer model to filter spam based on 5 MX layers that have different finctions at different levels. Our MX structure is as follows:
dummy0.junkemailfilter.com - 10
mx.junkemailfilter.com - 20
mx.junkemailfilter.net - 30
mx.junkemailfilter.org - 40
dummy1.junkemailfilter.com - 50
The dummy0 layer is the primary MX and it is for the most part a dead IP address. It is listened to on our main spam filter group through a firewall so if we have an IP that absolutely must send mail on the lowest numbered MX we can open it up for them. For everyone else it create a retry barrier. Real servers will retry, spam bots don't retry. So we get rid if a lot of spam this way.
The next level is our primary server group. It consists of a main email server, a spam assassin server, several DNS servers for DNS caching and feeding our hostkarma lists, and servers for processing spam on the back end and feeding data to other blacklists and hash servers.
The third level is our backup ring. This is in case of massive failure at the main data center the remote server groups handle the load. This level is also used for short term failures such as server rebooting or if we shut down key systems for diagnostics. Sometime there is a routing problem on the Internet where Verizon can't send packets to Level 3 so all Verizon servers are cut off. And it is used for spikes in email/spam traffic.
There are times when the main server cluster can overload. All of a sudden the load levels goes up 20 times normal. In cases like this instead of allowing servers to shut down they change modes so that the primary group focuses on getting as much good email through as possible while letting the backup ring process messages that are more likely to be spam. The primary server responds with "come back later, we're not ready" messages which tells the sending server to go to the next level hight in the ring. The backup ring gets the email and processes it.
[ sign me up! ]
|
