We serve some of our assets directly out of s3 and while it is convenient it is not the speediest way to deliver content. The crew over at Viximo worked out how to bolt Varnish on the side of Apache so that they can cache their S3 content and I was so smitten with the idea that I wanted to adapt what they worked out for our configuration so I asked Chris Chiodo reveal the secret sauce. Below are the configuration files I munged from what he generously shared.
Nginx
This is pretty straightforward, what I’ve done is made varnish an upstream server and am intercepting any content in photos, avatars, kit, or caboodle and passing it the request to it.
upstream varnish {
server varnish01:7000 max_fails=3 fail_timeout=30s;
}
location ~ ^/(photos|avatars|kit|caboodle)/ {
proxy_pass http://varnish;
}
Varnish
This was my stumbling block until I talked to Viximo, the problem was how I defined the backend and that for whatever reason it did not like or AWS did not like the request to the bucket-name.amazonaws.com.
backend media {
.host = "s3.amazonaws.com";
.port = "80";
}
sub vcl_recv {
set req.url = regsub(req.url, "^", "/bucket-name");
set req.backend = media;
set req.http.host = "localhost";
remove req.http.X-Forwarded-For;
remove req.http.X-Forwarded-for;
remove req.http.X-Forwarded-Host;
remove req.http.X-Forwarded-Server;
set req.http.X-Forwarded-for = "127.0.0.1";
set req.grace = 30s;
lookup;
}
sub vcl_fetch {
set obj.http.X-Varnish-Url = req.url;
// set a 1 day ttl for avatars
set obj.ttl = 1d;
set obj.grace = 30s;
if (!obj.cacheable) {
pass;
}
set obj.prefetch = -30s;
deliver;
}
That’s it. Simple and it works.
Good job! Thank you!
i’m about to build something that needs this very kind of tooling, and since this is the second time i’ve gleamed something useful from your site – one could not possible forget the header photo – i thought i’d leave a note and say thanks …
Thanks a lot!
I was experiencing problems with bucket.s3.amazonaws.com approach too.
I think the problem is in varnish, because when using only nginx to cache content there is no problem. You can check it out here:
http://kpoxit.blogspot.com/2008/12/how-to-save-60-month-with-20-lines-of.html
Nginx relies on a filesystem for caching. And this is a huge trouble when dealing with lots of small images (avatars, etc). This is why I am trying varnish now.
Current nginx versions have caching functions built-in, so is varnish even necessary with the latest nginx versions?
Could you eventually post an example for a current version?
We switched back to Apache to use passenger (haven’t felt the urge to switch back to nginx because that would just make me dizzy) but I am sure that caching will be coming back up on my radar and I’ll make sure to take a look at how nginx handles it.
nginx is less capable of negotiating specifics with the origin server. varnish is the best in class for header rewriting to control the client’s behaviour too. nginx won’t have any problem with lots of small images, it’s designed to work with that – if that is the only issue I would use nginx and be done; it’s the best reverse proxy and it’s the best http server, no question. varnish is not an http server at all, but if you need to specify expiry, max-age and other headers, and want to control this for sites and directories and want to be able to use file extension, regular expressions and normalizing algorithms for site names, you need varnish.