Varnish
De Hegyd Doc.
Sommaire |
Configuration et commandes du daemon
- Configuration
Elle se fait dans le fichier "/etc/default/varnish", nous pourront utiliser les paramètres suivants :
-a <[hostname]:port> # listen address -f <filename> # VCL file -p <parameter=value> # set tunable parameters -S <secretfile> # authentication secret for management -T <hostname:port> # Management interface -s <storagetype,options> # where and how to store objects (file for file backend and malloc for memory bakend)
- Commandes
Test de la configuration et la recharge si valide (n'affecte pas le cache) :
service varnish reload
Test et chargement séparé (ce que fait un reload pour nous !) :
varnishadm vcl.load varnishadm vcl.use
Purge d'éléments du cache :
varnishadm -S /etc/varnish/secret -T :6082 "url.purge .*" varnishadm -S /etc/varnish/secret -T :6082 'purge req.http.host == "blog.hio.fr"' varnishadm -S /etc/varnish/secret -T :6082 'purge req.url ~ .(xml|txt)'
Le language VCL
Rappel : Le code VCL par défaut est toujours présent ! Il est simplement ajouté à nos sections !
Subroutines
Ci-dessous les différentes section configurable du cache en utilisant le lagnuage VCL.
- backends & directors : Configuration des serveurs de backend. Les backends et directeurs sont nommés, il peut y avoir plusieurs backend et/ou directeurs.
- vcl_recv : Appelé des qu'une requête est entièrement reçue et parsée par varnish. Cette routine permet de décidé si l'on doit traiter la requête, de quel manière et sur quel backend si nécessaire. L'objet requête ("req") est accessible et altérable dans cette routine, typiquement pour ajouter, supprimer et modifier les headers et cookies.
- vcl_fetch : Appelé après qu'un document ait été récupéré avec succès d'un backend. Permet de modifier les headers de la réponse avant son insertion en cache. L'objet de requète est encore accessible et il est possible de l'utiliser sur un autre backend (réponse du backend non satisfaisante, en cas de 404 par exemple ...). L'objet de réponse du backend ("beresp") est aussi disponible, c'est cet objet qui sera stocké en cache.
- vcl_hit : Called after a cache lookup if the requested document was found in the cache.
- vcl_miss : Called after a cache lookup if the requested document was not found in the cache. Its purpose is to decide whether or not to attempt to retrieve the document from the backend, and which backend to use.
- vcl_timeout : Called by the reaper thread when a cached document has reached its expiry time.
- vcl_deliver : Appelé juste avant de renvoyé la réponse au client. Permet de modifier, ajouter ou supprimer des headers via l'objet de réponse ("resp").
Actions
Actions les plus courantes :
- pass : When you call pass the request and subsequent response will be passed to and from the backend server. It won’t be cached. pass can be called in both vcl_recv and vcl_fetch.
- lookup : When you call lookup from vcl_recv you tell Varnish to deliver content from cache even if the request othervise indicates that the request should be passed. You can’t call lookup from vcl_fetch.
pipe : Pipe can be called from vcl_recv as well. Pipe short circuits the client and the backend connections and Varnish will just sit there and shuffle bytes back and forth. Varnish will not look at the data being send back and forth - so your logs will be incomplete. Beware that with HTTP 1.1 a client can send several requests on the same connection and so you should instruct Varnish to add a “Connection: close” header before actually calling pipe. deliver : Deliver the cached object to the client. Usually called in vcl_fetch. esi : ESI-process the fetched document.
Requests, responses and objects
Le language VCL permet principalement de manipuler 3 structures de données : La requète provenant du client, la réponse du backend et l'objet stocké en cache.
- req : The request object. When Varnish has received the request the req object is created and populated. Most of the work you do in vcl_recv you do on or with the req object.
- beresp : The backend respons object. It contains the headers of the object comming from the backend. Most of the work you do in vcl_fetch you do on the beresp object.
obj : The cached object. Mostly a read only object that resides in memory. obj.ttl is writable, the rest is read only.
Backends
- Configuration d'un hôte avec test :
backend hostname1 {
.host = "XXX.XXX.XXX.XXX";
.port = "80";
.probe = {
.request =
"GET / HTTP/1.1"
"Host: localhost"
"Connection: close";
}
}
backend hostname2 {
.host = "XXX.XXX.XXX.XXX";
.port = "80";
.probe = {
.request =
"GET / HTTP/1.1"
"Host: localhost"
"Connection: close";
.timeout = 120s;
.window = 5;
.threshold = 2;
.interval = 5s;
}
}
Directors
- The random director: The random director takes one per director option .retries. This specifies how many tries it will use to find a working backend. The default is the same as the number of backends defined for the director. There is also a per-backend option: weight which defines the portion of traffic to send to the particular backend.
- The round-robin director: The round-robin director does not take any options.
- The client director: The client director picks a backend based on the clients identity. You can set the VCL variable client.identity to identify the client by picking up the value of a session cookie or similar. The client director takes one option - retries which set the number of retries the director should take in order to find a healthy backend.
- The hash director: The hash director will pick a backend based on the URL hash value. This is useful if you are using Varnish to load balance in front of other Varnish caches or other web accelerators as objects won’t be duplicated across caches.
- The DNS director: The DNS director can use backends in three different ways. Either like the random or round-robin director or using .list: This will specify 384 backends, all using port 80 and a connection timeout of 0.4s. Options must come before the list of IPs in the .list statement.
director WebSiteA round-robin {
{ .backend = hostname1; }
{ .backend = hostname2; }
}
director WebSiteB random {
{ .backend = hostname1; .weight = 1; }
{ .backend = hostname2; .weight = 2; }
}
ACLs
Vous pouvez définir des acl afin de restreindre l'utilisation de fonctionnalité. Ci-dessous, l'exemple d'une purge :
# Who is allowed to purge....
acl local {
"localhost";
"192.168.1.0"/24; /* and everyone on the local network */
! "192.168.1.23"; /* except for the dialin router */
}
sub vcl_recv {
if (req.request == "PURGE") {
if (client.ip ~ local) {
return(lookup);
}
}
}
sub vcl_hit {
if (req.request == "PURGE") {
set obj.ttl = 0s;
error 200 "Purged.";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
error 404 "Not in cache.";
}
}
Mode "Grace"
When several clients are requesting the same page Varnish will send one request to the backend and place the others on hold while fetching one copy from the back end.
If you are serving thousands of hits per second this queue can get huge. Nobody likes to wait so there is a possibility to serve stale content to waiting users. In order to do this we must instruct Varnish to keep the objects in cache beyond their TTL. So, to keep all objects for 30 minutes beyond their TTL use the following VCL:
sub vcl_fetch {
set beresp.grace = 5m;
}
sub vcl_recv {
set req.grace = 1m;
}
You might wonder why we should keep the objects in the cache for 5 minutes if we are unable to serve them passed 1 minute ? Well, if you have enabled Health checks you can check if the backend is healthy and serve the content for as longer.:
if (! req.backend.healthy) {
set req.grace = 5m;
} else {
set req.grace = 1m;
}
Saint mode
Sometimes servers get flaky. They start throwing out random errors. You can instruct Varnish to try to handle this in a more-than-graceful way - enter Saint mode. Saint mode enables you to discard a certain page from one backend server and either try another server or serve stale content from cache. Lets have a look at how this can be enabled in VCL:
sub vcl_fetch {
if (beresp.status == 500) {
set beresp.saintmode = 10s;
restart;
}
set beresp.grace = 5m;
}
When we set beresp.saintmode to 10 seconds Varnish will not ask that server for URL for 10 seconds.
== Troubleshooting ==
Trace it under varnishlog, /var/log/syslog, /var/log/messages.
When Varnish won’t start:
Try to start varnish by:
<pre> varnishd -f /usr/local/etc/varnish/default.vcl -s malloc,1G -T 127.0.0.1:2000 -a 0.0.0.0:8080 -d
Notice the -d option. It will give you some more information on what is going on.
Logs
To see a real-time log dump (in a system-wide Varnish configuration):
varnishlog
By default, Varnish does not log to any file and keeps the log only in memory. If you want to extract Apache-like logs from varnish, you need to use the varnishncsa utility. Every poll is recorded in the shared memory log as follows:
0 Backend_health - b0 Still healthy 4--X-S-RH 9 8 10 0.029291 0.030875 HTTP/1.1 200 Ok
The fields are:
0 -- Constant
Backend_health -- Log record tag
- -- client/backend indication (XXX: wrong! should be 'b')
b0 -- Name of backend (XXX: needs qualifier)
two words indicating state: "Still healthy", "Still sick", "Back healthy", "Went sick"
4--X-S-RH -- Flags indicating how the latest poll went
4 -- IPv4 connection established
6 -- IPv6 connection established
x -- Request transmit failed
X -- Request transmit succeeded
s -- TCP socket shutdown failed
S -- TCP socket shutdown succeeded
r -- Read response failed
R -- Read response succeeded
H -- Happy with result
9 -- Number of good polls in the last .window polls
8 -- .threshold (see above)
10 -- .window (see above)
0.029291 -- Response time this poll or zero if it failed
0.030875 -- Exponential average (r=4) of responsetime for good polls.
HTTP/1.1 200 Ok -- The HTTP response from the backend.
Management console
You can access Varnish admin console on your server by:
- Your system uses a secret handshake file
varnishadm -T localhost:6082 -S /etc/varnish/secret
- Your system does not have a secret handshake file
telnet localhost 6082
Purging the cache
This will remove all entries from the Varnish cache:
url.purge .*
Loading new VCL for the live varnish daemon: You can dynamically load and parse a new VCL config file to memory:
vcl.load <name> <file>
e.g. vcl.load newconf_1 /etc/varnish/newconf.vcl vcl.load will load and compile the new configuration. Compilation will fail and report on syntax errors. Now that the new configuration has been loaded, it can be activated with:
vcl.use newconf_1
