A little while ago I was asked to solve a problem that somebody was having with Upstart, and I realised that people weren’t understanding how things were actually working and were just muddling along when doing event matching in jobs. This is unfortunate, because it hides some of Upstart’s true power, so I thought it high time I actually explained this.
Let’s start with a simple example. Fire up any Linux distribution with Upstart 0.6, Ubuntu or Fedora current releases will do, and create a file named /etc/init/example1.conf with the following content:
start on surprise
This is pretty simple, it’s a job that does nothing except declare that it’s started when the surprise event happens. We can demonstrate that works by emitting the event ourselves and checking the status of the job before and afterwards:
root@angrybirds:/etc/init# status example1 example1 stop/waiting root@angrybirds:/etc/init# initctl emit surprise root@angrybirds:/etc/init# status example1 example1 start/running
Nothing too surprising after all, I hope. The job did indeed start on the surprise event, and would now be running if we’d actually told Upstart to run something.
Incidentally I’m often asked why there isn’t a single list of events anywhere, that’s because you can match any event you like as long as you know something emits it. Events are supposed to come from all manner of sources. I do try and document them though, try running man 7 startup on your system to see an example of an event’s man page.
If events were just names, they’d be pretty boring. Events can also have attached environment variables, and these get put into the environment of any job’s process started by the event. Here’s /etc/init/example2.conf:
start on weather
script
echo $KIND > /tmp/weather
end script
This will now run a small shell script that outputs the $KIND environment variable to a file. This isn’t set anywhere, but we can pass it in the event.
root@angrybirds:/etc/init# cat /tmp/weather cat: /tmp/weather: No such file or directory root@angrybirds:/etc/init# initctl emit weather KIND=RAIN root@angrybirds:/etc/init# cat /tmp/weather RAIN
Ok, these are just examples but there are plenty of useful events on your system right now which carry environment variables such as which network interface just came up, and so on.
If you wanted to only run on a certain type of weather, you might think to check the value of $KIND within the script; you could do that, but it’s inefficient, ideally you don’t want your script run at all. Fortunately we can match the environment of an event in the job easily enough, here’s /etc/init/example3.conf:
start on weather KIND=snow
Hopefully you’ll figure that this one will only start if it’s snowing, and you’d be right:
root@angrybirds:/etc/init# status example3 example3 stop/waiting root@angrybirds:/etc/init# initctl emit weather KIND=hail root@angrybirds:/etc/init# status example3 example3 stop/waiting root@angrybirds:/etc/init# initctl emit weather KIND=snow root@angrybirds:/etc/init# status example3 example3 start/running
Events can have more than one environment variable, and you can have more than one match:
start on weather KIND=rain INTENSITY=heavy
The matches are actually globs, so you can use * and ? in there and as well as =, there’s obviously !=.
One useful use for the latter is in the stop on stanza, as well as being available for the job’s processes you can also use these in other stanzas within the job. Here’s a cute example for /etc/init/example4.conf:
start on weather KIND=rain or weather KIND=snow stop on weather KIND!=$KIND
This one takes a bit of explaining. First of all to start the job we match the weather event with $KIND set to either rain or snow. Now we supply a condition to stop the job, and we also match the weather event with a given value of $KIND except this time we match what looks like itself.
In fact this expansion of $KIND is the value that variable had when the job was started, not the value in the new event. It says to stop the job if it stops raining, or stops snowing depending on which of the two started it. Most importantly, if an event simply repeats the same kind of weather, but maybe with a different intensity, the job carries on running (but it doesn’t have its environment updated – UNIX can’t do that).
root@angrybirds:/etc/init# status example4 example4 stop/waiting root@angrybirds:/etc/init# initctl emit weather KIND=rain INTENSITY=heavy root@angrybirds:/etc/init# status example4 example4 start/running root@angrybirds:/etc/init# initctl emit weather KIND=rain INTENSITY=light root@angrybirds:/etc/init# status example4 example4 start/running root@angrybirds:/etc/init# initctl emit weather KIND=sun root@angrybirds:/etc/init# status example4 example4 stop/waiting
Ok, last fake example before we get onto the fun bits. Remember the example from above:
start on weather KIND=rain INTENSITY=heavy
Upstart lets us shortcut this a little, the environment variables are specified in an order on the initctl command-line and if we know what that order is, we can just assume what variable is in that position. So as long as we know a weather event always has a KIND followed by an INTENSITY, we could shortcut that to:
start on weather rain heavy
If you’ve used Upstart at all, you’ve seen that shortcut before. A lot. You may not have even realised it was a shortcut at all, and that’s what I hope to fix here.
Here’s an example of where you’ve used that:
start on started dbus
You should hopefully now recognise that started is the name of the event there, an dbus is simply the value of its first argument, whatever that might be. Remember I mentioned that events have man pages? Take a look at man 7 started, which is the man page for this event.
It documents which environment variables are attached to the started event, and most importantly what order they come in.
started JOB=JOB INSTANCE=INSTANCE [ENV]...
So really when we wrote the previous, we were just using a shortcut to specify:
start on started JOB=dbus
You might wonder what difference this makes. A good example of how to exploit this is the stopped event. If you look at it’s man page (man 7 stopped) you’ll see it has a large number of environment variables specifying not only which job stopped but the reason for it stopping. One of those is the exit signal, for example.
Now you know that you’re just matching the $JOB environment variable, it’s obvious that you don’t have to! You can match any other environment variable or variables in the event, or none at all.
Here’s how to run a script if any other job on the system exits with a segmentation fault:
start on stopped EXIT_SIGNAL=SEGV
I said you didn’t have to match any variables, just like in the first examples we didn’t, there’s a neat use for that with the job events. The starting event blocks the named job from actually starting until anything run by it is started; or, in the case of jobs marked task, finished.
Here’s a little job that runs every time another job is started, and blocks that job from actually starting until the script finishes.
start on starting
task
script
....
end script
Useful both for debugging and performance analysis.
Now for the really neat bit. So far we’ve concentrated on the environment variables that come from events, and those that Upstart puts into the job events. But we can influence these in rather useful ways.
Firstly we can declare a default value for an environment variable in a job, if no alternate value is given in the start event or command, then this default value wins:
start on mounted
env MOUNTPOINT=/tmp
script
....
end script
This script will run for each occurrence of the mounted event, and will hopefully get the value for $MOUNTPOINT from that event. But should the value be missing from the event, or the script be started manually by a system administrator, a default value is provided.
This isn’t a false example, that’s from the job on your system that cleans up the /tmp directory on boot. The default value wasn’t there in earlier versions of Ubuntu, and this had a rather disastrous side-effect when run by hand.
Ok, we can set the values of environment variables from a job, and we don’t have to match the job name in the usual job events. We can combine these two facts in a very interesting way when we can export the value of a job’s environment variable into its job events.
Here’s the first job:
env AM_A_DISPLAY_MANAGER=1 export AM_A_DISPLAY_MANAGER
This sets the default value of $AM_A_DISPLAY_MANAGER, but this isn’t a variable we ever expect to be supplied by an event so it just gets passed into the environment of its processes. It’s not that useful either on its own.
The export line is the useful one, it adds the value of the named environment variable to the job’s events. That is the starting, started, stopping and stopped events.
Now, in another job, we can do:
start on started AM_A_DISPLAY_MANAGER=1
This is run when any job is started that has that environment variable in its events. In other words, we can tag classes of services so we don’t have to list every single one.
And because everything in Upstart is the same fundamental type of thing, this can work in the opposite direction. For example we can put in our job:
env NEED_PORTMAP=1 export NEED_PORTMAP
This means our events will have NEED_PORTMAP=1 in them, now remembering that the job waits for the side-effects of the starting event to complete, we can now write in /etc/init/portmap.conf:
start on starting NEED_PORTMAP=1
So we can implement a dependency-based init system with Upstart, an event-based init system.
I look forwards to finding out what else you can do with it.
(reposted from http://upstart.at/2010/12/03/event-matching-in-upstart/ – post comments there)