Bummer! This is just a preview. You need to be signed in with a Pro account to view the entire video.
Configuration Management with Puppet for Developers47:52 with Rachel Andrew
In this session Rachel will introduce you to configuration management using Puppet and why this is helpful, even for small infrastructures. You'll learn how using a tool like Puppet can make it simple to create development and staging environments that replicate production. This session will give you practical advice for things you can implement immediately in your own work. You'll also take away plenty of resources to take your learning further.
[MUSIC] 0:01 Okay, morning. 0:15 Obviously not a popular subject. 0:16 Nobody wants to know about this stuff, the designery things going on elsewhere. 0:18 I am going to talk a bit about config management with Puppet. 0:23 I've got a whole bunch of stuff ranging from real kind of getting started 0:28 with this stuff right through some of the more interesting stuff that we're doing. 0:33 So, I am kind of hoping there will be a bit of something for everyone, no matter 0:36 whether you've not touched this stuff before, or whether you've done a fair bit. 0:39 So I'm Rachel Andrew. 0:44 I am from the UK, as you can probably tell. 0:46 And the reason I'm talking about this stuff is really due to my day job. 0:49 I have a content management system. 0:53 That's our product. 0:55 That's what we sell. 0:56 And it's self-hosted. 0:57 It's written in PHP, MySQL, and people download it and 0:59 install it on all of their terrible shared PHP web hosting. 1:03 So I kinda know an awful lot about terrible shared PHP web hosting. 1:07 But I also get to see in support a lot of the problems people are having with that, 1:11 and also with their own local environments. 1:15 Cuz most of the stuff that actually we deal with everyday in support doesn't 1:21 really have anything to do with our product, it has to do with the fact that 1:25 people develop in ways that do not match their live environment at all. 1:29 And so they build things, quite happily, sometimes taking months to do so, and 1:34 then they deploy it live and everything breaks, and 1:37 they have this whole catalog of issues which come into support. 1:40 And that's just because they were developing in a way that just didn't 1:42 mirror what they were going to end up with live. 1:45 So, we see all sorts of problems. 1:51 We see an awful lot of people developing on live sites or in subfolders of them. 1:53 We also get people developing in subfolders locally. 1:57 And we just get this thing where people set up local environments that are so 1:59 different to the eventual live server that they just don't work. 2:04 We also get people who work in teams so 2:08 everyone has got a different development environment and they kind of have this 2:10 constant battle between each other as to whose environment wins. 2:13 And then they go live and then it all breaks again. 2:17 So a lot of this stuff isn't particularly exciting. 2:21 And I talked to people who probably should know better a lot of the time. 2:24 Clever people who are developing in the kind of most scrappy of ways and 2:27 deploying stuff in the most scrappy of ways. 2:31 And you're like, why are you doing this? 2:34 And they're like, well we haven't got time to sort it out. 2:35 But yet sorting this stuff out gets you a huge amount of time back, 2:37 which you can then use for more interesting things. 2:41 And as well as talking about workflows and 2:43 helping out customers with workflows, I also use all of this stuff live myself. 2:46 We are not software as a service. 2:50 Our product is self-hosted, yet 2:52 we've got that kind of infrastructure that you need to sell software. 2:54 So we have servers that run support systems, and 2:57 we have servers that sell software and deploy license codes and things. 2:59 And all of that is managed by Puppet and has saved me an awful lot of time where 3:03 I used to be SSHing into individual boxes and running PERL scripts and things. 3:09 So, I wanted there to be a bit of something for everyone in this cuz I know 3:15 that when I talk about this stuff people are at very different levels with it. 3:18 So I've got a really simple way to get started with Vagrant and Puppet, for 3:22 people who haven't used this stuff before. 3:26 And then, moving on to a bit of stuff about writing your own manifests and 3:28 your own modules in Puppet. 3:32 And then just some kind of ideas about how you might take this knowledge further. 3:34 How you might take it into production, or how you might use it in interesting ways, 3:37 to do things other than just, you know, setting up SSH users or whatever. 3:41 So first steps with all of this. 3:48 If we assume that we need a bunch of things and we want to be able to develop 3:52 multiple websites on a local system, we'd like to know that the live server and 3:56 our local server, local environment, actually support the same things, so 4:00 we're not going to have any crazy things happening when we go live. 4:05 We'd like to be able to deploy our site and 4:10 have confidence that what's on the live server is the same as our local version. 4:12 And we'd like everyone who works on the site to have the same development 4:17 environment so we're not creating these problems for ourselves of people sort of 4:20 not knowing whether someone else is using the same thing as them. 4:24 Now a lot of this is PHP based, because I am a PHP developer, but 4:30 really it doesn't matter. 4:34 You can use this stuff with anything. 4:35 I did a beginner level version of this stuff for Smashing Conf, and 4:38 I ran a survey to ask people what they were using for their local development 4:41 environments, and that was a sort of a design-heavy audience. 4:45 So I asked, one the answers was, are you using things like MAMP, or WAMP, 4:50 these things that set up a lump stack on your local computer. 4:54 And 63% of people were using these things, there was also a big bunch of people who 4:58 were just developing on the live server and kind of hoping stuff wouldn't break. 5:02 There's a lot of people using these packages. 5:06 I mean they're better than not having a local development environment but 5:09 they kind of fall down because the aim of a development environment 5:12 should be that it's identical to the live server, and these things aren't. 5:17 They create a sort of development environment that allow you to maybe 5:22 install WordPress, or Perch, or Craft CMS, or whatever you're using. 5:26 But it's not actually going to mirror any sort of live server. 5:30 And certainly for PHP, this is really important because PHP does the hand stuff 5:34 off to the host operating system. 5:39 So quick example of that which catches people out and who come into support all 5:42 the time is in PHP is a couple of methods of formatting dates, if you don't know.. 5:45 You can use the PHP date function. 5:51 You can also use STRF time, which is neat because it allows you to format the date 5:53 and use the locale that the server's set in. 5:58 So if you ask for a French date or 6:01 whatever you are gonna get the date words actually spelled out in that language. 6:03 That's pretty neat. 6:07 But it passes its functionality back to the host operating system so 6:09 on Windows a whole bunch of this functionality just doesn't work. 6:12 Your dates just won't appear. 6:16 This obviously baffles people. 6:18 So if you happen to be developing Windows you might think, well this doesn't work, 6:20 and whereas most people are deploying to Linux and it would be fine. 6:25 So that's just one of a whole bunch of things that happen if people are using 6:28 Windows to develop and then going out to Linux. 6:33 There's also obviously differences between PHP versions or you know, 6:39 the versions of any language. 6:42 Quite often, people are running fairly up to date versions of PHP locally, 6:45 then they deploy to their client's terrible web hosting and 6:48 it is running PHP 5.3 and things break. 6:51 And a lot of PHP modules, even really common ones, are completely optional, 6:54 such as, like, how you connect to your database. 6:59 That might not be there on the live server. 7:02 So all of these things cause problems when people go live. 7:04 And that's why I try and 7:08 encourage people to use virtual machines rather than these packages. 7:10 Because you can set up a virtual machine that mirrors even the most terrible of web 7:16 hosting if that's what you want to do. 7:19 The most limited of web hosting, or the best web hosting or whatever you want. 7:21 And you can then create the exact same configuration of that live server locally. 7:26 And lots of people have used virtual environments even if they haven't 7:31 run VMs for development. 7:34 Mac users are often using them to run Windows on their local machines. 7:36 And so it's exactly the same sort of thing. 7:40 We're just using a Linux virtual machine for development. 7:42 And there are now some really good tools for doing this. 7:46 I've been doing this for a few years and the whole sort of setup has got so 7:49 much easier. 7:52 So if you've never done this before, you need a couple of free bits of software. 7:54 VirtualBox, which you can download from this really beautiful website, here. 7:59 The software however is better than the web site. 8:03 And the second thing is Vagrant. 8:07 Vagrant is a tool that manages VMs. 8:11 So you basically just install these and then you need to configure and 8:15 provision a virtual machine, which if you haven't done before, 8:18 other people have sorted this out for you and that's what we'll first look at. 8:22 There are specific packages that people have built for different environments, for 8:27 instance, for WordPress. 8:30 This is something which I don't actually know how you say this, 8:32 it's kind of like P-U-P-H-P-T. 8:35 Puffit, maybe. 8:38 On the About Us page, it says the P is silent, so you can take that as you will. 8:39 But this is essentially something that allows you build, 8:44 configure and build a project for Vagrant. 8:48 So you don't have to kind of worry about all the nuts and bolts yourself. 8:51 So there's a bunch of different options in the side bar. 8:57 I mean it was originally just a lump stack thing for PHP. 8:59 You can now pick all sorts of stuff, Python and Node and all sorts. 9:02 So you choose what languages you want. 9:06 I wrote up, actually, I'm gonna go through some of the different options for this. 9:13 But I actually wrote up a quick tutorial, how to go through the sort of options for 9:18 basic lamp setup, if this is something you haven't looked at before so 9:21 I don't have to go through everything blow by blow. 9:25 A quick tip for using PHP shared hosting, it seems that most 9:33 people don't seem to know actually what is on their shared hosting. 9:36 And in fact if you ask your host what's on your shared hosting, they'll probably 9:39 tell you something different to what's on your shared hosting is what we discover. 9:42 So using PHP info, is before you start configuring your virtual machine, 9:46 find out what is on your server. 9:50 if you stick that into a file and run it on your server, 9:52 you'll get all of the information about what PHP is doing, all the PHP modules, 9:55 the version of PHP that's installed. 10:00 Other things to check, and obviously the version, any modules, things like GD. 10:02 Stuff like post max size, and upload max size. 10:08 What is the size of files that you can upload? 10:10 I constantly get people coming in to support saying works on my machine, 10:12 went live, now can't upload anything. 10:16 And it's just because they've got their, you know, 10:18 the configuration on the live server at some shared host. 10:21 They've set all the limits down really low to stop your from loading massive files. 10:24 And so things just don't work anymore. 10:27 Just that sort of stuff. 10:30 And max input vals, ridiculous setting that the PHP project, that, 10:31 say if you get a big form. 10:36 If that's configured really low, it just won't submit. 10:37 And that's obviously incredibly baffling to people who've created a nice big form. 10:39 So there's things like that you can check on the live server. 10:44 And then you can replicate them locally so there's no nasty surprises. 10:47 If you're using a virtual machine, sort of one of the sort of sticky problems and 10:56 one of the things that used to be quite difficult was that you're basically 11:01 running an entire operating system with its own file system on your local machine. 11:04 An issue is that you need to be able to access the files. 11:11 You want to be able to access files on your local machine from a virtual machine. 11:15 You don't want to be having to you know, 11:18 copy files in to the virtual machine to run them from the web server there. 11:20 So virtual box and they will kind of sort this all out for you and 11:24 allow you to share folders. 11:29 So you can have your local edit files in your sites folder on your Mac, or 11:31 whatever, and you can use virtual machines, to access them and 11:36 you can display them through the web server. 11:38 So you're configuring your sites here. 11:43 You can actually set up, and you can map files. 11:45 And I usually, you know, map things to var dub, dub, dub, on the on the VM there. 11:49 If you're using a Mac, in effect is a good idea, 11:54 it will speed things up quite substantially. 11:57 You can also also host multiple virtual hosts on one VM, what we see 12:08 an awful lot in support is people who are developing their sites in sub folders. 12:13 Even when they're using virtual machines or they're using mound for 12:18 one or whatever. 12:20 Instead of creating virtual hosts, they'll just have one virtual host, 12:21 you know one host, so local host maybe locally or 12:26 whatever and then they'll create sub-folders 12:28 which means that when they go live they have to move everything up into root. 12:31 And things break, like oh where's my CSS gone? 12:35 Because they've shifted things around. 12:38 So you can create multiple V-hosts here which allows you to 12:41 have everything with its own route which then solves that problem. 12:45 When I am creating VMs, it kind of depends on how I'm working. 12:51 What I tend to do is set up one which mirrors the server site, 12:55 you know I've got several servers with various things. 12:59 So I set up a VM which has got all of the same sites on. 13:01 Vagrant also has a nice functionality where you can sort of set up 13:05 clusters of VMs if you've got things that need to talk each other, 13:07 like a search server or whatever. 13:10 You can set those up and run them together. 13:12 Which works quite nicely. 13:14 So once you set all your options in that tutorial that I linked to earlier, 13:21 that sort of goes through, in more detail, the different options. 13:24 You can then download a zip, and 13:26 put it somewhere on your computer, and unzip that. 13:28 And then, don't do things like this over conference Wi-Fi, 13:33 cuz it doesn't please them. 13:36 But if you go to the command line, change into the unzip directory, and 13:37 run Vagrant up. 13:41 And the first time you do this, it takes a while. 13:43 Because Vagrant has to go and fetch a base box. 13:46 It's got this concept of base boxes, 13:49 which is essentially an operating system configured with a bunch of stuff 13:52 already which it will go and fetch and retrieve and then use that. 13:57 When you run new virtual machine, if you use the same base box, then you won't have 14:01 to have the step of downloading it because you've already got it, 14:06 it'll just copy it and use that again. 14:08 So that's quite useful it sort of manages those for you. 14:10 So once you got the box up and running you should be able to go in a browser 14:17 to the domains you configured when setting up the virtual host and view your sites. 14:20 And that just sort of works really nicely. 14:24 So these are pretty much the commands you need to know to use Vagrant. 14:31 You can take a box up or down, and if you halt a machine it will preserve state so 14:34 everything will remain as it was. 14:39 If you set up a database it'll stay there. 14:42 And if you do vagrant destroy it will take it right back to the beginning again. 14:44 So, you will, obviously, lose files that are on your host computer. 14:48 But, if you've configured something, and the typical thing that you build inside of 14:52 the VM is the MySQL data base, for instance. 14:56 If you do vagrant destroy, that will get deleted. 14:58 So if you've got data in there that you care about, 15:01 you're going to want to MySQL that, export it before you do vagrant destroy. 15:03 But I have machines which I bring up and down and use sometimes for 15:07 months before I destroy them. 15:12 It'll preserve everything unless you actually do a destroy. 15:14 You can get into the box with sh, by just doing vagrant ssh and 15:18 then you're in at a command line inside the server and 15:22 it'll just act pretty much like any Lenox at the command line. 15:25 That's a nice tool called Vagrant Manager. 15:35 It was originally just for the Mac, they now have a Windows version. 15:39 That's a really nice tool. 15:43 It let's you do all sorts of things with Vagrant and keep you out of command line, 15:44 if you prefer not to be in the command line and even if you are. 15:49 The other thing it does is it shows you, 15:52 in the taskbar, how many vagrant virtual machines you've got running. 15:54 So when you're wondering why your computer has gone incredibly slow. 15:58 And you realize that you've got, like, eight of the things, 16:02 running away in the background, which happens to me quite a lot. 16:04 It just sort of alerts to the fact that they're there, cuz you can kinda get quite 16:07 a lot going on a decent mac before it starts to really impact performance. 16:10 They're fairly good, but yeah it sort of lets you know what you got running. 16:14 So the nice thing is that even just doing this in this simple way you can share 16:21 this package, this project with anyone in your team. 16:26 You can see I have configured this. 16:29 This is how we're going to develop and you can hand it over to someone else. 16:30 They can also use it and you've both got the same development environment there. 16:34 Which is great because you don't have this problem of people 16:38 kind of fighting amongst themselves with development environments. 16:41 So if you've never used Vagrant before, that's a really great place to start, 16:48 because it gives you everything configured. 16:51 And you know then that what you're doing is gonna work. 16:55 If once you start playing with this stuff by hand, there's lots of moving parts, you 16:58 don't know whether you've done something wrong or whether stuff just isn't working. 17:02 This does just seem to work. 17:06 I've had lots of people who have never used vagrant before get started with this 17:07 and it does just seem to work pretty well on most people's machines. 17:12 So that's all pretty cool and if you're doing, 17:20 I don't know WordPress projects are fairly simple projects. 17:23 That may well be all you ever need. 17:27 However, I'm gonna talk now a bit more about the actual technologies that are at 17:30 play here and how we can start to write our own stuff and 17:35 do interesting things with this. 17:38 So I'm gonna roll back and look at a very simple project for Vagrant. 17:42 It's probably something a lot more like what you'd create yourself. 17:47 Because obviously a tool that we've been looking at. 17:52 That's for everything. 17:55 It's trying to do everything for everyone. 17:56 It's got an awful lot of configuration. 17:58 If you poke around inside the pocket manifest and 17:59 things, there's an awful lot of it. 18:03 This is the opposite, this is really, really simple. 18:05 I put this together, it's on GitHub. 18:07 I put this together just to help some of our customers, 18:10 say this is kinda how you get set up with Vagrant and with puppet. 18:12 So it's very, very simple. 18:18 There's nothing specific to our product about it it is just a very simple 18:20 lump stack. 18:22 So as I have mentioned already Vagrant is simply a tool for 18:29 managing virtual machines. 18:32 So we need to tell Vagrant exactly what sort of machine we're creating. 18:34 And so we do this with a Vagrant file, 18:38 every Vagrant project needs a vagrant file in the root it looks a bit like this. 18:40 As you'll see if you look inside any downloaded project you'll see 18:45 something like that. 18:48 And I've commented that it's so that people can kind of modify it a bit for 18:49 their own set up. 18:52 If the Wi-Fi's holding up you can actually go and have a look at this project on 18:53 GitHub So the top of the file 18:58 is some settings which I use when setting up the boxes, IP address and so on. 19:03 The stuff I want to concentrate on is right at the bottom of the file, 19:07 it's where we choose to use puppet to provision the server. 19:10 So provisioning is just setting up and you can use all sorts of stuff, 19:14 vagrant has got its own very simple provisioning language you can use puppet. 19:18 It's not the only one you can use. 19:22 You can also use Chef, if you've come across Chef or you can use Answerable. 19:24 But you need to provision the box, 19:28 you need to say what you want on the server, generally. 19:30 I mean you might want to create some sort of bare metal box, for some reason, but 19:33 most of the time you want to set up other stuff. 19:36 So, here, I'm enabling Puppet. 19:40 And I'm pointing it to some stuff that Puppet needs to know where it is, 19:42 some manifests and the main file, and some modules. 19:45 And also, this higher config, which I'll talk about in a minute. 19:49 So if you've not come across Puppet, 19:56 Puppet is essentially a configuration management solution. 19:58 It lets you define, in code, the state of a server. 20:01 So it'll let you define, in code, packages that should be installed, 20:05 like Apache or PHP, services that should be running. 20:09 So, yeah, like Apache, or any other sort of service, files or folders, permissions. 20:12 And you can set up things like Chron jobs and 20:18 so on, which is a useful thing to be doing with it. 20:20 So you could create a VM, create a bare method VM and SSH into it, 20:25 a vagrant SSH and start installing stuff. 20:29 So I'm an old Debian nerd so I used apps. 20:32 So you might just install on Debian or Ubuntu like this. 20:36 So if you use young on some of the red hat base things. 20:40 I install packages like that and just set yourself up. 20:44 And do that every time you set up a VM. 20:47 And then obviously you need to create your databases and 20:49 create your V-host configs just like you would maybe on a live server. 20:53 So you could do that but that would take some time. 20:57 But instead with Puppet, I create files and these files can be checked in to Git. 21:00 And they can be reused every time I create a virtual machine. 21:06 So when I create a VM, I can just run these with a provisioner. 21:10 I can create these files. 21:12 I'm just writing these in my editor Sublime text, like any other code. 21:15 And I'm also safely away from the command line when I'm doing it. 21:18 So I can't make any stupid mistakes. 21:21 I can just write it in here, and 21:22 I can check over what I'm gonna do before I deploy it. 21:23 So I don't need to go SSHing into individual servers just to get them setup 21:28 to start to do some development. 21:32 I can vagrant up a server and I can start to do some development. 21:34 And that's really what I want to be doing. 21:37 I don't want to be messing around with servers all day. 21:38 So if you look at the Puppet Labs website. 21:44 Because they have, they have two versions of the project, 21:49 they have a sort of enterprise version and then they have open source Puppet. 21:51 And a lot of the stuff on the website is very, 21:55 very geared towards enterprise Puppet. 21:57 And it's very geared towards people running Puppet 22:00 to manage big infrastructures. 22:02 But if you dig around you can start to find the actual docks and 22:05 things, which are the same no matter what you're doing. 22:10 So there's a whole bunch of new terminology if you've not used Puppet 22:13 before. 22:15 So a manifest is basically a file that contains Puppet code. 22:17 It'll have a .pp extension, and that will have Puppet code. 22:21 Puppet has a DSL which is its own sort of language that it writes, that you write 22:25 the code in, but it's very straightforward to use if you've used anything else. 22:31 A resource is anything that needs configuring. 22:36 So a resource might be a file, or it might be a package like Apache. 22:38 It could be something you need to be running, task running or whatever. 22:43 And so, recourses have types. 22:47 So you have file and package and Chron for Chron jobs. 22:49 And then you have modules. 22:53 And modules are just a collection of manifests, templates, 22:54 and other files organized around a particular purpose. 22:57 So, you might have a module for configuring Apache. 23:00 Or, a module for 23:03 doing something completely custom to your setup that just applies a bunch of stuff. 23:03 So we can take my simple Apache module as an example. 23:13 That module is stored under Puppet modules in my Vagrant project. 23:16 And, there's two manifests init.pp and vhost.pp. 23:19 And that's the kind of structure of it. 23:22 So all modules need an init.pp manifest. 23:32 And when we include the module, that runs. 23:35 So we're saying here that we want the Apache package, the Apache 2 on Debian. 23:38 And we're saying, ensure present. 23:45 We want it to be there. 23:46 And then we want to make sure that it's running. 23:48 And the we want to enable the rewrite module, because I think I might need that. 23:52 Now, something to note here, 23:59 what tends to baffle developers when they first encounter Puppet. 24:01 Is that you kind of expect that that will run from the top to the bottom. 24:07 But, it might not. 24:13 Because Puppet is used to describe the end state of your machine. 24:15 It is not describing how it gets there. 24:20 Because it doesn't know when it arrives at your machine, what state it's already in. 24:23 All it has it do is make sure it's there. 24:27 So although they are making changes at Puppet labs for 24:30 how dependencies are ordered, 24:35 I think manifests are gonna tend to run top to bottom far more than they used to. 24:36 If you want things to happen a certain order, 24:42 you need to declare it which is what I'm doing here. 24:44 When I've got like require a package Apache 2, 24:46 I'm saying don't check that Apache is running. 24:48 Until you've made sure Apache exists. 24:51 Because that's not gonna work. 24:54 So I'm using require there. 24:57 You can also kind of chain resources. 24:58 There's a few ways to deal with resource ordering in Puppet, but 25:00 it's something to kind of be aware of. 25:03 That things may not run in the order that you expect them to run. 25:05 It's not sort of procedural from the top to the bottom. 25:09 And if you want to know a bit more about this, this is a post from Gary Luisa who I 25:13 think works for Puppet now but he writes a lot of stuff about Puppet. 25:17 If you're doing this stuff, he's a great person to follow. 25:21 He writes all sorts of interesting things. 25:25 So yeah, so generally Puppet doesn't care about execution order. 25:28 It's probably a good idea to believe it never cares about execution order and 25:31 actually declare what order you want things to happen in. 25:34 If you get a whole load of errors when you run stuff, 25:39 it tends to be because things are happening in the wrong order. 25:41 So another thing we might want to do with Apache is setup the virtual hosts, 25:47 the web sites we are running. 25:50 So we do this with a manifest called vhost.pp. 25:52 And so when this is called it sets up the virtual host. 25:55 So at the top I'm setting up some parameters and 26:01 I can then use those when setting up the file resource. 26:04 And the content of that file resource is actually a Ruby template 26:11 with a .ERB extension. 26:15 And, I'll show you where we get the values that we put into that in a minute. 26:17 So, we can, with that, if we do this template, content is template, that will 26:21 use a template, and will actually replace in any values that we pass into it. 26:25 And so that template looks a bit like this. 26:30 And if you've ever set the V-host in Apache that should look familiar. 26:33 It's basically the V-host there and you can see there's some placeholders there 26:36 for the things we're going to replace in. 26:40 Now I'm not a ruby developer at all but 26:42 the amount of Ruby that you need to do to work with Puppet is very, very limited. 26:44 Most of so I'm a PHP and a PERL person. 26:49 So I sort of manage. 26:53 I tend to put this default template, 26:55 V-host management Puppet if I am managing anything on a server with 26:57 Puppet I will tend to stick something like that at the top. 27:00 Because it means that if anybody else SSHs in and thinks I'll just change this, 27:03 if you're working on a live environment. 27:06 On a live environment Puppet runs on some of them every 30 minutes and 27:08 some of them more frequently than that. 27:12 What will happen is if Puppet comes across a file that it manages that 27:15 has changed it will put it back to the way it's expected to be. 27:17 So someone's SSH has made a change it'll just go back. 27:21 So I tend to make sure that configuration files that Puppet is looking after, 27:25 I make sure that it says. 27:29 So anyone SSHing in will realize that it would be a very silly idea to make that 27:31 change because it's gonna get rolled back. 27:33 So that's the module. 27:41 And we have other modules for things like PHP and MySQL and so on. 27:43 So if we go back to the Vagrantfile you can see that I am telling Vagrant here 27:47 where those modules live, so they could really be anywhere in any project. 27:50 And I also pointed to my sites.pp, 27:56 and that's a special manifest that just kind of kicks off the whole process. 27:57 So site.pp doesn't do much. 28:04 What it really does is include all of my modules. 28:06 The stage stuff up there, 28:09 is essentially just another way of managing resource ordering. 28:11 I want Base to run first because it stores some resources that other modules need. 28:15 For instance, it edits the app sources list 28:20 to make sure I've got things there that I'm then gonna need. 28:23 So when we Vagrant up our project for the first time, or 28:26 you can run Vagrant Provision to provision a machine you've already Vagrant upped. 28:29 That includes all the modules and Puppet starts setting up the server. 28:34 So we know how to write modules. 28:43 And we know how to get Vagrant to use those modules. 28:46 So the final piece of the jigsaw really is how do we get configuration data 28:49 into those modules. 28:54 And, what people kind of used to do. 28:55 When I first came across Puppet what people were doing was putting all of their 28:57 configuration data into the manifests themselves. 29:01 So you'd end up tracking to get a whole bunch of stuff passwords and 29:04 things that were specific to individual sites. 29:09 And as a developer I'm looking at this and I'm thinking this is really crafty it's 29:12 not nice to have all of this specific data kind of peppered through my manifest. 29:16 It's potentially a security risk and passwords and things there. 29:20 And I'm gonna have to go and 29:25 search right through them every time I set up a new server. 29:26 So I started poking around and I discovered Hiera. 29:29 And Hiera is becoming now very important in the sort of world of Puppet and 29:33 it's great. 29:38 So Hiera allows us to create hierarchies. 29:40 You can specify multiple backends for Hiera. 29:44 Which, a back end is just a data store. 29:47 And the default stuff tends to be in XAML. 29:50 Which is what I'm using because that's what you tend to find in examples. 29:53 However, you can also use JSON. 29:56 And you can write backends for Hiera. 29:59 So, people have written backends for all sorts of other things as well. 30:01 We use JSON Live because we can then exploit it from other things and use it. 30:06 But if you're writing your own I think Jamil is quite nice because it's kind 30:11 of human readable. 30:14 It's easier to read than JSON. 30:15 So this is just the default configuration file and 30:17 that's just pointing to my own data. 30:22 So this is my config.yml which has my specific data in it, 30:26 and I have some settings at the top there, sort of things about php. 30:32 PHP modules and I've also got my V host and 30:36 some databases that I'd like to create. 30:39 And they're all there in the YAML So 30:41 we can have a look at how this actually works. 30:44 So it's pretty simple, we want to set up MySQL, so MySQL needs a root password, 30:48 I'm just local on vagrant so generally in vagrant if you have a password you set 30:52 it to vagrant because you know what it is. 30:56 It's just saying you bring it up and down. 31:00 So I'm setting my MySQL root password here to vagrant in the higher config. 31:02 So this is the in it.pp inside the MySQL module and 31:09 we're using that root password and we just request it by name in a higher function. 31:13 And so now that loop password is set to vagrant, and then we can use that. 31:18 So this gets us away from this problem of having all of our modules in manifest full 31:24 of specific information that is for one setup. 31:28 So you're not limited to just these simple variables like this. 31:34 You can also pass data structures around. 31:36 So back in the YAML we've got this list of PHP modules here. 31:40 And that, it's quite nice to that those there in the YAML because then if you want 31:45 to set up something which doesn't have GD for instance or you know doesn't have curl 31:49 or mirror some terrible hosting environment that you're trying to mirror. 31:53 You can just take them out or you can add in some other ones and 31:57 that's the name of the PHP modules. 32:01 So, in the php module init.pp we use that information. 32:03 So, higher a php module grabs that, turns it into a list and 32:07 we can just pass that directly into the package resource type because that will 32:10 take a list of packages and it just makes sure they all are present. 32:13 And we're also using some those setting as well from the top there you can see 32:19 that I set up. 32:22 And the virtual host we looked at earlier using the template. 32:28 Now that information also comes from Hiera data. 32:31 And we want to be able to set up a number of sites that have 32:33 a number of parameters each. 32:37 So, we can't just use a simple list like we did with the PHP modules. 32:38 So that's the YAML I have got a couple of sites configured there. 32:43 So I have got a special module I have named boot strap. 32:49 And I get those sites with the Hiera Hash Function. 32:54 I can then pass that into a special puppet function called create_resources. 32:58 And that will create several resources at once. 33:02 So you're passing the name of the resource you're creating. 33:05 So in this case it's the vhost1, and 33:07 also then the sites which is that they have a hash of all that data. 33:10 So in our case that creates two websites but it could create two hundred. 33:16 On our demo server quite frequently it will be creating, yeah, 150 sites at once. 33:20 So I'm doing the same thing with the mass-grow databases. 33:26 And so that's the manifest that creates individual databases and 33:34 it's just the same as the thing that creates individual websites. 33:37 So, if I want to set up a new VM I can clone that repository and 33:44 I can edit the vagrant file give it an IP address and a project name. 33:50 I can edit the configure YAML to set up the sites and databases and I vagrant up. 33:53 And then it just kind of works, and that's nice. 33:58 And the nice thing about that is that my entire development environment is 34:01 described in text files. 34:05 And so it's reproducible. 34:07 So you can commit that, or you can create development environments. 34:09 You can stick them in get. 34:14 You can store them alongside your project files if you want to. 34:16 And you can share them around. 34:20 Your Mac blows up and you have to send it into the geniuses and 34:22 it's gonna be there for two weeks. 34:27 Apple actually broke my Mac, they had to replace it in the end. 34:29 So I had no Mac for three weeks, I was working on various different ones. 34:32 But it didn't matter because I could just grab a copy of my development environment, 34:34 faker it up and use it. 34:39 This issue, it was always development environments, just with one thing. 34:41 Everything that I have is in the cloud. 34:45 Everything's up there. 34:46 But it was development environments that were such a pain every time I had to 34:47 switch machines. 34:50 And this has just totally gotten rid of that problem for me. 34:51 You can share this stuff around, and 34:56 if you use Hiera, the next thing about that stuff is it's very easy to edit. 34:58 So you designers, and developers who are less happy about doing server type of 35:03 stuff, can easily just edit the config file, and off they go. 35:07 There's not a huge amount of stuff they've go to be doing there. 35:12 So that gets us a working lump setup. 35:20 So hopefully you've got your virtual hosts there pointed at site files on 35:23 your Mac drive. 35:27 Obviously you'd need to then import a MySQL dump of your database and 35:29 then you should be ready to go. 35:32 But there's no reason why you couldn't automate all sorts of stuff. 35:34 So I'm going to just talk through something I do which deals with a very 35:40 specific problem to us not because you're likely to want to do the same thing, 35:43 but it might give you some ideas of the sorts of stuff you can be doing with 35:48 these things. 35:51 So, for Perch, we've got three demos. 35:54 And so, every time we do a Perch update, or at least every few updates, or 35:56 is anything major, we want to update our demos, and they're live, so. 36:00 Now what do I have to do? 36:07 Cuz when we provision a demo on the live server, those demos, 36:08 it's essentially like setting up a little shared hosting environment 36:11 with a sort of fully configured site that someone can poke around in. 36:14 Cuz what I didn't want with our CMS was to have this thing. 36:18 You know when CMSs have a demo and everyone can log in to the same demo? 36:21 And so you're like trying something out and somebody else is changing it and 36:24 you can't figure out whether that's the site of the problem or you or whatever. 36:27 So we actually set these things up. 36:31 So what I needed to be able to do was to create these demo packages and 36:35 get them up onto the server. 36:40 And this was becoming a real pain because they were never getting updated. 36:41 We put the template files and things out on GitHubs so 36:46 people can have a look at them. 36:49 And we also have a database dump out on GitHub which has placeholders in so 36:51 that if someone wants to get that to put it in with their copy of Perch they can 36:55 try everything out locally as well. 36:58 So, when I want to update our demos, what I need to do is I need to get the stuff 37:00 off GitHub which is just the files and the site pages and bits and pieces. 37:05 I need to combine those with Perch call, 37:10 the call of our software, and any add ons, because they go based as well. 37:12 So I need to combine those two things. 37:17 I then need to do the upgrade, which is going to change the MySQL file, 37:20 and I need to commit back to GitHub to get all those files, but 37:26 also that data base dump with everything replaced back out again so it's available. 37:29 And I also need to then create Ruby templates, and so on, 37:35 that are used in our live demo server, also with placeholders in. 37:37 Now, doing this by hand, right nuisance. 37:40 So, I do all of this by Puppet in a vagrant virtual machine, locally. 37:44 The first thing I need to do. 37:50 Those files are in Git. 37:51 What I wanted was when I my virtual machine I wanted puppet to check some 37:53 files out of Git and I was about to write this, but 37:57 I instead I went to the puppet forge and I discovered that there was a module 38:00 called VCS repo which will check some things out of Git for you. 38:05 Now you can see there is a little flag on this, supported. 38:10 Puppet Labs have got, anyone can submit modules they have written 38:14 to the Puppet Forge and there's loads of really great ones. 38:18 But Puppet Labs have a certain set of them which they support. 38:21 And basically they support those for their Enterprise customers. 38:25 So if you're using one of the supported modules, you can be pretty 38:28 sure that it's gonna be quite good, and it's not going to cause you any problems, 38:31 because they're using it for their enterprise customers who pay quite a lot 38:34 of money to be private enterprise customers. 38:37 So they're worth looking out for, 38:39 I mean there's loads of great stuff on the puppet forge, but 38:40 this is one of those supported modules. 38:43 You can install them at the command line if you're actually 38:46 in a machine that's running Puppet. 38:49 There's a Puppet module tool. 38:51 But you just want to grab the package and 38:53 put it into your project if you're working locally. 38:55 So, here we go. 39:04 I'm using, as a resource, I've now got a new resource VCS repo, and 39:05 that's the path to a repository. 39:08 And say, you know, ensure it's present, the provider's GIT. 39:10 And the source is the actual repository, the live repository, there. 39:13 and that will just grab the files from that repository and 39:17 sort of check them out which is pretty neat. 39:19 So I've got a section in my higher config that details each of the three demo sites, 39:23 and so one of the settings is the Git repository for the site files and 39:27 database. 39:31 So that's it there and you can see the repo right there. 39:33 So that's the YAML for one of those three sites. 39:39 You see I'm doing things like saying which add ons the site needs, 39:42 and the licence key path to the actual SQL export. 39:49 SO when I create my database and virtual host resources, 39:53 much as we did before I also create a resources perch demo deploy passing in 39:57 the array of demos and the related information. 40:02 And so here we get the files from get and each site has that database done and 40:09 I'm just doing some string replacement on it. 40:13 You can exec from this function you can just execute stuff so 40:17 whatever you like really. 40:22 It's not really the ideal way to do stuff, I'm going to try and encourage you not to 40:24 do lots of exec things, there's better ways to generally do things. 40:28 But if you just want to do a bunch of stuff locally, 40:31 that's often just kind of the path of least resistance. 40:33 And you can just exec, database import some things as well. 40:36 So I have a local file store which has my up to date copies of perch core and 40:44 the apps and I use those I add perch, the software on to the files. 40:48 I then have a ruby template for 40:52 the perch conflict which needs data which is the license key and 40:54 database conflict added and I want another create resources Puppet 40:56 function to create all of the add-ons which we need for each specific site. 41:00 And the manifest just copies those to the local file store. 41:04 So after visioning, I've got these three running sites I can log into. 41:11 I can make sure they're upgraded. 41:14 I can then go to the command line and run a script that creates the packages for 41:16 upload to the demo server. 41:20 And you just, there's a furnish called Puppet apply. 41:21 And that lets you just apply any Puppet manifest at the command line. 41:24 So that's a kind of, say, a very specific little thing that I need to do. 41:31 And I'm going all this in a Vagrant machine. 41:34 Actually, when the things go out onto the demo server, 41:36 they're also deployed by Puppet. 41:38 And so, because I know that everything is using the same templates and 41:40 the same MYSQL. 41:44 I know it's gonna work all the way around, and I can just round-trip these things. 41:45 It works surprisingly well. 41:50 And certainly better than the sort of sketchy mess of PHP scripts and 41:52 things that I was trying to do it with before. 41:56 So a lot of this stuff has been in the safety of our virtual machines, but 42:01 you might want to start taking this stuff to production. 42:05 Not the stuff we've been talking about really so far, 42:07 because a lot of that's very simple, it's not worried about security, 42:09 it's just worrying about getting local VMs up and running. 42:12 But we can start doing this stuff live. 42:15 If you look at the Puppet site, you'll find a lot of the stuff talks about people 42:20 running Puppet with a Puppet Master. 42:24 And basically, that's the server which has, or you load all your modules and 42:25 manifest to one server. 42:29 You connect other servers to it. 42:31 And then the Puppet Master basically hands out the right configuration to 42:33 the servers that are connected to it. 42:37 That's a very nice way to work if you've got a whole bunch of servers. 42:38 But it's not really the way you have to work. 42:42 It's not the way that a lot of people work. 42:44 You can also run Puppet as stand alone or Masterless. 42:47 In which case you just put the modules and manifest on each server and 42:49 you just run them. 42:53 So you can check them out of get onto the server, and just run them. 42:54 So you can run puppet from the command line with Puppet apply. 42:57 And you could also, you know set up a chron job and 43:00 just run things periodically. 43:02 And the important thing is you don't have to do 43:04 everything with puppet to start using puppet. 43:09 You can do very small things with it, things that are just annoying. 43:11 Stuff that you keep having to remember how to do so, one of the first 43:15 things that I started doing, was creating user accounts with the correct privileges. 43:20 If you often have set up servers, and you have a bunch of people who have to have 43:25 privileges on those servers, rather than having to attach in and 43:28 set all that up, you can have that defined with Puppet, and just run it. 43:31 And you know that server has got all the stuff there, their keys, and 43:34 what have you or just ready installed. 43:37 If you develop sites for 43:40 clients, and you've got like one of these VMs that you use for 43:44 a staging server, you can use Puppet to 43:48 set up the virtual hosts, and get rid of them when you don't need them anymore. 43:53 Because puppet is maintaining state if you had say a list of hosts, 43:58 you could say if they're in the file, if they're in the higher config, create them, 44:02 if they're not there, get rid of them. 44:08 And so you don't end up with things lying around on the server that you don't 44:10 need anymore and if you're going to use 44:12 Puppet to manage Apache in that sort of way, this is a really good module. 44:16 The Puppet Labs Apache module, you can do just about anything with Apache with this. 44:21 It's really, really good, and save you a whole bunch of time. 44:25 And again, it's a supported one. 44:28 So it's all checked out and you don't have to worry about it too much. 44:30 But of course all the code is just there. 44:33 You can poke around and see what it's doing. 44:35 So you can schedule regular Puppet runs to check that services are running and 44:40 restart them if not. 44:43 We have a misbehaving message cue that just sometimes stops working and 44:45 Puppet can start it up again. 44:48 Because I have got puppet hooked into a simple system for 44:50 alerts using a service thing called Scout. 44:54 If Puppet fails cause it can't restart that I get an alert. 44:59 And so I can go and figure out what on earth has happened this time. 45:03 So you can use Puppet just to check that stuff is running that you need to 45:06 have running. 45:09 You can use file resources to check that your files and folders are there and 45:12 if their permissions are correct. 45:15 So that's really useful if other people can access the server and 45:16 might change things you know to have completely open access or whatever. 45:19 So you can use Puppet to enforce the security policy 45:23 which is really useful if you need to be say, PCI compliant. 45:26 In fact, anecdotally, people are getting their audits for 45:30 PCI compliance and the auditors are just looking at the Puppet manifests, and 45:34 saying, yep, that all looks fine because they can see that everything's in code. 45:39 And I can see that that's how the server's set up. 45:44 So that's really useful if you ever have to deal with that sort of stuff. 45:45 It means you can allow people to edit configs. 45:52 If you were in a small agency, 45:54 say, and you are the one person who knows how to deal with the servers. 45:56 Rather than taking everyone's requests for new things that need setting up, 45:59 you can teach them how to edit the Hyra and 46:03 then someone can just check out those changes make sense before deploying them. 46:05 But it means that you know, everybody can actually be involved 46:09 in setting up their own stuff without needing privileges on live servers. 46:11 Cuz the configuration can always be edited it can be checked into Git. 46:15 It can be reviewed like any other codes that you have. 46:20 Often those modules are using live, can just be used locally and 46:27 that makes sure that you're environment stay exactly the same everywhere. 46:30 So that's a whole bunch of stuff. 46:35 And if you start looking at this stuff, it kind of looks fairly overwhelming. 46:37 When I started looking at this stuff it was quite overwhelming and 46:41 I've had years of systems experience. 46:44 You know I am not really administrative but I've always been the person, 46:46 that stuff has fallen on my head. 46:49 But if you have never used any of this stuff. 46:53 You know give it a go from a simple level. 46:55 If you've used configuration tools before try writing your own stuff. 46:57 If you've used puppet before have a look at Hyra it is totally game changing. 47:01 You know maybe teach other people in your team how to do some of this stuff. 47:05 And really, just improve one thing about workflows. 47:12 And it can be incredibly overwhelming. 47:17 But a lot of this stuff is actually very simple, you know, 47:19 if you're just dealing with one thing at a time. 47:22 And you can really start to chip away at things that do just eat up time with it. 47:24 And for me, as someone who runs a very small business, has an awful lot to do. 47:29 That's been really really useful, and has changed an awful lot of things. 47:33 So thanks very much of listening, and 47:37 I'm very happy to talk about this at any point. 47:39 Thank you very much. [APPLAUSE] 47:41 [MUSIC] 47:45
You need to sign up for Treehouse in order to download course files.Sign up