Using the Requests Library

[MUSIC] 0:00 Working on the web. 0:04 It seems like there's always an API that you need to integrate with for 0:04 your next project. 0:08 Maybe you want to use the New York Public Library's API, or 0:09 Marvel's superhero database. 0:12 Or maybe you just need to tie into something more mundane, 0:14 like your local bus service or Akamai. 0:17 Regardless of the web based API, the super powerful and 0:19 popular request project has you covered. 0:22 You'll need to install it with pip install requests. 0:25 Let's see how to use this excellent tool. 0:28 I think that using the request library is a pleasure. 0:30 Mostly it's because it has a pretty straightforward API. 0:33 So let's play with it a little bit. 0:36 I want to make a get request against a server, and 0:37 the server I'm going to be using is this httpbin.org, 0:41 which is an awesome, awesome service for testing out HTTP stuff. 0:46 So yeah, I'm going to use this for testing out stuff. 0:52 We're going to use this for all of our requests except for one. 0:53 So, what do we have to do? 0:57 Well I'm in the python shell, and what I need to do is, I need to import requests. 0:59 So I have it, of course we have to import to the library. 1:06 So to make a get request, we actually have to use the get method. 1:10 So we're going to say r=requests Dot get, and we're going to get 1:14 httpbin dot org slash get, which is this end point. 1:20 You see right here, it's an end point for testing get data. 1:26 Okay? Cool. 1:30 So we got it. 1:31 So what do we have? 1:31 Well, what we have Is a response object, and you can see that our 1:32 response object has a status code of 200, its in the brackets there, 200. 1:37 We can check that as an attribute as well, and we get back 200. 1:41 And we can actually do r.ok, and it comes back as true. 1:47 The cool thing here is that you don't have to check against all of the 200 status 1:51 codes, because those are all, like, good status codes. 1:54 You can just do r.ok and it says, yeah everything’s cool, everything’s fine. 1:58 All right, and then let’s look at our headers. 2:01 Let’s see what headers came out. 2:05 So there’s our headers, and it’s basically a dictionary, so 2:07 we can see all of our different headers. 2:10 Like let’s see if we can do headers.keys. 2:12 Okay, r.headers Content-Length. 2:16 We're working on the web, right? 2:23 And on the web, you encounter APIs. 2:24 And most of the APIs are going to take and give back JSON data. 2:27 So how do we deal with getting back JSON data from an API through 2:32 the requests library? 2:36 Sadly we can't get interesting json from httpbin without doing some extra work 2:38 ourselves and, you know, we don't want to do that extra work if we can help it. 2:43 So we're going to use the public GitHub API for viewing of it. 2:47 Let's hit that one up real quick. 2:52 So we're going to do requests.get 2:55 https://api.github.com/events. 2:59 Okay, cool. 3:04 So now, I have another response object that's the same as my 3:05 last response object. 3:09 It's got a status code and all that stuff. 3:11 But his one has a payload of trace on data. 3:12 So let's look at that in it's raw form, and we'll do r.txt, 3:16 and oh my god, so much text. 3:20 Right? 3:22 That's a giant, giant string. 3:23 Now, we could import the json library and convert this from a string to 3:25 python list of dictionaries, but that seems like a lot of work, doesn't it? 3:30 We just went over being lazy. 3:35 So, the PO behind the requests library want us to be as lazy as we want to be. 3:37 They don't want us to do much of extra work so 3:42 they gave us a really handy method that's called JSON. 3:45 We can do r.json, and we get all this as JSON data. 3:48 Let's make a new variable here, timeline equals r.json. 3:53 And then let's do timeline zero, and then let's get the ID key. 3:58 And there we go we've got this ID 275 so on. 4:05 If you're working with a JSON API a lot this dot JSON method is a huge time saver. 4:08 There is also a content. 4:16 If we look at that, it's the same thing, basically. 4:20 In this case, we have text. 4:23 If you're working with binary content, like images, audio, pdfs, what 4:24 you'll think of as being a graphical file, or anything that isn't just plain text. 4:28 The content comes across as bytes, not text so you 4:33 can take those bytes and turn them into your PDF or your SVG image, or whatever. 4:38 That's beyond the scope of this workshop, we'll talk about working with images 4:43 in Python in another workshop, but it's good to know that it's there. 4:46 At some point, you're going to want to post data to a server. 4:50 For example, you'll eventually need to create a new object through an API and 4:54 usually that requires a post request, and some sort of payload of data. 4:58 Like, we got the json payload earlier. 5:02 Now you need to send out a payload. 5:05 Requests makes this pretty simple, thankfully. 5:07 Let's make a payload. 5:10 Usually these are dictionaries. 5:11 We're going to say content and I really like requests. 5:13 And then we'll say user ID is number one fifty two. 5:19 All right, nothing special there, just a plain, everyday python dictionary. 5:23 So now, let's do R equals requests dot post and we want to send it somewhere. 5:29 We're going to send it to httpbin.org/post. 5:36 And we're going to say params=payload. 5:39 And when we get that back, then we can see that everything's still good. 5:46 All right. 5:55 We got all the good stuff that we want. 5:55 And what's cool is we posted it we don't see in our headers the stuff that came in. 5:58 But if we look at our and we do our .text I think. 6:05 Yeah. So, we can see here we have these args, 6:08 an insight args we have, actually let's do json. 6:13 There we go. 6:15 So we can see here, 6:16 we have args, and we got content requests, and the user ID 52. 6:19 So that was what went across, they send that data back to us. 6:24 So that's pretty nice. 6:28 Sometimes though, you need to submit data inside your payload, 6:32 where your key represents a list. 6:37 So, this is a little bit tricky, so, I wanted to point this one out specifically. 6:41 So, lets make a new payload. 6:45 Payload. Actually, you know what, 6:47 I'm going to go ahead and exit out of this. 6:48 And clear. 6:52 Get back in. 6:52 Okay, so let's do a new payload here and we're going to say posts. 6:57 We're going to mark it as being a list. 7:02 I mean sort of, kind of. 7:04 Right? 7:05 Then, we're going to give it numbers one, two, three and four, five six. 7:06 So, those are our posts that we care about. 7:11 You know what let's do this as a delete request. 7:14 So now we would do r equals requests.delete. 7:18 http://httpbin.org/delete. 7:21 And params equals payload. 7:26 And now if we do r.json and 7:28 we look for args and we've got posts one two three, 7:35 four five six, what if I do r.headers? 7:40 Yeah, that doesn't go back in headers. 7:44 So, that's great. 7:46 We're able to send our information through and 7:47 then we know that it works because it comes back to us. 7:49 Good URLs don't change, but every once in a while things can't be good URLs. 7:53 So what if your endpoint, your resource that you're posting to, getting from, 7:59 whatever, has moved? 8:03 Well, requests actually handles this pretty well out of the box. 8:06 So, let's just do a pretty basic one, 8:09 request.get http://httpbin.org 8:14 And we're going to say redirect, and we want it to redirect us three times. 8:20 All right, so now we look at the status code, and it's a 200. 8:25 So our redirects worked, or it didn't happen. 8:29 But we're going to assume that our redirects worked [LAUGH]. 8:33 So now let's look a r.history. 8:36 Look at that. 8:38 We've got three response objects because we went to three different things. 8:40 So that's great. We got redirected three times, 8:45 we told it to that's what happened and 8:47 in the end we got a 200 because that's what we wanted. 8:49 What if we didn't want to allow any redirects? 8:52 What if we wanted to stop it before the redirects happened? 8:54 Well we can do that really simply and you may be at, why would we want to do that? 8:58 Well maybe you've got all your URLs stored in a database, right? 9:02 A URL comes up as being a redirect or as being bad, you want to change that URL. 9:05 So let's go back up here and we will do this one again, but 9:11 this time we're going to say allow_redirects equals false. 9:15 So no redirects. 9:21 And if we look at r.statuscode we get a 302. 9:24 And if we look at the headers, then we can see that it's sending us to 9:27 relative redirect two, which makes sense, we're on three, we'd go to two, go to one. 9:33 Then we'd be there. 9:36 And we can see that, yeah, so the 302 doesn't come through here, but 9:38 we have the status code. 9:43 So, that's cool. 9:44 We can see where it's trying to send us to. 9:45 We can then take that location, update our database, 9:47 follow that again maybe, make sure it doesn't redirect as well, and keep going. 9:51 But that's all business logic, that's not part of this. 9:54 And the last thing I wanted to talk about with requests is doing authentication. 9:58 What if you need to authenticate before you can get to the resource? 10:02 Now, I would say this is more likely to be done when you're 10:06 trying to request a website, a webpage, and you're not trying to hit an API. 10:10 APIs generally do authentication through public and private keys. 10:15 But a webpage you'll often find it's behind, 10:19 say, http basic or http digest authentication. 10:21 Both of those authentication methods work about the same, but 10:25 http basic authentication sends your password in plain text and 10:30 http digest authentication hashes it before it sends it out. 10:34 Digest is a little safer, obviously. 10:38 But, it seems like you're encounter basic more often than you encounter Digest. 10:40 People just don't always think about security. 10:45 But, the cool thing is, we can do both of these right inside requests. 10:48 So, let's see how to do our basic one first. 10:52 From requests. 10:56 .Auth, import 10:58 HTTPBasicAuth, and 11:01 you know what, let's go ahead and import HTTPDigestAuth too, 11:07 because I know we're going to use that in a minute. 11:13 Okay. 11:18 So r = request.get("http://httpbin.org /basic-auth/user/password. 11:18 So here, if you look over at httpbin, look at basic auth, 11:26 we supply the user and the password that we want them to come up with. 11:32 And then we're going to say our auth equals HTTPBasicAuth. 11:37 And then we have to supply our user name and our password. 11:46 All right. 11:52 So let that run through and then we check out the status code. 11:53 Our status code is 200. 11:55 So our auth worked. 11:57 If we were to try that again and 11:59 we'll say like password2 and now we look at status code. 12:02 We see that it failed. 12:07 We got a 401, which is a forbidden. 12:08 So, cool, that's basic auth. 12:10 You supply your user name and your password as auth equals HTTPBasicAuth and 12:12 you go to whatever the URL is that requires basic auth. 12:17 You may be surprised to know that the digest auth is pretty similar. 12:20 So, let's try that one. 12:24 request.get('http://httpbin.org/digest-au- th/auth/user/password') and 12:26 our auth is HTTPDigestAuth, again, with user and password. 12:36 Okay. Only thing we changed here is the digest 12:45 stuff and our url changed a little bit. 12:47 All right? And r.status_code is again 200. 12:51 And again if we wanted to try this with a bad password, then we'd get a 401. 12:53 Request has support for authentication through OAuth and 13:01 people submitted extra packages for 13:04 authentication through Kerberos and NTLM and also some other ones. 13:07 I'll have a link in the teachers notes to request 13:12 docs which has much more information about all of this. 13:15 And even stuff we didn't cover the request library 13:18 is quickly going to become an indispensable part of your work flow. 13:21 If you do a lot of work with APIs or HTTP requests. 13:24 If you're going to be using for a large work flow, be sure and 13:27 check out these session object that request provides. 13:29 It will make working with cookies and 13:32 a lot of other persistent parameters a lot lot easier. 13:34 If you liked this format, be sure to let us know through email and the forum. 13:38 Also, let us know what other topics you'd like covered in future workshops. 13:41 Thanks for watching, and I'll see you next time. 13:45