Back in the good old days of the Internet, if we wanted data,

we had to view it on Web pages.

Now, however, many sites provide a Web API that shares their data.

Sometimes, we can use these APIs to directly access information,

without having to scrape the data.

I'd recommend looking to see if the site you are wanting to scrape

offers an API for the information you need.

It can be a big time saver.

Let's take a look at how we can get data from the World Bank, using their API.

There are many instances when using an API is great.

Sometimes, though, scraping results from an API is useful as well,

especially if the API documentation isn't super helpful.

Let's take a brief look at one technique we can use to get and

process data from an API.

In this case, we'll look at The World Bank API.

It's actually very well documented, which provides us with some extra knowledge

as we go about trying to scrape things.

If we look here, at the Developer Information overview page,

it provides information about how to get started, and what the API provides.

Let's look here,

at the Country Queries section, to see what information we might explore there.

It looks like we could use this information to get some generic

information about the countries of the world.

For example, if we wanted to do some high-level data exploration about

income level in regions of the world, let's use this request format here,

look through some ISO codes, and get some information that we could explore.

We won't be doing any actual exploration of data in this course, but

check the teachers' notes for more information.

Let's take a look at the information we get from a country with a lot of horses,

like Ethiopia.

I know their ISO code is ETH, so let's put that into the request format.

So we can copy this, Let's create

a new tab, and we'll do ETH.

It looks like we're getting back the same information as the documentation stated,

and its in XML format.

That's great, we can handle that, we'll use Beautiful Soup to parse this XML,

and get the name, region, and income level.

This could be used, for

example, to generate a histogram chart of regions of the world and income levels.

Lots of options for data visualization, here.

Let's go back to our code, and create a new world_bank.py file.

We don't need it inside the spider.

World_bank.py, and we'll start with our imports.

So, from urlib.request import urlopen.

We're going back to Beautiful Soup, so bs4 import BeautifulSoup, and

we'll be using a csv file of ISO codes, so we 'll want to import csv as well.

Let's define a function to get the country information, get_country,

and we'll pass in our country code, And

just like we've done with Beautiful Soup in the past, we define our HTML string.

It's urlopen, And

it's that request format string that we saw just a moment ago,

worldbank.org/v2/countries/, and we'll use the string formatter,

country_code, and let's bring this down to a new line.

Next, we define our soup object.

So, we pass in our HTML, and for our parser,

since we're dealing with XML, we can use an XML parser.

Scraping XML is pretty straightforward with Beautiful Soup.

If we look at the results we got for Ethiopia, we want to get three fields,

wb:name, wb:region, and the wb:incomeLevel.

Let's go ahead and define those.

Country_name is soup.find( 'wb:name' ),

Region, ( 'wb:region' ),

and income_level, soup.find(

'wb:incomelevel' ), and it was all lowercase.

Now, let's print that information out.

Here's a good example of a time when we can use the get_text method.

Get_text, and we'll print the region,

get_text, And the income_level.

Now, we can loop through the ISO codes, and pass them to our get_country method.

So, if __name__, == '__main__':,

Let's bring that up on the screen a little bit,

I've included a file of ISO codes that we can open up and read.

So, file, country_code,

oop, country_iso_codes.csv,

want to read that.

Now, iso_codes, then, will be our reader, File, and our delimiter is ",".

Now, we can loop through our file, and get our information.

for_code in iso_codes, and we want to pass in our code into our get_country

method, And we want the first one from the list.

Now, we can run world_bank.

And it looks like I made a mistake back up here,

it wasn't all lowercase, it's actually incomeLevel.

Let's try it again, and we get all of our expected data.

Again, we could do something else here,

like saving the information to a csv file or database.

Check the teachers' notes for more resources on that.