In our last lesson, we talked about fixing or removing data from our dataset.

Now we're going to use our practice dataset for

some real examples of finding and dealing with missing data.

We'll be using the version of the files you saved at the end of the first video.

But you can also download the version I'll be using from the links in

the teacher's notes.

This also links the code I'll be using.

Go ahead and pause this video while you download any files you need and

set up your programming environment.

Ready?

Great, let's look at our missing data.

Some data may already be missing while others maybe be removed later due

issues you uncover during the data cleaning process.

In this lesson, we use both the pandas and numpy libraries.

And now we load in our data.

In this lesson, we use both the Demographics, and BodyMeasures data.

Now, let's take a look at our missing data.

Some of the first questions you might ask,

are how many elements are missing from each column?

And what percentage of the data is missing from each column?

We can get the number of good values in each column

using the count method on our data frame.

We can get the total number of rows by finding the length of the index.

Subtracting those two numbers will get us the total number of missing elements in

each row.

There are a lot of columns, but

we can use the head method to show only the first few.

We can display the amount of missing data as a percentage by dividing by the total

number of rows, then multiplying by 100.

We can also look at how much information is missing from each row.

Using the DataFrames' isnull method,

we can add up the number of missing entries in each row.

Now we'll use the len function to get the number of columns.

Dividing the number of missing elements in each row by the number of columns then

multiplying by 100 gives us the percentage of data missing from each of the rows.

As we mentioned in the previous lesson, any rows or columns with

a large number of missing information will probably need to be fixed or removed.

But remember, these are only the entrance mark as not a number.

There may be other entries we also want to remove.

Let's take a closer look at the column for students datas.

According to the code book valid answers are, in school.

On vacation from school, between grades, neither in school.

Or on vacation from school, between grades.

Refused, and don't know.

The last two responses, refused and don't Know, are not missing entries.

But in most cases,

they aren't meaningfully different from truly missing data.

We can remove those values from our dataset

if we want all of our missing data represented by not a number.

Going back to our code, let's use the same method we used before to get

the percentage of missing data in the student status column.

About 67% of the data is missing.

We can get the number of refused answers by taking a sum of the number of entries

equal to 7.

And for don't know responses, we add up the number of entries equal to 9

Now let's print out our results.

There are no refused answers, but 2 who responded that they don't know.

Let's remove these entries so

that all missing information in this column is marked as not a number.

First, find all entries in that column greater than 3 since 3 is the largest code

that contains meaningful information.

And we set all values in that column at those indices to be NumPy dot not

a number.

And we look at all the unique values in that column,

we should see not a number 1, 2, or 3.

And there we have it.

No entries for 7 or 9.

Let's re-run our code that shows how much information is missing.

We don't have to change anything so we can just copy and paste it from before.

Those two entries didn't increase percentage of missing data by much.

We can verify again that there are no more entries of 7 or 9 in that column.

We also need to remove incorrect entries where the wrong type of data has

been entered.

Let's look at the data type for the first few demographics called demo, and

body measures called BMX.

Before printing out the first few columns,

let's have Python print the name of the data file so our output is easier to read.

Then we can look at the d-types properly to see the type of each column.

We're only interested in the first few columns so

we use the head method to limit the display.

And we do the same for our body measures data.

The sequence column should exist in all files using the same number for

each participant across the files.

But in demographics it's stored as integer data.

While in body measures it's stored as an object or string type.

Pandas has a to numeric function that will convert a data frame to numeric values.

Let's use it to convert the sequence column in body measures to integer data.

We use the downcast argument to specify,

that we want our data converted to integers.

But when we run it, we get an error.

It looks like someone entered the person's name,

Jane Doe, instead of their ID number.

We had an error because Pandas doesn't know how to convert a name into a number.

The two numeric function has another argument called errors

that tells it how to handle any errors that runs into while converting the data.

If we tell it to coarse the error,

it will insert not a number anywhere that the conversion fails.

No errors this time.

Let's see what the type is now.

Here's an example of why it's always good to double check that your code is doing

what you expect.

The type still isn't correct.

It's not longer an object type, but

now it's floating point instead of the integer values we wanted.

This is because the not a number entries are considered floats and not integers.

If this were only data file, we could probably generate a new ID number for

each of the participants.

As long as the other data in those rows were still okay.

But here,

the values in the sequence column need to match across different files.

And we have no way of knowing which ID number they should have had.

We can remove all rows with invalid IDs and

then we'll be able to convert the column to integer values.

First, we use NumPy's as NaN function to get a Boolean series telling us whether

each element in the column is not a number.

We can drop those rows by selecting the inverse of the Boolean values.

And now, let's try converting the column to integers again.

There we have it, the column is integer data now.

We could also remove any data containing sensitive information, like full names,

social security numbers, or credit information.

In this server, we have marital status for everyone aged 14 and above.

But we may want to remove that information for everyone considered a minor, which for

our purposes means anyone under 18.

So first, lets locate all rows with the age under 18.

Using the count method, we can see how many entries we have for minors.

It looks like there are 1,244 entries for marital status for minors.

To get rid of these data, replace all these entries with

not a number then print out the new count for those rows.

Now there are no entries in the marital status column for anyone under age 18.

Finally, remember to save your modified data set using the to_csv method, so

you can use it in future lessons.

Great work, we've just learned a lot about missing data in our dataset.

Before moving on, I have more practice exercises for you.

The teacher's notes have a link to the practice notebook for this lesson.

This notebook has instructions for two practice exercises.

First, identify the column in demographics

with the highest percentage of missing information.

Second, remove all the pregnancy status information for minors like we did for

their marital status.

Don't forget to save your data again when you're finished.

When you're finished take a moment to relax.

We have just covered a lot of information.

There's a lot that goes into dealing with missing data.

In the next video, we'll be looking at what to do when we have the same data

recorded in our data set multiple times.