Let's take our first look at the dataset we'll be using as an example in this

course.

Our dataset is a modified version of the 99 to 2000 National Health and

Nutrition Examination Survey.

The original data is from a series of studies used to determine the health and

nutrition of both adults and children in the United States.

The data is split into multiple files, and contains information from both

interviews and physical examinations from a sample of the population.

The collected data includes demographic, socio-economic, dietary,

medical, and physiological information, from over 5,000 people.

This should give us plenty of data to explore.

The researchers who conducted the study have already done some cleaning to

the dataset.

For this course, I've modified the data so that it contains examples of the different

types of problematic data we discussed previously.

Now let's take our first look at the data.

You can download the data files from the links provided in the teacher's notes.

I'll be using Jupyter Notebooks for this course.

But feel free to use whichever environment is most comfortable for

you as you follow along.

You can also download the Jupyter Notebook containing all the code

that I'll be using from the teacher's notes.

Take a moment to pause this video and download those files.

Remember, you can slow down or pause the video whenever you would like more time to

work with the code or data files.

First, we need to load our data and extra libraries into Python.

We use the pandas data analysis library for Python,

that provide some convenient tools and data structures.

If you'd like to learn about pandas,

there are learning resources available in the teachers notes for this video.

Start up your Python environment in the same folder as the data files.

Once our Python environment is set up, we can import the pandas library and

load in the data using the pandas read CSV function.

This loads the data into a table like dataframe object.

So first we'll import the pandas library.

And then we load in our three data files.

First, Demographics, Then,

BodyMeasures, And then Occupation.

You can use the describe method,

to get a summary of the data in each numeric column.

This summary will tell us how many values are present, the mean and

standard deviation, the maximum and minimum values, and the 25th, 50th, and

75th percentiles for each column.

Keep in mind that it will skip all columns that aren't numeric, such as text entries.

The head function will show you the first few rows of all columns in the data frame.

These are large tables, and

you may not be able to see all of the columns when displaying the table.

You can look at a smaller number of rows and

columns by specifying the column names using the LOC or LOC indexer.

Rows are specified first, with a single colon, meaning select all of the rows.

Colums are specified after a comma with either a colon, indicating all columns, or

a list of the column names that we want to select.

We're going to pick five columns and all of them have some weird names.

The code book for the demographics dataset will tell you what each of them means.

For example, we can see that SEQN is the column for the respondent sequence number.

This is a fancy way of saying that each person they interviewed was given

a unique ID number.

The code book also has other useful information about the data like who it was

collected for and what values are possible.

There's a link to each of the code books in the teacher's notes.

So we use the LOC method and use the colon to tell it to select all the rows.

Then we give it a list of our five columns.

SEQN for sequence number.

SDDSRVYR the data release number.

RIDSTATR interview an exam status.

RIDEXMON the six month time period in which the data was collected.

And RIAGENDR, the person's gender.

And then we use the head method so that only the first few rows are shown.

You can also specify a section of the table by row and column number using ILOC.

Remember that indexing starts at 0.

So the first column is number 0.

As with LOC indexing, rows are specified first and

columns are specified after a comma.

So let's use iloc to select the first four rows and the first five columns.

According to the code book, the sequence column uses a distinct number for

each participant.

Which is the same across all the data files.

We can combine multiple data frames using the merge function.

The first two arguments are the two data frames to be merged.

The on keyword tells the function which column values

should be matched between the two data frames.

The how argument specifies an inner, outer, left, or right join,

similar to how databases are joined.

Using inner we'll only keep entries for people who have data in both tables.

While an outer join we'd keep all rows,

even if a person is present in one dataframe and not the other.

Let's merge the demographics and body measures dataframes.

We'll join them on the sequence column and

use an inner join to keep only those examples present in both dataframes.

If you want to save your merged data frame to a CSV file,

you can use the to_csv method.

Using index=false tells pandas not to add an additional column

containing an index for each row.

Take some time to continue exploring the dataset.

Make sure you also take a look at the code books for the files we're using.

The code books will tell you what the column names mean,

what information they contain, and what values we should expect.

Keep in mind that we aren't using the original data files.

In particular, I changed many columns in the demographics file from numeric

to text entries to give you practice working with text data.

So far, we've only glance at the demographics file.

Try doing the same things with the body measures and occupation files.

Practice selecting different rows and columns from the dataset.

Try merging the three files into a single dataset.

Compare the size of the dataset when using inner and

outer joints to merge the data frames.

If you want to be able to easily rerun your code at a later date,

you can put it in a script file or Jupyter Notebook.

There is one more quiz after this video.

And then you'll have completed the first stage of this course.

Congratulations, in the next stage, we'll continue working with this dataset to

practice finding and collecting dirty data.