[MUSIC]

Hi, am Alyssa, a computer programmer and data analyst.

Welcome to the Cleaning and Preparing Data course.

There are a lot of resources available for learning how to analyse data.

But most assume your dataset has already been cleaned.

In this course you'll get hands on experience

learning how to clean a dataset.

Also called data cleansing, repairing, or scrubbing.

This is an, an important step to making sure your data is relevant and accurate.

Data cleaning is typically the biggest time commitment

in any data analysis project.

Often taking up as much as 80% of your time and effort.

As you go through this course, we'll explore a practice dataset, and

discuss the problems we can find during data scrubbing.

We'll talk about how to deal with these problems, and

how to decide which data to include or exclude from our dataset.

We'll use hands-on examples with a real dataset so

you can practice your new skills and get real-world experience.

By the end of this course, you should be able to prepare your own dataset for

use in a data analysis project.

You can adjust the speed of each video if you need to slow it down.

And feel free to pause a video so you can practice what you just learned.

It's also a good idea to read the teacher's notes for each video for

additional information and resources.

Remember to check the prerequisites for this course.

These videos will assume you're familiar with programming and

Python, as well as the NumPy and Panda's libraries.

If you don't know Python, NumPy or Pandas yet, check the teacher's notes for

links to courses that can teach you what you need to know.

Now that we've covered the course essentials, let's get started.

Data cleaning, preparing, or scrubbing is the term for

fixing any issues in a dataset.

So that all data is present, relevant, and correct.

You usually do this when a dataset is first collected, or

when multiple sources of data are merged into a new larger dataset.

It's also an ongoing process as you can discover new problems with the data

at any point in your analysis.

A common way of arranging data is in table format,

like you would see in a spreadsheet or database query.

In this set up each row represents and example or observation, and

each column represents a feature.

A feature is a descriptive or measurable characteristic such as Length,

Crew capacity, or Lightspeed capability of fictional vehicles.

Each examples contains all the feature values related to a single unit

such as the Millennium Falcon.

Data preparation is a very important step before trying to get any information

out of a dataset.

The results of your analysis can only be as good as the quality of the data itself.

Any mistakes in the data will cause mistakes in your results.

A common way of phrasing this idea is garbage in, garbage out.

There are many sources of data available on the Internet.

While some of these datasets have already been cleaned.

You will likely need to create your own dataset to meet the needs of your specific

problem and questions.

You may need to gather raw or unclean data from various sources.

Or you may be given a dataset that has been collected, but not cleaned.

Even if a cleaned data set is available,

it's often beneficial to look at the raw data yourself.

And understand the steps taken during the data cleaning process.

Cleaning a dataset involves many decisions on how to deal with

problematic data entries.

And you may want to deal with them differently than the person who originally

clean the dataset.

I've included resources for finding data as well as further reading on data

cleaning in the teacher's notes for this video.

The rest of this course we'll walk you through the process of scrubbing

your practice dataset.

In our next video,

we'll outline the types of problems you will find incorrect throughout the course.