Data Analysis with Python - Full Course for Beginners (Numpy, Pandas, Matplotlib, Seaborn)

Welcome to our data analysis with Python tutorial. My name is Santiago and I will be your instructor.

This is a joint initiative between Free Code Camp and remoter. In this tutorial, we'll

explore the capabilities of Python on the entire PI Data stack to perform data analysis,

we'll learn how to read data from multiple sources such as databases, CSV and Excel files,

how to clean and transform it by applying statistical functions and how to create beautiful

visualizations will show you all the important tools of the PI Data stack pandas, matplotlib,

Seabourn and many others. This tutorial is going to be useful both for Python beginners

that want to learn how to manage data with Python, and also traditional data analysts

coming from Excel tableau, etc. You learn how programming can power up your day to day

analysis. So let's get started.

Welcome to our data analysis with Python tutorial My name is Santiago and I am an instructor@remoter.com

an online Data Science Academy. This tutorial is a result of a joint effort by remoter and

Free Code Camp, and it's totally free. It includes slides, Jupyter, notebooks and coding

exercises. Let me tell you a little bit more about remoter were an online hands on Data

Science Academy. We specialize in data science, including data analysis, programming and machine

learning. We have a complete course catalog and we're adding more content every month.

If you're interested in learning data science or data analysis, check us out. As part of

this joint effort between Free Code Camp and remoter you can get a 10% discount in your

first month by using the following discount coupon. Let's quickly review the contents

of this tutorial. In the description of this video, we have included direct links to each

section, so you can jump between them. This is the first section and we are going to discuss

one is data analysis. We'll also talk about data analysis with Python and why programming

tools like Python SQL and pandas are important. In the following section will show you a real

example of data analysis using Python. So you can see the power of it will not explain

the tools in detail. It's just a quick demonstration for you to understand what this tutorial is

about. The following sections will be the ones explaining each tool in detail, there

are two more sections that I want to especially point out. The first one is section number

three Jupiter tutorial. This is not mandatory, and you can skip it if you already know how

to use Jupyter notebooks. Also the last section Python in under 10 minutes. This is just a

recap of Python. If you're coming from other languages, you might want to take this first.

If that's the case, again, you can use the links in the video description to jump straight

to it. All right now let's define what is data analysis. I think the Wikipedia article

summarizes perfectly the process of inspecting, cleansing, transforming and modeling data

with the goal of discovering useful information, you forming conclusions and support decision

making. Let's analyze this definition piece by piece. The first part of the process of

data analysis is usually tedious. It starts by gathering the data and cleaning it and

transforming it for further analysis. This is where Python and the PI Data Tools Excel.

We're going to be using pandas to read, clean and transform our data. Modeling data means

adapting real life scenarios to information systems using inferential statistics to see

if any pattern or model arise. For this we're going to be using the statistical analysis

features panelists and visualizations for matplotlib and Seabourn. Once we have processed

the data and created models out of it, we'll try to drive conclusions from it finding interesting

patterns or anomalies that might arise. The word information here is key. We're trying

to transform data into information. Our data might be a huge list of all the purchases

made in Walmart in the last year, the information will be something like pop tarts sell better

on Tuesdays. This is the final objective data analysis we need to provide evidence of our

findings, create a readable reports and dashboards and aid other departments with the information

we've gathered. Multiple actors will use your analysis, marketing sales, accounting executives,

etc. They might need to see a different view of the same information. They might all need

different reports or level of detail what tools are available today for data analysis.

We've broken these down into two main categories, our managed tools, our close products, tools

you can buy and start using right out of the box. Excel is a good example. Tableau and

luchar are probably the most popular ones for data analysis. In the other extreme, we

have what we call programming languages or we Call them open tools. These are not sold

by an individual vendor, but they are a combination of languages open source libraries and products.

Python R and Giulia are the most popular ones in this category. Let's explore the advantages

and disadvantages of them. The main advantage of close tools like Tableau or Excel is that

they are generally easy to learn. There is a company writing documentation providing

support and driving the creation of the product. The biggest disadvantage is that the scope

of the tool is limited, you can cross the boundaries of it. In contrast, using Python

and the universe of PI Data Tools gives you amazing flexibility. Do you need to read data

from a closed API using secret key authentication for example, you can do it? Do you need to

consume data directly from AWS kinases, you can do it a programming language is the most

powerful tool you can learn. Another important advantage is a general scope of a programming

language. What happens if Tableau for example, goes out of business. Or if you just get bored

from it and feel like your career is taught you need a career change? learning how to

process data, using a programming language gives you freedom? The main disadvantage of

a programming language is that it's not as simple to learn as with a tool, you need to

learn the basics of coding first, and it takes time. Why are we choosing Python to do data

analysis? Python is the best programming language to learn to code. It's simple, intuitive,

and unreadable. It includes 1000s of libraries to do virtually anything from cryptography

to IoT. Python is free and open source. That means that there are 1000s of PI's very smart

people seeing the internals of the language under libraries. from Google to Bank of America,

major institutions rely on Python every day, which means that it's very hard for it to

go away. Finally, Python has a great open source spirit. The community is amazing, the

documentation, so exhaustive, and there are a lot of free tutorials around checkout for

conferences in your area, it's very likely that there is a local group of Python developers

in your city. We couldn't be talking about data analysis without mentioning r r is also

a great programming language. We prefer Python because it's easier to get started and more

general in the libraries and tools it includes. R has a huge library of statistical functions.

And if you're in a highly technical discipline, you should check it out. Let's quickly review

the data analysis process. The process starts by getting the data where is your data coming

from? Usually it's in your own database, but it could also come from files stored in a

different format, or a web API. Once you've collected the data, you will need to clean

it. If the source of the data is your own database, then it's probably in writing shape.

If you're using more extreme sources like web scraping, then the process will be more

tedious. With your data clean, you'll now need to rearrange and reshape the data for

better analysis, transforming fields merging tables, combining data from multiple sources,

etc. The objective of this process to get the data ready for the next step. The process

of analysis involves extracting patterns from the data that is now clean and in shape. Capturing

trends or anomalies. statistical analysis will be fundamental in this process. Finally,

it's time to do something with data analysis. If this was a data science project, we could

be ready to implement machine learning models. If we focus strictly on data analysis, we'll

probably need to build reports communicate our results, and support decision making.

Let's finish by saying that in real life, this process isn't so linear, we're usually

jumping back and forth between the step and it looks more like a cycle than a straight

line. What is the difference between data analysis and data science? The boundaries

between data analysis and data science are not very clear. The main differences are that

data scientists usually have more programming and math skills, they can then apply these

skills in machine learning on ETL processes. The analysts on the other hand, have a better

communication skills creating better reports with stronger storytelling abilities. By the

way, these Weiler chart you're seeing right here is available in the notes in case you

want to check out the source code. Let's explore the Python and PI Data ecosystem, all the

tools and libraries that we will be using. The most important libraries that we will

be using are pandas for data analysis, and matplotlib and Seabourn for visualizations.

But the ecosystem is large and there are many useful libraries for specific use cases. How

do Python data analysts think if you're coming from a traditional data analysis place using

tools like Excel and Tableau you're probably used to have a constant visual reference of

your data. All these tools are point on Click. This works great for a small amount of data.

But it's less useful when the amount of records grow. It's just impossible for humans to visually

reference too much data, and the processing gets incredibly slow. In contrast, when we

work with Python, we don't have a constant visual reference of the data we're working

with. We know it's there. We know how it looks like. We know the main statistical properties

of it, but we're not constantly looking at it. These allows us to work with millions

of records incredibly fast. This also means you can move your data analysis processes

from one computer to the other, and for example, to the cloud without much overhead. And finally,

why would you like to add Python to your data analysis skills aside from the advantages

of freedom and power theories, another important reason, according to PayScale, data analysts

that no Python and SQL are better paid than the ones that don't know how to use programming

tools. So that's it. Let's get started in our following section will show you a real

world example of data analysis with Python, we want you to see right away what you will

be able to do after this tutorial.

We're gonna start this tutorial by working with a real example of data analysis and data

processing with Python, we're not going to get into the details yet, the following sections

will explain what each one of the tools does, and what is the best way to apply them combining

and the details of them. In general, this is just for you to have a quick on high level

reference of our day to day processes, data analysts, data managers, data scientist using

Python. So the first data set that we're going to use is a CSV file that has this form, you

can find it right here, under the data directory, the data we're going to be used is this, I

have just transformed it into a spreadsheet. So we can pretty much look at it from a more

visual perspective. But remember, as we said in the introduction, as data analysts are

not constantly looking at the data, right, we don't have a constant visual reference,

we are more driven by the understanding of the data right in the back of our head, and

we understand how what the data looks like, what's the shape of it. And that's what it's

conducting our analysis. So the first thing we're going to do is we're going to read it

this CSV into Python, and you can see how simple it is just one line of code gets us

the CSV read into byte, then we're going to give a quick reference. And this is what the

data frame that we have created looks like data frame is a special word is a special

data structure, we use independent tool. And again, we're going to see that in detail in

the pan this part of this tutorial. The data frame is pretty much the CSV representation,

but it has a few more enforced things like for example, each column has a strict data

type. And we will not be able to change it to tetra, it's a better way to conduct our

analysis, the shape of our data frame tells us how many rows and how many columns we have.

So you can imagine that with these amount of rows, it's not so simple to again, follow

a visual representation of it's like, it's pretty much infants crawling, in this point

100,000 rows. But the way we work is by immediately after we load our data we have we want to

find some sort of reference in the shape and the the properties of the data we're working

with. And for that we're going to do first an info to quickly understand the columns

we're working with. In this case, we have date, which is a date time field, we have

day, month year on that are just complimentary to date, we have the customer age, which is

uninjured, which makes sense right? age group, you can say it's right here. It's age group

youth, customer gender, we have an idea again, of the of the entire data set, we know the

columns we have, but we also know how large it is. And we don't care what's in between,

we will be cleaning it probably, but we don't need to actually start looking row per row,

right just with our very limited eyes, we have a better understanding of the structure

of our data in this way. And we're going one step further, we will also have a better understanding

of the statistical properties of this data frame with a describe method. For all those

numeric fields, I can have an idea

of the statistical properties of those. So for example, I know that the average age of

these data set is 35 years old. I also know that the maximum age in this case if these

Or is the sales data is 87 years old, I know the minimum is 17 years old. And again, I

can start building right if my understanding of this that physical properties of it. So

in this case, the median of my age is very close to the mean. So this is telling me,

all is telling me something, and the same thing is going to happen for each one of the

columns that we are using.

For example, we have a negative profit here, and we have very large values here are these

correct, is maybe there's a mistake, again, it's by having a quick statistical view of

our data, we're going to be driving the process of analysis without the need of constantly

looking at all the rows that we have. It's a, it's a more general holistic overview.

So we're gonna start with unit cost, let's, let's see what it looks like. And we're going

to do a describe only if you need coast, which is pretty much what we had right here. In

the previous in this line, what we did was for the entire data frame for the entire data,

in this case, we're just focusing in the unit coast, cost, sorry, column, the mean, the

median, all fields, we know already pretty much from this, and we're gonna quickly plot

them, we're going to use these tools to visualize them. And it's the same tool, it's paying

this that it's using on top, right? It's using matplotlib. So the visualization is created

with matplotlib. But we're doing it directly from pandas. And again, don't worry, this

is all explained in pandas lessons. So this is unit costs, right is what this is the box,

but we have just created, we have the whiskers that mean that shows us the the first and

third quartile, the median. And then we see all the outliers that we have right here.

So we see that our product study is around $500 is considered to be an outlier. And the

same thing if we do a density plot, right. So this is what it looks like. We're going

to draw two more charts, right, in which we're going to pretty much point out the mean and

the median, right in the distribution charts. And we're going to do a quick histogram of

the costs of our products. Moving forward, we're going to talk about age groups with

the age of a customer. And at any moment, we can always do something like sales sort

here to give a quick reference, we know that the the age of the customer is expressed in

actual years old they were but also they have been categorized with three, four, actually

four age groups, seniors, youth, young adults and adults, right. So they we have given categories

were creative, right to better understand these groups, and we do that with values.

Value counts, we can quickly get a pie chart out of it, or we could get a bar chart out

of it. As you can see, right here, we're doing an analysis of our data, we see that adults

right here are the largest group in our for our data at least. So moving forward, what

about a correlation analysis? What is a correlation between some of our properties, we will probably

have high correlation for example, between profit and unit cost, for example, or order

quantity, that's kind of expected, but that's all something that we can do right here. This

is matrix right of correlation showing in red high correlation. So order quantity, and

unit cost or where is profit right here. Profit is right here. So we see high correlation

with unit with cost with profit. Now with profit, actually, it's the opposite blue is

high correlation, I'm sorry, the diagonal, which is blue, is correlation is equals one.

So high correlation is blue. And we see that profit has huge correlate has a lot of correlation,

positive correlation with unit cost and unit price. And negative correlation is with dark

red. So we again can have a quick idea. Let's see, for example, here profit, it has negative

correlation with order quantity, which is interesting, right? It's we wouldn't dig deeper

into that, of course, the profit has a high correlation positive with revenue, right?

And again, it's just a quick correlation analysis. We can also do a quick scatterplot to analyze

the customer age and the revenue right to see if there is any, any correlation there.

Right? And the same thing for revenue and profit. This is obvious, right? We can we

can quickly draw a diagonal here, right. So there is a lot Linear depth and dependency

between these variables. So a form a few more box plots, in this case, understanding the

profit per age group, right, so we can see how the profit will be, will change depending

of the customer's age, and a few more box plots. And we're creating these these grid

of year customer age, unit costs, etc, for multiple things. So moving forward, something

that we can quickly do when we're working with Python, especially within this is Drew

shape or data or derive it from other columns, right. So this is pretty common in Excel,

we can create these revenue per age column, if you're here in Google spreadsheets, you're

going to do something like revenue, per age, and you're going to do something like equals,

right? Equals revenue, divided, I don't remember if this correct formula we're using, but just

for, for you to have a reference. And we're going to pretty much extend this whole thing.

There we go, Oh, well is processing, and I have 100,000 rows. So you can see how slow

it is, I let's compare that just to the way Python works, I'm gonna execute this thing.

It was instant, you know, extremely fast. And it was all calculated seems that we have

the same results as expected. same results as expected. And we can quickly plot both

the in a density plot and in a histogram, as you can see, right there, now that revenue

parade is going to be relevant. In any case, it's just to show you the capabilities of

what we can do. Let's annual analyze, well, we're gonna create a new column, which is

calculated cost is the total, the total orders the total, the quantity of the order, times

the cost, right, extremely simple formula, very fast process. And we're gonna get right

here, how many rows had a different value than what was provided by cost? So what we're

doing right here is like, we're quickly checking if the cost provided by the data set, at some

point doesn't align with the actual cost we are calculating. So is there any mistakes

that were made by the I don't know the original system, or people doing a data entry, if these

new column is different from cost, we want to know about that. And that doesn't happen.

So again, quick, quick, regression plot. In this case, it's very obvious that there is

some linear dependency between calculate cost and profit. So more formulas, in this case

costs part cost plus profit. So we're going to adding a little bit more, there is no difference

with the revenue and the calculated revenue that we are having. So that all makes sense,

we're going to do a quick histogram of the revenue. We can, for example, on 3%, to all

the prices that we are using, we need to increase prices. How are you going to do that? Well,

it's very simple with Python, we're just going to do increase everything by point 03. And

now all the prices have changed.

What else we're going to be able to do quick filtering, let's get all the sales from the

state of Kentucky right. So these are all the sales from the state of Kentucky, we can

get only the average of the sales by these age group on only revenue, right. So these,

all these filtering options, and extremely simple to get with Python. In this case, we

say, give me all the sales from these age group, and also from this country, right,

and we're gonna get the average revenue from these groups that we are selecting. And again,

to modify the data, we can make just a few quick modifications, like in this case, we're

going to say, all the sales from country right to revenue, we're going to increase it by

1.1. I don't know why, which is doing it arbitrarily. It's just for me to show you how it works.

So far, so good. Again, we've done a couple things, you don't need to know about the details,

we will actually go through that in the NumPy independence sections in this tutorial. So

just for you to have a quick reference of it. There are exercises associated with these

given lectures. So if you want to pause right now and get into the exercises, that's going

to be very helpful. We're going to move forward now with the second lecture in which we will

be using a database this Akila database and we're going Be erasing data, instead of from

a CSV file, as we did before, we're going to read data now from a database. Reading

data from a SQL database is as simple as it is from an Excel file or a CSV file, as we

were doing with our previous example. And once you've read the data, that's we're going

to do now the process is the same. So what we have right here is a query a SQL query,

if you don't know about SQL, you can check our courses or other courses online. Basically,

we're pulling the data from the database. This is one of the advantages of Python, it's

not, there are connectors for pretty much every database provided out there, Oracle,

Postgres, MySQL, SQL Server, etc. In this particular example, we're going to be using

MySQL. So once you construct the query, and you pull the data from the database, then

the process is the same, we have just converted these outside data into a data frame that

we can use with our Python skills. The first step, as usual, is to check the shape information

description of our data of our data frame. In this case, we want to, again understand

the structure of it. So we want to know how many rows we have 16,000, we want to know

a little bit more about our rows, we want to know about a little bit more about our

columns, and how many rows how many records we have for each one of them and the type

of each one of these columns. And we also want to have a better statistical understanding

of our data. So we do a quick describe, and we have more details about it. If we want

to focus in individual columns, right, we can just do that by in this case, we're gonna

focus in film rental rate, right, pretty much how much you pay to rent a film. Um, we're

gonna see the kind of distribution we have, we can call it distribution, it's pretty much

a categorical field in this case, but basically, the rentals are divided into three main categories

are prices, zero 99 299 499. So that's these box plot these pretty much perfect, never

seen in real life plot box plot gives you those prices. And move forward, we can also

check very quickly a categorical analysis, understanding the distribution of rentals

between cities, so we have two cities. And it's pretty much even as you can see right

here, creating new columns and reshaping the data for further analysis, etc, is relatively

simple. In this case, we're going to analyze their return in rentals, right, which, which

films are going to be more profitable for the company div, dividing the rental rate,

how much we charge, divided by the cost, how much it costs us to acquire the film. So in

this case, we can see the distribution of that, right. So most rentals are here in the

beginning. And then we have more profitable rentals, were making up to 60% above the rental.

And we can quickly analyze the mean and the median fit right to have a quick idea of all

that.

Finally, selection and indexing, if you want to start focusing, if you want to go into

data, right, you want to zoom in, you want to have a better understanding. So you start

filtering, in this case, we can filter by customer, but if you want to do it per city,

if you want to do it per state, if you want to do it per film, per price category, etc.

It's very simple to filter to filter and zooming, which is one particular characteristic of

your data. So you can perform a more detailed analysis. So in this case, we have all the

the films are rented by the customer last name, Hanson, which doesn't mean it's the

same person. But again, it's very simple to filter dot. And here, we can do we can very

quickly see which ones are the price, the film's sorry, that have the highest replacement

cost, right. So basically, what we're doing is we're going to isolate those films that

have the highest replacement cost. And also we can see right here just for you to have

an idea, all the films that are in the category PG or pG 13. It's very simple to to filter

that data. So this is the process we usually follow. we imported the data, we reshape it

somehow create columns, there is an important process of cleaning up or not highlighting

this part of the tutorial, we're going to talk about it in the tutorial itself. There's

the process of cleaning, then reshaping creating new columns, combining data and creating visualizations.

This is the process, right? We're following here with our Python skills, but it's a tone

more to odd as you might imagine, from creating reports to running machine learning processes,

creating linear regressions, etc. For now, this is just a quick understanding of the

process. We follow. Now starting now we're gonna move forward with more details of each

one of the individual tools we're going to talk about. We're going to talk about Jupyter

notebooks. We're going to talk about NumPy. We're going to talk about pandas, we're going

to talk about mapa, lib, seaborne, etc. Starting now, right? The first thing we're going to

see is, what is this whole thing that I've been using this Jupyter Notebook, I want you

to now too, if you want, if you if you don't have experience with it, I want you to have

an idea of how it works. And then we're going to move forward the individual tools, NumPy,

pandas, etc. Remember, there are exercises also associated with this particular lecture.

So you can always go back again, and work with them. Once you get more a better understanding

of the tools we are using.

Before we jump into the actual data analysis course, and we start talking about Python,

pandas, all the tools, we're going to use import files, read data from databases, etc,

I want to show you the environment that we work with. It's our primary environment, it's

the tool that we use 99% of the time on its Jupyter Notebook, there are going to be different

terms here, I'm going to be referring to it as Jupyter Notebook. But as you are going

to see in this, in this part of the of our tutorial, you can see that Jupiter is actually

a whole ecosystem of tools. And it's a very interesting project. Jupiter is a free and

open source, again, ecosystem of multiple tools. And primarily, we're gonna talk about

first, what is a Jupyter Notebook. What you're seeing right here, and you're gonna see live

in a second, I can actually show it to you is this thing we're going to use. And we are

also going to talk about Jupiter lab. Okay, which is the evolution of the regular Jupyter

Notebook. So, I think this could be familiar to you already. Usually the questions in the

question is, what's the difference between Jupyter Notebook and Jupiter lab? Well, the

difference is that Jupiter lab is just a nicer interface on top of Jupyter notebooks. It's

not just the plain notebook. This is a notebook, but I'm scrolling right now. It's also the

addition of tree view, it's an addition of get tools, as an addition of command to lead

and multiple other things. You can open some files with a nice preview in it, etc. So,

Jupiter lab Jupyter Notebook, they are similar Jupiter lab easy, again, the evolution of

a Jupyter Notebook. And that's what we're using. Again, Jupiter is a free and open source

project. So anybody can install it, anybody can download it, it's very simple to get it

set up in your local computer. In this case, we're using something we call notebooks AI,

it's a project that provides Jupiter environment for free in the cloud. So you don't need to

install things locally, you don't need to put things in sync in your own hard drive,

right you That means you don't need to buck it up, for example, because it's just a service,

it's all worked in the cloud. So said that, I want to tell you that we have compiled a

very quick list of everything, we're going to talk in this part of the tutorial, in this

list of two, it's just a thread of with multiple, multiple hints of how to use Jupyter notebooks.

So after the video after the course, if you forget some of these concepts, you can always

go back to this to it, it's a quick reference for you to have. So let's get started. Why

do we use a Jupyter Notebook? Because it's an interactive real time environment to produce

our or to to explore our data and to do our data analysis. It's a tool you're gonna fire

commands, and it will immediately respond with something back. It's a very interactive

tool, when we're working with data analysis, and this is mainly main difference with some

other tools like for example, Excel, tableau, etc, is that we are not constantly looking

at the data, there is no visual reference, like for example you have in Excel, right?

So in Excel, you're constantly looking at the data, you have it in front of you, there

are 100,000 cells and you can stroll and see them. The problem is that that's not scalable,

right? It's like nobody can work with 100,000 rows in their, in their, in their mind, we

will always forget something. So the way we work with Python indeed, analysis is by always

having a reference of how our data looks like but always at the back of our head and we're

not constantly looking at it. We're like this person from the matrix, you know, the, the

the commander of the matrix that commands people to get get in and out. We're basically

telling people telling people that basically asking data, right asking questions to the

data, and having a picture in our mind of how that's going to work, we're not constantly

looking at it, we're just having a reference, or in our in the back of our heads of what

our data looks like. So that's why this tool is very useful. This tool is useful Also,

if you're just training your Python skills, and or their permanent language skills, because

what you're gonna see is it's just a regular Python interpreter. In this case, I can execute

some code, that's two one times, actually one plus three, there we go. And the result

is four. Right. So this is a Python is a fully featured Python interpreter. The good thing

is that again, it's going to respond to us pretty much immediately I create a command

and I immediately get a response, I can do something a print here, hello world. And I

immediately get a response, I can do Hello, world, times, times three. Again, it's a again,

a Python interpreter, a fully feature Python interpreter, but it's not being accessed from

a terminal, which you can write this is the good thing about Jupiter lab to have a terminal,

I can do Python, right. And I can do two, time three, and I get an answer back. But

this is not convenient to work with our data, we need something a little bit more interactive,

we can also mix with documents, that's going to be the advantage of a Jupyter. notebook.

So what what's the way we work with Jupyter notebooks, there are a few concepts, very

important concepts that we are going to follow a Jupyter Notebook is just a sequence of multiple

cells, okay, everything is a cell. And as you can see, when I click on these cells,

even if even if it doesn't look like being a cell, it is, you will see that these blue

thing right here, right is pretty much following me because I'm clicking on the cell, and I'm

selecting that particular cell. Everything happens within a cell, if I want to execute

some code I can do, again, one plus five, and to get a result or a result back, right,

that's, that's how it works. So I'm creating a cell, I'm deleting a cell, I create another

cell again. So it's everything happens with a cell, and I'm going to tell you how to add

the cells, how to remove them how to execute code, etc. The interesting thing about a cell

is that it can either be Python code, or any other programming language you're using in

this case is a Python data analysis course. It can be Python code, as we're we were doing

before one plus three, this is Python code, or it can be what we call markdown, okay,

which is a formatting

format, right? To create text, that will be a render with sort of HTML ID at the output.

So in this case, this is what the source code of the markdown looks like in markdown, any

line that starts with this part, it's going to be a title, in this case, it's going to

be the largest, the biggest title you can have is just one pod, and then you keep adding

to reviews the size in this case, level three title. And then you can have for example,

this is a quote this is bold, this is it Alex, this is a link, right? So let me actually,

I could copy the cell and open the source code. There we go. So this is a link right

issue, issue is created or it's rendered as a link. So markdown, what is is that is a

text formatting tool, right or protocol, we could say that in this case, we just specify

us we have some

some rules to use in our in our text, and markdown knows how to interpret them and format

right or return a formatted document after them. So for example, here, we have green

divider, which is a picture and we know it's a picture because it starts with an exclamation

marks. And that's that what you're saying right here. So again, a cell can be either

Python code, or it can be markdown. markdown is an entire thing on its own. You can get

any tutorial online free, it's it's fairly simple to get started with. And it's also

very important because when you're formatting your reports, right, when you're creating

your reports, you want them to look pretty, you can use markdown for not and what we're

going to see later So you can export these notebooks and they will generate PDFs, right.

So this whole thing can be a PDF or an or an HTML page. So after you're done with your

data analysis, you can hand over to whoever asked for the analysis, a PDF report, which

is pretty neat. So moving forward, again, any cell is going to be either markdown, or

it's going to be code right here. So these ones code, and you can switch the modes, you

can say, this LS code, or actually, let's make it markdown. So right now, if it's a

code, it doesn't doesn't matter, or just, it's not executing anything, because the cell

is interpret as markdown. So now, I'm switched back to code. And now it works. Again, I said,

Sure. It can also be raw, but to be honest, we don't use raw very often. So again, you

have this this general cell type, this cell we're using, what type is it? Is it code is

it markdown, you can switch it with these with the selector right here. So a few more

things that I have to tell you right away, so you can start internalizing them, and it's

gonna take some time to get used to it. But once you get used to it, you're gonna move

very fast in your data analysis with Python Jupyter notebooks. The first thing is, as

you're seeing right here, every cell has been given an execution number. So any, the cells

will be moved, right, they will be moving around, you will be moving them around. But

you will always know which one executed before another one. And that's because every execution,

you run will be assigned an execution number. In this case, this is the seventh time I have

executed code. If I execute code again, for example, I don't know, two times two, this

is the eighth time that I've executed code. And if I move this thing, right here, if you're

reading this thing, top down, you will not be full, right? You will understand this thing.

It was moved, the cell was moved, the structure of the notebook changed. But these thing was

executed after this other cell, right? xact. And this is seven. So the execution order

is always preserved. So that's an important thing. Something else that you're seeing me

change the structure, and do things with the notebook without using any menu. And that's

because I know how to use keyboard commands keyword shortcuts to run most of these commands.

So for example, how can I add a new cell I have these is a markdown cell. This is a code

cell, if I need a cell before these one, what's what's that command that I'm going to issue

in order to create the cell, in this case, the command is going to be the letter A, I

just type A, and there is a new cell creative. How can I delete the cell, it can be two times

that the key two times the D key. And again, this is all these reference with built. So

for example, right here, whereas hit at some point, you can.

Here, you can type, you can press A to create a new cell, you can press B to create a new

cell, what we call below. So let me put something here, this is a reference. And I'm going to

put here the letter B and it's going to create a cell B below the currently selected one.

So the selection here is here in the blue, I hit let me delete this one, I hit B. And

again, it's going to create a cell below the previously selected one, if I hit a, it's

going to create a cell above that previously created one. So these, these are the mnemonics

of the creation. Something else and it's very important why when I'm in this cell, and I

hit the letter, a leader, literally it just hits the letter A in my keyword, no control,

no command, just a, it creates a new cell, and it doesn't type A inside the document,

right? So right here, if I type A, it's adding an actual a character in the cell. Why didn't

that happen before. And you're going to notice that when I change, when I'm going to call

a mode in a second, you're going to see that the content of the cell is grayed out, show

what now when I when I press on the letter A it actually creates to sell and it's not

adding content to the sell itself. If I go back again to the other mode, and I'm going

to give you a better explanation in a second. If I type anything, in this case, a it's actually

appended to the text within it. So this is my interaction to sell modes and this is very

important. The Jupyter Notebook is a mode base editor, right? So there are multiple

editors are, for example, vim or VI, vi, those are mode based editors, which basically, the

behavior of your work will change depending on the mode that it's currently activated.

So for example, in this case, I am in addition mode, because any character that I type will

be appended to the cell, A, B, C, D, etc. If I switch out of editing mode to what we're

gonna call command mode, I switch out of that mode. Now the cell is grayed out, and any

key that I hit, it's gonna do something different associated with that key. So A is going to

create a new cell above, B is going to create a new cell below, Double D is going to delete

this cell, right. So that's, that's the important part of Mo. That's one of the most important

parts in order to understand how to work with Jupyter notebooks, the mode that you're currently

working with, and there are only two modes, so it's fairly simple. This is command mode.

And we recognize command mode, because this cell is grayed out. When we get into edit

mode, there is a regular prompt, as you're saying before, the number one on the cell

is actually subjects of addition. So that's the way we can realize that, how are you going

to switch from modes, in this case, I'm in editing mode, if I'm using my mouse just pointing,

I can click outside, I'm gonna get out of the edit mode into command mode. If I point

inside and going back again, to the Edit Mode, but let me tell you something right away and

then say, we don't like to use our mouse, we don't like to point and click, because

that's very slow. We like to use our keyboard, we move very fast with our keyboard. So how

are you going to switch from, from editing mode back to command mode, that's going to

be with the Escape key to go from editing to command, edit as Escape key, it's going

to switch out of editing, but when mode. And if you actually want to make modifications

to the cell, basically, you want to get into edit mode, you're going to hit the return

key, that's going to get you into edit mode, again. So we have tackle multiple things are

writing, again, we said in Jupyter notebooks, we're going to use Python code very quickly

to interact with our data, we need a real time, you know, I'm asking a you're answering

type of editor. That's what the Jupyter Notebook is. The Jupyter Notebook has these two modes,

edit and, and command mode. And then the cells which is pretty much everything is the most

important, it's a fundamental part of the notebook, the cell is going to have two types

can be either code, or it's going to be markdown, right. And now I'm going to start showing

you more features. And I'm going to show you, I'm going to show you the most important commands.

And of course, how the what the keyboard shortcuts for those commands are, so you can move freely.

And and and work with Jupyter Notebooks in the most efficient way. So let's get started.

First of all, for for from the most important commands is moving right. So navigating, it's

very simple to navigate, just use your arrow keys, up and down, up and down. And you're

going to move around in your notebook. If you wanted to switch the type, right going

from markdown to code, etc, you can switch use these drop down or you can press the specific

key is to switch to either markdown or Python. So for my markdown, you're gonna switch sorry,

hit the M key, that's going to make it markdown. For Python, you're going to hit the Y key,

that's going to make it Python code. So M and y are going to switch you back and forth.

Keep an eye on the selector you're going to hit y m y m is going to switch it from code

to markdown.

What else how can you execute code once you are within your typing code and you want to

execute it, there are two types of executions you can run. The first one is going to keep

the selection the currently selected an active cell is going to stay the same place you are

and that's going to be my by keeping press the Ctrl key and hitting return that's going

to run decode on the cell there the prompt or the current selected cell will remain being

the same. So I'm running this thing a couple of times already on this selection or the

currently highlighted cell stays the same, I can change that by using shift return. So

I keep the shift key pressed. And I hit return and is going to execute the code. But it will

immediately switch the prompt or the currently selected cell to the following one. And that's

useful when you have multiple cells, you want to execute one after the other. So you can

keep hitting shift, return, return, return return, and it keeps you moving right from

top to bottom. Alright, so Ctrl return or shift return to change the execution is the

same is just what's going to happen with the currently selected cell. We already saw how

to create cells with the A key, we create a cell above with B key we create a cell below.

To delete a cell, you're going to hit the D key, the D key two times one after the other

very quickly, dd is going to delete these the cell. What happens if you made a mistake,

and you want to undo the previously issued commands? Well, the mnemonic here is going

to be Ctrl Z, you know the mnemonic, it's not the command, it's going to be Ctrl Z,

you only need to press the Z key, you know, you don't need Ctrl Z, and it's gonna undo

whatever you did in your previous command. Alright, so a B, D deletion, and then Z to

undo the all the commands were saying they all have a correspondence in this toolbar

or in this command palette. So for example, right here, I could run this code by pressing

these play button right here you see it, the execution is changing. There are multiple

ones and you can search them if you don't remember right here. And the neat thing about

it is that you actually have the shortcuts to issue the same command. So let's say you

don't remember how execute and stay stay in the same cell, or move whatever you can search

for run. And you can see what's the name, and what's the actual command that you have,

right there, right. So you can, at least for your first ad or a month working with Jupyter

notebooks, you will usually need to go back to these commands, and try to remember the

the quick shortcuts. And with time and practice, those will just come naturally. So moving

forward, what else, we have a few other commands, in this case, we have something to cut and

paste the cell somewhere else, just cut and paste, that's going to be x to cut it, or

you can also use the scissors here, x to cut it. And to paste it, you can use this button

or actually these buttons sorry, or you can just press the V key V is going to paste it

wherever you're currently standing it. So I'm going to cut it, I'm going to remove it

from here, and I'm going to paste it below there. Or you can also copy it. So instead

of cutting it, you can press the C key just going to cut, sorry, copy. And then you can

actually say where you want to paste it. In this case, we have duplicated the same cell.

And it looks something interesting here, the execution count remains the same. So again,

there is like this unique identifier for your executions, which means that you know, when

and where something was executed. Moving forward, we're going to use some code here, we're going

to import some tools, you can see some characteristics or advantages of Jupyter notebooks and why

we use it so often compared to, for example, the regular Python terminal.

One very important thing is visualizations, we as data analyst, we're constantly getting

data on expressing it through images, or animated animations, right. But most commonly, images.

The main library we use in Python is model live. And model lib is a first class citizen

in Jupyter notebooks, which means that you can just run the figures from matplotlib.

And they will just show up directly in your notebook without the need of doing anything.

Crazy. So can you imagine showing these these beautiful picture in this terminal? That's

that's very hard, of course. So again, that's one of the main advantages of a Jupyter Notebook.

Moving forward, what we're going to do is we're going to first we're going to get some

data from a public API. So there is these crypto watch service, which basically has

crypto information, Bitcoin, ether, etc. And you can check the docs, we can actually open

them. It's gonna give you market data Tesla. You can check the docs and How you can get

in this case it's BTC Bitcoin to euro, sexual see if we can change it to USD USD price.

There we go. So this is the current price of bitcoin results, surprise, etc. And we're

actually going to do markets do we have crack and BTC USD, let's do, let's actually issue

the same query we're going to use which is open high, low, close Oh h LC. And don't worry,

this looks ugly. But this is actually what we're using. There's a list of results write

for all different candles, we call them, we get the idea of the open price, close price,

high price and low price. So we're going to issue those, we're going to issue these requests

to the internet to these API, the crypto the crypto watch API, so you can get information

about bacon to do some analysis, I say they can, you can actually get it from ether for

for ether for author different types of crypto or currencies. So the function we're defining

is get history, get historic price, it's a very simple function that uses pandas is one

of the most important tools, we're going to be using this course. And the requests library,

which is also very famous library for Python. And what we're going to do here is we're going

to get Bitcoin on ether prize for an entire week. Right. So from ferreted that the second

February sorry, February 25, up to today, right? So depending on when I'm shooting this

video, and we're gonna get a quick reference of the prices open, high, low, close. So in

this case, we have four information per hour. Okay, so this is something you can actually

change in the in the, in the request you're making to the API, you can reuse the candles

eyes. In this case, we're keeping it per hour. So we have by the hour information about Bitcoin,

in this particular market, which is bitstamp. Here, we have these day these day, and these

are right, when I'm in the morning, open, close, highest price and lowest price, and

also the volume that was operated within this time period. And we're gonna immediately plot

the price. So we see that in these time, which I think is an entire day, we the price dropped,

it's actually a few days, like an entire week, the price dropped from $9,600 below, right

9000. So it was a pretty significant drop. Let's see ether highperformance. We have here

all the records, and how it moved. So this is what I tell you that when you're doing

data analysis with programming tool like Python rar, you're not constantly looking at the

data. So what I'm showing you right here are the first five records, we actually have.

Let's do that. We actually have 169. Records, okay, 169 Records. And this is per hour. So

if we do 169 hours divided by 24 hours, we have seven days, right? So we have seven days

of data 169 Records, and then we have a little bit more information keeps this to go. I'm

gonna get to that in a second. But basically, this is one I tell you 169 Records, to be

honest, something you could be saying in a spreadsheet. But I want you to get the concept

here. We're not just looking at our data, we have it in our brain, we know what did

it we know what shape it has. We know how many records it had, we know information standard

deviation, what's the mean of that? Right? So close price was the standard deviation,

right? What's the the average, the mean, the median, right? So we have information about

our data. It's sitting behind, you know, in our brain, but we're not looking at it. And

that's because with a very simple example, with only 169 Records, but in real life, we're

dealing with millions of records, so it's impossible to see it. Have you ever tried

is crawling in an Excel spreadsheet through millions of records. It's crazy. It's not

possible. It's just unusable. So that's again, the way we work with data analysis in Python

and R and other tools. We don't constantly keep an eye on the data. We know the shape

of it. And we just have these quick references like show me the first five records. I mean,

the last five records, show me this chunk here down there, but that's it. So again,

these are the visualizations we're creating on Jupyter notebooks. Again, it's just very

simple to get the plot done right there. We're going to also see in Jupyter notebooks, a

few other pretty neat things. The first one is that we can use another library, which

is called bokeem. And the difference is that boakye will have charts that are interactive.

So I'm moving it right here, it has JavaScript. And it's interactive, you look back again,

to what we had here. This is a static chart, it's just a PNG, you can actually export it

as a PNG, there is nothing you can do with it. With bokeem, it's actually a dynamic,

dynamically generated interactive charts. So I can, I can zoom in piece of data, right,

I can move it around, I can just do whatever I want with it. I can refresh and reset it

to whatever it was. And it's a dynamically generated chart. The difference is, if you're

working with data, dynamically in your analysis, sort of in your exploration, then boek is

a planning tool because you can zoom in, right, so what's going on here, let's, let's look

at these things. If we're working on a mean, reverting strategy, for example, we see a

high volume, we see a low volume, the mean is going to be here. So we see some mean reversion

in there. It's very interesting. If you need to, for example, export a PDF, export a huge

HTML file, then static images are going to be probably better. So that's the difference

between them. To be honest, model lib is a lot more popular than bogey, we use model

live a lot more because it's we actually have a few other tools like seaborne that make

it very easy to access and use it. What else Jupyter Notebooks work very well with some

Excel, Excel files with all the file formats csvs, XML, Excel files, etc. And that's also

the the availability of Jupiter lab. So Jupiter lab can immediately interpret and opens his

v files can open with some extensions, XLS files, XML files, JSON files has a very nice

editor and tree view for Jason. So the Jupiter lab environment combined with Python Jupyter

Notebooks will give you a good idea of Jupiter in general. So in this case, we have just

saved I'm not going to execute these you can try it out. But you can execute and run what

we have just done and export this crypto file us an Excel spreadsheet. So you can just click

on here and you can basically download it, you're going to open it and see what has

There we go.

So let me reduce the size of this thing. There we go. So you can see that we have just exported

to spread two sheets, in this case, Bitcoin on ether, right? With the data that we had

in our previous notebook, right. So that's all again, the combination of Jupiter, the

combination of Python and the combination of Jupiter lab, which are tools just work

very well together. So we're gonna keep moving forward, in this video, this tutorial, I'm

talking about more data analysis, in general, we're going to talk about Python, we're going

to do a quick review of Python. Maybe when we when I was running these commands, you

felt you felt a little bit lost what I was doing with it. So we're gonna do a quick review

of Python and all that. And of course, we're gonna get directly deep into data analysis

with pandas with some other tools, I want to tell you something before we finish this

chapter. And it's not, it's very important for you to get familiar with data analysis,

with sorry, with Jupyter notebooks, because you're going to spend a ton of time with it.

And it's a very, very valuable skill that you can get if you get proficient, comfortable

with Jupyter notebooks, you know, like creating cells, deleting cells, cutting, pasting, moving

things around, etc. For you to generate reports Jupyter notebooks are going to be excellent.

So keep an eye on it. Keep practicing, it's the only way to learn it to the to the analysis.

Keep practicing it, keep open the command Polat. So you can always want if you forgot,

how can it caught a cell? Well, there is here it is command x, right? It's gonna just tell

you upfront, keep an eye on it, keep working with it and practicing it. And once you get

familiar with Jupyter notebooks, you're going to move very, very fast. Remember, they have

these nice list of compiled commands and reference you can always access if you need extra help.

And we're going to keep moving forward now with more data analysis.

Now it's time to talk about NumPy, one of the most important libraries in the Python

ecosystem for data processing. In general, it's the one that got pretty much everything

started. And if you trace back NumPy, it, it's a very old developed library. 20 years,

maybe it's it's an extremely popular library and important library, I'm not gonna say popular.

And I'm going to explain why in just a second. But it's a very, very important library in

the Python ecosystem for data processing. NumPy is a library that will lead you it's

a numeric competing library, it's just to process numbers to calculate things with numbers.

And that's it. So NumPy has a very limited scope, we could say, and this is an on purpose,

a very simple library, when you look at it, and when you look at the API, which is very

consistent, by the way, why is NumPy so important? Well, in Python, numeric processing, and just

pure Python processing numbers is very slow. Okay, Python is not slow as itself compared

to other programming languages. But when you go down, right to very deep levels of performance,

when you are processing large amounts of data, right, and you need to squeeze, even, you

know, that tiny bite at the end of your pipeline, you need to squeeze every flow up from your

CPU, then Python is not the right tool for non Python as as a pure python programming

language. NumPy is actually solving that NumPy is a very efficient numeric processing library

that sits on top of Python, and gives you the same API as you're going to work with

with just writing Python code, as you're saying here. But low level, it's going to be using

high performance, numeric computations and, and arrays of numbers and representations,

etc. That's it. That's it for pi NumPy. It's extremely simple from from an API perspective,

but it's extremely powerful. Why did I say that? It's not so popular. But yes, it's so

important. Well, because in reality, we don't usually employ NumPy directly, you will not

see yourself using NumPy. So often, but you will be using other tools in Python, like

for example, pandas, and matplotlib. And they are all working on top of NumPy. They're all

relying on relying on NumPy for their numeric processing. So that's why NumPy is so important.

So the for, at least for this part of the tutorial NumPy. I'm going to divide it into

pieces. The first one is going to be a very detail, low level explanation of how NumPy

works, why we need to use NumPy. And what are the differences between different bite

sizes for numbers, we're going to talk about integers. But this is going to apply for decimals

and data types also. And why you need a very low level, optimize to us number. Now you

can, you can skip this part, you're going to find in the description of this tutorial,

the precise moment in time. So you can just skip and go directly to the second part, which

is when we actually start using NumPy. And I show you how to create arrays, how to make

computations, etc. So for now, we're going to divide it in two parts, we're going to

start first with the low level explanation which you can escape if you want, because

it's not going to be crucial, you can easily use NumPy. Without it. We have found that

for some of our students, it's it's important to understand the low level basics of it,

especially if you didn't have a computer science background, it can help you get you know,

raise your right your level of understanding of computers, and how to make your computations

more efficient. But don't worry if you if you don't want to go through that now it's

fine. You can skip this part and come back later or any other at any other moment. You

don't need the ease to use NumPy seriously, you don't need it. It's going to be beneficial,

but you don't absolutely lead so you can just skip and come later. So with that said, let's

actually go into into a deep

understanding and explanation of how computers store integers, numbers in memory and what

are bytes bits etc. In order to understand why NumPy is so important. We have to go back

again to the basics. What are numbers, how they are represented in computers, etc. As

you might know already a computer can only process ones and zeros bits, it can't process

numbers or just decimal numbers to be more correct, sorry, it only can process ones and

zeros. A computer is just always storing and processing ones and zeros. It's a binary machine.

Your memory is the central place around the random access memory in your computer is the

the central place where your computer is storing the data that it's actively processing, right.

So you have, for example, a hard drive, which stores long term data. But the computer can

process data directly from your hard drive. Before doing that, it has to load it into

your ram into your random access memory again, usually, right a computer is going to have

what eight gigabytes 1632 doesn't matter. Let's say you have eight gigabytes of memory,

that at some point is going to translate to number of bits that your computer can store.

So if you follow, if you follow these we have right here, you can see the total number of

bits available in a regular computer with eight gigabytes of memory. Why is this important?

Because again, the objective of these of these tutorial is the objective of this bar, at

least is to explain how you can squeeze out of every single bit you can in your computer,

right? How can you make it more efficient? For your numeric processing, both in storage?

use less memory for the same data? And also how to make it faster, right for your calculations.

So in terms of physical storage, or actually memory storage, right? How can we make it?

How can we optimize to use the least amount of memory for this given problem? That's the

objective of optimizing it, we need to understand how numbers decimals or sorry, integers into

decimal numeric system are represented in binary, right. So these table right here shows

you the first nine numbers, 01234, etc. and their binary representation. In your computer.

Let's say you want to store the age of user age of a user, which is 32. You can't store

32 in here, because your computer again doesn't know about decimals, it only knows about binary.

To do that, you will need to find the correct representation in ones and zeros of 3030.

All right, sorry, two, which is not this one, to be honest, I'm just making it up as we

go. But again, you need to know the correct binary representation of these number in norther.

To store that data, how can you know that? Well, there is this whole binary arithmetic,

right? There's a whole part of math dedicated to binary doesn't matter for now. But I'm

going to just drive the intuition of it so you can have a better understanding. And if

you're interested, you can dig deeper later. So basically, any decimal number needs to

be stored in a binary format, which of course only steaks ones and zeros. And what we usually

do is just we keep increasing zeros and ones in positions, right. So in this case, we have

the number zero, the number one, that's fine. Once we need to store the number two, winning

now to increase the number, the position right here we need to increase, right, so we need

to go from two to one zero, we'd go to the number three, it's one one, and then we need

to go to number four, we need to increase positions again, because we only have two

symbols, zero and one. So as you're seeing right here, up to these level, we need only

one position. Up to this level, we need two positions. This level, we need three positions.

And these levels going to need four positions. And you'll see how the size of each of these

is increasing. And it has a

an explanation behind it that we're going to see in a second. So the question is how

many decimal numbers you can store with n bytes and bits, sorry, bits. So let's say

we have n bits. And let's say n is equals to three. That means that you only have three

positions, right three bits, how many total decimal numbers, you can store with it? Well

we can store 000 we can store zero, we can store 100 we can start stores are you one

zero, right? So in this size, we can store up to here, we can store up to seven numbers

111 is equals to seven was, once we've filled all the positions, right, we've reached the

limit, right? The largest number, the largest binary for this amount of symbols or positions.

That's the number seven. So these means that with three numbers, you can start from zero

from zero, here, zero up to one, one. In total, you can store eight decimal numbers, here

you have eight decimal numbers 012345678, total decimal numbers from zero to seven.

The

equation if you want behind this is as follows. If you have n equals three, and it's, in order

to know how many decimal numbers you can store with those bits, it's two to the power of

n, in this case, is total a bit. So if we go back into our drawings, we said that with

three bits, we can store up to eight decimal numbers. And again, the equation is two to

the power of n is going to give you how many decimal numbers you need. You can always do

the opposite process using logarithm and get how many bits you're going to need to create

to store a given decimal number. I'm, I'm not going to get into that. So we don't complicate

it. But again, the math behind it is extremely simple. So now, moving forward, we're going

to delete this whole thing. Moving forward. Why is this important? When you're working

with your data, when you're doing your data analysis, you know what, what data you're

what type of data, you're working with their own numbers, but numbers only usually have

a connotation behind, right? So let's say that you have here it's a table of people,

and you have the total net worth of the person. And also you have the age of the person. The

age is a value that will range between what zero, right? Just born

to,

I don't know, 120, we can say I don't know, what's the maximum age registered right now,

the oldest human being but zero to 120, it seems, seems reasonable. In your other column

net worth for this person, the range is it's completely difference. We can go from something

like $0 up to, I don't know $60 billion, I think Mark Zuckerberg or Jeff Bezos or one

of those. So we go from zero to 62 billions in this case, if there are dollars, what happened

if this is a highly devaluated currency? Would we have to go to trillions, right? So these

two even though they're just plain numbers, and we can say they are integers, even though

these are pulling numbers, they have an integers, they have a different connotation, and they

will need different requirements in terms of storage size, right? So if we say that

nh goes from zero to 120, we don't need so many. So many bits to store it in memory,

right? We can do the math, actually, how many bits Do we need in order to store 120 100?

And what do we say 120. Right? Well, if you do the math, you will see that two to the

power two to the power of seven is 128. So if you have if you have seven bits here, seven

bits, you're going to store from zero, up to 1111111, which is actually 127. Okay, these

number, all ones, seven ones in binary is equals to 127. in decimal, in total, we can

store 128 numbers 00 matters, up to 127. So that means that for our column right to column,

age, here age, we only we can use the size of the men We need to use is going to be seven

bits per user, or costumer or person, whatever. What about these number right here, if we

have to go up to a couple billions? Well, in that case, the numbers a little bit more

complicated, we're going to need, for example, we can say 64, or 3232. It's actually 64,

probably, but with 32 bits, right, you can store up to from zero up to these volume.

So again, I don't know about the currency we're using or anything, so we can assume.

But here, we need 32

bits in order to store that. And now you can do the math, how many how much memory space

you need, in order to process this data? How many records Do you have, if you have only

1000 Records, that's not significant. You can use whatever, you can use 64 bits here

to store the age, and you're not going to have a problem. But what happens if you have

more what happens? What happens if you have the entire population of the earth, you have

7 billion records here 7 billion records, then every bit that you're saving in these

columns is going to be important? Because he's going to take a ton of data. And of course,

you have a ton more columns, right? What happens if you are processing trillions of records

from financial transactions, right, you want to be very, you want to be very efficient

and optimize every single bit, you can't. And that means again, selecting the correct

number of have a bit per the columns you're currently processing. So so far, so good,

again, when there's 10, that the the number in decimal that we need to store has a correspondence

with emits, right? eight bits is one byte. And the more we can optimize that, the less

memory we're going to use for our obligations. Where does NumPy come in place? Why are we

talking about data in these NumPy lessons? Well, they're right. The idea is that NumPy

is a library that will lead you has a very advanced numeric processing, in order to let

you select the number of bits you want to take for an integer. Even more, let's say

you for forget about NumPy, you want to process this thing with pure Python. So you x equals

five, for example, working with Python, you create a number, we're storing age as a five,

how many bytes? How many bits? Do you think the simple variable takes in memory? How many?

Well, in reality, even though we think it should be around, what, three, three bits,

eight, let's say to be simple, too simplistic. In reality, for Python, this is going to take

around 20 bytes. Okay, so we are wasting a ton of memory in order to store this number.

And why is that? Well, because Python is a high level, object oriented programming language.

The reasoning behind it is that Python is simple to write, write simple to also read

and, and, and code on top of it. But again, in order to create that simplicity, in its

syrup, all the numbers in objects, which have all these attributes, that if you know, advanced

Python, you're going to recognize that are not necessary. So these is taking a ton of

memory. And a regular, very simple number in Python ends up consuming 100 times more

memory than what it should be consumed. And this one NumPy comes in place in NumPy, you

can create numbers that are for example, you can control the size, in terms of bits, you

can say I want to create a number that has only eight bits. And that's it, that you're

going to create a one byte integer, and you're very precise and how much memory it takes,

you can create a number that it's actually need a little bit more space, we're going

to do NP int, and we can hear us a talkie, you're going to get auto completion 6016 bit

or eight or 32 or 64, right. So we can actually be a lot more precise in the number of bits

that we need. And this is extremely important for again, our high level processing. On top

of that, NumPy is our array processing library at NumPy is 99%, about processing a race constantly

processing erase the data structures we have in Python, the built in data structures we

have in Python, for example, the list dictionary, they are not optimized for high level computing.

So if you have a list of numbers in Python, let's say you have, I don't

know, l equals 3224, right, you have three numbers in your list. In Python there, it's

not guaranteed that the least they'll the list is gonna contain all the numbers, three

to four in contiguous positions is gonna, it might put them in separate positions in

memory. On top of that, you can't rely on advanced CPU directives and instructions for

processing matrix matrices, sorry, because Python, again, is wrapping these things in

objects. So there is no access to these high performance, low level instructions with NumPy,

that changes because when you create an array NumPy, you say, I want to create an array

of three numbers, and they are all into eight, then imposition forget about this is not these

are not bytes I am, I'm using these drawing as a general representation of memory. So

in that case, in NumPy, when you create these three element, int, eight array, it's going

to create those three elements in contiguous positions in memory, three to four, and they

are only going to take that amount of memory the police said they were going to take and

on top of that, we can rely on a bunch of very efficient low level instructions from

your CPU for matrix matrix calculation, this is something that it's a little bit more advanced.

And it's something that has exploded in the past 10 years CPUs with more with richer instruction

sets, and the same thing for GPUs, you might have heard, especially with machine learning

and all that we need, we need fast array processing, when we are storing features and weights and

all that's a topic for for different story. But again, the idea is we need right a ton

of week, sorry, we can use all these important and very efficient, low level directives from

our CPU, which makes our computations a lot faster. So again, as a recap, you don't need

to know all these to work with NumPy. That's the first thing. Second, you don't need to

get extremely, extremely conscious about all the numbers you use. At the beginning, you're

just going to use NumPy as it is, and you're going to use just the default types that it

picks in 38 cents or in 32. In 64, that's fine.

But then, with when you get into bottlenecks, when you're working with with larger amount

of with more amount of data, then you might need to get into the details of that size

of the integers that you're using. And this all applies to float. So I'm just using integers

because it's simpler. But it's all applies to floats. So again, NumPy, the main advantage

is that it's it has built in very fast and I raised kit, take advantage of CPU instructions

for matrices and arrays and all that. And it also has a very efficient representations

of numbers, right are not the regular objects of Python. Again, recap, you don't need a

list. If you want to get into more details, I recommend you to get a little bit more understanding

about binary arithmetic, and how numbers are uncomputable architecture, how numbers are

stored in memory, etc, especially for floats and all that's a completely different representations.

So with that said, we're going to see now how we actually use NumPy without worrying

so much about the low level details. And that's the beauty of NumPy. So we have already done

our low level explanation of binary arithmetic, why unknown vice important and all that if

you skipped it, that's perfectly fine, you will not need it. The reasoning was to include

was that if you're in this tutorial, you're probably looking for fast and efficient options

to process large volumes of data. And that's when all those things come into play. So let's

without further ado, let's just get started and start using NumPy as a library. So again,

as I told you, a NumPy is a very simple library for array, processing and numeric powers.

To sing, it has a few objects, numbers, floats, integer floats, arrays, and that's it. And

it's very simple, but it's extremely powerful. So, in NumPy, we're going to create these

arrays, which look a lot like Python lists, but there are going to be significant differences.

The first one is, of course, performance. If you go to the previous part, when we were

discussing the binary representation of an array of numbers, in Python and NumPy, you're

going to see the difference between them. So in this case, we're creating two arrays.

And you will see right that the creation is extremely simple. The only thing that changes

we need to add this NP dot array, and then we're passing in this case, a list of numbers.

This is something we will usually be reading from external sources. Now, how can you access

individual elements of a NumPy array is this works in the same way as with a Python list,

you can say give me the first element, give me the second element. And it's zero index,

like, again, in a Python list. Slicing works the same way. So in this case, up a zero to

something, a one up to three rights, just getting low level, right, on high level of

the index, negative indexing, and steps, they all work in the same way as with a Python

list. So if you know how to use a Python list, you will know how to use a NumPy array. There

is one new thing right here so differently from a Python list. And it's what it's called

multi indexing. Let's say you have a, an array this case B, and you need to extract three

elements out of it, you need the element of the first position, third position and last

position, you can just type B of zero, B, A to B at minus one, or, and this works, this

also works for a list. Or you can use again, mod the indexing, which is from B, I want

to select the elements in zero to n minus one first element, third element on last element,

right, so you pass an int, another list containing the indices of the elements that you want

to select. And in this case, the important part is the result. It's another NumPy array,

it's not just individual elements, you're creating another NumPy array, which again,

if you're processing, it's gonna be a lot faster. So arrays have types associated. And

this is related to what we were speaking before. As a NumPy array is a continuous is continuously

assigning memory, the NumPy library needs to know what's the type of the object you're

storing, you can't just or you know, anything, a string a number within it, because it will

not be able to

provide performance and optimizations for arrays or non consistence insights. So for

example, when we create these arrays only had injures by default NumPy selected in 64,

is because of the platform, it's a 64 bit platform, you can tune this, and you can select

us, we're going to see other sizes in a second, when we created the array B that contain decimals

or floats, it assign a different type, which is float 64. Again, the default type is always

six, at least in this platform that is 64 bits, it's going to be float 64 and integer

64. You can always change that you can say Actually, I want these, even though these

are all integers, I want you

to

create them using a float type, or as we saw in our previous video, we can say it should

be actually type integer x. So smaller integers, for performance, for performance for better

performance. Alright. So moving forward, we were also going to see a few other types like

for example, strings on the regular objects. But as you're going to see this, there is

no point of storing these things in NumPy NumPy, stores numbers date Booleans, but not

a regular individual objects as we're seeing right here. There is a way to store strings,

it's perfectly valid and it has its own time. Its own type sorry, and it's related to the

Unicode representation memory etc. But again, NumPy is usually used for numeric processing.

So the idea of NumPy arrays is we can create multi dimensional arrays we can create the

what we had created before. This is a one dimensional array right? Just one dimension,

you can create matrices, which in this case are two dimensional, we have two rows and

three columns. And NumPy has a ton of attributes and functions to work with multi dimensional

arrays. So the first thing we're going to see is the shape of an array, which is two

rows by three columns, how many dimensions it has, it has one vertical and one horizontal,

we have two dimensions. And what's the total size of the array in this case, the total

size is six, the total number of elements we have, let's go one dimension. Further,

let's create a three dimensional object, a three dimensional array, which is basically

a cube. In this case, for B, we have that the shape is two by two by three, the number

of dimensions is three, and the size is a total count of elements. 12, you always have

to be careful when you're creating these multi multi dimensional arrays. If the dimension

dimensions don't much, like in this case, right here, where we have this second list

that only has one less than bits in it, then the dimensions will not match. And it will

just tape you they'll use sorry, that the array is of type objects. And the shape is

only two only has two elements, these one element, and there's another element. So in

this case, we've done we've done it wrong, basically. And you have to be careful when

creating these these objects by hand. So how can you index and slice matrices? We've done

it for a one dimensional array. So we were selecting elements, individual elements, give

me the first element, give me the second element cetera? How can we do it with a matrix with

a matrix, what we're going to do is going to be very similar to what we did before.

The difference is that now we have to account for multiple dimensions when I do give me

a at one, is it the column add one, or is it the row at one? Well, as you can see, it's

the row. So this is going to be right here. 012. Right. And there is also another dimension,

right? So this is 012. In terms of index, index positions for our slicing. So here,

how can you get the first element, the first element of this second? rope. In that case,

you're going to first select the first row, the sorry, the second row, and then select

the first element. And that's what you get number four. But there is a better way, which

is by using the multi dimensional selection of NumPy. In this case, you're going to say

from this matrix, I want to select and here you're going to pass a in this case, you're

going to pass dimension one dimension to dimension three, dimension four, etc, right. And these

are selectors for each one of those dimensions that you're passing. In this case, we say,

for a row level one, the element, the position one second element, and for a column level,

we want the first element in it. And it's the same thing as we did before. The advantage

of this index and keeping in mind and remaining it is that it will also let you add slicing,

right, so you'd say I want to select every thing from dimension one, which is rows. So

in this case, you say from zero up to two is these two ones, the two is not included

upper limit the same as as Python. And then

you can also pass other other dimensions, you say, I want to select every row, that's

fine. But then I want to select from column level, I only want to select the elements

up to two. So these two and these two, and the two, right, so 124578. These all works

as intuitive as it gets. Remember this syntax is the important that you need to keep in

mind. Moving forward for modification, you can say I want to assign these new array to

this entire row, right? So if the dimensions match, that is going to work now 10 is equals

it's added to the second row, or you can just use what we call usually an expand operation.

We're just going to say for row number two, I want to assign the number 99 and NumPy is

going to take care of expanding it into this corresponding array, given the number of dimensions

that you have So so far that selection, it's simple, we're going to see also is that NumPy

has a huge advantage of containing a ton of operations you can perform on top of your

arrays and matrices, your multi dimensional arrays in general. So the first one is the

all the summers basic methods we have. So given an array, all these methods are already

built in the sum, the mean average, right, standard deviation, variance, etc. And that

also works for matrices. So in this case, we can get the sum the mean standard deviation,

or we can do it per axis. So this is very useful, we can get the, the here, let's compare

these two, there we go, we can get the some of these, what is this first column, the second

column or the third column, we can get it the first row, second row and the third row.

So it's either this dimension, this dimension one, or it's a vertical dimension, which is

x equals one, right? So per row per column. Or, if you have more dimensions, you can just

keep increasing the number of this answers. And that's just going to work as expected.

Broadcasting vectorized operations, this is a fundamental topic that we're going to talk

about. And it's going to be extremely related to Boolean arrays. And these are a few new

things that you have to keep in mind with working with NumPy. And now we're going to

talk about vectorized operations and broadcasting, which can be a counterintuitive topic at the

beginning, but then you're going to understand how much sense it makes. It's one of the fundamental

pieces of NumPy. We've seen how NumPy works in a very general way we saw the multi dimensional

arrays and all those advantages. But you might be thinking, I mean, I don't need another

library just to complete the summer domain. When I show you the vectorized operations

and broadcasting part, this is going to make a little bit more sense of why NumPy is so

important. So to get started, we're going to have these array, which is a right, that's

just very simple array vectorize vectorized operations are operations performed between

both arrays and arrays and arrays and scalars, like in this case right here, which are extremely

fast, they're optimized to be extremely fast. In this case, what we're going to do is we're

going to sum the entire array plus 10. And what it means we're going to see an example

of what happens without with Python.

But what it means is that let me show you the results, that each one of the elements

within the array will be applied the same operation. So usually, that's the concept

of vectorizing an operation you have the number and then this operation is applied to each

one of the elements in here are actually in these other one, right, so here and here and

here. And here to result in these new array, the operation is expressed at an array level,

right, we say a plus 10. That's it. But then again, internally, this is broadcast said

to each one of the individual elements within the array. And this gives me how a plus 10?

Well, a times 10, for example, which also in this case is we're playing the times 10

operations to each one of the elements in the array, resulting in a new array with the

result of that operation. And these resulting in a new array is very important, because

as we're going to see, NumPy is an immutable first library, it will not any operation,

you performing an array will not modify it, but it will return a new array, if we check

the status of a, you're going to see that the elements are the same, it has never changed,

we are creating a new array and returning it. There are ways to override these behavior

if you want. And this they all these operations were performing these way always have the

interface of plus equals minus equals times equals etc, which will indeed modify their

rights. In this case, we're making a broadcasting operation, adding 100 to each one of the elements

in this array. And now this operation was immutable. A was modified and did it hasn't

returned a new operation. If you remember from your pure Python skills write the correspondence

of vectorized operations are list comprehensions, in which you're expressing an operation for

each one of the elements in your collection. Right. So that's a list comprehension. It's

a it's pretty similar to what we're doing with NumPy. The main difference is that this

is all optimized and extreme. It's extremely fast. So, the operations are these vectorized

operations are reduced broadcasting doesn't need to be only between arrays and scalars

can only be between arrays and arrays. So in this case, we have a and we have B and

showing you right here. And we can do something like a plus b. And what you're saying is that

if there is a correspondence, right, so zero plus 10, one plus 10, two plus 10, right?

Let me, let me do it in this way. 110 210 and 310. There we go. And that's the result

that we get right here. So these for these to work, you of course, need the arrays to

be online and to have the same shape.

But when that does work, then the operation is extremely fast in memory. And it's aligned,

it's a vectorized operations with seen so far. Why is this topic of vectorize operations

so important? Well, because of the following, which is bull in a race. And this is a very,

very, very important thing. If you don't completely get it now, I asked you please, to go and

check the exercises we have for this lesson, because we're gonna use it a ton. And we're

gonna, we're gonna see that in pan, this, the same syntax, the same primitives of Boolean

arrays, a play apply, and we're going to use the same things. So why are Boolean arrays

similar to vectorize? operations? Well, all these operations we've had performed here

are just arithmetic operations, mathematical operations, plus something times something,

etc. If you look at the operators that you have in your programming language, it's it's

not only mathematical operators, like plus or minus, or times, you also have Boolean

operators. And the question now is going to be what happens when you apply Boolean operations,

when you apply Boolean operators to it. So given our right, we had, what ways we had

to select different numbers. For example, in this case, we need the first and last element,

we do zero and minus one. That's, that's the way we saw with NumPy. We also saw the traditional

Python one, right, so we can say a zero, and also want to get a minus one. So this is the

first, the first way of selecting these elements, we know there's a second way with multi index

selection. And there is a third way and this is new, which is with Boolean arrays right

here. So in this case, we're gonna say I want to select the elements in this order, right?

And you're gonna pass either true or false if you want to actually select the element

or not, right, so if you have four elements, you have to pass four Boolean values, saying,

I want to select this element, I don't want to select these ones. I mean, I don't want

to like this element. And I do want to select this element right here. So I want the first

one, and the last one, and the result will be the same 030303. So so far, it's it's nothing

terribly new, right? So this is new, but it's not extremely complicated. We are showing

you a brand new way of selecting data, you can select regular Python multi index, or

a Boolean array. Now, you might be thinking, well, I manually write true false false, true,

true false, for I don't know how many records you have a million records, this is not scalable,

right, you will not say to write all the strong forces. But this is actually very important,

because these arrays are the ones that are the result of broadcasting Boolean operations.

So we saw again, regular arithmetic operation like this, but we also have it for Boolean

operations. So we what happens if we ask a greater than or equals to the number two,

right, and array A is this right here is 0123, then the result is false for zero, false for

one, because they are not greater or equal to do true for number two, of course, and

two untrue for number three. So all the individual elements that match this condition will have

true and false. In other cases, this is the power of Boolean arrays, we will be able now

to combine these operations. So now we can do a greater than or equals to two, right

that a equals A being greater than or equals to the number two. The

advantage of this is just filtering, we're filtering No, no numeric arrays very quickly

with a very familiar syntax a greater than equals to and we just provide that as the

index of the operation. It's pretty much What is happening right here? We're saying use

these Boolean array. It's a willing list, right? is a Python list with Boolean, to filter

or sorry to select elements based on that. But the question is, how do we construct that

list of Boolean? Well, in this case, we have constructed it by including a predicate by

including a condition that needs to be matched. The result, again, is filtering. It's a query

method, you're looking, looking up some data, you're saying, Give me all the elements that

match this condition. So you can say, for example, these values can be of course calculated,

you can say, give me all the elements that are greater than the mean. Or you can actually

provide other Boolean appraiser operators like for example, all the elements that are

not greater than the mean. So that means they're less or equals and the mean, or you can also

include all their Boolean operators like or, or, and so or n and in NumPy, are expressed

with a pipe or an ampersand ampersand. Because we can't use just the regular or and then

in Python, we can, but it's a good choice, they've selected this. So again, this is the

concept of Boolean arrays, we are going to construct these arrays that artist Boolean

representations or Booleans, based on conditions, right, so we have this matrix, and we're gonna

say I want to select these one, and these one end is one, etc. So in that case, this

is the result right here. This is the result of that. And we can generate a dynamic Boolean

array, we never manually type all these right, we don't sit and say true, false false through

etc. We just Run Query filtering option, a Boolean operation, which results in a Boolean

array. And now we can use it as filtering. So again, the idea here is that the operations

we saw in broadcasting before, a timestamp are also defined for Boolean operators. Boolean

operators return Boolean, a race, which can be used in filtering, that's the idea of all

of it. And you can even combine these operations, you can say, A equals zero, or a equals one,

a less or equal to two. And it's also divisible by zero, you can combine all these queries.

So now it looks a lot more powerful than when we were doing before. So moving forward, talking

about linear algebra very quickly. And this is we're approaching the end of the NumPy

lesson. The part the important part of of linear algebra is that NumPy already contains

all the most important operations for it already optimized with low level semantics, it's going

to be extremely fast, adult product cross products, and all that transposing majors

is all that works as expected. And again, these might be very important, specially,

for example, machine learning, etc. It's it's extremely important. And finally, to wrap

up what we saw in our, in our binary explanation at the beginning, what you might have escaped

is the difference in sizes between NumPy and Python, the differences in terms of performance

between them. So in Python, a regular number, this is just a regular engine in Python, that

total size is 28 bytes in order and just let this thing for a second. The total number

of bytes, not bits bytes that you need in Python to store a simple number, as the number

one is 28 unit 28 bytes to store just the number one is extremely,

super space consuming, right? It's not very efficient, larger numbers will even take more

bytes to store them. What's the size of the integers? Well, we've seen it we have, for

example, we can create integers with eight bytes. We can create integers with one byte

right which were something like here we have np.int eight will already know how many bytes

has only one byte, right, but you can have control of how many bytes or bits write your

numbers will take. And you can see here the difference between the size of an integer

in Python which is extremely large 28 byte on NumPy and also the difference in performance.

Let's say for example, we want here you have the ultimate difference in size of lists,

which is also significant. But I want to focus on performance, we have two elements two,

we have one list that has the first 1000 numbers, I will have a NumPy array that has the first

1000 numbers, we're going to perform the same operation in both of them. Let's use the Python

one. First, we're going to do the Python one first. In this case, we're, we're squaring

all the elements in the list, okay, the elements A squared, and then we're summing all the

operations might so we express it at saying, create a new list, x times x, sorry, squared,

4x, nl, and then some everything, how much time it takes 321 microseconds, we're gonna

do the same thing with NumPy, we're gonna say NP dot sum, a square. And you're gonna

see that it's a lot faster in the NumPy perspective, then the Python perspective. And these are

all very, very tiny, tiny operations with small numbers. What happens if we add more

numbers, let's add two more numbers here. That's odd. Two more numbers here. And we're

going to do the same two operations. So as you see here, that that the units have even

changed, we're still in the microsecond layer here with NumPy, we've gone to the millisecond

layer in Python. So as the size of your objects increase, NumPy will prove to be extremely

fast compared to Python. So there are a few other functions you can see here, for example,

extracting normal, random numbers, etc. I'm going to live let these for you to look, if

you're interested in them, I remember you have the exercises, which can help you solidify

all the concepts we discussed. And we're going to move forward now to work with pandas, we're

going to see also visualizations are gonna keep moving forward this data analysis with

Python tutorial.

Now, it's finally time to talk about pandas is the most important library that we use

for data analysis in our day to day basis with Python. It's a library that will aid

in the entire process of your data analysis project, you're going to start getting the

data, step one, getting the data from multiple sources, like databases, Excel files, CSV,

files, etc. That's all gonna get into pandas, you're going to be processing the data, right?

So you're going to be combining merging, doing different types of analysis, you're going

to be visualizing the data, right, so a bar chart, you're going to be visualizing the

data with pandas, and you're going to be creating reports, you're going to be also doing simple

statistical analysis, you're going to be doing machine learning close to it, with the help

of other libraries, but everything from the platform that provides the pandas library,

it's, again, one of the most important libraries in in in the data analysis data science ecosystem

with Python. pandas has recently released the version 1.0. So we are talking about a

very mature library. It's been around for a long time now. And again, it's the primary

library that we use in Python for data analysis and data science. So I'm going to do a quick

introduction to the data structures of pandas house, and we're gonna understand how they

work. So you can start building right the phone, we're gonna start building the foundations,

I need you to be very familiar with the way the data structures from pandas are processed.

And then we're going to move into other things like reading files, grouping data, etc. So

to get things started, we're going to talk about the first data structure to pandas house,

which is this series. In reality, pandas has two main data structures that it uses all

the time, and it's the series under the data frame. The data frame is the one you will

probably be more familiar with. It looks just like an Excel table. But we're gonna start

first with a series. Okay, so just stay with me here. We're going to talk about a series

for a second. In this case, we have important pandas, and we have also imported NumPy. As,

as you might imagine, as I told you before, in the NumPy part of this tutorial, we're

saying NumPy is fundamental for data analysis because every other library pandas, matplotlib,

they all sit on top of NumPy and you can see it right here. We're gonna be using some features

from NumPy within this lesson, too. So these is a series in pandas, what you see right

here, it's The concept of a series is this ordered sequence of elements right? Or indexed

right with they are all indexed by a given index, of course. And you might think that

this looks a lot like a Python list, right? So in this case, we're storing the population

of countries, right in millions of inhabitants. In this case, it's jevelin. g7. pub is because

we're getting the population of the Group of Seven, you can console the Wikipedia page.

But basically, we are storing population in here in this series. And again, it looks a

lot like a list, but we're gonna find a ton of differences in here. So the first one is

that the the series has an associated data type. And this is something we saw in NumPy,

when a NumPy array couldn't hold different types of objects, we were all we were only

having one type of object. In this case, it's float 64. So all the numbers of the series

will be of type float 64, the underlying data structure, the 10, this is using to store

these objects is a NumPy array. So a second difference we see very quickly is that zeros

can have a name, right. So now when we display the series, we see that it has a name. Now

it might not make a ton of sense. But once this series is part of a data frame in the

form of a column, then the name is going to make a lot more sense. So moving forward,

again, we saw that A has a type. And again, this is because the backed the data is backed

by a NumPy array that you can always consult, you can check values of a series. And you're

going to get the array that it's backing up that pandas series, right, so you can see

that it's a NumPy array.

Once you have these series, we were just consulting here, design pop, you can in you can select

elements as you good in a regular list, right? So for example, give me the first element,

give me the second element, the last element, etc. And that's because a series inherently

has an index, similar to list a list when you create a list in Python, right? So if

I create L equals a, b, and see, but there is something wrong here missing, quote, this

list, we don't say it right. But the idea is that there is an index here, zero, this

is one, and this is two, right? In the pendous series, this is a lot more explicit, each

element has an associated value within it. And you might think that is pretty much the

same thing. They're all they're both the list on the series, there are both sequences, they're

ordered sequences of elements. But we're going to see that there is a fundamental difference,

and is that we can arbitrarily change the index of a series. So by default, when we

created it, we didn't assign any indices. So by default, it was a range index from zero

up to n minus one elements. But you can actually arbitrarily again, say, what is the index

of your series. And in this case, these data structure these series has now these indices

that we're seeing right here. Why is this important? Because now we're going to be referring

to these values, not by a sequential position, but by a name, but by a label by the index,

which has a meaningful name for us humans. Okay. So now, these thing looks a little bit

more like a dictionary we could say, than a list, we started thinking that a series

was similar to list but now, we can think that a series is limit similar to a dictionary.

But wait, don't get me wrong here. The series has a fundamental trait, and it's that it's

still ordered something that didn't happen with. With dictionaries, dictionaries in Python,

are not ordered, actually, in python 3.7. They're ordered, but we shouldn't be thinking

that they are ordered their unordered data structures. In this case, a series is in the

order. So it has both those advantages. It's ordered candidates always before friends,

that's as we decided to create it, but also it has names or labels or keys associated

with the values as a dictionary. So this is creating the series from scratch. Right? All

these methods, you can see you can create a series bypassing the index it doesn't have

To be a two step process where you first created the series, and then add the index, in this

case, you can do everything at once. And the indexing is now going to be done by those

indices, right. So those labels that make up the index will be used to index specific

data. So g7 pop, we see has these countries with these population. And now, before the

index, we were saying, I want to get what's the population of Canada, and then we had

to remember, what was the position of Canada, oh, it's the first help countries, we have

to do g7, pop zero. With the index, now we can just consult what's the population of

Canada, what's the population of Japan. And as you can see, the syntax is the same as

with a Python dictionary, it's just pretty much same, you pass the key and is going to

get the value. So again, summary, the advantage of a series is it's it's a ordered sequence

of elements, backed by a NumPy array, very efficient very fast. But it also has

an index that can take any labels we pass, so it's going to make it a lot better for

indexing, you can steal when you have a series, you can still get the elements by the sequential

ordering. After all, it's a sequential data structure, and doesn't matter if you have

in an index, you can still say, Hey, I know we have on the index. But if you want to get

the last element, or the first element or the second element, you're going to do that

by using the attributes, I look at it and say to this series from this series, I'm going

to ilok locate by sequential position, these element the element in position zero or the

last element. And that still works as expected series also support multiple indices as we

saw with NumPy. So in this case, we can get two elements out of two, three n elements,

you can pass multiple indices. And the same thing happens with more with sequential multi

index series also support range or selection or slices. But there is a fundamental difference

here, this is very important here attention, there's a fundamental difference with Python,

and it's not in Python, the upper limit of a slice is not returned. So from our list

that we created before, if I do l, up to number two, I don't get the index See, right, so

this is zero. This is one, this is two, two is not included in our pandas series, the

upper limit is indeed included. So if when you asked from Canada up to Italy, Italy is

in the result. Okay, so this is something to consider when using index selection in

pandas, I think this is still valid, it's very, I understand the reasoning behind it's

just different from Python. So, you should remember, Boolean arrays, which was a topic

we discussed in our previous lesson of NumPy. Boolean arrays is still a thing in pandas,

the difference is we instead of saying Boolean arrays, we should say Boolean series right,

the idea is that we will be able to perform operations on top of series. So for example,

right here we have mathematical operations on top of series in this case, we have the

zero D seven pop, which as I told you the beginning is in millions of inhabitants. If

we want to get the series of interest units, we will need to do Jessamine pop times 1 million

and there we go now is in terms of units these operations right these vectorized operations

the bras these broadcasting operations can also be performed with Boolean operands. So

instead of a multiplication, a summation and subtraction, etc. We can add we can use a

Boolean operators. So in this case, we get asked

what

are the countries that have more than 770 million inhabitants we will receive receive

their assault is a bull in aerates, Nebraska, right? Well, let's hear it you know, but basically,

it's the same concept of with us with a NumPy Boolean array. Canada, friends, they do not

have more than 70 million inhabitants in Germany does have seven more than 70 million inhabitants

here. 80 on the same for Japan, so Japan Here is the same on the same for the US, the US

also have past more than 70 million inhabitants. So again, the Boolean array or Boolean series

in this case, works in the same way, as with NumPy. And selection also applies. So I can

now select, I can say, give me from these series g7 pop, all the countries that have

more than 70 million inhabitants, the value is more than 70. So now, again, we are building

filtering, we're building a query language if you want on top of pandas, we're selecting

data based on this condition. Remember, when if you ever have trouble remember all these,

the idea is that you can always track down the way this index is being built. In this

case, we are it's not that the selection knows anything, these first election knows anything

about how to select countries with more than 70 these operation was performed first, which

resulted in these series. And now this series will be indexed by these array, this Boolean

array. And the result is as you can see it, and again, these operations can be run with

calculator methods, and all the operators we saw in our previous lesson, which was not,

which was or this irregular pipe, or, and amberson, which is the and all these can be

applied in any order you want. So if we read this thing, which is complicated in purpose,

it's worth saying give me all the elements that are above the mean, minus two standard

deviations or below the mean, actually, above the mean, and here was below the mean, or

if this isn't correct, but it doesn't matter. It's just an OR operation between two ends

of the it's actually, it's above the mean, minus the standard deviation. So we are applying

this operation or right, that operation we have before so they're not the or, and the

and they all work with Boolean selection as well. The operations we saw from a mathematical

perspective mean in in statistical operations, we saw a NumPy. Some mean, average standard

deviation, we're actually using standard deviation before, they're all still relevant in this

case, but also you can use traditional NumPy functions with our pandas series, because

again, a panda's series is internally backed by a NumPy array.

So this is all the same, as you can see, here is an example that it's a little bit more

clear, we're getting all the countries that have more than 80 million inhabitants, and

all the countries have less than 200 million inhabitants. So it has to be above 80. But

it also has to be below 200. Okay, or in this case, we say either above 80, or below 40,

or below 40. Right. So that's with the OR operator or the NOT operator. Modifying series

is relatively simple. Whenever you have a value, you can just assign it all together.

In this case, we're saying Canada is now 40.5. I don't know why we just wanted to do it.

This is by index, you can also do it by sequential positions. So in this case, we're going to

say the last country should have 500 now. So we're going to see a right here, the last

country has 500 now, or you can also modify elements based now bool and selection. So

you can say all the countries that have less than 70 million inhabitants, all these from

our previous query, all these will now be 99.9. So as you can see, it has changed all

these countries. So this the assignment works by direct indexing, or also works by Boolean

indexing. And this is going to be extremely important when we are cleaning data. So let's

move forward and start talking about data frames now before you have exercises in for

series, and also for data frames, so I recommend you to check them out. So talking about data

frames, this is what a data frame is going to look like. It's pretty much the same thing.

us an Excel table. So this was our series and this is going to be our data frame. It's

a table. So it looks a lot like an Excel spreadsheet. And actually, it's very common to create pandas

data frames out of CSV files, which are tables basically. And I'm going to create it we created

with these data frame object I created. There you go, these are data frame. And as you can

see, right, it has columns that we have assigned. In this case, we were designing the columns,

and we have rows of values right below each one of these columns. Why? What's the similarity

with with series, and it's not a data frame column will be basically a series. So we can

think a data frame is a combination of multiple series one per column, we're going to assign

an index to the data frame the same way that we did with our series. So in this case, this

is our data frame. Sorry, right here. This is our data frame that has the index, right?

And it has the columns as we had before, what columns Do we have, what's the index of the

data frame, these are all attributes that you can consult, there are a couple of very

interesting methods from data frames that we use all the time. The first one is the

info method. That's going to give you quick quick information about the structure of your

data frame. Right. So it's going to tell you what columns you have population GDP surface

area, HDI continent. And it's also going to tell you the types and how many no values

you have, it's actually telling you how many non null values you have. But we use these

when we're cleaning data to quickly then define those columns and have missing values, we

can check for the size of the data frame, we can check for the shape. And this is similar

to a matrix right, a two dimensional array in NumPy is pretty much a data frame. And

also similar to info the voice again, to check a summary of the structure of the data frame,

we can also use this cribe, which is going to give you a summary of the statistics of

the data frame. And in this case, what we see is that for each numeric column, only

those columns are numeric continent is not here, for example, this is continent so you

can see the type is object is a string, basically, all the numeric columns, we're going to have

summary statistics for them. So for example, for population, how many elements we have,

what's the mean, right? What's the average Romney, what's the standard deviation, the

minimum, the maximum, and in between a couple of percentiles 25th 50th and 75th percentiles.

So this is quick summary statistics. And we do this a lot. So keep in mind, his crime

method is very popular.

As you could see, in the in the info method, the columns have associated types, okay, so

this is very important. They continent is an object that means that it's basically a

string HDI is a float and surface area is an integer. And that's because NumPy, pandas

is automatically with through NumPy, is automatically recognizing the correct type to assign to

each one of the columns. This is similar to what we saw with a series in which the series

contain natural datatype, a series was part of a given data type. So that's something

you cannot change. And in this case, checking value counts, you can have a quick reference

of the types of your series. So moving forward, how will we we will be selecting data from

series Well, there are a couple of methods. And this might be a little bit confusing.

So what I'm going to do is I'm going to skip and just going to give you a quick reference

first, and then you can read if you want through the process we follow here, given a data frame,

and this is just two quick rules, given a data frame, you're going to select by index

using the lock attributes. So the lock attribute is will let you select individual rows. So

for example, when I get Canada and that's the value of Canada, when I lock attribute

will let you select similar to the series, the row by sequential position. So let's say

We want to select the last row. In this case, it's the United States of America. So again,

look lets you select a select rows by route by index, give me the row under this index,

I log will let you select rows by sequential position, give me the last row, the first

row, the second row, etc. And finally, without using lock without using a lock, just by saying

the f up something, you are selecting that column, give me the entire give me a V and

tire column population right here, the entire column population. So what you're seeing here,

first, first of all, this is a quick reference dot dot Lok will give you an element by index,

I look we'll give you an element by position, I wrote by position and just doing df, on

some things gonna give you the element, the column sorry that you are passing. So it's

like, both look on I look, look and I look work in a horizontal ladder, give me this,

while bf art, whatever works in in a vertical montanus, which is getting you a given row.

But something more interesting here is that all the results, these one and these one and

these one, they're all series, what are being returned our series. So that's what we saw

before. And the way it works is first, if we focus in this last example, we're going

to see that it's pretty standard, just these series right here was is a one return I remember

it has a type and everything. So that's, that's fine. If If we ask for a row, like in this

case, we can get for example, here easily. There you go. The result is also series. But

what you can see here is that this thing is kind of transposed in a way dot here was the

volume of this year is population is here, and GDP is here and surface area is here HDI

on continent on here you have volleys. So it's it's again, it's it's being transposed,

right from vertical to horizontal, in our regular series manner on the index of this

series is extracted as the name

that the column hot. So in this case, the name right there is the value of the index

that it had. So you can read more about it right here. But I just want you to remember

these rules don't lock you select by index dot I lock you select by sequential position,

the F at something you go by column, there are times when these might not apply. So or

not want to apply, there will be some issues. So for example, if your rows if your index

is numeric, you might have issues with these form or dot form, just respecting these three.

For now, it's gonna get you any element you want to get either by row or column. So from

what we've seen, the oldest slicing also works as expected. So we can get, for example easily,

or we can get friends up too easily. So the upper limit is included. But again, it's built

look and we select by indices from France to Italy, we can also do the second dimension

similar to the way we worked with NumPy, we can do second dimension here. And we can get

all the countries that are from France, or to Italy, including Italy, but only the population

column or population and GDP. So here you can see the second dimension being applied

at the concept of of multiple dimensions in selection being applied also to famous for

ilok. It works in the same way that in that then multi index and the slicing. So we get

for example, from one to three right in sequential positions. In this case, the upper limit is

not included. So that's another difference from what we have. And we can also do multi

dimensions we can say give me the countries from one to three and the column should be

0123 should be the third column, the fourth column, the column under index three which

is HDI, so that also works as expected. And again, recommended, always use Look, I like

to select rows and just use the naked data frame to select columns as we saw before.

Now moving forward, conditional selection Boolean arrays will series, whatever you want

to call it. This also works for data frames. And it's very important, it's a way to filter

data, it's a way for us to consult the data when the when, when it so in this case, what

we have is, we want to select all those countries, which the population is greater than 70. Okay,

so all the countries that have more than 70 million habitants, similar to what we were

we did with a series, but in this case, we want to do it with a data frame. So what you're

going to see here is that we're going to construct a Boolean series as we did in our previous

video, right? So every country with more than 70, false false, true false. And we're going

to inject that result, that Boolean series in a dot lock selection, give me all the countries

which match here than that the true value in it. And remember, just this is kind of

mnemonics are a way to remember, the way pandas knows how to filter things is by matching

this index, right from the resulting series. With these index of the resulting data frame.

These are two different objects, they are completely different objects, but their index

much. So here, Japan, March, Germany March, so here, Germany, on Japan, they are the same,

and that's why that thing is working us expect that they This is just the first dimension,

which is give me these rows, you can also on the second dimension, saying give me these

column, or these columns, right. So that's steel, that's the awards us desire. So what

about dropping stuff, you can say, whenever you have data from you can say give me just

these pieces, or you can say drop the others, right, it's just pretty much the same. Dropping

is very simple, you can drop by index, drop this value drop Canada altogether, period,

or drop these indices can in Japan, or you can also drop columns, drop population, and

HDI as columns. These ways also have a more advanced usage, which is with access similar

to NumPy. I don't recommend them so much, but you can still use them and see them here.

So all the operations we've seen, so far, they're all working. The most important part

here is the broadcasting operation that we're going to do between series. So we're going

to create a new series crisis. And I'm gonna

show you what it looks like. So we have here crisis. And we're going to perform a broadcasting

operation between between these, I'm going to show you what this thing looks like first,

between that two, these data frame on the crisis. And the result will be that we will

subtract, I don't know what's this number 1 million, subtract 1 million for each volume

in here. And we're gonna subtract 0.3 HDI for each one of those. So what you can see

here is again, this alignment between columns and indices, the GDP here is matched with

these GDP and the HDI is much with these HDI. So there are two different objects, two independent

objects, these series and these data frame here. But when we combine them with an operation

like this, the the columns in this case are aligned GDP, and HDI and they work together.

So you're gonna subtract these value in all these column, let me remove this, you can

subtract these values in all this column for all these values, I'm going to subtract these

value here in these column for all these values. That's the way it's going to work. So moving

forward, what about modifying data frames? Now I wanna I want to show you something.

And that's when we were dropping stuff before. We were not actually modifying the data frame.

So here we did df dot drop Canada, but df still has Canada in it. And that's because

similar to what happened with NumPy these operations are all immutable. They are not

changing the underlying data frame. We are storing. We are storing we're creating new

data frames that store the result of the given operation. So in this case, you have to drop

Canada. The result is that the these new data frame but the underlying That iframe is not

changed. That's because again, they are immutable operations. 99.9 operations in pandas, it

are immutable, there are ways to change it, there are ways to make the changes permanent.

But for now, I want you just to think that everything is immutable. Whenever you want

to perform an operation, it's going to create a new series. If you want to keep track of

this, you will just need to do something like df two equals that, or even df equals, you

know, just to modify the current data frame. Again, there will be a way to not do that.

But we're going to save in a sec. So modifying series more explicitly, that affrontare modifying

data frame more explicitly, how can you create a new column? Well, very simple. Assign a

column, I said, let's say in this, this column right here, it says similar to say, here,

language. Oh, it's just read only. But if I say language equals, and I can just write

whatever I want. In this case, what we've done is that the language, let me show you

what Lynx had, in this case, was a tiny series, it didn't have elements for all the indices

in the data frames, but that doesn't matter. pandas will match all the indices of our chill

exist. And it will live like the rest. This na n is what we use for a blank. It's another

number from NumPy. We're going to talk more about it when we start doing cleaning data.

Data cleaning, sorry. So again, links France, Germany, Italy, you can see the volleys are

all up there. What happens if you want to change a value the language series already

exist, you want to change it or column or read exist, you want to change it. So in this

case, we're going to say df language equals English. So we're going to change it all together,

df now will be affected, and all the values of language will be English. How can you relate

How can you realize when there is an operation that is changing the underlying data from

the underlying series or than the line NumPy array, it's usually when you have an equal

symbol, remember, NumPy, we saw something plus equals, in this case, whenever you have

a plus and equals symbol is you're modifying the underlying data frame.

So for example, check this out, the Rename function or method of a data frame will let

you pass columns and indices to rename. So in this case, we want to change the United

States to USA, the EU, United Kingdom to UK and Argentina to AR, Argentina doesn't exist

in this data frame. But that doesn't cause a problem. And that's why we want to show

you, the US, UK were modified correctly, and HDI was modified correctly. And a PC which

doesn't exist, didn't cause any problems. Now, why am I showing you this because remember,

these operations are immutable. If I check what's the state of the data frame, we see

that the original data frame has not been changed HDI a steel HDI, it doesn't matter

if we renamed it before, it's still the same data from the same thing for days, indices,

all these operations are immutable. A few more examples of modifying data just for you

to look at. And something that is very common for us is creating columns that are combinations

of other columns. So again, this is read only, but you can you can imagine, that I could

do is hear something like for example, GDP per capita, right? If I go here, and I do

GDP per capita, GDP, p per capita, per capita, and here I say is equals to the GDP, this

column divided by this column, right? So I do something like B, B three, actually, C

three, C three, divided by b three, right. And then we would extend the values all the

way along here. In pen this, we could do something very similar. We can do just any column, we

can just perform operations, broadcasting operations between them, in this case, GDP

by population. And we can assign that series which is a result right there. So it's a series

we are going to assign that series to a new column. So GDP per capita Now, there you go

is now a column of our data for. Again, all these broadcasting operations are extremely

fast, they are backed by their NumPy array, and they result in a series. So very quick

statistical information, a few methods, right to do summary statistics. We saw them with

this crime method. But minimum maximums mean, median, all that works as expected. Something

that I want you to note here, if possible, is that with pandas, we have, I'm going to

change colors here, we're going to use red. With pandas, you have this concept of a data

frame, right data frame that has multiple columns, multiple rows. And these operations

are resulting operations are resulting in just one series. So in pandas, you have your

data frame, and you have your series. And we could say we have individual numbers. And

it's like always, the data frame is always resorting back to this, it's like some operations

will just return a series. And the series can be used in a data frame, right. So in

this case, these resulted in a series, but then we merely use the series to set the value

of a column. Right. So that's why understanding series is so important. So there are a few

more assignment exercises for you here. So you can check them out and complete them if

it's going to make a little bit more sense once you're working with it.

Finally, I want to give you a very quick introduction to reading the external data on plotting.

And to do that, we're going to use a few methods that are very popular in there, maybe we can

look them up very quickly here, we can say read CSV, use the read CSV function from pandas.

So these function, read CSV. And as we have read CSV, we actually have a few others read

sequel, read Excel, read XML, there are multiple adjacent or multiple ones, read HTML will

be able to automatically parse an HTML page and read it. So a few functions like these

like, what we're going to do with these read CSV, right here is the structure of it. A

few of these functions will let us import data from an external source into our pain

this workflow. So in this case, what we're going to read is these BTC market prize volumes,

so it's right here, if I open the CSV, this is what it looks like. It's the date of the

price taken a read and devalue the bread, the timestamp, and the value the timestamp

of the value no decide the price of bitcoin 2017. Now it's close to $9,000, I think. But

just note inside, but again, this is a CSV, and this is a CSV that we're going to be writing.

To do that, again, we're going to use this method read CSV, the method will automatically

parse the CSV, as expected. And there you go. And the process now will be for us to

start tuning it to get to the right point. So I'm going to show you a few customization

SP customizations, we can do with the receipt, read CSV function. So the first one, and sorry,

let me tell you first, we have a ton of attributes here. So we have a ton of customization to

do with read CSV, you will not remember all this, you will not remember everything out

of the top of your head. So don't worry, you can always go back again to the documentation

and just practice, it's going to come naturally. So the first thing, the first row of the CSV

was considered to be the column names. So in this case, this fine lesson have a column

name, let's say I add it, I'm going to do timestamp, timestamp price, you're going to

save it, I'm going to rearrange the file and re re read it. There you go. So by default,

pandas is assuming that the first line of the CSV is the rd columns. I'm going to go

back into what it was. Right, and I'm gonna show you again, that's the assumption that

pandas is doing. We're gonna Of course, of course, change that assumption, because in

this case, our CSV file does not have column names. So we're going to just say Heather

equals none. And this is when we start seeing the attributes that we're going to use from

the read CSV function, read CSV. When I do hether equals none for us going to be known.

That means don't infer don't read a header. Don't try to infer a header, a header from

the CSV file. And the columns are zero and one. So now I'm going to change the columns.

And I say, actually to be time something prize. And now what I'm going to do is show you the

first rows. So you're saying here that I have these df dot head method that I'm doing. That's

because this is a significantly large file. So we're going to say not not that long, but

at least it doesn't fit in my screen. What's the shape of the day CSV or the data frame?

It has 365 rows, and we have two columns. So we can do df the info, for example, to

have a little bit more reference about we have 365 values, there are no no values, and

price is actually float, that Tamsin is an object and we're going to fix that in a second.

I'm sorry, that the F that head on the F dot tail, are the methods we used to get either

the first and files or the end row sorry, or the last n rows, which are five rows, by

default, you can change that and say, Show me the last three rows, for example, that's

something you can do. And again, the types so the types is the timestamp in this case,

the timestamp column was not properly parsed as a date, he was parsed as an object as a

string, which we don't want. So we're going to use the function PD dot today time, something

we're gonna explore in more detail in the reading in the cleaning data cleaning course.

Part sorry, if it weren't tutorial, we're gonna use it today time function to turn these

column D f, the timestamp into an actual date. And now we're going to say, the F that timestamp

equals to this function resulting, and now everything looks as expected, there is one

more change that we want to do, we want to set the index of the data frame to be the

timestamp, because by doing so, we can quickly access price information led me see what was

the price of bitcoin in 2000 1709 29. And I make a mistake here, I forgot to do the

LLC. There you go. So we have the value of Bitcoin. On these particular date, forgot,

look, remember that to get value from a particular row, you have to do dot lock. There we go.

So we are getting Dodd's particular value. Because we've made a timestamp the index,

we get artists value directly from the index. So what happens if you want to turn this thing

into an automated script, for example, when I run this process, every day at 5am, whatever

we can, we want to read the CSV, strip the columns, rename them turn into timestamps,

etc. This is what we've done so far. Read the CSV without a header, create the columns,

turn it into a daytime timestamp into a daytime and assign it to the index. And that's the

result again, well, actually, the read CSV, oh, sorry, the read CSV method is so powerful

that it will let us do all these actions in just one call of the read CSV method, we there

are parameters that will let you customize the behavior to achieve the same results that

we did with four lines of code right here. So in this case, we're gonna say, read this

CSV, don't assign a header, that's something we do already or don't don't infer our header

from the first line. These are the column names. So we don't need an extra line, we

can just say these are the columns names. Oh, and by the way, the first column is going

to be the index of the data frame. Oh, and also part of the date. They've the index,

it's a date, so part of the date, and we have the same result as before. So now I'm going

to pro try and same thing. There we go. So you can see it's work. So very quickly pan

this plotting. Alright, so we're going to be doing here is I want to show you very quickly,

I don't know what's this thing is as a vertical scrolling. I want to show you very quickly

that you can create plots with Hannah's interest a breeze. It's so simple to create a block.

So in this case, what we're going to be doing is, given a data frame, you can always invoke

the plot method. And the plot method, what it's doing, it's using the map plot live library,

something that you can check if you want in the docs. But for now, it's not necessary

with these, we're going to be more than enough. What it's doing is just using, again, the

regular plug library, as you can see dimopoulos Library, which is part of the standard PI

Data stack. And again, for us to access using pandas is extremely simple, just df dot plot,

you're done, you can set the plot as you want, we're gonna see more details of matplotlib.

So don't worry too much about that later. So there is a more challenging example here

that I can just run very quickly, you can inspect the process we follow to fix the data.

But this is what we have, there we go.

And what you can see right here is the difference between the Bitcoin and ether in this period

of time right here, and they are both loaded in the same chart. And that's because this

is the resulting data frame, we have Bitcoin on one side, and we have ether on the other

side on we are plotting it right here, we're creating one plot with all of it. And we are

noticing these empty value right here. So what we can do is we can go from December

1 up to January the first these period, so we can select that period, is in that lock.

And we can just go ahead and plot it again. And this is what you see right here, the gap

that we're seeing. So again, this was the introduction to pindus. We have a real life

example of pandas following up. Also we have a little bit of data, more data cleaning on

reading all the interesting files and sources of data for in getting more data into the

pipeline, right. So the idea is going to be showing you how you can import data from Excel

from SQL and then do the actual processing and analysis.

Now it's time to talk about data cleaning, we have arrived to that point in our tutorial,

in which we have pulled the data, I've shown you how to manipulate it with pandas, the

beginning at least the introduction to data manipulation with pandas, and now it's time

to properly fix it. For the sake of brevity, we are skipping a few parts of the process

of data cleaning, especially you're going to find it in this first notebook that we

talked about basics, conceptual, missing data with Python with NumPy. And we're going to

miss a few other things. But I'm just going to mention them. pretty generic, pretty general

form. And then you can of course dig deeper, you can check our courses if you want to know

more about it. Usually when we talk about data cleaning, where it's in from a more conceptual

level, we're going to talk about a four step process. The first step is usually finding

missing data, which is the simplest problem to identify from a data set when something

is missing. So you have car sales data. And there is a car that has no name right? Or

there is a card has no price, right? So there is an number missing or there is a category

missing and there's a string missing. And of course, each one of those is going to have

a different meaning how to solve how to fix data set that is missing data, it can be very

simple. If you can just for example, drop the record, if you can fill the value, right.

So for example, the prices fill in these missing, you can fill it with the average value of

the sales data or something like that. Or it can be very complicated if the volume is

important if you can't move forward until you actually find that missing volume. And

it can involve something like picking up the phone calling your ETL team asking what's

going on that the data is missing. Or even if you're buying the data, you have to call

the vendor, ask them why their ID if you've you're paying for that and there is data mentioning

etc. So it can be a very political process. It depends what's your use case. But again,

from a technical perspective, identifying missing data and fixing it is going to be

extremely simple. Once you have fixed the missing values, then you start looking for

the data is assuming the data is not clean yet in this process of data cleaning. The

second step is when there are invalid values. So you have for example,

column that is price and there is a string within it right here. We're expecting only

numbers and there are strings in it. So then It's not going to be complicated to identify,

it's not going to be too complicated to fix it. But again, we're increasing the complexity

until a deeann of these data cleaning process, we're gonna reach problems that have to do

with the domain of the day you're looking right. So for example, you have a column that

is customer age, and there is a value that is 170. Right? So that is not an invalid value,

it's a perfectly valid integer. The problem is that given the domain, right, but speaking

about customer age, is highly unlikely that a customer is 170 years old, right? So in

that case, the vowel is completely valid, there is no missing data, there is no invalid

values, etc, is just about the domain. And this is when things get very complicated,

because in this case, that example of age is something that resonates with all of us,

we know about age of humans. But if you're working in a domain, if you're working as

a data analyst, in a domain that you don't know much about, right, then you might not

be able to judge if a value is invalid or not. If I am working in a biology lab, and

I have something like white cells count per milliliter of blood, I don't know what's what

it's a good value, or what's an invalid value, right. So it's, it's something you need to

know the domain. So that's usually the the most complicated part of data cleaning, when

you reach the limit of everything is valid, everything checks out. And now I need to make

sure that these value is valid for these domain that we're working. So again, this is the

spectrum that we're going to be revisiting today. So to get things started, the way pin

this works with no values is is it has four functions, which actually there are synonyms,

it's going to be it's going to be relatively simple, just trust me on that. There are a

few things first, everything that pandas does in the process of missing values, is related

to the way NumPy works. So again, we're skipping it, you can go to that notebook, check it

out by yourself. But it's extremely simple. NumPy has these objects and n not a number

to identify a missing value or no value in Python world to have the non value. But again,

in pandas and NumPy, we're going to use na n none on there, or in this case, at the beginning,

we have these two functions is no n is na, which are complete synonyms, we're going to

find also is no and no we have it isn't a and they're also complete synonyms. So no

n na for pan, this is the same. You can use the one you prefer. Sadly, I like is na because

it's the way I learned it. I think for my students I usually recommend is no, because

it feels more correct. And it feels more self explanatory. So you can use the one you prefer,

if you can use is no, I think that's going to be better. If you get used to ease in a

then you're going to be on my side, just do whatever you prefer. So again, it's no one's

gonna say true or false, depending if the value is no or none, right? And of course

not No, it's going to be or not na is going to be the opposite. So not na have not a number

is false, and not an A of three is true. If you get to this first notebook, you're going

to set all the false e values on the true fi values in detail in terms of Python, anything

that is not empty or non etc is going to be considered to be truthy. So anything you pass

here that again, is not an empty string or a no is going to be considered a true fi value.

So it's no not no or is it a and none an A, they both work also with entire series or

entire data frames, right? So it's not just for one of Valley you can pass an entire series.

And the result back is going to be if the series is if the series what values in this

series are either no or not no, depending what's the question you're asking either is

null or not null. So in this case, we say which one is of the series are no, this is

not, no this is not No, this is no so this is only true. And the opposite for the following

method we are applying are actually function. And again the same thing works with not

entire entire data frame. So something we do usually is if you look in to not know unknown,

a few hacks that we usually apply are the count on actually this be the sum of all the

no values or not no value. So we have this entire series, we can say how many not null

values we have. And if we sum those, not no values. In this case, we're going to get a

result out which is the entire the entire summary Have the nod no bounds we have asked,

and the same thing is gonna happen if we say is no. So if I do here is no, some, we're

gonna get how many novels we have? And it's pretty much the opposite of this question

is no. And the way it works is in Python bullions are pretty much integers, they're ones and

zeros. So every true Val is going to count as one and every four is going to count as

zero. So if you ask for the sum of a Boolean series, you're going to get out the result

of the number of truths that are available in that series right. So, in this case, we

have to know values we ask how many knows value we have is know that some we get two

out, you can use these tricks to filter the data with a series. So in this case, we can

say give me all the values that are not known. Right? Just not know. Also, something interesting

is that both for data frames are for series. The not not no is no isn't a not an A methods

also, sorry, functions also work as methods. So in this case, we can say instead of PV

dot know, we can say s.is, no load s, that is no. So now, it gets a little bit more,

a little bit simpler. But if the final objective of these core as equals alzarri, s selecting

only the boundaries are no no, was to drop the null values, then there is a simpler form,

which is dropping, okay, so in this case, we can say s dot drop in a, and we're basically

invoking the same thing that is happening here, we're missing we're just excluding sorry,

all the missing values in the series or the data frame, because this also works for data

frames. So what's the one, one important thing to remember here is that all these methods

are immutable, we are not actually changing or modifying the original series, the underlying

series is not being modified, there is a new series that is returned. So if I invoke s,

again, this thing has is not modifying their series, you're creating a new series, and

that's the one that hasn't, that doesn't have the missing values. Everything we've said

also works for data frames. So right here, with these on a frame, we can say how many,

right? The first thing usually is to start with an info method, right? So we have info,

and we see that there are in total, four entries, four rows, we can also do a shape, if we need

more information about the structure of our data frame. So there are four rows, four entries

in our index, column A has only two no no values. So that means there are two values

that are actually no no, sorry, no, there is column B that has three nought non null

values. So that means that one value must be known, and that's for column B, again,

so usually info gets you very close to understand the structure of your data frame and how many

values there are missing. The same thing happens with some, we can just do df.is, null isn't

a and then some, we're gonna get a quick reference of how many null values we have in that given

data frame. Drop in a works in the same way, but there is a significant difference. The

way drop in a works in a data frame by default is by dropping any row that has at least one,

no value. So these row has no value dropped, these row has no value dropped, these row

has two new values dropped, this is the only one that it's not being dropped, right. So

it's very harsh in that respect, you can change that to make it to the column only, only keep

the column that has no no values, and that's by switching the axis equals to one. And there

is also a way to select a subset or thresholds. So only delete rows that have less than three

valid values. For example, in that case, you're going to use something like the strategy of

the drop in a you're gonna say, drop the columns, the rows, sorry, are the columns because it

is also works for columns that have all the values and no, or drop. The This is the default

behavior, drop all the rows that have any value in an NA or specify a threshold, which

you mean by basically saying, I need this amount of valid values in order to keep the

rope it's the way it works. Now, which ones to drop is which wants to keep based on the

fresco. So once you have identified it No values, it's extremely simple to clean them

to sorry, fix them. So the first method we're going to see is fill in a within a particular

value, we're going to say from this series, I want you to fill the blanks or fill the

missing values with or fill the anaise. fill them with numbers zero in this case. So these

two are numbers zero, or, of course, you can use any statistical method you want. In this

case, we can use the main. Remember, this is not altering the series, the original series

is still the same, we're not changing it, it's creating new series because all these

methods are immutable.

The following method is or this the following way This method works is by passing a method

which is for field or backward fields, these are the possibilities. And basically the way

it works is it's overflowing, all the values top down, at least in Fairfield, right starting

here, it's dropping this value here, dropping this volley here. And dropping now three here,

as this thing is a nun, it gets replaced. So this thing is three now, which gets throw

up here. And now this thing is three again. So that's what we have right there. And of

course, backward fields works in the other way, starts with four and moves, it moves

it here and then moves here, etc. You have to be careful when using these. Because if

you have no no values at the beginning or the end, then you're gonna end up again, with

no values because there is nothing to fifth forward, right, this is the first volley you

have India. And all we've seen also works for Donna friend. So both boggler fail for

field or both in terms of rows for feeling, right, so we have, we have these, these data

sets. So we do for field row base is going to be one to here too. And then five. So that's

going to be for field x is one, if you use for field x zero, then it's a vertical filling,

right? It's going to go here, one 130 30. So that's for the column, that is y here,

one 130 30. So it's either for filling in, in, sorry, this direction for failing, or

it's going to be in this direction, depending on the axes that you are passing. And actually,

let me we're going to put the correct forms with axes equals zero, it's going to be columns,

it's going to be visit direction with axes equals one, it's going to be row based. So

it's this direction, right? So we had a no volley here, that got fail in this way. Okay,

moving forward, we what else we have, we have here, checking for values. And we've pretty

much seen this already, you can use the is know, the sum method to get how many values

you have. And there is also any an old, which will give you very quick. These are usually

called Boolean tests, you can say ask if there are any values are valid, or all the values

are valid is just to build more complicated queries. So so far, so good. So the process

we said was at the beginning, we were fixing missing data, missing values, there is nothing

in there. We have read a data frame, where's our data frame right here? We have read our

data frame from CSV from a database, and the value is missing. No, there is a hole in it.

So we have quickly identified it with isn't a or is no, we were able to drop the ones

we didn't want to keep dropping a or we were able to fill the volume we wanted to fill

fill a name that was simple isn't a drop in a fill in a what happens when you're cleaning

data that actually has a value, so there is no nothing missing.

But those warnings are invalid. So for example, here, the sex column is a categorical column

that only accepts an on f. d on question mark, those are invalid they are, it's very simple

to see an invalid value here because it's completely out of the scope. The same thing

as we have, for example, question mark in the age column where we have we have a string

in the age column, it's very simple to identify that, how we're going to clean those. Let's

start with sex first, because it's simpler in this case, the first check we can do is

with either unique or with volley counts, I'm going to use value counts. We've seen

this method before. It's a quick summary of all the unique values you have. And in this

case, volley counts also gives you a total count for those values. How can you fix them?

Well, there is a replace method which is extremely intuitive. You can just replace in this case,

we're changing all of these two F's and The End two M's, and it can work in multiple columns.

For those volleys, that again, we said were more complicated to fix, like, in this case,

we know age, in this case, is 290. And we know because we know the domain, that 290

as an invalid age for a human. So we will need usually in those cases, we're going to

need more complicated fixing, and it will involve more programming, that's the reality,

you have to be better coding. In this case, we know that these volley is invalid, because

it's probably an extra zero. So all these values, you're pulling a CSV with ages, and

there are a total of 180 290 32 320, for example, invalid values out of 100, right in the 100

places. And that's because there were typos when they were creating the ages. So how are

you going to fix that? Well, in this case, it involves a little bit more programming,

we're dividing everything by 10.

So also, something that may be useful is dealing with duplicates. And we need to first define

what's going to be a duplicate value. So this is, this is usually a little bit more political,

if you want, you have to define what's going to be a duplicate. In this case, we have a

series that contains ambassadors, and each, their master is the index, the country of

the ambassador is going to be the value, right? This is usually the important part. The rating

here says the word conducting a party, and we want to invite one Ambassador per country,

we don't want to repeat ambassadors, ambassadors. So in this case, what's going to happen is

that these two in our humanize at least, we can click clearly and quickly see that these

two belong to the same country. And these three belong to the same country. But here

again, we have to define which ones are the duplicate, if you want, and which ones are

not duplicate. So for example, maybe we can say the first one is duplicate, or we can

say the last one is duplicate. So this is the first one not duplicate, or actually can

say this, the last one is one, and when I bite, it's not to duplicate. So we're going

to have political rules if you want for each one of those. So let's see the duplicated

method and the way it works by default. By default, duplicated method is going to return

true for duplicate for all the it's it, I'm going to invert it, it's going to not treat

it as a duplicate as the first instance that it says. So the method is actually walking

top down right now saying, Do I have friends? No, I don't have friends. I'm going to keep

it here. Because it's the first time I see friends. Do I have the UK? No, I don't have

the UK, it's just gonna keep it here. Then it sees the UK again realizes the UK is already

there, too. It's already present. So this one is going to be considered a duplicate.

Italy is here, it's fine. The first occurrence of Germany, it's fine wrightstown, Germany,

but then it says Germany two more times. And it realizes that Germany was there. So those

are now duplicates, right. So the way it works by default, we can change that and change

it to last to the last element is not considered to be duplicate, and the other two are considered

to be duplicate. And the same thing here. Kim, here is the one consider duplicate. So

it's either top down or bottom up depending the way the parameter you're passing, it's

either keep default or keep last, or you can be a little bit more harsh on say everything

duplicate it is actually to be needs to be considered duplicate. So these two are duplicates,

and these three are all duplicates, as you can see, right there. Similar to the duplicated

method, which pretty much tells you which values are duplicated, it's it helps you identify

them, you also have the drop duplicates, and in this case, what this method is going to

do is basically the same thing as before, but dropping all the values are checked for

true, right if the method is if the value is missing, it's gonna just drop it. And the

same rules apply default, last and false. For subsets in this case, we have Ace, we

have multiple, we have multiple players in the data frame. But what happens is that these

player Colby is present three times for humanize we see Kobe three times. What is going to

happen here is that the The way we're going to think about duplicates is by understanding

the correct subset that we should check. In this case, Coby plain as sn SG is duplicated

two times but COBie, playing us in SF could be considered a different player if you want,

because maybe it's a different season, or it's a different, a different position they

played. So in that case, we need to pass What's this subset that we are going to consider

duplicate, only check for the column name, or check for the column name on or not check

for the column name, which is the default is going to check the entire data frame. And

when that happens, then these two are considered to be duplicate. So these one is a duplicate

with this rule, if we put keep last, sorry, keep false, both are going to be considered

duplicate. So this second occurrence is the duplicate one. And the last one is a completely

different row, because the the value in position is different. That's the way it works here.

Moving forward with more cleaning of values, we're going to talk about string handling.

And this is a very neat feature of panelists, that special types of columns will have special

attributes. So given the column type, so df info, which is an object, which is a string,

right, in pandas, that all the strings columns are going to have these special attribute

which is str, all the daytime columns, something we're not going to cover, but you need to

know, all the daytime columns have a.dt, Math attribute, all the categorical columns don't

have a.ca t cat attributes. And those attributes, str DT cart, they have a special methods associated

the domain of that column. All the methods associated with string are of course, we're

string handling, or the methods associated with DT r for data handling. So in this case,

we're going to review all not all very good subset of the string methods we can apply.

And something interesting is that all these methods have a very good have a lot of relevance.

And they're related to the ones in pure Python. So if you have a pure Python string, there's

a split method. There is a contains method or I don't know if there was a contain actual,

it's actually, I think it's the in operator, but there is a strip, and there is a replace,

right, so most of the methods under the str attribute in pandas have,

have an analogy in the standard library of string handling with Python. So starting at

the beginning, this data we have, I'm going to delete this this data we have, what we

are going to do is split the values right by an underscore. So in this case, that's

what we have, we have split all the volleys with that underscore, and we're going to use

the special attribute is expand, expand sorry, equals true. And what it's going to do, it's

going to create a data frame out of that. So we create a data frame with 70 columns.

And this is what we have now. So we can keep applying methods. So for example, contains

or content contains, regular or contains with regular expressions rights for you to see

the power of it, we can just strip replace, and we can do even regular expressions with

replacing so we could fix something like this question mark in a string, we could fix it

with regular expressions if you know how to handle them. And finally, something that is

going to be very helpful when you're doing data cleaning, is looking at the data from

a visualization perspective. data cleaning has a ton to do with statistical understanding

of your data to when a volume is considered an outlier. For example, it might be invalid,

and you want to claim it. So but that's a lot more about statistics. And this case,

I want to show you very quickly, the mottled leave library, I've been promising for some

some time now, the mapa lib library. So far, we've accessed it directly from pandas, from

pandas, or we're doing a data frame dot plot. It's these library mapper lib is the one backing

all those methods and we're going to see how to use it directly. Now. The model live library

has two important API's we're gonna call him one is the one that I don't prefer, which

is the global API, but it's the most common one. It's the one you're gonna find around

the global API. And the second one is the object oriented API. So it's around here.

And usually there are there are ways it's just two different ways of doing the same

thing. Okay. The global API is an API that it's in part inspired in MATLAB. It's been

around for a long time on sadly Most of the answers you find in Stack Overflow tutorials

and books will be using these global API. The way the word the one I prefer the most.

And I'm gonna explain you why in a second. It's going to be the object oriented API.

But I want to show you both. So you have a reference. If you follow me in this feeling

of preferring the object oriented API, you will always have to translate global to Opie.

Why is it considered a global API? Well, we have imported matplotlib.pi plot as PLT. So

we haven't imported the whole module, the whole Python module, depending how much you

know about Python programming is going to make sense or not. We have important the whole

module. And now what we're doing is we're invoking PLT dot figure. And finally, and

then we're going to do a title. And then finally, we're planning two things. We're plotting

x, our plotting x squared and minus x squared. And why is this global because we're invoking

functions that are at module level. And there is an object, the final plot, that it's being

modified by these very generalistic and global courts, right. So by by doing these call right

here, I'm modifying the final result of the plot. Let me show you a more complicated example.

So you see the problems with the global API.

If you look at these line, if you could delete everything, let's actually delete everything.

What is this line doing which plot is affecting, you do not know, there is no object oriented

way of saying in this second plot the plot on the right or the figure on the right, or

actually the sub plot on the right, I want you to plot this thing, you're just saying

it to the entire module. And depending the order that you set it, is where it's going

to land, that particular figure where it's going to land in which plot, it's going to

lend. Again, it's a global API. So we start saying, I'm going to create a figure, trust

me from So from now on, I'm going to start drawing on it, there's going to be the title.

And hey, by the way, it's going to have one row, it's going to have two columns. And I'm

gonna start drawing in the first plot these one right here, these one right here on the

left, okay. So now I have kind of activated if you want that plot, it's active. So now

I'm going to start drawing on it. So every action that happens after this line is going

to be affecting these blocks, these blocks, right. So then I plot x and x square, I plot

this vertical line, I put a legend, I set labels, etc. And at some point, I just stop

and say, Hey, now I want to switch the plot, I want to now start plotting. Sorry, I want

to start plotting here in this second one, because I have just changed that the first

line these one. Oh, sorry, the way it works is by saying the first row, second column,

but second plot. So now I want to start plotting in here, every successive line will affect

that line. And again, you can see that understanding a code, given the order that the order in

the sequence of lines is very hard. If you have to debug a report that has a plot that

takes 100 lines, then you have to keep in your brain, what's happening top down, a different

approach is going to be the object oriented approach, in which we're creating a figure.

And we're creating axes. So in this case, we have in this case, we have right here,

one entire figure in red. And we have in here, purple, we have two axes. So these axes one,

and this is access to so we have two axes. We're going to create those using an object

oriented approach. And we're going to keep references to them. So we're going to say

later, to these blocks to these artists, sorry, I want to plot something. And that will be

very explicit, it's going to be an object oriented way. So the first thing is creating

the figure on DCE. The axis in this case, we have just one axis, that's it, but you

can have more and then you say in this axis, I want to plug this thing in this axis, I

want to pull up that thing, etc. When you have multiple axes, so I could show you. I'm

going to go back again to that in a second. But In this case in which we have four axes,

right, so we create one figure. And it has four axes, we do it with this subplots, method

saying and rows and columns. Now we say to the axes number one, I want to put this thing

to axis number two, I don't want to put that thing, right. So it's 1234. And now it's a

lot more explicit, it's not depending on the order, I could change this order, that doesn't

matter.

They're that the results are gonna be the same oxes number four has yellow, regardless

of the position that we're following. So the map will live. And now that we have clear

out the differences in both API's, maple leaf has this very simple plot function, or method,

depending on sugar enter global, that we'll plot something you specify. In this case,

we're passing all the values in x and all the values in y. And in this case, we're passing

a given line style, this can change with these type of syntax, you're saying, I'm plotting

this thing in X, I'm blowing this thing in y second parameter and why. And I want you

to use a straight line, it's a straight line, yes, with this marker, the dot and in green.

So this is if you are very familiar with it. If you're very familiar with my bullet you

can use to send links in other games, you can just say line style market marker, sorry,

color specific keyword arguments for each one of those. So do we only have line plots

in APA live? No, of course not. We have a huge variety of plots. And by the way, there

is another one here, if you want to see more events are grids, you can create these grids

and put different things in it. And again, not only land plots, one good example is a

nice scatterplot. So basically, we're plotting X and Y correlation. And there is also our

value, our color map, right. So given the volume, there is going to be a change in color.

So these kind of lets you plot three to four dimensions of your data, the volume x, the

volume, y, the size of the bubble, and the color of the bubble. So where you're pretty

much encoding four dimensions in just one figure, right. So in this case, we're just

using two different scatter plots, there's more information here, we can also block histograms,

that we've very quickly seen that with pandas with pandas is, is very simple with just plot

type histogram, current histogram hist, actually, you can look it up in our previous lessons.

So just go back into the index in the video. And the histogram is extremely simple just

takes the valleys you're plotting and how many bends you want, or some more advanced

arguments here, like the alpha level, etc. But it's simple. And similar to the histogram,

you can also create kernel density estimator diagrams, which is very similar to distance

to simulate if you want a continuous distribution. You can combine these plots if you want, in

this case, we are creating the plots were plotting a histogram. And they were plotting

the lines and they were plotting our changing limits. But that's pretty much it. And you

can also create bar plots, right? So in this case, we have PLT dot bar, or here we have

two bars are stacked, right? That's the different way to look at it. And finally, check in outliers.

You can always plot histograms or box plots, right? So box plots are also a nice feature

to have in here. So this was all with data cleaning, we're gonna keep moving forward

this tutorial, I want to mention one more thing here. And it's there are notes here

for kind of a task that you can follow with data cleaning, which where we are identifying

where indentifying missing values in given positions with is known as an A. And right

here, we're looking into more detail about some statistical properties of the data, in

case we need to clean it. Okay, so this is little bit more events. And it's it's related

to the concept of cleaning data given the domain. So the statistical analysis can tell

you that this value is an outlier. For this distribution, the value might be valid. So

for example, a human being is 90 years old. That's, that's valid, that's a valid age.

But if you're analyzing data about high school students, and a human that it's not a year

soul, it's going to be completely invalid or it's going to be an outlier in that distribution.

And you can treat it as such You valid valid and clean it out, remove it, for example.

So that's, that's deal a little bit more with the whole statistical analysis you can follow

here, it's a little bit more advanced for the scenario. So let's move forward with the

rest of the videos.

Now it's time to get into more advanced features of pandas to import external data. So we've

seen already in our real life example, the way we can import data from CSV files, and

from SQL databases, right, we had actually those two lessons, the objective of these

part of the tutorial is to show you how you can improve or get into more advanced use

cases of importing data. So we're going to start for example, with csvs, and text files.

And again, you've seen it already. But here, we're gonna give it an extra twist. So I'm

going to show you more advanced features. And for special use cases, txt files, CSV

files, is, conceptually speaking, a CSV file is a text file, it's just human readable text,

right? That it's encoding information. The idea for CSV file is that it's tabular. Right?

So it's a plain text file that contains tabular data in it, and it's separated. csv stands

for comma separated, but it can be separated can be anything, we can see more examples

later. But basically, the idea is that it's a text file that it's tabular into in a tabular

format. So though, both CSV files and text files will be read with the same method. So

to get things started, I want to show you the basic way we import will read data from,

from from external sources using Python without even starting yet, with pandas. So you don't

need to know this, it's usually it's usually productive if you want for data scientists

or data analysts to understand a little bit more how fire reading and writing works in

computers, because there are multiple, multiple concepts align, here, they evolved, operating

systems processes your language, right, it's not same thing to read a file with our or

with Python or with another language. So there are multiple concepts here. And even though

pandas in this case can make it simple, very simple to read and write data, you can get

a little bit of a more advanced use case, if you know the internals of again, both the

operating system processes on your language. So this the way we read data with a reader

file, sorry, using pure Python, we use a function open. And in this case, we're using a context

manager, just a security feature, again, related to to the advanced usage of reading and writing

files. But it creates a file pointer, right. And with a file pointer, you can then use

the very simple API x point post. But they but that pointer, which is something like

red line, red lines, read a number of bytes or characters, or you can just even trade

FP as an iterator, just do a four line in FP. But basically, we're going to do something

like this, we'll start reading data from top to bottom, just a month to, I don't know,

we hit I've given in this case, we're doing it just for a couple of lines. What else we

can, it gets very difficult when you're reading text files to process them, because it's usually

hard to parse the structure of the file. So it's not the same thing to have a funnel that

is separated by comma separated by colons separated by pipes, spaces, etc. So you're

gonna see that once you want to get a little bit more, I don't know a little bit more with

an advanced usage, right, or a little bit more fancy your calculations and and the way

you parse the data, it's gonna, it's gonna get harder. So that's why we're going to use

pandas, or I'm going to show you in a second, this is the module that is part of, of Python.

So this is the file that we're going to be reading. It's the XM review file, and I'm

going to open it. And even though it doesn't look like a CSV, it isn't either CSV. The

difference is that here the separator is the greater sign, it's not the comma, it's a greater

sign. That's going to be what marks the elimination between different fields in our CSV file.

So we're gonna use the CSV module. And

the way right here to parse the data using that module is by passing a special delegator,

right? So that's gonna be the type of work you might need to do when you're parsing the

data. It's not the same thing to have that limiter dates a greater sign. It's not the

same thing to have numbers for example, that are enclosed in quotes. All those things right

will change the way you work on all days is going to be abstracted away by the pandas

module. So to get things started, again, with pandas, at least, pandas has multiple read

underscore something methods that will work for different sources, right. So we saw already

have read sequel we've seen read CSV, there's also a read HTML to directly parse information

from a table, it's literally you can just you pass a website's going to read information

from a table, or read Jason read more advanced formats like pocket, or Stata, etc. And, again,

each file format will usually have a correspondence in pandas, it's, I've never had the chance

to rewrite my own stuff. To be honest, the same thing is going to happen for something

like Excel, which might need external modules, it's not directly provided by pandas, but

by installing those modules, you can easily incorporate Excel files in your day to day

work. So the read CSV file methods already has a ton of parameters. So this day, the

main characteristic of all these rate something methods, given the amount of possibilities

you're going to have with these files, there exist a ton of different ways to customize

the method invocation. Alright, so again, CSV files, we saw, there are multiple things

happen. csv is a passage that have a header don't have a header, different delimiters

different and closing of strings or numbers, multiple things, blank lines, etc, multiple

things are going to happen. And that's all you're able to customize all that with the

read CSV method. So this is the reference of all the

attributes you can pass to it, usually something that I do, and I do this very often, and I

use pandas a lot, and I still do something like read CSV, and I get the documentation

right here, to look into the, the parameters that I think I need to pass to my particular

use case. So keep an eye always in the docs, because it's impossible to remember all the

parameters in the CSV. So in this case, what we're gonna do is something very interesting

is we're gonna parse a CSV file, but it's not located in this computer, it's not locally

available in the computer. The CSV file is these one right here, which actually is the

source, if I get the raw version is this thing. So this is CSV file, what I could do here

is download the file, right, so just do File, Save, get the CSV file on my computer uploaded

here, right, so just copy and paste here, drag and drop it here. But actually pain this

has this nice characteristic that it will read a CSV that it's either locally as we

did with BTC market price, or you can also do it remotely, it's automatically going to

download the content of those files. And it's going to provide, it's going to save it in

memory for further usage. So there's a very neat feature. And again, this is the the CSV

file that we are using. And again, the same thing, if it's a local file, it works in the

same way. So a few features you've seen already, in this case, we can do Heather known, if

you don't want to treat the first row as a header. Or what about missing values, we can

treat some of these values like a question mark, or like an exclamation mark, or dash

etc. us not a number, not a value, right, so it's a missing value. And now any of these

values we have passed, will be transformed into another number for easier and easier

process cleaning, we can pass names, which is going to be basically the column names

for each one. And we can also specify column types, as you can see, right there. So now

the types are going to be float. And object. We've done this already in one of our lessons,

we are parsing the time and there you go. So putting all together, we get to these advanced

forms of reading csvs where we're passing column names were passing types, were asking

to read dates, were passing no values, Heather's etc. So this is a pretty common thing we are

doing. So what about XM review, if we try parsing this thing, we get this very ugly

format. In this case, they put the parameter to specify the what we used to call delimiter

in CSV is now set from separator so the separator, it's going to be the greatest sign and that

just works as it needs. So, a few more examples you can check on here the most important part

is following right, the documentation to find those particular use cases that you are having

so for example, some Like skip blank lines, or whenever there are like empty rows at the

beginning, right. So if you have empty rows at the beginning is something you can also

say skip rows. So you don't need to parse those out, it's not going to break, etc. So

that is all part of the read CSV file. And to finalize these part, at least csvs, I'm

going to tell you something that applies to pretty much every other data format. As you

have a read something method, there's going to be a to something method, it's basically

the process of writing. So you can do read CSV, or you can do to CSV. So these CSV that

we imported from the external source and the remote source, we can just do to CSV and it's

going to store it locally. Alright, and there are multiple options also to pass the CSV

delimiter, or actually the separator, if you want to include a header if you want include

an index, etc. They're pretty much the same as the other one. But the idea is that for

every read something method, there's gonna exist a to something method that it's basically

the process of writing. So let's move forward with a few more data formats. And interesting,

we're gonna get to read directly HTML pages in just a couple of minutes. And now it's

time to read data from databases. We have already done that in our real example with

Panis part of the tutorial. But I want to show you a little bit more details details

for you understand how data is being processed in case, this is a common scenario for me

importing data from databases. So the libraries you will need first thing, depending on what

database engine, you're using Postgres, MySQL, Oracle, etc, you will need to install different

libraries. But the API's, once you have installed, those libraries are going to be the same.

There's actually p Ep from Python that actually defines the interface for databases, libraries,

unpin, this can work with pretty much any any database of these SQL common database

that comply with that interface. In this example, we're going to use SQL lite because the database

right here, there's nothing, no server to connect, etc, is extremely simple to get started.

And the example we're going to use, or the danavas example we're going to use is actually

different one from our previous video is reading in the previous one, we were using circular,

in this case, we're going to be using chinuch, which is smaller both in structure and in

size. So it's going to be a little bit simpler. So to get things going here, the same thing

that we did with our previous part, that was how to read data from files, I show you how

to actually read data using Python. So forget about pandas for a second, I told you, if

we go back again, to the beginning of time, there was no pain this, this was the way we

were writing, finance, open FP, FP, the red lines, etc. So I now want to show you what

predates to pin this, what was the default way to read data before paying this, which

is with the regular again, interface from Python. So the way it works is we're gonna

import SQL lite three, we're gonna create a connection. And now with this connection,

we have these common interface that again, it's common for pretty much any other database

that you're used to. And the default behavior is we're going to create a cursor. And we're

going to execute queries using that cursor. In this case, we're going to execute a regular

Select star from employees limit, Fox will want to have five, five records out of the

table employees, once you have executed a query, it's like they're waiting, you can

do a fetch all to get all the results of that query. And here are all these results. As

you are noticing this is the result is a list of tables. So it's not extremely useful. Now,

if you combine it with pain, this you can just create a data frame out of that info.

And we're close. It's not perfect, but we're close. So let me show you now before we were

gonna close it Kurt Dickerson on the connection. Let me show you now how we work with pandas.

With pandas we have as we have a read CSV method, we also have a read, see as read SQL

method, and in this case, what this method is going to receive is the first parameter

is going to be the query that we're passing and the second parameter is going to be the

connection. That's the object the connection object to actually issue the connection by

panelists. So it gets a simple as writing the query. And now everything has been imported

into a data frame, including column names and all that if you want to get a little bit

fancier, you can either specify the index column, there's going to be use, of course

as a index, and also what types to parse for a specific column. So now we have pretty much

all the work down. So we're going from something very manual as processing things with a coarser

etc, which might also be as low to using pain this to do Actually imported data from the

database. There is actually a caveat here that I'm going to tell you is kind of a very

deep detail of the way pandas works, and is that the read SQL method is actually a shell

for two other methods, read SQL query and read SQL table. Alright, so right SQL table

on read SQL query, when you're using read SQL, it's actually kind of forward in the

work to either query or table, or an SQL query is the default behavior, what we've done so

far, so in this case, it's just going to issue a query and the connection is going to read

it for you. In contrast rate SQL table is can I read an entire table, you just pass

a name, and it's going to automatically give you all the information for it. So in this

case, all the column names, etc. So it's a lot simpler to read an entire table, the only

thing to keep in mind is that to use this method, you need to install these libraries,

SQL alchemy, and the connections generated from it. So in this case, we create an engine

on we create a connection objects, and now we can pass an actual auction object sorry

for pandas to do it. So again, it's pretty much the same, if you find yourself doing

Red Star from this table, Red Star from that table, it's a lot easier just to write SQL

table, and that's going to do it just advance. As we saw that read CSV files hard to CSV,

sorry, read CSV method had a to CSV method, the same thing happens with read SQL, there

is a read SQL and the results are to SQL, what's what it's going to let you do is get

the from the database and write it down into a database table directly. So it's going to

also receive the connection, right? So to SQL, it's gonna receive what he will name

of these data frame, what table name is going to be, and a connection object. Now something

to keep in mind is that to SQL has an important parameter, which is what happens if the table

already exists, that in the default way, it's going to fail, just going to throw an error

when you are trying to save data to a table. And this makes sense, because as data analysts

were usually reading data and processing it, we're not so much writing it. So we want to

meet make sure that it's not by mistake. But if you do actually want to write data, you

can just change this parameter if exists something like replace or append. Usually, we're writing

to intermediate intermediary table tables, again, you can choose either to replace the

whole concept of the table, be careful here, or to append, write, just write it a dn of

the current table. So that's just for to see. So this was the way to read data from databases,

of course, we're not touching on anything like SQL and all that, that it's a lot more

advanced, it's just for you, if you already know SQL, if you're already working with databases,

you can pretty much copy and paste what we're doing here. And you're gonna, you're gonna

get your data import into Python. So let's move forward to read some HTML files. And

now very quickly, I'm going to show you how to read tables or data frames directly from

HTML web pages. To be honest, this is a simple method is going to be just read HTML, but

it depends a lot on the structure of the web page. So if it's not well structured, or the

tables are not correctly created, you're going to have issues and you will have to do a ton

of data cleaning. In my experience, whenever I try to parse a table from a well structure

site like Wikipedia, or some stats site, it usually works very well. And it's a very quick

way of hacking. You know, whenever you have questions, you know, like, I don't know, I

need to know the GDP of countries. Instead of looking for a GDP data set, you can just

go to Wikipedia page, there is usually a table there, you can directly parse it and you are

done. So again, it's it's a relatively simple way to get some data for quick hacking and

exploration. The way it's going to work is we have these HTML creative. It's just for

testing purposes. To get started, usually, of course, you will try to read something

from a live website. So you're going to pass the URL to the read HTML method. And the read

HTML method will download the content of the page and parse it. Let's suppose we have the

the content already the HTML, and this is what it looks like. This is a exactly the

same HTML we have on top, I'm just displaying it here in a book. And what we're gonna do

is we're gonna invoke the method, read the HTML. And the read HTML method is going to

parse the entire HTML and look for multiple tables, not just one site will potentially

have multiple tables, even if you don't see them. The is a common way to structure things

in HTML to use tables. That's why it's going to pause multiple tables. In this case, we

stored them all in a DFS, multiple player like multiple data frames. And we see that

there is only one. So in that case, we're just going to get the first data frame. And

it has correctly parsed what we had before just working in the same way.

The same is going to happen with for example, things for headings and all that if the table

doesn't have a header, it's gonna automatically right understand.in that case. So that's pretty

much as we know it already. In this case, what you're going to see is what I told you

before about data cleaning process that these table does not have a header like the previous

one that has a T head to head attribute, in this case, a header is just another row. So

that's why read HTML is going to have issues and you have to provide a little bit extra

information. So let's see another more realistic example. And we're going to parse data directly

from a website, let me tell you here, just just for educational purposes, you always

need to understand if you have if the data is public, so you can actually parse it. Again,

for Wikipedia, at least what I do, the content is created comments, so you can get a hand

on it. There. What we want to show you here is that a very complicated table that has

multiple headers, etc. So that's why we're using this example. So we're gonna get the

URL, and we're gonna directly do NBA tables. Equals read HTML, the only table in this page

is this one, the large one. So that works. And now we're gonna get NBA is going to be

that and we see that the all the players in this case have been parsed. What about something

else, let's actually open this page right here to Wikipedia, for the Simpsons. And here,

we will probably find several tables. See, we have one right here, this one. So I'm going

to import it. We have 27 tables, again, you don't see it. You don't see them, sorry, but

they are there. And the most important one is the one we care is these one right here.

So the problem you're gonna have with this table is that each using both columns, pans

and rows pans. So in this case, this column here is pans for one to three columns. And

these row here stands for 123, at least three rows. So that column spans results in these

very ugly data frame, and you will need a little bit of extra cleaning. So that probably

you're going to find with HTML tables that usually there are things that are not well

formatted for machines that are formatted for humans. So for example, in this case,

we have this header repeated, when you parse this data, you're going to find that every

20 rows, there is going to be header row, and you will have to clinic every for in this

case, to enter rows, you will need to drop it you will do something like df the drop,

let's see, actually, if we can see it haven't tried this, but let's just do it like that

head, and you're going to find 25 Records now. So here, record 22, we find that, Heather,

so what we're going to do is you will need to do something like df the drop df dot drop,

range 22 starting in 22, up to the F the shape, zero, right, these many rows plus one plus

one and every 20 rows, I don't know this is going to work, just run it. Hope didn't it

didn't even work. It didn't compile. Oh, this is NBA actually. There you go.

So maybe it works, you can check it. But what I'm going to say is, again, there is some

cleaning to do because HTML pages are optimized for humans, not for machines. So usually,

it's going to take a little bit more time. The good news is that there is usually a service

associated that you can consult. So for example, there is a Wikipedia API that you can use

instead of a page. But again, sometimes just easier to pull the data directly from Wikipedia.

So that's it. You can also write data to CSV or of course or HTML. That's pretty much the

standard. As we've said, this is up all we had for the read data portion. And we're gonna

move forward now with a few other methods, especially what we call data wrangling. We're

going to do a little bit of grouping and keep moving forward with our tutorial. We have

decided kind of last minute to our final source of external data that it's going to be an

Excel file. It's just a common Excel files, you know it, because we imagine that you might

come from an Excel backgrounds, you can just export the data you have in your Excel files,

Excel spreadsheets, and load them into Jupyter Notebook and start working with them with

him this so you can try things out and kind of draw the pearls in between Excel and what

you do with pandas and Python. So the first thing is, an Excel file is not a text file.

So if you try getting the content of it, it's not a text file, it's not so simple to parse

it. So that's why it's gonna require external tools that they already installed in notebooks

AI, there might be a student's holding goal up, but it depends on your computer, how you're

going to install it. So just keep in mind that there might be issues when importing

data from Excel, if they if there is low compatibility between the library you're using another spreadsheet

version you're using. But without those without getting into those details, there is read

Excel method, which pretty much takes care of everything for you has different parameters,

like the finding the the sheet that you're reading from, of course, the path, etc. So

we're going to start reading these file, which is products file that has three sheets, products,

descriptions, and merchants. And it's actually something we use in an Excel file to sorry,

in our data analysis, from Excel to pandas course, to show how to merge data and all

that. And from this file, what we're gonna do is just read Excel. And what you're gonna

see is that it reads the first sheet of the Excel file, I mean, a data frame is just corresponds

to one sheet only, right? And the first one is product. So that's what we are writing.

There are different behaviors for it, you can change the way you parse, Heather's etc,

you notoriety defining and specific index, that's pretty much everything we have seen.

So far, it's selecting specific shifts is simple, just pause the sheet name, and you

can share the rate story either products, merchants, whatever is available in the current

Excel file. There is another format or a new specific class that it's a little bit more

advanced. But it's the Excel file class. So it's not, as we were doing here, right, Excel

directly is going to read thought Excel file into a data frame, but you're going to instantiate

this Excel file class, with the parameter being the file name. And now these files gonna

have just a reference of everything you have. In this case, we can do for example, sheet

names, it's going to tell you how product descriptions merchants, there's a little bit

more explanatory data analysis. So let's say you can't use Excel to actually see the contents

of the Excel file, this is going to be helpful, you're going to first parse the Excel file,

get the sheet names, and a little bit more of an understanding of it. And now we can

say from these files we have previously parsed right here or instantiated, we can parse the

product, the product sheet, and that's going to get you that that frame. And the same thing

is going to happen with all the parameters weekend pass, they are the same as read Excel.

Finally, you can see that the results are to excel file. And it works pretty much the

same way as to CSV, and decide if you pass an index or not. And also you can define if

you're going to pass a sheet name or not, are just going to be the default one. So as

you can see, getting your data into a from an Excel file into a CSV, data frames array

is extremely simple. There are more customizations to do, let's say all your file is shifted

array, either rows or columns, you can change that with Star row or column that's going

to work, too. So that's pretty much the only thing we need. If your writing process is

a little bit more complicated. Like for example, you want to write specific sheets in our multi

sheets. Excel file, you can use what we call an Excel right and it's also part of fantasy,

you instantiate the rider, and then you can start the ride process saying which shades

you want to ride with each one of those, that friend. So again, reading and writing data

from on to Excel files is relatively simple. It all depends on the libraries are installed.

It depends on on what libraries you have in your current environment, if it's windows

or if it's a Linux slash slash mark, the documentation of PD dot read Excel

might have more details for the given platform that you have. So let's see if it names per

document, if it's not here, it's gonna be in the pandas documentation, but there might

be a requirement For each one of the platforms, that pan This

is supported. So just check it out, check for your own for your own platform if you're

in Windows, Mac Linux, how to get those libraries installed.

So in case you're just getting started with Python, and you might come from another language,

the objective of this quick section is to show you Python. Ideally, in under 10 minutes,

I think it's going to take a little bit more. But there's a very, very, very quick reference

of Python, again, just the high level features of the language, how to use it, how to code

functions, how to import modules, variables, data types, collections, etc. You can just

scroll through this notebook, if you want to take less time, I will be providing an

explanation on top of all the topics, but there's a very good reference of the entire

language. So to get things started, Python is an old language period. It has card, it

has caught more attention in the past five to 10 years. But it's a very old language.

It's even older than Java. It's up here in 1990s. And it was created by this person good

by Guido van Rossum. And it's an important actor in our ecosystem he is used to be I

think he still the one deciding discussions etc, when it comes to defining features of

the language, etc. Python is a high level interpreted dynamic language. And this means

a tone actually, if we read these entire sentence, interpreted high level, general purpose, this

is basically high level programming language, it's object oriented. And it also includes

functional attributes or functional features like functions as first class objects, etc.

And it also, of course, it supports imperative programming. And it has a wide variety of

applications, you can do web development with Python, you can do scripting, it's a lot use

for system development for configuring machines in general. And of course, you can also do

data science, it has multiple applications has a couple of interesting features like

indentation, for defining blocks, etc, that make it and very good language to get started

with programming. So if Python is your first language, you should be comfortable with it.

It's a very good idea for me, it wasn't my first language. And I hope it was, it wasn't.

But I, I have taught people programming with Python as their first language. Seriously,

it's always been very good for them, because Python doesn't have weird things like my have

in JavaScript or Java. So it's a very concise language and consistent language to be honest.

So let's get started very quickly. First of all, when you're going to install Python,

your own computer or you can use notebooks AI or Google call up. But if you're installing

in your own computer, you might see that you can install either Python two, or Python three,

or actually, if you're reading tutorials online, etc, you might see Python two and Python three,

the reality is that Python two was deprecated in 2020, so the you cannot you should not

use it anymore. There are still ways to install Python two, but it was deprecated. So you

shouldn't use Python two, you should stick with Python three, which is the evolution

of the language. So ton of fixes from Python to the bay where, where things happen in the

language and used to confuse beginners. So that's no longer a problem. Python three,

again, is what you should use, you will read in multiple tutorials, etc. What they are

using Python two, you should try using Python three, and sometimes the code will break,

but the changes to fix it are not very hard. So to get things started here, I will be drawing

the problem of this and with regular syntaxes. For example, this is the way you will define

a function in for example, JavaScript. And it's also very similar to something like C

or Java based languages, the function keyword, curly braces, etc. So I will be drawing a

parlors and with these sort of languages. So to get things out of the way to defined

function in Python is in this way. And the main characteristic of this language is that

the way we're going to define blocks is by Using different indentation levels. So this

is a valid function in Python def is the key where we use the name of the function the

parameters it receives. And the way to define that the body of the function is by just indenting.

Everything one level to the right. Usually, this is just for spaces.

Another example is an if else statement. So if this thing happens, do that if else do

something else, right? This is JavaScript. In Python, again, it's defined by indentation.

If this thing happens, we indent one level to the right, do this else do something else,

if there was another if statement here, if I don't know, language, ends with something

like I don't know, three, then do something else. Print pi, three, for example. So we're

indenting everything to the right, every time we start a new block, whenever the block finishes

is just when you go back again, print this as first block, right, that's the way it's

going to work it by indenting. Our blocks, this is very good, because first, we don't

have debates of where we should place the curly braces. And also, because it makes it

a lot more readable, it's a lot easier to read these code because there is obligated

obligatory indentation to even make the code work to. So you can see that's that's just

how it works. How we're going to make comments in Python, just by using the number pad symbol,

there we go. And the way to define variables is just by specifying the name. So it Python

is a language that you don't need to declare variables, you just declare and define everything

and just one pass, you know, you find a variable, as it goes. Python is dynamically dynamically

typed. But it's also strongly typed. And these might kind of cause confusions. But basically,

you can assign variables to any value you want. And you will see that collections etc,

are heterogeneous in terms of types, etc. It is a very dynamic language. Talking about

types, I'm going to show you the most important types that we have in Python, especially we

have numbers, of course, integers, we don't have so many like, like, you might find that

other languages, like different precision cetera, we have integers, there is also the

concept of Long's that has changed with Python two. To be honest, on Python three, to be

honest, we use just integers, that's the way we work. It's a, it's a smart enough type

to save storage when needed. So that's, that's good. And it will also have floats, right,

which is the regular float type for floating point arithmetic in other languages. And of

course, it suffers if you want from strange behavior from float floating point arithmetic,

like in this case, you can prevent that by using the decimal module, which, as you can

see, doesn't suffer from from this issue. So numbers, we have integers floats, and we

also have decimals, strings are just a type str, and they are defined literal, right,

as in this in the st, you can see right here, you can just type the string as it goes. There

is a difference between there was a difference already in Python two, between Unicode and

strings, etc. In Python three, that has all been fixed. So we Python three, this is all

Unicode. And there is the concept or the difference in terms of the concept of something being

the type. The Unicode code points as it's this string, and the underlying encoding will

turn it into binary. So in Python three still have we have a few ways to differentiate between

whether it's a binary string or whether it's a text based string. For you shouldn't worry

about it, I just want you to know, if you're writing a Python tutorial, for example, you

might find a difference between Unicode strings and regular strings, which is, is no longer

something that we should be worrying about. If you have a string that it's too long and

it expands multiple lines, you can always write it using three quotes can be double

quotes or single or single quotes.

So just to create multi line strings is extremely simple. Boolean there are two Boolean type

do Boolean objects are unique, right? It's kind of a single tone which is the true or

false objects. For example, They are of type Bo. There is also the concept of No, in Python,

which is none, we don't have no, we have none, but it serves pretty much the same purpose.

In Python, everything is an object. So even this strange, strange objects, like none will

have an associated class, if you want, everything in Python is an object. So all these types

of you have seen. So for example, we have this string, which is H of a string. The type

is str, you can use the int, str float bool types, right, but it's the result of the type

also as function. So in order to cast in this case, a string into in order to cast a string

into an integer, you will use it you will do it using the end function, which is the

same thing that you get with these, for example, so this is the same as this, as you can see,

what we have to show. So functions again, death is the key word we use, we don't use

function, we use death I, you can use define, as a mnemonic, the name of the function parameters

are optional, and finally have the return keyword, you should always include a return

you usually 99% of the time, the function should return something. Because that's going

to be the result assigned once we invoke the function just this is pretty regular. If your

function doesn't return anything explicitly, if that means if you haven't written down

a return statement anywhere in your function, the function will still return something so

that the fact that you haven't included a return statement explicitly doesn't mean that

the function is not returning anything implicitly, actually, it is returning something, it's

returning none, right. So by default, if you don't include a return, Python will do this.

Just for you to know a function always returns something as specified parameters and passing

parameters is pretty standard. Python has some advanced features with parameters like

for example, variable length arguments, we can pass as many arguments, we want to make

it very dynamic keyword arguments, named arguments, etc. So all their ethic operators, you know,

already, the shin modulus, in this case, were doing a power its operation, all this is pretty

standard. And the same thing happens with all our Boolean operators greater than greater

or equals then etc, there are type checking. So this is when we have the strongly typed

feature, even though Python is dynamically typed. It is the types are enforced. In this

case, you cannot compare a two with this doesn't make any sense. And Python is going to complain

about that. So this is an example of an error in Python. The exception type error was raised

on the same thing with bolens and not on or operators. As we saw before control flow is

defined by the indentation so every new block is defined with an indentation level. Python

includes if else and also l F, which is very convenient. And this is an example If this

happens, Elif, Elif, etc. Python does not have a switch statement. For example, loops,

how are you going to loop through something in Python loops on lists,

or

collections in general, are very interconnected. Because in reality, when you're looping the

Python, you're not doing a regular in Python, we don't have something like in, in Java,

you're gonna have something like int i equals zero.

What else I

it's been decades. And I this is I haven't coding in Java. So I, I don't know, minus

10, less than 10 less than 10. And here we do I put last There you go. So we don't, we

don't have these in Python. We have a way to mimic it. But we in Python, we always eat

iterate over a collection. So what we're going to do is we're going to create a range elements,

and we're going to iterate over it. So the way it works is very close to one other language

is going to be a for each. Alright, so in this case, we have all these elements and

we're going to do for name in names, that's it. And at any moment, the name is going to

be associated with an element in the list. while loops are part of the language, they

are usually discouraged in favor of for loops. If something can be coded with a for loop,

it should be coded with a for loop and not a while loop. Because as you might know, already,

these my trigger or these might result in an infinite loop if you're not checking the

conditions correctly. So the collections we have in Python, are the fundamental ones,

the primitive ones, the most important ones are first the list Python is we do a heavy

usage of lists. And it's just a heterogeneous data structure. So you can put anything in

it. And actually, all these collections are heterogeneous, you can mix volumes as you

want. And in this case, we have three elements that we have added one string, one integer,

one string, and one Boolean. And let me say something here. Even though pythons, Python

supports mixed types in the collections, it doesn't mean that you should do it. To be

honest, we should, you should usually avoid mixing types in collections, because that

means we don't, we don't know what we're putting in it, right. So it's, we should be consistent.

So it's possible, revisit your code, if you have too many different types in it. I'm checking

the length length function accessing elements is by by zero indexed, and we use square brackets.

So in this case, give me the first element given the second element. And also we can

index starting from the from behind from the end. So in this case, minus one, minus two,

minus three. So in this case, minus one minus two, again, give you different elements, you

can check the operations associated with all these elements. Very quickly, a list is L

dot append, we're going to append the new element. So the list now has that element

at the end. And we can check if that element is part of the list in this case is true in

this case is false. topples are similar to lists, they are also sequences, but the main

difference is that they are immutable, there is no way to add new elements to a tupple,

or remove elements from a tupple once it has been created. So in this case, we have created

a list with three elements. Now tupple, sorry, with three elements, we can access it, we

can check if something is in it in the same way that we did with a list. But in this case,

with a tupple. Again, you cannot modify it tupple never changes, you can't add elements

to it. Another important data structure is a dictionary. In Python, a dictionary is a

key value, right and mapping, it's similar to an object in JavaScript or hash table in

in, in Java, it's a key value mapping type. And in this case, we are going to associate

values to names. So you can see this, the way I like to explain it is if you create

a topo list, right? So let's say we're going to create a list, out of all these elements,

give me one second, we're going to create a list. There we go, we're gonna copy these

elements. And we're gonna associate that to our list. There you go. So these are a list,

we could very well store the information about our customers in a list, right? That works.

I mean, I can get it done. The problem is that whenever I need to access information

about this list, we're going to say, for example, I don't know I want to give me the email for

this

customer, I have to remember the position that the email is located so in this case

is going to be position number one, if these information grows, instead of having 1234

values or four pieces of information for our user, we have 100. Right, then it's gonna

be very hard to access those individual volleys. So that's why we create dictionaries, dictionaries

are collections of values. The important part is on the right, the important part is the

value. But they are instead of just indexed by the precision, we give them arbitrary names,

we tell them very explicit names. This is the name, this is the email. This is the age.

And this is if they are subscribed or not. So once we create these dictionary, we can

access those values by the name, give me the email of these user or is the age present

of the user is the last name present of the user in the user in the user dictionary. So

again, it's a way to store information associating later In order to make it simpler for us later,

let me delete this. And I move four sets sets are very common data structure, he is when

you're learning about a collections and, and and yeah, the instructions in general, it's

not so common in too many languages. I mean, it's not very popular in Python, we use it

often because it has a very interesting feature, first of all, and it's something that I forgot

to tell you about dictionaries, both sets and dictionaries, are what we call unordered.

data structures, you never know, the order of the elements. In Python, with recent versions,

there have been changes, which make Python dictionaries ordered. But for now, I'm going

to say you shouldn't rely on it, you should think your dictionaries as they are completely

unordered data structures, and the same thing for sets, sets are, it's a bag that contains

elements, you know, it's a big bag, you keep throwing elements inside of the set, there

is no orphan in it. And what's gonna happen with it, you're gonna odd elements, for example,

to the set, or you're going to remove elements to the set. And there is one important thing

that makes this set so useful, and it's the membership operation, I'm gonna

write it down here, membership, ship operation, there you go. So you can access these notebooks

later.

So in the membership operation, the the, the process of checking if something now, nine

in s, the process s of checking this is extremely fast, it will be called oh one. And this is

because as you might have seen here, when I created this set, I included a couple of

repeated elements, 333, write 11179, the resulting set doesn't have those repeated elements,

these are two features of the set, the set will only contain unique values. And by the

way, it's implemented behind the scenes will make dot these unique values are extremely

simple to check whenever you pass these membership operation is extremely simple, or sorry, is

extremely performant. It's very fast, different from for example, a list. So keep it in mind

sets are very, very useful when you're checking for members. So again, as I told you before,

we're going to iterate over collections with the for loop. So in this case is if we have

a list, it's going to be for element in list. There you go. If you have a user dictionary,

use a dictionary, sorry, in this case user, we're going to the default iteration is by

key, we're going to get for name email age subscribed, and we have to extract the value

out of the of the dictionary, we could also do for value in user dot values. Oh, there

you go. Or you can iterate over both key and value with items. Key. And value. There you

go. So each iteration in in in Python is very readable to put it in a way. And again, remember,

we're always using the for loop that assumes that you're iterating over a collection, we

don't have the for Ei equals zero equals zero, I equals zero, i less than 10. i plus plus

we don't have that right in Python, we can simulate it with for i in range. Five, for

example. Print. We've got simulated with the range function, which generates pretty much

those elements. Something that you might have heard about Python is that it has a huge library

of built in modules, right that you can just import and just gonna work. There are so many

things already coded in Python, that it makes it very simple for you to create something

on top. Do you want an a library for I don't know security cryptography Math, numeric processing

NumPy, right? machine learning web development, creating games through is pi game, do you

want to create a graphical user interface, whatever you want to do, there is usually

a library that has already been coded and will make your job easier. On top of that,

the bill team is down there library, right, which is already included with Python, it's

not third party. In this case, it's already created by the Python core team. It's a huge

library, so many modules. And the way it works is by importing this module, so this is the

way we work with packages and modules, there are differences between modules and packages,

third party ability, and this is a little bit more advanced. But again, this gives that

random number generator, it's already built in. And you can check the docs

right here.

exceptions, whenever you do something that doesn't work. So in this case, we say, if

the age is greater than 21, but age is a string, it's an it's not an integer, this is going

to fail. We can catch exceptions before they happen, that's going to be with a try and

accept lock. Right. In that case, if this fails, if anything here fails, these blocks

going to be kicked in. And you can catch the exception without the program fail failing.

And you can be more explicit about the error aspect. So again, this is just an introduction.

It might be useful if you're coming from another language, especially to keep this notebook

as a reference. We're going to be using Python a lot, of course, and it's a great language

if you want to do scripting, work development, of course processing with data, data analysis,

etc, visualizations, machine learning, Python is just great. So I hope this tiny tiny reuse

lesson helps you port your knowledge from other languages into Python. And that's it.