Data science I

pandas

pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language (pandas-website).

Let’s say we want to investigate the weather data recorded at the Frankfurt International Airport from the year 2015. The data is available in the tabular CSV format:

Read data

First, we want to read the data:

The pandas function pd.read_csv(...) opens the csv file and reads its contents.

• The parse_dates argument tells Python to interpret the values in the time column as dates
• The index_col argument tells it to use this column as index.
• The sep argument tells it which character it should use for separating columns
• The na_values argument tells pandas, which values should be interpreted as NaN (missing) values. In our case, missing values are just blank cells ("")

pandas has many more readers for a variety of file types, e.g. Excel, SQL, HDF, HTML, etc. The full list of supported file types and the respective readers can be found here.

DataFrame

All data is stored in the tabelle object. This object is a so-called DataFrame, the basic pandas object. You can think of it like a spreadsheet.

To be more precise, it is a two-dimensional size-mutable, potentially heterogeneous tabular data structure with labeled axes (rows and columns).

The DataFrame object offers many functions that allow you to work with the data conveniently.

For a first glimps at the data, you can call the head() function which prints out the first 5 rows of the table:

To get an overview of the value distributions of the variables within tabelle, you can call the describe() function. It generates descriptive statistics that summarize the central tendency, dispersion and shape of a dataset’s distribution, excluding NaN values:

The dtypes field of tabelle shows you all column names and the data type of the respective column:

If you want to know, how many columns and lines the table consists of, use the shape attribute:

Let’s say, you want to sort this table by visibility values in descending order.

You can sort a pandas DataFrame by the values of a specific column, using the sort_values() function:

Selection by label

Let’s assume, we are not interested in the complete year 2015 but only in August. We can select the respective rows with the loc attribute.

When the index of the DataFrame is a range of dates (like in this case), you can select a row ( = measurement date) by specifying a date (as string):

If you want to select a specific date range, you can do so by providing a start and end date (notice that the specific dates do not have to match the date values in the “time” column):

What if we are only interested in the measurements between 12:00 and 15:00 for each day? Unfortunately, for this task we cannot use the loc attribute. However, the between_time() function helps us:

Now we know, how to select rows in time series (=measurements). But what if we are only interested in one specific variable, e.g. cloud height?

You can select single columns by:

Note that selecting a single column returns a Series object and not just the column as a list for example. In principle, a Series is a one-dimensional DataFrame and it provides more or less the same functionality.

For example, you can use the count() method to count the values within a column/Series:

As you can see, the returned value of 10210 is smaller than the total table row count of 17520 because NaN values are excluded automatically.

There are many more methods provided with the Series class but we will not go into further detail here.

If you are interested to get a simple list of the values of the column use the .to_list() method but most of the time it is better to use the .values attribute which returns the values as an array (don’t worry about what an array is right now. We will cover that in the next chapter):

You can select multiple columns by:

You can also select columns and rows together within one call:

Selection by position (index)

Just as with the label selection, you can select values via their index using iloc:

Boolean indexing

What if you don’t want to clip the table to specific times but to other conditions? Let’s assume you want to select all rows, where visibility was measured to be below 1000m.

You can do this easily by specifying the conditions within the [ ] brackets:

You can also use multiple columns to select data after specific criteria. Let’s say we are only interested in rows with visibility values below 1000m and temperature values above 5 °C:

Operations

Besides its many selection options, the DataFrame class provides many functions for calculating statistical measures, e.g. the average of each column:

You can do the same for single columns (Series):

And of course, you can combine all of the above:

If you want to apply specific operations to your data that are not covered by the standard functions of pandas, you can do so by using the apply() method. This works for single Series or along an axis of a DataFrame object:

But what if we want to do an operation on multiple columns at the same time and save the result in a new column?

Let’s say we want to calculate the difference between air temperature and dewpoint values and save the results to a new column named “temp_dewpt_diff”.

We can achieve this easily via:

Treating missing (NaN) values

All of the methods shown above exclude missing values in their calculations by default. If you do not want to exclude NaNs, you can do that with the skipna or dropna keywords (in most methods).

For a complete overview of how pandas treats missing data and where you should be aware of implicit behavior, have a look at the pandas docs.

Resampling

Another often needed functionality when working with time series data is provided by the resample() method. You can use it to resample your data to different temporal resolutions.

The resample() function returns a Resampler object which in turn provides many functions for the specific resampling method.

For example, we can average the Frankfurt Airport data on a 3 hour interval:

But we are not limited to calculating the mean over these intervals. We can also calculate the sum or any other function we like:

.sum() and .mean() are convenient to calculate the same statistic for all columns of the data frame. Let’s say we want to calculate the mean of the temperature but the maximum of the cloud height. Instead of selecting each column into a new variable and applying the corresponding function to each we can use the .agg() method of the data frame.

We can also resample to a higher temporal resolution than the original data set and provide an interpolation technique for time steps between known measurement points:

Write data

If you want to save your data, you can use one of the many writers of pandas. A full list of available writers is given here.

So far, you have learned how to read, manipulate and write data in tabular shape. But what if you want to get a visual representation of your data?

Although pandas has the ability to plot the data in graphs as well, we use specific plotting libraries for this purpose. There are many plotting libraries for virtually any data visualization need. Some of these libraries can be used regardless of the specific application, but many of them are intensely focused on accomplishing a particular task. A good overview is given on the FusionCharts Blog.

Here we will focus on the most basic (and oldest) plotting library called “matplotlib”.

matplotlib

is a very powerfull plotting library. This brings some complexity with it, which sometimes can be a little confusing.

So before we get to the actual plotting we have a look at how a plot is structured in general and the vocabulary associated with its elements.

For more details on matplotlib basics have a look at the Usage Guide.

The tutorial and examples pages are also worth looking at.

Source: https://www.machinelearningplus.com/wp-content/uploads/2019/01/99_matplotlib_structure-1.png

Source:https://matplotlib.org/_images/anatomy.png

Line plots

Let’s start with a simple example. We want to take a look at the time series of the temperature data in a simple line plot:

This gives us a nice little line plot with the temperature time series depicted as blue line. However, the plot is a little small, the axes are not labeled and there is no legend. Let’s fix that…

This looks better. We can see that we do not have to worry about the details of the axes and axis of the plot since pyplot takes care of it for us. Each time we do a function of pyplot it just reuses the axes from the first plot call.

However, let’s say we are not interested in the daily temperature fluctuations.

Additionally while manually constructing the xticks labels works fine, it is a little annoying. Especially since the index of the dataframe (which is used for the x-axis) already contains all information we need and just the formatting is wrong. Fortunately, matplotlib has some convenience functions available in its date submodule we can use if we switch to the “manual-axes” approach.

So let’s resample the data to daily means, format the dates and plot it again…

You can also put multiple lines into one plot:

And if we don’t like the colors or the line types, we can choose different ones by providing the c and linestyle arguments to the plotting function:

What if we want to plot 2 variables in the same plot that have different units, e.g. temperature and air pressure values?

For this, we can use the twinx() function. It creates a second subplot that shares the Y-Axis with the first subplot:

Normally, subplots are, however, used to plot multiple figures next to each other in one plot. For example, we can create 2 subplots “ax1” and “ax2”, that depict temperature and wind speeds next to each other:

If you want to save your figure, you can simply call the savefig() function instead of show():

Histograms

To get an overview of the distribution of temperature values, we can also plot a histogram of the data:

Again, this plot is nice and simple but we can do better…

We can also plot multiple histograms within one chart, e.g. the temperature values separated into January and August:

And we can draw multiple plot types within one plot. Let’s plot the winter and summer temperature counts as lines and the total temperature counts as bars:

Scatterplots

Another often used plot is the scatterplot.

Let’s use it to plot temperature vs. air pressure values:

We can also plot multiple scatterplots on top of each other to depict more information within one plot.

If we do this with temperature/pressure measurements separately for January and August, we can see that air pressures below 1000 hPa were only prevalent during January but not during August and that the air pressure value range was much larger during January than during August:

There are many more options that you can tweak in order to get a nice looking ready to print plot. And of course, there are many more plot types (e.g. bar plots, box plots, 3d plots, …) We won’t cover all of them here but you can have a look at the matplotlib documentation.

Also, there are many more plotting libraries out there which cover all kinds of useful plotting routines, e.g. for plotting wind data…

Windroses

Using the windrose extension, you can easily plot the wind data in a typical wind rose chart:

Info

• Actually the plotting functions in pandas use matplotlib in the background too. It just makes smart default assumptions to hide many of the cumbersome matplotlib tasks.
• There are other high level plotting libraries like:
  • Holoviews
  • hvplot (which is based on holoviews)

  They wrap matplotlib and many of the libraries you can see on the FusionCharts Blog. They make it easy and fast to explore data interactively because you don’t need all the “trouble” of plot configuration like with matplotlib.

So if you are still in the knowledge discovery and exploration phase using one of the high level libraries instead of matplotlib can save you a lot of time, but if you need to configure every detail of your plot there is often no way around the latter.