Python II

Intermediate Python

Training Session Date

Tue. Oct 20, 2026 – 6:00 pm

This notebook contains content for the Python II training session. We will cover the following topics:

Working With DataFrames: Data transformation
Inferential Statistics
Visualizations
Saving and exporting
Advanced Topics and Examples
Appendices

Download the Code and Dataset

You can download this notebook and open it in any Jupyter-compatible IDE, such as JupyterLab, VS Code, Positron, RStudio, or Spyder.

Ensure that you are running at least Python 3.11 and that the necessary package dependencies (pandas, numpy, pingouin, scipy, matplotlib, and others noted throughout the notebook) are installed.

PythonII_Training.ipynb

Python_Data.csv

Using Google Colab

This workbook can also be run in the cloud using Google Colab, with no local setup required. Google Colaboratory (Colab in short) is built on top of Jupyter notebooks and has added functionalities that are found in more conventional IDEs like PyCharm, RStudio, Spyder, etc.

Colab is entirely cloud-based and the free-tier even allows for some free GPU and TPU processing capabilities. Additionally, it can connect with your Google Drive storage for accessing and storing data.

Introduction

Jupyter Notebooks

Earlier called the IPython notebook (hence the .ipynb extension), Jupyter notebooks are an open-source web application that allows you to create and share documents that contain live code, equations, visualizations, and narrative text. They are widely used for data cleaning and transformation, numerical simulation, statistical modeling, data visualization, machine learning, etc.

Jupyter notebooks are a great way to learn and practice Python, as they allow you to run code in small chunks (called cells) and see the output immediately. You can edit a Jupyter notebook in your browser or in a program (called an IDE, short for Integrated Development Environment) like VS Code or RStudio. You can also run Jupyter notebooks in the cloud using Google Colab.

If you want to install Python and Jupyter notebooks on your local machine, we recommend you use an “environment manager” like Anaconda, venv, uv, or pyenv, which can help prevent package conflicts as you install more Python libraries in the future. If you don’t have Python installed on your machine, you can follow this guide to install Anaconda.

Common shortcuts

Jupyter Notebook cells have 2 modes, ‘Command’ and ‘Edit’.

To switch from command to edit mode press Enter. To switch from edit to command mode press Esc.

Execute cell: Ctrl+Enter Execute cell and move to next cell: Shift+Enter (This will add a new cell below if none exists) Execute cell and add new cell below: Alt+Enter

In command mode: - insert cell above: a - insert cell below: b - change cell type to Markdown: m (Ctrl+mm in Colab) - change cell type to code: y (Ctrl+my in Colab) - delete cell: dd (Ctrl+md in Colab)

Setup

Google Colab Setup

If you’re running this notebook in Google Colab, run the cell below to mount your Google Drive (you will need to grant Colab permission to access your Drive). This gives you a persistent place to store the workshop dataset between sessions — upload the Python_Data.csv file you downloaded above into the Colab Notebooks/pyworkshop folder in your Drive, creating it if it doesn’t already exist. If you’re running locally, just make sure Python_Data.csv is saved in the same folder as this notebook.

try:
    from google.colab import drive
    IN_COLAB = True
except ImportError:
    IN_COLAB = False

if IN_COLAB:
    drive.mount('/gdrive', force_remount=True)
    workshop_dir = '/gdrive/MyDrive/Colab Notebooks/pyworkshop'
else:
    workshop_dir = '.'

import os
print(f'Looking for the workshop dataset in: {workshop_dir}')

Looking for the workshop dataset in: .

Magic Commands

Jupyter notebooks allow you to execute certain operations outside of the Python environment. Often, we might need to directly access the system terminal for installing new packages and make other changes outside the environment. These are called magic commands and begin with !.

You can even turn an entire cell to access an outside environment by specifying it with the %% in the first line. Every subsequent command within the cell will refer to the external environment and not Python (without using !). For example, this is one way you could use the bash shell to inspect the workshop folder (this cell is illustrative and is not evaluated, since its output depends on your own folder structure):

%%bash
ls -l pyworkshop # Check the contents of the workshop folder

file_path = os.path.join(workshop_dir, 'Python_Data.csv')

Working With DataFrames: Data transformation

A majority of data analysis in Python will involve using the pandas library to deal with dataframes. It provides similar functionalities to R dataframes.

Let’s first import the GSS 2018 data.

import pandas as pd
my_data = pd.read_csv(file_path)

While working with data sets, we often need to create new variables or recode the current variable into new values. This can also be easily done with the help of pandas.

Creating a new variable

To create a new variable in Python, you can directly use varname = to specify this new variable.

But to create a new variable in a dataframe, you have to use the object name and the dataframe to specify the name of the created variable.

For instance, you need to type my_data['birth_year'] = rather than 'birth_year' = so that the system understands the newly created variable is an element of my_data rather than a stand-alone variable.

Let’s compute people’s birth_year based on their age.

my_data['birth_year'] = 2018 - my_data['age']

# use describe() to see the descriptive statistics
my_data['birth_year'].describe()

count    2341.000000
mean     1969.028620
std        18.060879
min      1929.000000
25%      1955.000000
50%      1970.000000
75%      1984.000000
max      2000.000000
Name: birth_year, dtype: float64

You can also check whether the new variable has been created successfully by previewing the dataframe.

# Display all the columns
pd.set_option('display.max_columns', None)

my_data.head()

	id	age	year	wrkstat	hrs1	marital	degree	sphrs1	postlife	sex	SIBS	agekdbrn	othlang	born	neisafe	hsbio	hschem	tspac	nextgen	grass	hrlax	colsci	fmi	numpets	trust	tvhours	sop	ssi	sphrs2	educ	moos	faos	realrinc	wordsum	rhs	Male	pres	birth_year
0	1562	NaN	2018	working fulltime	50.0	1.0	high school	40.0	yes	1	3.0	NaN	no	yes	very safe	NaN	NaN	NaN	NaN	NaN	2.0	NaN	63560.0	1.0	can't be too careful	1.0	64.0	82.0	NaN	14.0	53.0	NaN	45400.0	3.0	NaN	male	46.0	NaN
1	570	NaN	2018	working fulltime	45.0	1.0	bachelor	NaN	NaN	2	3.0	NaN	no	no	somewhat safe	yes	yes	about right	agree	NaN	4.0	no	NaN	NaN	can't be too careful	1.0	44.0	56.0	NaN	16.0	NaN	59.0	NaN	8.0	NaN	female	75.0	NaN
2	1473	NaN	2018	working fulltime	40.0	3.0	junior college	NaN	yes	1	5.0	23.0	no	yes	somewhat safe	yes	no	too much	agree	legal	0.0	NaN	14755.0	NaN	can't be too careful	2.0	NaN	NaN	NaN	14.0	48.0	NaN	14755.0	7.0	good	male	35.0	NaN
3	1963	NaN	2018	working fulltime	40.0	1.0	graduate	NaN	yes	1	7.0	NaN	yes	yes	somewhat safe	yes	no	NaN	agree	NaN	4.0	yes	NaN	NaN	NaN	1.0	27.0	14.0	NaN	20.0	27.0	47.0	NaN	5.0	NaN	male	64.0	NaN
4	419	NaN	2018	working fulltime	35.0	1.0	junior college	40.0	yes	2	5.0	23.0	yes	no	very safe	NaN	NaN	NaN	NaN	not legal	2.0	NaN	24970.0	4.0	can trust	1.0	46.0	52.0	NaN	14.0	42.0	49.0	24970.0	3.0	NaN	female	47.0	NaN

Recoding variables

Recoding variables with pandas is simple.

The recode procedure transforms an existing variable into a new variable by changing, rearranging, or consolidating the values of the existing variable. It allows the user to:

Change an existing categorical variable into a new categorical variable with a smaller number of categories/levels.
Change an existing continuous variable with many unique values (e.g, income, age, etc.) into a categorical variable with fewer levels (for example, if recoded, the variable income will have the values low, medium, and high, while the variable age will have young, adult, and senior as values).
Code missing values.
Replace miscoded values with correct values.
Create a dummy variable based on a cutoff value.

To recode a variable, one or more of Python’s logical operators are used. You may refer to our Python I Notebook for common logical operators.

For instance, if you want to recode age based on the following rule:

age <= 21: non-adult;
age > 21: adult

You can use the np.where function:

import numpy as np
my_data['age_group'] = np.where(my_data['age'] > 21, "adult", 'non-adult')

# You can use value_counts() to get the frequency of each value
my_data['age_group'].value_counts()

age_group
adult        2251
non-adult      97
Name: count, dtype: int64

If the original variable is categorical, we can use replace function to recode the values.

For example, let’s check the variable ‘sex’. Let’s recode it using the following rule:

1 –> 'Male'
2 –> 'Female'

my_data['sex'].value_counts()

sex
2    1296
1    1052
Name: count, dtype: int64

my_data['sex'] = my_data['sex'].replace({1:"Male", 2:"Female"})

my_data['sex'].value_counts()

sex
Female    1296
Male      1052
Name: count, dtype: int64

Exercise 1

Recode marital using the following rule:

1: 'MARRIED', 2: 'WIDOWED', 3: 'DIVORCED', 4: 'SEPARATED', 5: 'NEVER MARRIED', 9: np.nan

# Note: notice how we recode 9 into missing values here

# Use isna() to detect where a value is missing and then sum() to get total number of missing values in the column
my_data['marital_labeled'].isna().sum()

np.int64(2)

Inferential Statistics

!pip install -q pingouin

import pingouin as pg
import scipy.stats as stats

Contingency table and chi-square test

The crosstab() function creates a contingency table.

A contingency table is used to show the joint distribution of cases over two or more categorical variables. The basic syntax to produce a contingency table,mytab = pd.crosstab(var1, var2), will result in a table with var1 in the rows and var2 in the columns.

Let’s create a contingency table using the variables marital_labeled and sex.

# Contingency table
# A contingency table is used to show the joint...
# ...distribution of cases over two or more categorical variables.
# The basic syntax to produce a contingency table is
# mytab = pd.crosstab(var1 (index), var2 (columns))
# For example, var1 = marital and var2 = sex

mytable = pd.crosstab(my_data["sex"], my_data["marital_labeled"])
mytable

marital_labeled	DIVORCED	MARRIED	NEVER MARRIED	SEPARATED	WIDOWED
sex
Female	242	530	335	43	145
Male	161	468	335	32	55

If you wish to see the percentage, you can do it by specifying the value of normalize to index or column or all.

# To see the percentage
mytable_percentage = pd.crosstab(my_data["sex"], my_data["marital_labeled"], normalize = 'all')
mytable_percentage

marital_labeled	DIVORCED	MARRIED	NEVER MARRIED	SEPARATED	WIDOWED
sex
Female	0.103154	0.225916	0.142796	0.018329	0.061807
Male	0.068627	0.199488	0.142796	0.013640	0.023444

If you would like to run statistical tests and analyses like Pearson’s χ2 test on the two variables, we can use the chi2_independence function from the pingouin library.

expected, observed, stat_vals = pg.chi2_independence(my_data, x='sex', y='marital_labeled')
stat_vals
# print(f'Expected values: {expected}')
# print(f'Observed values: {observed}')

	test	lambda	chi2	dof	pval	cramer	power
0	pearson	1.000000	37.270945	4.0	1.584012e-07	0.125990	0.999621
1	cressie-read	0.666667	37.585898	4.0	1.364070e-07	0.126521	0.999654
2	log-likelihood	0.000000	38.432921	4.0	9.122259e-08	0.127939	0.999729
3	freeman-tukey	-0.500000	39.277019	4.0	6.106354e-08	0.129336	0.999788
4	mod-log-likelihood	-1.000000	40.322018	4.0	3.712969e-08	0.131045	0.999844
5	neyman	-2.000000	43.116519	4.0	9.787517e-09	0.135510	0.999932

Exercise 2

Find out if there is a significant association between country of birth born and speaking a language other than English othlang.

# Your code here
# Step 1: perform test


# Step 2: interpret the result

Independent samples t-test

An independent samples t-test determines whether there is a significant difference in the means of a continuous variable between two groups.

For this example, we’ll test whether there is a significant difference in the responses between men and women sex and how much time they have to relax per day hrlax

Before performing a t-test, we need to compare the variances of two samples. Different t-test methods make different assumptions about whether or not the variances of the two samples differ. To test for equality of variances, we will perform a Levene’s Test using the stats.levene function of the scipy library.

lev, p = stats.levene(my_data[my_data['sex']=='Male']['hrlax'],
                my_data[my_data['sex']=='Female']['hrlax'], nan_policy='omit')
# Notice that we include an addiitonal argument to handle missing data

print(f'The p-value of the Levene test is {p}')

The p-value of the Levene test is 0.39502551500197014

The Levene method tests the null hypothesis that the variances are equal. Because the significance of Levene’s test in this example is larger than 0.05, we fail to reject the null hypothesis, and the assumption of equal variance is met. In such a case, the statistics for equal variances assumed should be used.

We’ll use the pg.ttest function to test whether there is a significant difference in means between the two groups. Notice that it includes a parameter called correction. It is recommended to set it to auto, which will automatically use Welch T-test when the equal variances assumption is violated.

# Running an independent samples t-test

pg.ttest(my_data[my_data['sex']=='Male']['hrlax'],
         my_data[my_data['sex']=='Female']['hrlax'],
         correction='auto')

	T	dof	alternative	p_val	CI95	cohen_d	power	BF10
T_test	3.19478	1383.326605	two-sided	0.001431	[0.18, 0.76]	0.170868	0.892126	9.194

Exercise 3

Compare average income of male and female using the variable sex_new and realrinc and interpret the result.

# your code here

# step 1: get the mean of realrinc for both female and male separately



# step 2: perform t test



# step 3: interpret the result

ANOVA

A one-way ANOVA (analysis of variance) is used to determine whether there is a statistically significant difference in the mean of a continuous dependent variable among more than two groups.

For this example, we’ll test whether there is a significant difference in means in the hours spent watching TV tvhours between different marital statuses marital_labeled.

We will again need to test for equality of variances among groups using the stats.levene function.

We can then use either pg.anova or pg.welch_anova functions.

lev, p = stats.levene(my_data[my_data['marital_labeled']=='MARRIED']['tvhours'],
                      my_data[my_data['marital_labeled']=='WIDOWED']['tvhours'],
                      my_data[my_data['marital_labeled']=='DIVORCED']['tvhours'],
                      my_data[my_data['marital_labeled']=='SEPARATED']['tvhours'],
                      my_data[my_data['marital_labeled']=='NEVER MARRIED']['tvhours'],
                      nan_policy='omit')

print(f'The p-value of the Levene test is {p}')

The p-value of the Levene test is 0.0001026876401388878

# Conducting a one-way ANOVA

# If the Levene's test was not significant, that is, equal variances are assumed:

aov = pg.anova(data=my_data, dv='tvhours', between='marital_labeled')
print('One-way ANOVA\n',aov)


# If the Levene's test was significant, that is, equal variances are not assumed:

welchaov = pg.welch_anova(data=my_data, dv='tvhours', between='marital_labeled')
print('\n Welch ANOVA\n',welchaov)

One-way ANOVA
             Source  ddof1  ddof2        F     p_unc       np2
0  marital_labeled      4   1549  8.33143  0.000001  0.021061

 Welch ANOVA
             Source  ddof1       ddof2        F     p_unc       np2
0  marital_labeled      4  217.388472  7.64166  0.000009  0.021061

While the ANOVA procedure determines whether differences exist among the group means, post hoc tests are needed to determine which means differ from one another. Which post hoc test you use depends on the homogeneity of the variances.

# Conducting post-hoc tests

# Tukey's HSD

# pg.pairwise_tukey(data=my_data, dv='tvhours', between='marital_labeled').round(3)

# Games-Howell test

pg.pairwise_gameshowell(data=my_data, dv='tvhours', between='marital_labeled').round(3)

	A	B	mean_A	mean_B	diff	se	T	df	pval	hedges
0	DIVORCED	MARRIED	3.130	2.580	0.551	0.201	2.734	410.082	0.051	0.220
1	DIVORCED	NEVER MARRIED	3.130	3.014	0.116	0.240	0.483	621.075	0.989	0.037
2	DIVORCED	SEPARATED	3.130	3.128	0.002	0.645	0.003	44.881	1.000	0.001
3	DIVORCED	WIDOWED	3.130	4.029	-0.899	0.338	-2.662	249.378	0.063	-0.284
4	MARRIED	NEVER MARRIED	2.580	3.014	-0.435	0.177	-2.450	662.939	0.104	-0.166
5	MARRIED	SEPARATED	2.580	3.128	-0.549	0.624	-0.879	39.442	0.903	-0.236
6	MARRIED	WIDOWED	2.580	4.029	-1.450	0.296	-4.890	159.862	0.000	-0.597
7	NEVER MARRIED	SEPARATED	3.014	3.128	-0.114	0.638	-0.179	42.969	1.000	-0.035
8	NEVER MARRIED	WIDOWED	3.014	4.029	-1.015	0.324	-3.132	221.964	0.017	-0.315
9	SEPARATED	WIDOWED	3.128	4.029	-0.901	0.680	-1.325	55.055	0.677	-0.261

Exercise 4

Test whether the income differs significantly among different education levels using the variable degree and realrinc.

Then perform ANOVA test and interpret the result

# your code here


# step 1: conduct Levene's test



# step 2: perform ANOVA



# step 3: interpret the result

Correlation Matrix

Correlation is a measure of the linear relationship between two continuous variables. A correlation coefficient ranges from -1 to 1. A positive correlation coefficient indicates a positive linear relationship (i.e., as one variable increases, the other tends to as well). On the other hand, a negative correlation coefficient indicates a negative linear relationship. A correlation coefficient of 0 indicates there is no linear relationship between the two variables.

# Get the correlation matrix between age, realinc, hrs1, and tvhours

my_data[['age','realrinc','hrs1','tvhours']].corr()

	age	realrinc	hrs1	tvhours
age	1.000000	0.182487	-0.086195	0.212004
realrinc	0.182487	1.000000	0.225945	-0.132785
hrs1	-0.086195	0.225945	1.000000	-0.150086
tvhours	0.212004	-0.132785	-0.150086	1.000000

# Correlation matrix with significance levels

my_data[['age','realrinc','hrs1','tvhours']].rcorr()

	age	realrinc	hrs1	tvhours
age	-	***	**	***
realrinc	0.182	-	***	***
hrs1	-0.086	0.226	-	***
tvhours	0.212	-0.133	-0.15	-

Linear Regression

Linear regression is used to assess the association between one or more independent variables (continuous or categorical) and a continuous dependent variable.

We’ll use the pg.linear_regression function to fit a linear regression model to test the association between the family income realrinc and other independent variables.

# Simple or bivariate regression with age as the only IV

lm1 = pg.linear_regression(my_data['age'], my_data['realrinc'], remove_na=True)
lm1

	names	coef	se	T	pval	r2	adj_r2	CI2.5	CI97.5
0	Intercept	8620.201916	2516.279825	3.425772	6.313763e-04	0.033301	0.032588	3683.978081	13556.425751
1	age	368.136546	53.863408	6.834632	1.238923e-11	0.033301	0.032588	262.471892	473.801200

# Multiple linear regression with age and occupational prestige score as IVs

lm2 = pg.linear_regression(my_data[['age','pres']], my_data['realrinc'], remove_na=True)
lm2

	names	coef	se	T	pval	r2	adj_r2	CI2.5	CI97.5
0	Intercept	-16646.417617	3186.708681	-5.223702	2.029766e-07	0.126316	0.12502	-22897.864927	-10394.970306
1	age	285.289614	51.554707	5.533726	3.762295e-08	0.126316	0.12502	184.153436	386.425792
2	pres	635.072654	53.004688	11.981443	1.661409e-31	0.126316	0.12502	531.092013	739.053296

Visualizations

Histogram

A histogram visually displays the distribution of a continuous variable. In Python, the plot.hist function works as shown below.

# Graphics
# Histogram
my_data['age'].plot.hist(title = 'Age Distribution',
                         color = 'green',
                         bins = 20)

Bar graphs

A bar graph can display the frequencies or distribution of a categorical variable. First, a data table must be created for plotting. In this case, we name our data table sexcount, then plot the bar graph of the sex variable by using the plot.bar function.

# Bar Graph
sexcount = my_data['sex'].value_counts()
sexcount.plot.bar(title = 'Sex counts',
                 color = 'blue')

A grouped bar graph is a bar graph that uses an additional variable to group the plotted data. For example, you can create a grouped bar graph displaying sex and marital status. If you would like to plot the grouped bar plot of sex and marital variables, you will need to first use the crosstab command to get the frequencies and then use the plot.bar function.

# Grounded bar graph
groupbar = pd.crosstab(my_data['sex'], my_data['marital'])

groupbar.plot.bar(title = 'Sex vs. Marital',
                 color = ['sandybrown','salmon','skyblue','teal','orchid','cornflowerblue'])


# Named colors https://matplotlib.org/stable/gallery/color/named_colors.html

Exercise 5

Create a grouped bar chart to show the number of respondents across the variable degree and sex. The x is degree, y is the frequency and the group is sex

# your code here

# step 1: get the crosstable



# step 2: plot the grouped bar chart

Scatterplot

A scatterplot plots the “positions” of two variables in dimensions x and y. Let’s plot the the relationship between age and weekly working hours.

# Scatterplot
my_data.plot.scatter(x = 'age', y = 'hrs1',
                     title = 'The relationship between age and the weekly working hours')

Using Matplotlib library for advanced visualizations

When it comes to more complex visualizations, or when you want to customize your plots, you can use the Matplotlib library. Matplotlib is a comprehensive library for creating static, animated, and interactive plots in Python. It is a powerful library that can be used to create a wide range of plots, including histograms, scatter plots, bar charts, pie charts, and more.

import matplotlib.pyplot as plt

Scatterplot

# create a scatter plot where x being age and y being hrs1
plt.scatter(my_data['age'], my_data['hrs1'], color = 'blue')

# create a scatter plot where x being age and y being hrs1, also differentiating the color by gender
colors = {'Male': 'blue', 'Female': 'red'}

plt.scatter(my_data['age'], my_data['hrs1'],
            c=my_data['sex'].map(colors), label=my_data['sex'])

Line chart

# group by age and calculate the mean of hrs1 and mean of sphrs1
hr_by_age = my_data.groupby('age').agg({'hrs1':'mean'}).reset_index()

plt.plot(hr_by_age['age'], hr_by_age['hrs1'])
plt.show()

Multiple charts

hr_by_age = my_data.groupby('age').agg({'hrs1':'mean','sphrs1':'mean'}).reset_index()
hr_by_age = hr_by_age[hr_by_age['age'] != 99]

plt.plot(hr_by_age['age'], hr_by_age['hrs1'], label = 'hrs1')
# add title
plt.title('The relationship between age and the weekly working hours')
# add x and y axis labels
plt.xlabel('Age')
plt.ylabel('Average Weekly working hours')
# add points
plt.scatter(hr_by_age['age'], hr_by_age['hrs1'], color = 'blue')
# add a new line for sphrs1 using dotted line
plt.plot(hr_by_age['age'], hr_by_age['sphrs1'], label = 'sphrs1', color = 'red', linestyle = '--')
# add points for sphrs1 using a different triangle marker
plt.scatter(hr_by_age['age'], hr_by_age['sphrs1'], color = 'red', marker = '^')
# add legend by color
plt.legend()
plt.show()

plt.figure()
plt.subplot(211) # plt.subplot(nmk) where n is the number of rows, m is the number of columns, and k is the plot number
plt.plot(hr_by_age['age'], hr_by_age['hrs1'], label = 'hrs1')
# add title
plt.title('Self weekly working hours')
# add x and y axis labels
plt.xlabel('Age')
plt.ylabel('Average Weekly working hours')
# add points
plt.scatter(hr_by_age['age'], hr_by_age['hrs1'], color = 'blue')
# add some blank space between the two plots
plt.subplots_adjust(hspace = 0.5)

plt.subplot(212)
plt.plot(hr_by_age['age'], hr_by_age['sphrs1'], label = 'sphrs1', color = 'red', linestyle = '--')
plt.title('Spouse weekly working hours')
# add x and y axis labels
plt.xlabel('Age')
plt.ylabel('Average Weekly working hours')
# add points for sphrs1 using a different triangle marker
plt.scatter(hr_by_age['age'], hr_by_age['sphrs1'], color = 'red', marker = '^')
plt.show()

income_by_age_gender = my_data.groupby(['age','sex']).agg({'realrinc':'mean'}).reset_index()
income_by_age_gender

	age	sex	realrinc
0	18.0	Female	2497.000000
1	18.0	Male	1475.500000
2	19.0	Female	3015.714286
3	19.0	Male	2421.333333
4	20.0	Female	1437.666667
...	...	...	...
139	87.0	Male	NaN
140	88.0	Female	NaN
141	88.0	Male	NaN
142	89.0	Female	NaN
143	89.0	Male	54480.000000

144 rows × 3 columns

income_by_age_male = income_by_age_gender[income_by_age_gender['sex'] == 'Male']
income_by_age_female = income_by_age_gender[income_by_age_gender['sex'] == 'Female']

plt.plot(income_by_age_male['age'], income_by_age_male['realrinc'], label = 'Male')
# add title
plt.title('The relationship between age and the income by different sex')
# add x and y axis labels
plt.xlabel('Age')
plt.ylabel('Income')
# add points
plt.scatter(income_by_age_male['age'], income_by_age_male['realrinc'], color = 'blue')
# add a new line for sphrs1 using dotted line
plt.plot(income_by_age_female['age'], income_by_age_female['realrinc'], color = 'red', linestyle = '--', label = 'Female')
# add points for sphrs1 using a different triangle marker
plt.scatter(income_by_age_female['age'], income_by_age_female['realrinc'], color = 'red', marker = '^')
# add legend by color
plt.legend()
plt.show()

Saving and exporting

If you wish to export the data as a .csv file, you can use df.to_csv (export_file_path, index = False, header=True) to save your data set.

my_data.to_csv('exportmydata.csv', index = False, header=True)

Advanced Topics and Examples

While Python can handle most common statistical analyses, it is first and foremost a general purpose programming language. To truly leverage the unique advantages and flexibility of Python, knowledge of some additional programming concepts is necessary.

Control Flow - Loops

We often need to execute a set of commands/operations repeatedly. Instead of writing out the same set of code multiple times, we can encapsulate the piece of code within a looping code block.

There are 2 types of loops in Python.

The for loop is used when the number of iterations is predetermined and fixed before the looping block is executed. The loop ends when the last iteration is completed.

The while loop is used when the number of iterations is not predetermined. The loop ends when a specific condition is met. Hence, the number of iterations is not fixed.

a_num = [2,4,6,2,8,2,6,7,3,1]

for i in a_num:
  print(i)

for i in range(10):
  print(i)

c = 0
while c<len(a_num):
  print(a_num[c])
  c+=1


# Sometimes it may be necessary to enter the loop at least once and check the
# exit condition at the very end instead of the beginning
c = 0
while True:
  print(a_num[c])
  c+=1
  if c>=len(a_num): break

Functions

Functions are a set of well-defined and self-contained instructions that accomplishes a particular task. These are smaller modules of code that can be invoked from anywhere within the rest of programming code. Any Python command that is followed by () is a function.

Python functions start with the keyword def which means you are defining your own functions.

Next, it is followed by the function name and a set of parentheses, within which the function inputs are specified. After the inputs are specified, you go in to the function body. Note that the indentation always matters and your function body cannot be aligned with the keyword def. At the end of your function, if you need your function to return some outputs, you also need to specify what you want it to return. Functions in Python usually look something like this:

def FunctionName(input1, input2, input3):
    
    FunctionConditions
    
    return output

Now let’s look at an example of recoding the age variable into three categories, Young, Middle Aged, and Elder. To create this new variable, we only need to know the original numeric age values.

Now, we create our own function called transform_age which takes the numeric value of age as the function input. Within this function, the function body is a series of conditional statements of our coding schema. Since the recode variable is a string variable, we ask this transform_age function to return the corresponding categories.

Once we have this function, we can use the apply command to apply this function to the age variable and get what we want. We can use the unique() command to check that our transformation is indeed what we hoped for.

# When we have more than 2 categories (e.g., age)
# We will need to write a function to implement

def recode_age(age):
    if age > 75:
        return 'Elder'
    elif age >45:
        return 'Middle Aged'
    else:
        return 'Young'

# Check if function works
print(recode_age(80))
print(recode_age(50))

Elder
Middle Aged

# Add a new column for the new categories
my_data['age_cat'] = my_data['age'].apply(recode_age)
my_data['age_cat']

0       Young
1       Young
2       Young
3       Young
4       Young
        ...  
2343    Elder
2344    Elder
2345    Elder
2346    Elder
2347    Elder
Name: age_cat, Length: 2348, dtype: str

# get the unique values in the column
my_data['age_cat'].unique()

<StringArray>
['Young', 'Middle Aged', 'Elder']
Length: 3, dtype: str

# Count frequencies
my_data['age_cat'].value_counts()

age_cat
Young          1084
Middle Aged    1079
Elder           185
Name: count, dtype: int64

Appendices

Appendix A: Download a Copy of this Workbook

PythonII_Training.ipynb

Appendix B: Example of Combining Python Concepts

The topics covered across Python I and Python II cover most of the fundamental concepts, commands, and data structures that underlie every Python module, library, or program.

Let’s look at an example of how we can combine all of these concepts to accomplish more complex tasks by scraping Wikipedia pages and generating wordclouds.

This example relies on access to Wikipedia and the bs4, wordcloud, and nltk packages. The code below is not run in this workbook, but you can run it on your machine if you have internet access and the necessary packages installed.

# Load necessary libraries
# Corresponding documentation websites provided
import requests # https://requests.readthedocs.io/en/latest/
from bs4 import BeautifulSoup as bs # https://beautiful-soup-4.readthedocs.io/en/latest/
from wordcloud import WordCloud, STOPWORDS # https://amueller.github.io/word_cloud/index.html
import matplotlib.pyplot as plt # https://matplotlib.org/stable/index.html
from nltk.corpus import stopwords
import nltk # https://www.nltk.org/
import re # https://docs.python.org/3/library/re.html
import pandas as pd # https://pandas.pydata.org/docs/

# Download NLTK stopwords
nltk.download('stopwords')

# Function to get text from a Wikipedia page

def get_wiki_text(url):
    # Websites often require headers like user agent, cookies etc. to mask automated crawling activity.
    # Use this Google search query to find your current browser agent string https://www.google.com/search?q=my+browser+user+agent
    # There are more parameters that can be included in the header https://requests.readthedocs.io/en/latest/user/advanced/
    header = {'User-Agent':'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:143.0) Gecko/20100101 Firefox/143.0'}

    response = requests.get(url, headers=header) # Sending an automated request and storing the data
    soup = bs(response.text, 'html.parser') # Parsing the response systematically
    paragraphs = soup.find_all('p') # Storing only text content
    text = ' '.join([p.text for p in paragraphs]) # Combining all paragraphs into a single document
    return text

languages = [
    'Python_(programming_language)',
    'R_(programming_language)',
    'JavaScript',
    'Java_(programming_language)'
]

# Dictionaries for storing text and wordcounts for each language as key:value pairs
language_word_counts = {}
language_texts = {}

for language in languages:
    # Create the Wikipedia URL
    url = f'https://en.wikipedia.org/wiki/{language}'

    # Get the text content
    text = get_wiki_text(url)
    language_texts[language] = text

    # Clean text - remove numbers and special characters
    cleaned_text = re.sub(r'[^a-zA-Z\s]', '', text)

    # Count unique words that are at least 3-letters long
    words = cleaned_text.lower().split()
    words = list(set(words))
    word_count = 0
    for word in words:
        if len(word) > 3 and word not in stopwords.words('english'):
            word_count += 1

    # Store the word count
    language_word_counts[language] = word_count
    print(f"{language.replace('_(programming_language)', '')}: {word_count} words")

# Function to generate and display wordcloud
def generate_wordcloud(text, title):
    combined_stopwords = STOPWORDS.union(set(stopwords.words('english')))
    wordcloud = WordCloud(
        stopwords=combined_stopwords,
        background_color='white',
        max_words=100,
        width=800,
        height=400
    ).generate(text)

    plt.figure(figsize=(10, 5))
    plt.imshow(wordcloud, interpolation='bilinear')
    plt.axis('off')
    plt.title(title)
    plt.show()

# Generate word clouds for each language
for language, text in language_texts.items():
    display_name = language.replace('_(programming_language)', '')
    generate_wordcloud(text, f"Word Cloud for {display_name}")

from scipy.stats import chi2_contingency

# Create a simple contingency table for two words - programming and object
words_to_check = ["programming", "object"]
contingency_table = []

for language in languages:
    # Clean text and convert to lowercase
    cleaned_text = re.sub(r'[^a-zA-Z\s]', '', language_texts[language]).lower()

    # Count occurrences of each word
    programming_count = cleaned_text.count(" programming ")
    object_count = cleaned_text.count(" object ")

    # Add to contingency table
    contingency_table.append([programming_count, object_count])

# Create a readable DataFrame for display
language_names = [lang.replace('_(programming_language)', '') for lang in languages]
contingency_df = pd.DataFrame(contingency_table,
                             index=language_names,
                             columns=words_to_check)

print("Contingency Table:")
print(contingency_df)

# Perform chi-square test using scipy library
chi2, p, dof, expected = chi2_contingency(contingency_table)

print("\nChi-square Independence Test Results:")
print(f"Chi-square statistic: {chi2:.4f}")
print(f"p-value: {p:.4f}")
print(f"Degrees of freedom: {dof}")
print(f"Interpretation: {'Word usage differs significantly across languages' if p < 0.05 else 'No significant difference in word usage across languages'}")

Appendix C: Common external libraries and frameworks across various domains

A non-comprehensive list of useful Python libraries with documentation links for further exploration.

Data Analysis & Statistics

pandas: Data manipulation and analysis
NumPy: Numerical computing with arrays and matrices
SciPy: Scientific computing (statistics, optimization, signal processing)
statsmodels: Statistical models and hypothesis testing
pingouin: Statistical analyses with simple API
pyMC: Bayesian statistical modeling and probabilistic machine learning

Data Visualization

Matplotlib: Basic plotting library
Seaborn: Statistical data visualization based on matplotlib
Plotly: Interactive visualizations
Folium: Interactive maps

Machine Learning & AI

scikit-learn: Classical machine learning algorithms
TensorFlow: Deep learning framework by Google
PyTorch: Deep learning framework by Facebook/Meta
Keras: High-level neural networks API
XGBoost: Gradient boosting implementation
Hugging Face Transformers: State-of-the-art NLP models

Natural Language Processing

NLTK: Natural Language Toolkit for text processing
spaCy: Industrial-strength NLP
gensim: Topic modeling and document similarity
TextBlob: Simplified text processing

Web Scraping & APIs

Requests: HTTP library for API calls
Beautiful Soup: HTML/XML parsing library
Scrapy: Web scraping framework
Selenium: Browser automation

Physical Sciences & Engineering

Astropy: Astronomy and astrophysics
Biopython: Biological computation
MDAnalysis: Molecular dynamics analysis
PyMOL: Molecular visualization
RDKit: Cheminformatics and computational chemistry
PyCaret: Low-code machine learning for scientific data
scikit-image: Image processing for scientific data

Bioinformatics & Genomics

Biopython: Processing biological data
scikit-bio: Bioinformatics algorithms
Pysam: Reading/writing genomic data formats
Bioconductor: (via rpy2) Genomic data analysis

Finance & Economics

pandas-datareader: Access to financial data
pyfolio: Portfolio and risk analytics
TA-Lib: Technical analysis library
Zipline: Algorithmic trading
arch: ARCH models for financial time series
statsmodels: Time series analysis
scikit-learn: Predictive modeling for financial data

Geospatial Analysis

GeoPandas: Geographic data handling
Rasterio: Raster data access
Shapely: Manipulation of geometric objects
PySAL: Spatial data science
Cartopy: Cartographic tools

Deep Learning Applications

OpenCV (cv2): Computer vision
librosa: Audio analysis
Transformers: NLP models
fastai: Vision, text, tabular data
Detectron2: Object detection
Kornia: Computer vision

Web Development

Django: Full-featured web framework
Flask: Lightweight web framework
Streamlit: Data apps with minimal code
Dash: Interactive analytic web applications
Gradio: Create UIs for machine learning models

Other Useful Libraries

tqdm: Progress bars for loops
pytest: Testing framework
Dask: Parallel computing
Ray: Distributed computing
SQLAlchemy: SQL toolkit and ORM
Pillow: Image processing
PyQt/PySide: GUI development
pygame: Game development

Download the Code and Dataset

Introduction

Common shortcuts

Setup

Working With DataFrames: Data transformation

Creating a new variable

Recoding variables

Exercise 1

Inferential Statistics

Contingency table and chi-square test

Exercise 2

Independent samples t-test

Exercise 3

ANOVA

Exercise 4

Correlation Matrix

Linear Regression

Visualizations

Histogram

Bar graphs

Exercise 5

Scatterplot

Using Matplotlib library for advanced visualizations

Scatterplot

Line chart

Multiple charts

Saving and exporting

Advanced Topics and Examples

Control Flow - Loops

Functions

Appendices

Appendix A: Download a Copy of this Workbook

Appendix B: Example of Combining Python Concepts

Appendix C: Common external libraries and frameworks across various domains

Data Analysis & Statistics

Data Visualization

Machine Learning & AI

Natural Language Processing

Web Scraping & APIs

Social Sciences

Physical Sciences & Engineering

Bioinformatics & Genomics

Finance & Economics

Geospatial Analysis

Deep Learning Applications

Web Development

Other Useful Libraries