A guide to structuring your group projects
Summary: In your group projects you will solve a data science problem end-to-end, pretending to be recently hired data scientists in a company. To help you get started, we've prepared a checklist to guide you through the project. Here are the main steps that you will go through:
- Frame the problem and look at the big picture
- Get the data
- Explore and visualise the data to gain insights
- Prepare the data to better expose the underlying data patterns to machine learning algorithms
- Explore many different models and short-list the best ones
- Fine-tune your models
- Present your solution
In each step we list a set of questions that one should have in mind when undertaking a data science project. The list is not meant to be exhaustive, but does contain a selection of the most important questions to ask. We will be available to provide assitance with each of the steps, and will allocate some part of each lesson towards working on the projects.
Our expectations
To streamline the grading, your group must submit a single Jupyter notebook, structured in terms of the first 6 sections listed in this guide (the seventh will be presented to the class). You are welcome to adapt code from the web (e.g. Kaggle kernels), but you must reference the original source in your notebook.
In addition to clean, well-documented code (i.e. functions with docstrings etc), your notebook will be judged according to how well each step is explained (using Markdown). The main goal is to simulate what it is like to work as a data scientist, where communication is arguably as important as the ability to extract insights from data.
The analysis in the Jupyter notebook will be evaluated according to a rubric similar to the assignments:
| Critical Task | Needs Improvement | Basic | Surpassed |
|---|---|---|---|
| Computation: Perform computations | Computations contain errors and extraneous code | Computations are correct but contain extraneous/unnecessary computations | Computations are correct and properly identified and labeled |
| Analysis: Choose and carry out analysis appropriate for data and context | Choice of analysis is overly simplistic, irrelevant, or missing key components | Analysis appropriate, but incomplete, or important features and assumptions not made explicit | Analysis appropriate, complete, advanced, relevant, and informative |
| Synthesis: Identify key features of the analysis, and interpret results (including context) | Conclusions are missing, incorrect, or not made based on results of analysis | Conclusions reasonable, but is partially correct or partially complete | Make relevant conclusions explicitly connected to analysis and to context |
| Visual presentation: Communicate findings graphically clearly, precisely, and concisely | Inappropriate choice of plots; poorly labeled plots; plots missing | Plots convey information correctly but lack context for interpretation | Plots convey information correctly with adequate/appropriate reference information |
| Written: Communicate findings clearly, precisely, and concisely | Explanation is illogical, incorrect, or incoherent | Explanation is partially correct but incomplete or unconvincing | Explanation is correct, complete, and convincing |
Grading split: The group project accounts for 50% of the final grading and is split equally between the notebook (25%) and the presentation (25%).
Submission deadline: Thursday, May 28, 2020 before 23:59:59 CEST (Notebook + presentation recording)
Presentation date: Thursday, June 4, 2020 (Discussion of group projects with questions)
Deliverables
The teams have to submit two deliverables before the submission deadline: 1) a notebook and 2) presentation video.
Notebook
The notebook contains all the code to explore the dataset, train the final model and documents each step clearly. If code is copied from another codebase such as Github or Stack Overflow it must be properly referenced.
Presentation
The presentation video should be 15min long and should highlight the problem you are solving, interesting things you found in the data and the step involved in building up your model. On the presentation date we will discuss the presentation and ask questions about your project and submissions.
Some examples
The Kaggle competitions page has hundreds of examples where people have applied machine learning to solve a variety of problems. Below are a few examples that you might find useful:
- Exploratory data analysis
- Regression: Comprehensive data exploration with Python
- Model building
The is also the excellent Kaggle Learn resource that you might find useful too.
1. Frame the problem and look at the big picture
- Define the objective in business terms
- How should you frame the problem (supervised/unsupervised etc.)?
- How should performance be measured?
- How would you solve the problem manually?
- List the assumption you and your team have made so far
- Verify your assumptions if possible
2. Get the data
- Find and document where you can get the data from
- Create a workspace. We like to structure our project folders as follows:
my-awesome-project
├── data
│ ├── external <- Data from third party sources.
│ ├── interim <- Intermediate data that has been transformed.
│ ├── processed <- The final, canonical data sets for modeling.
│ └── raw <- The original, immutable data dump.
│
├── notebooks <- Jupyter notebooks. Naming convention is a number (for ordering),
│ the creator's initials, and a short "-" delimited description, e.g.
│ 1.0-ltu-initial-data-exploration.
│
├── reports <- Generated analysis as HTML, PDF, LaTeX, etc.
│ └── figures <- Generated graphics and figures to be used in reporting
│
└── requirements.txt <- The requirements file for reproducing the analysis environment.- Once you and your team have agreed on the folder structure, we suggest creating a new virtual environment as follows in the root of
my-awesome-project. - Get the data
- Check the size and type of data (time series, geographical etc)
3. Explore the data
- Create a copy of the data for explorations (sampling it down to a manageable size if necessary)
- Create a Jupyter notebook to keep a record of your data exploration
- Study each feature and its characteristics:
- Name
- Type (categorical, int/float, bounded/unbounded, text, structured, etc)
- Percentage of missing values
- Check for outliers, rounding errors etc
- For supervised learning tasks, identify the target(s)
- Visualise the data
- Study the correlations between features
- Study how you would solve the problem manually
- Identify the promising transformations you may want to apply (e.g. convert skewed targets to normal via a log transformation)
- Document what you have learned
4. Prepare the data
Notes:
- Work on copies of the data (keep the original dataset intact).
- Write functions for all data transformations you apply, for three reasons:
- So you can easily prepare the data the next time you run your code
- So you can apply these transformations in future projects
- To clean and prepare the test set
- Data cleaning:
- Fix or remove outliers (optional)
- Fill in missing values (e.g. with zero, mean, median, ...) or drop their rows (or columns)
- Feature selection (optional):
- Drop the features that provide no useful information for the task (e.g. a customer ID is usually useless for modelling).
- Feature engineering, where appropriate:
- Discretize continuous features
- Add promising transformations of features (e.g. $\log(x)$, $\sqrt{x}$, $x^2$, etc)
- Aggregate features into promising new features
- Feature scaling: standardise or normalise features
5. Short-list promising models
We expect you to do some additional research and train at least one model per team member. You use a Random Forest model, but each team memeber has to investigate one additional model. So you may want to investigate the following alternatives for regression:
- Linear Regression
- Extra Trees
- Histogram-based Gradient Boosting Regression Tree.
- Multi-layer Perceptron regressor
- Elastic-Net
These additional models don't need to contribute to your final submission but there should for the individual models should be annotated with the student who created it. Each section should have the student name annotated. Each student should understand his model to the point where you can explain how it works and answer simple questions about it.
- Train mainy quick and dirty models from different categories (e.g. linear, SVM, Random Forests etc) using default parameters
- Measure and compare their performance
- Analyse the most significant variables for each algorithm
- Analyse the types of errors the models make
- Have a quick round of feature selection and engineering
- Have one or two more quick iterations of the five previous steps
- Short-list the top three to five most promising models, preferring models that make different types of errors
7. Present your solution
- Document what you have done
- Create a nice 15 minute video presentation with slides
- Make sure you highlight the big picture first
- Explain why your solution achieves the business objective
- Don't forget to present interesting points you noticed along the way:
- Describe what worked and what did not
- List your assumptions and you model's limitations
- Ensure your key findings are communicated through nice visualisations or easy-to-remember statements (e.g. "the median income is the number-one predictor of housing prices")
References
- Hands-On Machine Learning with Scikit-Learn and Tensorflow, Appendix B, A. Géron
- Cookiecutter Data Science
Dataset
The dataset for the competition consists of a regression problem for car pricing. The data was scraped from eBay on contains several features, such as the age of the cars and the model type. In total there are 19 features in the dataset. The target for the competition is to predict the car price from these features. In total, the dataset contains 371'528 samples which are split into a train (2/3) and test (1/3) sets. The training data contains the feature a and label columns whereas the test set only contains the feature columns. The goal of the competition is to train a model on the training set and then use this model to predict the labels on the test set. A more detailed description of the data is available on the competition website.
Kaggle
The competition is organized as a Kaggle challenge. The data is available on the Kaggle page and you have to upload you model predictions on the Kaggle page. Your results will automatically be evaluated and you will see your scores as well as the scores of the other teams on the leaderboard. Note that the test set is split into two parts: 70% is used to evaluate your predictions every time you upload them. The remaining 30% of the test set are not evaluated until the competition finishes and is used to calculate the final score. This split aims at avoiding teams improving overfitting their models to the test set score. The number of team submissions per day are limited to 20. Therefore, make sure you distribute the work such that you can evaluate all your ideas during the competition.
Signup
Go to the Kaggle website and click on Register to create an account. Once you have set up your account go the the competition page and click on Teams. Invite your fellow team member and name your team according the the MS Teams names.
Submission
To upload your predictions store them as a csv file (see the sample_submission.csv for reference) and click on the Submit Predictions button on the competition page. Upload your submission in the dialog and add a short description of the steps that led these particular descriptions. After a few minutes you should see your score under My submissions and if its your best run also on the Leaderboard.
import pandas as pd
from pathlib import Path
datapath = Path('../data/')
train = pd.read_csv(datapath/'train.csv')
test = pd.read_csv(datapath/'test.csv')
train.shape, test.shape
train.head()
median = train['price'].median()
median
submission = pd.read_csv(datapath/'sample_submission.csv')
submission.head()
submission['Predicted'] = median
submission.head()
submission.to_csv(datapath/'median_submission.csv', index=False)
You can now upload this file on the Kaggle competition webpage under Submit Prediction.
The general procedure to fit a scikit-learn model and make a submission looks as follows:
# train model
X_train, y_train = train.drop('price', axis=1), train['price']
my_model.fit(X_train, y_train)
# predict on test set
y_pred = my_model.predict(test)
# create submission
submission['Prediction'] = y_pred
submission.to_csv(datapath/'median_submission.csv', index=False)