WARNING: All lectures of the First Semester of the academic year 2020/21, until 31/12/2020, will be provided exclusively remotely, through the Teams team named “599AA 20/21 - BIG DATA ANALYTICS [WDS-LM]” (https://bit.ly/35yJ65c).

ATTENZIONE: Tutte le lezioni frontali del Primo Semestre dell’a.a. 2020/21, fino al 31/12/2020, verranno erogate esclusivamente in modalità a distanza, attraverso il canale Teams “599AA 20/21 - BIG DATA ANALYTICS [WDS-LM]” (https://bit.ly/35yJ65c).

Instructors - Docenti:

• Luca Pappalardo, Fosca Giannotti
  • KDD Laboratory, Università di Pisa and ISTI-CNR, Pisa
  • http://www-kdd.isti.cnr.it
  • luca [dot] pappalardo [at] isti [dot] cnr [dot] it
  • fosca [dot] giannotti [at] isti [dot] cnr [dot] it

Timetable (http://bit.ly/unipi_timetable_2020)

• Monday 16:15 - 18:00 Aula WDS/1
• Tuesday 16:15 - 18:00 Aula WDS/1

Team Registration: build up teams of 3 or 4 students and register your team here, by September 27th: https://forms.gle/rbsV4dF6RuAnCBWz9

For students without a team: send an email to Luca Pappalardo to notify that you are without a team by September 30th.

Only for the registered teams, express your preference for the datasets by September 30th https://forms.gle/HVheaScCgQJw4o616

Dataset assignment: at thie following link, each team can find the dataset assigned for the project –> https://bit.ly/33eTfC9

Instructions for mid term 1: The first mid term presentation (data understanding and project proposal) will be on October 19th (BigProblem, Global, MMG, I TeamIDI) and October 20th (Bei Dati Acrobatici, Malucs, AMS Group).

• presentation: prepare a presentation describing the data understanding and a proposal of the problem you want to solve. Motivate your decisions and choices (e.g., which variables you delete, how you deal with missing values and noise, the new variables you created, if you integrated your data with external datasets, etc.). The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
• report: the report must be done in latex, using this template: latex_template_bda.zip. It must be a maximum of 5 pages long. Summarize the data understanding and describe and motivate your project proposal. A zipped folder (.zip file) containing the .tex file, the .cls file, the .pdf file, and the files of all figures must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
• code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
• Google form: upload the material by October 18th using this form: https://forms.gle/h2SAKFmkdXv4itiU6
• name the files using the format midterm1_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm1_iteamidi_presentation.pdf, midterm1_beidatiacrobatici_report.zip, midterm1_amsgroup_code.ipynb

Instructions for mid term 2: The second mid term presentation (model(s) implementation and evaluation) will be on November 16th (BigProblem, Global, MMG, I TeamIDI) and November 17th (Bei Dati Acrobatici, Malucs, AMS Group).

• presentation: prepare a presentation describing the models you tried (e.g., Decision Trees, SVMs, etc.), the baselines used as a comparison (e.g., DummyClassifiers), how you perform the hyper-parameter tuning, the evaluation technique used (i.e., holdout, repeated holdout, cross validation), and the metrics chosen to evaluate the performance of the models. Motivate your decisions and choices (e.g., which evaluation metrics you chose, how you deal with the unbalancing of the dataset). The presentation should last 20 minutes (+ 10 minutes questions). The presentation must be sent through the google form (see below) in pdf format;
• report: the report must be done in latex, using the same template as the first mid term: latex_template_bda.zip. It must be a maximum of 10 pages long, including the part of the report regarding mid term 1. Summarize the model construction and evaluation and motivate your choices. A zipped folder (.zip file) containing the .tex file, the .cls file, the .pdf file, and the files of all figures must be sent through the google form. In the report, put the name of the title of your project and the names of the members of your team.
• code: the python code in .ipynb format (Jupyter Notebook) or .py format used to generate the computations and the plots must be sent through the google form. Please document adequately your notebooks using the markdown language.
• Google form: upload the material by November 15th using this form: https://forms.gle/kr3uq2PyyMqraRn78
• name the files using the format midterm2_teamname_type, where teamname is the name of the team (do not use spaces, use lowercase only), type is the type of the file (i.e., presentation, report, or code). Examples: midterm2_iteamidi_presentation.pdf, midterm2_beidatiacrobatici_report.zip, midterm2_amsgroup_code.ipynb

Paper presentation:

• each student will present, during a talk of most 7 minutes, a paper on Big Data Analytics. The presentations of the papers are scheduled on November 23rd and 24th. The presentation should last 7 minutes (+ 3 minutes questions).
• Express your preference for 5 papers here: https://forms.gle/B9rCmpJ8jnQ4vzWN8. We'll take into account your preference as much as possible. Fill the form by Nov 3rd. I'll assign you the dataset within Nov 5th.
• During the presentation (with slides) you should highlight the following aspects: the data set used, the feature engineering and/or selection (if any), the problem addressed, the models/algorithms used to solve the problem, and finally the explanations of the model constructed (if any).

Examples of projects from past years:

• Credit Risk Prediction, final report: credit_risk_prediction_heloc_case.pdf
• Ted Talks, final report: dataworms_project_ted_talks_report.pdf

In our digital society, every human activity is mediated by information technologies, hence leaving digital traces behind. These massive traces are stored in some, public or private, repository: phone call records, movement trajectories, soccer-logs and social media records are all examples of “Big Data”, a novel and powerful “social microscope” to understand the complexity of our societies. The analysis of big data sources is a complex task, involving the knowledge of several technological and methodological tools. This course has three objectives:

• introducing to the emergent field of big data analytics and social mining;
• introducing to the technological scenario of big data, like programming tools to analyze big data, query NoSQL databases, and perform predictive modeling;
• guide students to the development of a open-source and reproducible big data analytics project, based on the analyis of real-world datasets.

In this module, analytical methods and processes are presented thought exemplary cases studies in challenging domains, organized according to the following topics:

• The Big Data Scenario and the new questions to be answered
• Sport Analytics:
  1. Soccer data landscape and injury prediction
  2. Analysis and evolution of sports performance
• Mobility Analytics
  1. Mobility data landscape and mobility data mining methods
  2. Understanding Human Mobility with vehicular sensors (GPS)
  3. Mobility Analytics: Novel Demography with mobile-phone data
• Social Media Mining
  1. The social media data landscape: Facebook, Linked-in, Twitter, Last_FM
  2. Sentiment analysis. example from human migration studies
  3. Discussion on ethical issues of Big Data Analytics
• Well-being&Now-casting
  1. Nowcasting influenza with retail market data
  2. Predicting well-being from human mobility patterns
• Paper presentations by students

This module will provide to the students the technologies to collect, manipulate and process big data. In particular the following tools will be presented:

• Python for Data Science
• The Jupyter Notebook: developing open-source and reproducible data science
• MongoDB: fast querying and aggregation in NoSQL databases
• GeoPandas: analyze geo-spatial data with Python
• Scikit-learn: machine learning in Python
• Keras: deep learning in Python

During the course, teams of students will be guided in the development of a big data analytics project. The projects will be based on real-world datasets covering several thematic areas. Discussions and presentation in class, at different stages of the project execution, will be performed.

• 1st Mid Term: Data Understanding and Project Formulation
• 2nd Mid Term: Model(s) construction and evaluation
• 3rd Mid Term: Model interpretation/explanation
• Exam: Final Project results

14/09 (Mod. 1) Introduction to the course, The Big Data scenario lesson1_introduction_to_the_course_bda2021.pdf

15/09 (Mod. 2) Python for Data Science and the Jupyter Notebook: developing open-source and reproducible data science

• How to install Jupyter notebook: https://jupyter.readthedocs.io/en/latest/install.html
• Python notebooks: http://bit.ly/bda2021_notebooks_1

21/09 No Lesson (Election Day in Italy)

22/09 (Mod. 3) Presentation of datasets for projects bda20_21_datasets_1_.pdf

28/09 (Mod. 2) Scikit-learn: programming tools for data mining (part 1): http://bit.ly/bda_notebooks_2

29/09

• (Mod. 2) Scikit-learn: programming tools for data mining (part 2): http://bit.ly/bda_notebooks_2
• (Mod. 1) Reproducing and Explaining Human Evaluations of Soccer Performance with Artificial Intelligence evaluting_soccer_performance_1_.pdf

05/10 No Lesson (SocInfo2020 conference)

06/10 No Lesson (SocInfo2020 conference)

12/10 (Mod. 2) Geopandas and scikit-mobility: managing geographic data in Python (part 1) bda2021_geopandas.zip

13/10 (Mod. 2) Geopandas and scikit-mobility: managing geographic data in Python (part 2) https://github.com/scikit-mobility/tutorials/tree/master/mda_masterbd2020

19/10 (Mod. 3) 1st Mid Term - first group of teams

20/10 (Mod. 3) 1st Mid Term - second group of teams

26/10 (Mod. 3) Discussion and group working on projects

27/10 (Mod. 3) Discussion and group working on projects

02/11 (Mod. 1) Nowcasting well-being with big data bda_wellbeing.pdf

03/11 (Mod. 1) Injury prediction in sports with AI bda_2020_injury_forecasting.pdf

06/11 (Mod. 1) Trustworthy data mining

16/11 (Mod. 3) 2nd Mid Term - first group of teams

17/11 (Mod. 3) 2nd Mid Term - second group of teams

23/11 (Mod. 3) Paper presentations

24/11 (Mod. 3) Paper presentations

30/11 (Mod. 3) Discussion and group working on projects

01/12 (Mod. 3) 3rd Mid Term - first group of teams

07/12 (Mod. 3) 3rd Mid Term - second group of teams

TBC

Big Data Analytics A.A. 2019/20

Big Data Analytics A.A. 2018/19

Big Data Analytics A.A. 2017/18

Big Data Analytics A.A. 2016/17

Big Data Analytics A.A. 2015/16