Session 4 · Clustering and Similarity

Objective

By the end of class, you can compare documents by vocabulary, group them with hierarchical clustering, and explain how k-means differs.

Before class

• Watch Orange’s Hierarchical Clustering and Document Clustering and Cluster Exploration tutorials.
• In DataCamp’s Introduction to R, complete Chapters 2 Vectors and 3 Matrices. Completion is checked through the classroom dashboard. No upload is required.
• Confirm that Demo 3 is pushed to GitHub.

Optional, if you want to get ahead

• Read Orange’s widget documentation for Distances, Hierarchical Clustering, and k-Means. It shows the current interface; the Hierarchical Clustering video uses an older one.
• Keep experimenting with your Demo 3 workflow. After your Bag of Words, add Select Columns and move division_number, president, location, and kind into Metas. Then connect Distances (cosine) and Hierarchical Clustering and see which speeches group together. Before you look, write down which ones you expect to group, and why.

See all R-foundation deadlines and marks →

What you bring from Session 3

Your Demo 3 workflow, which ends with a Bag of Words after Preprocess Text. Session 4 starts from it. Save a copy as your Demo 4 workflow before you change anything.

Session 4 uses a larger file, presidential-speeches-300.csv. It holds the 100 speeches you know plus 200 more, so that each president has 100 (corpus card). You preprocess it yourself, as in Session 2. Your groups can differ a little from the slides, because Kiwi versions split some words differently. That is expected.

What this session is about

Session 3 asked which words set one speech apart. Session 4 asks which speeches are alike, and groups them.

• Clustering. Grouping documents so that similar documents end up together, without labels given in advance.
• Vectors. Each row of the TF-IDF matrix is a list of numbers, one per word. Clustering compares these rows.
• Cosine similarity. How closely two speeches point the same way, whatever their length. Cosine distance is 1 minus the similarity.
• Hierarchical clustering. Joins the closest speeches step by step. The linkage sets the distance between two groups. We use Ward, which joins the two groups whose merged group stays the most compact. The dendrogram shows the joins, and cutting it gives the groups.
• k-means. One of the most common clustering methods. It puts each speech with the nearest of k centres, with k chosen in advance. We show it in class. It is not required for the demo.

The computer finds the groups. It does not say what they mean. You decide that by reading speeches from each group and using what you know about the data and the topic.

To review these ideas, watch the explainer video (new tab, 6.5 minutes). It walks through the 300 speeches, cosine similarity and distance, and hierarchical clustering, with the same numbers as the slides.

Laptop demonstration

Standard track · Orange

Extend your Demo 3 workflow. In the File widget, open presidential-speeches-300.csv instead of the 100-speech file. The Python Script and Preprocess Text then run on all 300 speeches, which takes a few minutes. Then add the clustering chain after your TF-IDF Bag of Words.

Bag of Words → Select Columns → Distances → Hierarchical Clustering

Connect at least two of Pivot Table, Distributions, and Select Rows with Word Cloud to Hierarchical Clustering. The settings for each widget are on the slides. k-means is optional.

BA2 review track · R

Complete this assignment in R. Orange is not permitted for assessed work. You preprocess the 300 speeches with your Session 2 script and then follow the steps from Session 3. You cluster the speeches, save a dendrogram and at least two more figures or tables, and write the README.

• Assignment sheet (PDF), with the steps and the code
• presidential-speeches-300.csv and stopwords-ko.txt, saved in data/

Demo 4 · Cluster and inspect documents

Due Oct. 16 at 09:15

Create demos/04-clustering/ with the following files.

• README.md naming your settings, showing your figures, and naming each group.
• workflow/ containing the Orange .ows workflow or completed R script.
• output/clusters.png containing the dendrogram with 3 to 5 groups.
• Two more figures or tables about the groups in output/, made with Pivot Table, Distributions, or Word Cloud (in R, the options on the sheet).

Writing the README

• Settings. Name the representation (TF-IDF), the distance (cosine), the linkage (Ward), the number of groups, and the label you chose for the dendrogram, with one sentence on why you chose that label.
• Figures. Show clusters.png and your two other figures or tables in the README. For each, write two or three sentences on what you did and what it shows.
• Groups. Give each group a short name and one sentence on what it contains. Base the name on its words and on two or three speeches you read from it. Say what the name does not cover.
• Optional. If you tried k-means, add a short note on what you found.

Before you submit

• The README names the settings (TF-IDF, cosine distance, Ward linkage, the number of groups, and the label).
• output/clusters.png shows the dendrogram with 3 to 5 groups.
• Two more figures or tables about the groups are in output/ and shown in the README.
• The README says what each figure shows and names each group.
• The workflow or script is in workflow/.
• The final files are pushed to GitHub.