Session 3 · Descriptive Patterns

Objective

By the end of class, you can represent texts as counts and TF-IDF weights, and use simple visuals to describe the corpus.

Before class

• Watch Orange’s current Bag of Words tutorial.
• In DataCamp’s Introduction to R, complete Chapter 1 Intro to basics. Completion is checked through the classroom dashboard. No upload is required.
• Confirm that Demo 2 is pushed to GitHub.

See all R-foundation deadlines and marks →

What you bring from Session 2

The speeches, the course script, and the stopword file in your data/ folder, and your Demo 2 workflow. It ends with clean nouns from Preprocess Text. Session 3 starts from that workflow.

What this session is about

Session 2 turned text into clean tokens. Session 3 turns those tokens into numbers.

• Bag of words. Each speech becomes a count of its words. Word order is set aside.
• Document-term matrix. One row per speech, one column per word, and the counts in the cells.
• Term and document frequency. How often a word occurs, and in how many speeches.
• TF-IDF. A weight that lowers words found in almost every speech and raises words that set a speech or a president apart.

We compare what raw counts and TF-IDF show about the same speeches.

Laptop demonstration

Standard track · Orange

Extend your Demo 2 workflow with two branches after Preprocess Text, one for counts and one for TF-IDF.

… → Preprocess Text → Word Cloud → Data Table → Bar Plot (counts)

… → Preprocess Text → Bag of Words → Word Cloud → Data Table → Bar Plot (TF-IDF)

Each branch ends in a word cloud and a top-ten bar chart. The settings for each widget are on the slides. Save the four images under the names in Demo 3 below.

BA2 review track · R

Complete this assignment in R. Orange is not permitted for assessed work. You build an R script that counts the nouns in each speech, weights them with TF-IDF, and saves the two word clouds and the two top-ten bar charts. The README has the same reflection as the Orange track.

• Assignment sheet (PDF), with the steps and the code
• presidential-speeches-100-processed.csv and stopwords-ko.txt, saved in data/ (needed only for route C on the sheet)

Demo 3 · Compare counts and TF-IDF

Due Oct. 9 at 09:15

Use (or create) demos/03-descriptive-patterns/ with the following files.

• README.md naming the corpus, your Preprocess Text settings from Demo 2, and the Bag of Words settings, followed by a short reflection. R users name their route and settings instead.
• workflow/ containing either one Orange .ows file or the completed R script.
• output/counts.png, the word cloud from raw counts.
• output/tfidf.png, the word cloud from TF-IDF.
• output/counts-top10.png, a bar chart of the top ten words by count.
• output/tfidf-top10.png, a bar chart of the top ten words by TF-IDF.

Reflection

Write one or two paragraphs in the README. Use your two bar charts.

• What do the counts tell you about the speeches?
• What does TF-IDF tell you that the counts do not?
• Name at least three words that rise or fall, and check in which speeches they occur.
• What did you learn about counting and weighting words?

In Orange, type a word in the search box of Corpus Viewer to find the speeches that use it. The Document Count column in the Data Table shows how many speeches contain it.

Before you submit

• The workflow or script opens from the repository.
• The README names the corpus and the settings, and includes the reflection.
• counts.png, tfidf.png, counts-top10.png, and tfidf-top10.png are saved in output/.
• Everything is committed and pushed.

The score records whether these items are complete. It does not reward a more sophisticated interpretation.