Sep. 25 · Tokens, normalization, and corpus choices
Session 2 · Text Preprocessing
Preprocessing decisions determine which parts of a text remain available for analysis.
Objective
By the end of class, you can apply and document a small set of preprocessing choices and explain how those choices change the text available for analysis.
Before class
- Complete the six assigned Swirl · R Programming lessons. Commit and push one completion screenshot per lesson to
r-learning/, usingswirl-01.png,swirl-02.png,swirl-04.png,swirl-06.png,swirl-07.png, andswirl-12.png. Each screenshot must show the lesson name and its completion message. - Watch Orange’s current Text Preprocessing tutorial.
- Read Text as Data in Six Steps.
- Confirm that Demo 1 is pushed to GitHub.
See all R-foundation deadlines and marks →
Opening survey
The ungraded questions review your Session 1 setup (committing, pushing, and pulling in GitHub Desktop), the Orange Text Preprocessing tutorial, and step 3 of Text as Data in Six Steps. Attendance is recorded separately from correctness.
Device-free lecture
We will introduce tokens, tokenization, stopwords, normalization, filtering, and language-specific choices through one small corpus. The lecture distinguishes what researchers remove, retain, or transform from what the software later counts. We then step presidential-speech sentences through the preprocessing app, which uses the same settings as the course script.
Laptop demonstration
- presidential-speeches-100.csv, the corpus (corpus card). You may commit it.
- ba3-korean-preprocessing.py, the script for the Python Script widget
- stopwords-ko.txt, the stopword file for Preprocess Text
data/ folder of your course repository.
If the script will not run. Download presidential-speeches-100-processed.csv, the same speeches with the
processed_text column already added by the script. Open it with File → Corpus, set processed_text as the text feature in the Corpus widget, and continue with Preprocess Text. Note in your README that you used this file.
Standard track · Orange
Build one workflow that takes the raw speeches to a clean word cloud.
File → Corpus → Python Script → Corpus → Preprocess Text → Word Cloud
Load the course script in the Python Script widget with More › Import Script from File, and copy its settings into your README. No programming is expected. Compare the raw and processed text in Corpus Viewer, then save the word cloud of the processed text.
BA2 review track · R
Complete this assignment in R. Orange is not permitted for assessed work. You build your own R script that runs Kiwi from R, preprocesses the raw speeches, and makes three word clouds: all speeches, commemorative speeches (기념사), and National Assembly speeches (국회연설). A short reflection goes in your README.
- Assignment sheet (PDF), with the steps and the code
- presidential-speeches-100.csv and stopwords-ko.txt, saved in
data/
Demo 2 · Preprocess the speeches and save a word cloud
Due Oct. 2 at 09:15
Create demos/02-preprocessing/ with the following files.
README.mdlisting the corpus and the preprocessing settings from the script.workflow/containing either one Orange.owsfile or the supplied R starter completed as an.Rfile.output/wordcloud.pngshowing the word cloud of the preprocessed speeches.
The BA2 review track submits three word clouds and a reflection instead, as listed on its assignment sheet.
Before you submit
- The workflow or script opens from the repository.
- The README lists the corpus and the preprocessing settings.
- The word cloud of the preprocessed speeches is saved as
output/wordcloud.png. - Everything is committed and pushed.
The score records whether these items are complete. It does not reward a more sophisticated interpretation.