No Previews in Pyongyang

Reform discourse and policy change in North Korea, 1987–2020

Steven Denney · Leiden University
Peter Ward · Sejong Institute
ECNK Webinar
European Centre for North Korean Studies, University of Vienna
4 August 2026
The question

In official North Korean sources, does reform talk precede, coincide with, or follow policy?

Everyone agrees the economy has been reformed repeatedly. What is disputed is how policy is made in Pyongyang, and what state propaganda does in that process.

BEFORE precede? reform talk rises ahead of the measure WITH coincide? talk moves together with the measure AFTER follow? talk rises once the measure is in place time policy change
Motivation

Why it matters

Comparativists see two kinds of propaganda Soft propaganda persuades and deceives. Hard propaganda intimidates and deters unrest. The policy function has been largely ignored. e.g. Carter and Carter 2023 · Huang 2015
North Korea watchers read it for policy Agitprop monopolizes the information environment, so analysts mine it for policy, elite dynamics, and the state of the country. Some see options being debated between party, cabinet, and military. e.g. Carlin and Wit 2006 · McEachern 2009
The timing has never been tested Prior quantitative work noted shifts in topics and tone, but read titles rather than whole articles, and never dated the response. e.g. Choi and Lecy 2012 · Hastings and Lee 2023
Motivation

What theory expects

Existing theory
Theory 1: Legitimation Regime decides internally in a closed fashion and policy content is not provided, just praise for the new policy and slogans. Carter and Carter 2023
Theory 2: Intimidation Regime uses propaganda to project strength and scare would-be dissenters without actual policy content. Huang 2015
Theory 3: Policy content The text does carry real policy content. Carlin and Wit 2006 · McEachern 2009
Our expectations
Precede The text airs reformist ideas before the decision.
Coincide The text arrives with the decision, aligning cadres and readers.
Follow The text explains the decision after it is taken.

Sources

Sources

The journal

Cover of Kyongje Yongu, issue 2020-2
Kyŏngje Yŏngu (경제연구) The regime’s economics quarterly, written for planners, managers, officials and academics. Paeans to great leadership sit alongside technical discussion of production and futures contracts. Yang 2009
Articles covering the period 1987 to 2020 3,036 articles, full text in DPRK orthography. Four issues a year, on a fixed schedule.
What is missing Three quarterly issues (1995-4, 2019-4, and 2020-3) were unavailable, leaving 133 of 136 issues. In addition, 82 articles contain only a title and author byline, without any body text.
Cover of the 2020-2 issue.

Data and methods

Data and methods

Building the measure

1
Prepare
The full text of all 3,036 articles, 1987 to 2020, in the North’s own spelling and vocabulary.
2
Seed
Our own list first: 55 reform terms from close reading, plus a matching retrenchment list. Machines only extend it.
3
Discover
An LLM, EXAONE 3.5 on the Leiden HPC, labels every article without seeing our list. Its disagreements with the list show where missing vocabulary hides.
4
Embed
No LLM here: a small statistical model of the journal maps which words keep company with our seed terms. Look-alikes become candidates.
5
Validate
Three agents built on a frontier LLM, separate from EXAONE, judge each candidate against our reform definition, six mechanisms that loosen central control. Of 32 candidates, 3 are added to the lexicon and 11 to an extended list used for robustness.
6
Measure
An expert review fixes the lexicon at 40 terms. Distinct reform terms per article, minus retrenchment terms, averaged per quarter: the outcome variable.
Data and methods

The reform list

40 anchor terms, drawn from close reading. Two obvious examples from three of the six mechanisms.

Enterprise autonomy
경영권 (management rights)
계획권 (planning rights)
Price and marketization
수요와공급 (supply and demand)
농민시장 (farmers’ market)
Incentives and distribution
번수입 (earned income)
물질적자극 (material incentives)
Data and methods

The policy timeline

Kim Il-sung Kim Jong-il Kim Jong-un 19901995 20002005 20102015 2020 Marketization 9 measures Opening 10 measures Retrenchment 5 measures July 1 measures, 2002 May 30 measures, 2014 Kaesong Industrial Complex, 2002 Currency reform, 2009 Mark size is the measure’s scope weight: minor (1) major (2) landmark (3)
The policy events tested in the analysis. Larger marks indicate broader reforms.
Data and methods

The time series

Reform talk over time. The red lines mark the nine marketization measures.

Net reform density, 1987-2020, with marketization measures marked
This is the outcome the estimator tries to explain.
Data and methods

The test

Three patterns are possible. Each one would look different on the calendar, so the data can tell them apart.

precede coincide follow talk rises first talk rises with it talk rises after −4 −3 −2 −1 0 1 2 3 4 a policy measure quarters, counted from the measure We estimate the response quarter by quarter to see whether reform talk comes before, with, or after policy.
Data and methods

The estimator

We estimate whether policy is followed by increases in reform talk, after accounting for persistence, era effects, and trends.

future reform talk (h quarters later) = α + βh·policyt + γ·reform talkt−1 + era + trend + error
βh is what we want to estimate For each horizon h, βh tells us how much reform talk increases after a policy measure, beyond what we would otherwise expect. We estimate one β for each horizon, h = 0, …, 8.
The gray terms explain the baseline They capture how much reform talk we would expect even without a policy measure: persistence from last quarter, differences across eras, and long-run trends. βh measures only what is left over.
The same test runs backward Directions swapped: does reform talk rise before policy? A preview would show here.
Two choices were made before we ran anything Bigger measures count for more, weighted 1 to 3 by scope. We look within the first year after each measure.

Findings

Findings

Discourse follows, never leads

Local projections in both directions
+0.29 Three quarters after a measure, the journal runs about a third of a standard deviation above its usual level. In plain terms: in an issue of roughly forty articles, about three or four of them pick up one more reform term than they otherwise would. Small per article, visible per issue, and gone by the next quarter.
The solid line shows the estimated response after policy. The dashed line tests whether reform talk appears before policy. The smaller rise at h = 6 is the clustered 2001–03 measures seen twice, not a second response (appendix).
Findings

Only marketization responds

Only domestic marketization is followed by more reform talk.

Three policy tracks, each against its own vocabulary
Only the marketization estimate is clearly above zero.
Findings

Where the estimated effects appear

The predicted rise in reform talk, three quarters after each of the nine measures.

The estimated reform-talk responses placed on the historical timeline
Predicted responses from the estimator, not observed reform talk.
Findings

Why three quarters is plausible

about nine months the measure this issue has already closed one issue later the line settles, doctrinal idiom two issues later articles drafted, vetted, typeset three issues later interpretive articles arrive four issues later back to baseline The publication cycle naturally predicts a response about three quarters after policy.

Reading the text

Findings

Text as data

What are the post-policy articles actually doing?

Pool the post-policy articles Every article published in the year after each of the nine measures, kept if it is actually about reform: at least two reform terms, more reform than retrenchment. 130 articles.
Count three functions, by machine A script counts words that justify the policy, explain how it works, or urge action on it, compared against the journal’s ordinary reform writing. Nothing is tallied by hand.
Read three cases closely The biggest measures, at the centers of the two response clusters:
1.  The July 1 measures, 2002
2.  General markets legalized, 2003
3.  The May 30 measures, 2014
This is our initial, human read of what the post-policy text says.
Next: a machine check. An LLM classifies the domestic-marketization content against the same categories. Done transparently and fairly, it removes our bias as readers.
Findings

Framing

Most post-policy reform talk explains or justifies policy rather than mobilizing readers.

Function shares of post-reform discourse
Findings

In the journal’s own words

Examples from the post-policy response identified by the estimator, both from the year after the 2003 legalization of general markets.

Justifying it · 경제연구 2003, no. 4
“경애하는 장군님께서는 최근 번수입에 의한 경영방법을 받아들여 모든 독립채산제기업소들에서 경영손실을 없애고 실리를 보장할데 대한 문제 … 조치를 취해주시였다.”
The respected General has recently adopted the earned-income management method and taken measures to eliminate losses and guarantee practical benefit at every independent-accounting enterprise.
Explaining how it works · 경제연구 2004, no. 1
“현 시기 지방별 물자교류시장을 합리적으로 조직하고 그것을 잘 리용하는것은 지방공업공장들의 생산을 정상화하기 위한 중요한 요구이다.”
Organizing regional material-exchange markets and using them well is an important requirement for normalizing production at local-industry factories.

Conclusion

Takeaways

No previews in Pyongyang

Reform talk comes after the policy Reform vocabulary rises about nine months after the state loosens the domestic economy, and it does not rise before.
Only for reform at home Opening to foreign trade draws no response, and neither does re-centralization.
The result is robust Three ways of measuring reform talk all find the same timing:
1.  Breadth: distinct reform terms per article
2.  Intensity: reform terms per length of text
3.  The deliberately generous extended lexicon
The finding
The journal explains and codifies reform after the fact, and only the reform that needs justifying.
Takeaways

What’s next

A new method, still calibrating An LLM helps discover and validate the reform terms, and a plain count of those terms in the text is what measures reform talk. It is a new way to do this kind of work. The pipeline, especially its LLM steps, is still calibrating.
The signal is strong, some details may change These are initial results. We think the timing finding is right, but specific counts and term choices could shift as we iterate on the codebook and classification.
Look closer at the six-quarter echo Reform talk rises most strongly three quarters after a domestic measure, and again, slightly weaker, at six quarters. Understanding what drives that second rise is next on the list.
Next: another round, then new sources A further iteration of the coding and lexicon with expert review, then the same pipeline applied to other DPRK sources over time: KCNA, Rodong Sinmun, Chosun Munhak, and others.

Appendix

Appendix

Positive in every era

Era-restricted subsample estimates
One estimate per subsample, all nested. Gray bars are wide because early issues are thin, not because the effect is weaker.
Appendix

The other wiggles

τ = −1
a bump just before a measure
The one that would matter if it were real, because it would look like a preview. Its band includes zero, and the backwards test finds nothing at any horizon.
h = 6
a smaller rise six quarters out
Not a second response. Two measures sit three quarters apart, so the sixth quarter after the first is the third quarter after the second. Drop the earlier measure and this rise halves while h = 3 does not move at all.
h = 8
a dip two years out
What a series looks like settling back after a temporary rise, not a policy effect running in reverse.
Appendix

How a term enters the lexicon

1
Evidence pack
Corpus frequency, reform-discrimination lift, era distribution, usage snippets. Judgment rests on usage, not dictionary sense.
2
Denotation judge
Does the term name one of the six reform mechanisms, or fall under an exclusion?
3
Usage judge
Skeptical by instruction. Ambiguity goes to the stricter bin.
4
Arbiter
Burden of proof on inclusion. Borderline terms go to the domain expert.

Of 32 candidates: 3 admitted, 11 to a robustness tier, 18 rejected.

Appendix

The marketization arm at h = 6

Domestic marketization arm, local projection at every horizon
Same clustering as the aggregate series: dropping the 2001-10 measure alone nearly halves h = 6 (+0.28 to +0.15) while leaving h = 3 untouched (+0.38 to +0.41). Not an independent marketization-specific response.