A cell’s gene expression and a stretch of character dialogue look like unrelated observations. Both invite the same question: what hidden state produced this observation, and where might that state go next? This course develops that question into a connected machine-learning workflow.
$$\boxed{\text{Observation}\to\text{Representation}\to\text{Geometry}\to\text{Trajectory}\to\text{Dynamics}\to\text{Fate}}$$We place cell differentiation and narrative progression beside each other throughout the course. Their evidence is different. Conventional single-cell expression measurements are destructive snapshots; they do not repeatedly observe the same cell. Dialogue logs preserve the actual order of turns within a session. That difference determines which transitions can be measured and which require assumptions.
Points alone, or points with time?
An illustrative 2D simulation. Lines reveal the simulator's known within-entity order; they are not inferred trajectories. Snapshot colors contain no time or branch labels.
Hide time in the diagram and a branching shape remains, but its origin and direction are not identified. Reveal entity IDs and time, and we know which points to connect. Lessons 1–7 largely study the first problem. Lessons 8–12 add sequences, population motion and outcomes.
Audience and format
The course is for junior CS undergraduates comfortable with Python arrays, matrix multiplication, means and variances, conditional probability and differentiation. No biology background is required. Lesson 2 reviews eigenvectors. Detailed SDE theory and causal identification proofs are extensions.
Lessons 1–10 take about 90 minutes, typically 60 minutes of explanation plus 30 minutes of exercises and discussion. Allow 100 minutes for lesson 11 and 110 minutes for the final integrated lesson. We repeat a structure: a concrete question, a small example, mathematical formulation, algorithm, Python implementation, biological interpretation and character-chat interpretation. These are original teaching notes, not transcripts of an external course.
The twelve lessons
| Lesson | Question | Methods | Shared project output |
|---|---|---|---|
| 1. Observations and hidden states | What does one row represent? | Observation models, identifiability | Data contract and evaluation split |
| 2. PCA | What is lost in compression? | Covariance, eigenvectors, SVD | Projection and reconstruction error |
| 3. Representation learning | How can a representation bend? | Autoencoder, VAE | Learned latent coordinates |
| 4. Manifolds | What does nearby mean? | kNN, Isomap, UMAP | Neighborhood comparison |
| 5. Diffusion and spectra | How do local links reveal global structure? | Laplacian, Diffusion Map | Diffusion coordinates and scales |
| 6. Clusters and communities | Where should state boundaries go? | k-means, DBSCAN, Louvain, Leiden | Partitions at different resolutions |
| 7. Trajectory inference | How do points become paths? | MST, Principal Graph, PAGA | Root-dependent pseudotime |
| 8. Sequential models | What changes when order is observed? | Markov, HMM, State Space Model | Transitions and hidden-state inference |
| 9. Velocity | Where is the state moving now? | RNA Velocity, vector-field regression | Next-state predictions and errors |
| 10. Population transport | Can we infer movement without matching individuals? | Optimal Transport, Sinkhorn | Marginal-constrained transport |
| 11. Continuous dynamics | Can a continuous movement rule be learned? | Neural ODE/SDE, Flow Matching | An integrated population flow |
| 12. Outcomes and causality | What can we predict, and what can we change? | Survival, Fate Prediction, Causal Inference | Censoring-aware analysis and effects |
The labs implement small CPU examples. They do not implement every named extension. For example, lesson 7 implements a centroid MST baseline, not PAGA. Each article marks the boundary between its executable method and conceptual extensions.
Run the labs
Download the complete lab ZIP. Extract it to a folder containing common.py, lesson01.py through lesson12.py, the runner and requirements. Use Python 3.12:
python3.12 -m venv .venv
source .venv/bin/activate
python -m pip install -r requirements.txt
python run_all.py --output outputs
On Windows, activate with .venv\Scripts\activate. Installing packages needs internet access; running the labs needs no external dataset, API key or GPU. UMAP’s first run may take longer because it compiles code.
python lesson02.py --output outputs
python run_all.py --lessons 2 5 7 --output outputs
Figures and metrics appear under outputs/. The recorded validation run includes Python and package versions. A fixed seed does not eliminate all operating-system or numerical-library differences. A sign flip in an eigenvector alone does not indicate an error.
One dataset throughout
You can also download the shared synthetic dataset as NPZ. Open it with numpy.load("branching-trajectories.npz", allow_pickle=False). It matches the generator’s default output.
The simulator observes 120 entities at 16 time points: 1,920 rows with 24 observed features each. A known two-dimensional branching state is transformed through nonlinear basis functions and a fixed mixing matrix, then perturbed with noise. The small size lets us examine behavior on a laptop. It is not intended as a realistic gene-expression or language benchmark.
| Field | Meaning and unit | Role |
|---|---|---|
X | 1,920 × 24 observation matrix | Model input |
z_true | Simulator’s two-dimensional state | Oracle visualization and evaluation only |
time, session, next_index | Time, entity ID, next-row index | Sequence analysis; final index is -1 |
branch | Row-level branch sign, −1 or +1 | Post-hoc evaluation |
train | Whole-session partition mask | 90 training / 30 test sessions |
fate | Session-level terminal branch, 0 or 1 | Outcome target |
duration, event | Follow-up duration and event indicator | Right-censored survival analysis |
baseline, treatment, outcome | Pretreatment risk, randomized assignment, subsequent continuous outcome | Causal exercise |
The duration clock starts after the full 16-frame observation window. It is separate from the trajectory clock time, which runs from zero to one. We do not pretend that frames after dropout were observed. Lesson 12 separates prediction of fate from an early prefix, follow-up survival analysis, and the randomized effect exercise.
The continuous treatment outcome is $Y=0.7B+0.8A+\epsilon$, with randomized $A$. Its population average treatment effect is 0.8, giving us a known target for estimation. This says nothing about the effect of a real prompt or biological treatment.
What counts as evidence?
| Goal | Check | Insufficient evidence |
|---|---|---|
| Representation | Held-out reconstruction and neighborhood preservation | Attractive separated islands |
| Trajectory | Root and neighborhood sensitivity, known-order agreement | One high pseudotime correlation |
| Dynamics | Next-step and multistep rollout errors | Arrows on training samples |
| Distribution | Marginal residuals, transport cost, held-out distribution | Claimed individual identities |
| Outcome | Session-separated Brier score, calibration and censoring | Accuracy alone |
| Causality | Randomization or explicit identification assumptions and uncertainty | State–dropout association |
Scaling and PCA must also respect the train/test split. For an early prediction task, inspect whether future frames influenced the feature representation. See the scikit-learn guide to data leakage for implementation details.
Final project: character conversations
Use conversations you have permission to analyze, stripped of identifying information, or publishable dialogues you create yourself. Choose one unit consistently: a message or a window of preceding turns. Every feature must depend only on text available at that time. A summary of the complete session would leak the ending into an early-state model.
If users or characters recur across sessions, splitting sessions may be insufficient. State whether evaluation separates users, characters or calendar time. Compare against a simple conversation-length baseline and inspect cases where a topic change is mistaken for an emotional transition.
Submit a one-page question and data contract, reproducible code, three central figures, two failure cases and a one-page limitations discussion. Assessment weights: question and data definition 20%, leakage-free evaluation 25%, methods and baselines 25%, interpretation and failure analysis 20%, reproducibility 10%. A biological project should consider donor and batch separation and distinguish lineage tracing from inferred paths.
Start with lesson 1: before learning an algorithm, decide what a row actually observes.