source: interviewstack.io

What is active learning and why is it useful when labels are costly? Describe three acquisition strategies (e.g., uncertainty sampling, entropy, BALD) and one practical challenge when integrating active learning into a production labeling pipeline.

Hints

!1. Acquisition functions determine which unlabeled examples to send to annotators!<

!2. Consider batching, annotation latency, and labeling cost as integration challenges!<

Sample Answer

Active learning is a strategy where the model selects the most informative unlabeled examples to send for annotation, so you get more performance per label. It’s useful when labels are costly because it reduces labeling budget by prioritizing samples that will most improve the model.

Three acquisition strategies:

• Uncertainty sampling (least confident / margin): pick samples where the model’s top class probability is low (e.g., 1 - max_y p(y|x)). Margin sampling picks smallest gap between top two class probabilities. Simple, fast, works well for many tasks.
• Entropy-based sampling: rank by predictive entropy H[p(y|x)] = -∑ p(y|x) log p(y|x). Captures overall uncertainty across classes rather than just the top prediction.
• BALD (Bayesian Active Learning by Disagreement): selects points that maximize mutual information between predictions and model posterior: I[y, θ | x, D]. In practice compute with approximate Bayesian models (e.g., MC Dropout or ensembles): choose x with high predictive variance across stochastic forward passes — targets epistemic uncertainty.

Practical challenge integrating into production:

• Labeling latency and model staleness: iterative AL needs a tight loop (select → label → retrain). In production, human annotation latency and batch labeling cause the model to train on stale data distribution or block serving. Mitigations: use batched acquisition with diversity to avoid redundancy, asynchronous pipelines (continual training with prioritized queues), warm-start models, and monitoring to detect distribution drift so selection remains effective.

Follow-up Questions to Expect

1. How would you combine active learning with human-in-the-loop quality control?