NCA-GENM - Multimodal Data (15% of the exam) - Section 3.4
Apply early, late, and intermediate model fusion techniques.
Compare early fusion, which concatenates raw inputs before encoding, with late fusion, which merges independent modality outputs, and intermediate fusion, which combines features mid-network. Choose the appropriate model fusion technique based on the alignment of modalities and the latency constraints of the application.
Model fusion
Practice question for this objective
lock_openFree sampleMultimodal Datahard
In a multimodal architecture, what fundamentally distinguishes an early fusion design from a late fusion design?
AEarly fusion combines the modalities at the input or low-level feature stage so a single joint model learns over the merged representation, whereas late fusion produces a separate prediction per modality and combines those predictions at the decision stage.check_circle Correct
BEarly fusion trains each modality with its own loss function, whereas late fusion shares a single loss across all modalities from the first layer.
CEarly fusion always uses attention to combine modalities, whereas late fusion always uses simple concatenation of raw inputs.
DEarly fusion can only be applied to two modalities at once, whereas late fusion is the only approach that scales to three or more modalities.
Distinguish early fusion from late fusion by the stage at which modalities are combined and how the model is structured. The early-versus-late distinction is defined by where the modalities meet in the pipeline: early fusion merges low-level features so one joint model learns over them, while late fusion trains per-modality models and combines their outputs only at the decision stage.
Why A is correct: Correct. The defining axis is the stage at which information from different modalities meets: early fusion merges before or near the input so one model jointly encodes everything, while late fusion keeps modality-specific models that each emit a prediction, fused only at the output.
Why B is wrong: This inverts the relationship. Late fusion is the design that keeps per-modality branches, and those branches may each carry their own objective, while early fusion shares one joint objective. The statement is tempting because it correctly senses that one design has more independence, but it assigns that independence to the wrong approach.
Why C is wrong: Neither mechanism is mandated by either approach: early fusion classically concatenates raw or low-level features, and late fusion combines high-level outputs by averaging, voting, or a small meta-model. The claim is plausible because attention is a common fusion mechanism, but it wrongly fixes one mechanism to each design and reverses where concatenation typically appears.
Why D is wrong: Both approaches extend to any number of modalities: early fusion can concatenate three or more feature vectors and late fusion can combine three or more predictions. The statement is tempting if one assumes concatenation grows unwieldy, but there is no two-modality ceiling on early fusion, so the distinction is fabricated.
Answers that look right on this material and are not. Each one is a distractor from a different question in the NCA-GENM bank for this domain.
The problem is modality collapse from unbalanced dimensionality, and the remedy is to project both streams to equal width before concatenation.
errorWhy it is wrong: Tempting because dimensionality imbalance is a real early-fusion failure, but it would not be cured by aligning timestamps, and the described symptom is mismatched moments rather than one modality being ignored, so it misdiagnoses the cause.
Intermediate fusion discards the smaller modality entirely, so only the larger, better-scaled modality contributes and instability disappears.
errorWhy it is wrong: Intermediate fusion retains both modalities through their own encoders rather than discarding one. The option is tempting because dropping a troublesome modality would indeed remove its scale problem, but that is not what intermediate fusion does, and it would forfeit the multimodal signal.
Early fusion: pad or truncate both raw streams to the same length, concatenate them, and pass the combined tensor through a single encoder with no modality-specific preprocessing.
errorWhy it is wrong: Early fusion requires combining raw inputs before any modality-specific processing. When input dimensionalities are highly heterogeneous, this approach is the one the team's experiments have already shown to be unstable, making it the wrong choice here.