Experimentation

During a QA benchmarking experiment, a researcher tunes hyperparameters by repeatedly evaluating candidate models on the held-out test set and selecting the configuration with the best F1. What is the primary risk introduced by this practice?

• AThe experiment requires substantially more compute because multiple full evaluation passes are needed.
• BF1 score becomes an unreliable metric when it is computed more than once against the same test corpus.
• CThe held-out test set effectively becomes part of the development signal, causing reported performance to overestimate true generalisation.check_circle Correct
• DThe model will overfit to the validation set rather than the test set, undermining the value of validation-based early stopping.

Why A is wrong: Tempting because repeated evaluation does increase compute cost. Wrong because increased compute is a secondary practical concern; the primary scientific risk is that the reported test performance becomes an optimistic, biased estimate rather than an unbiased measure of generalisation.

Why B is wrong: Tempting because it correctly identifies F1 as the metric in use and hints at repeated evaluation. Wrong because F1 remains a valid and reproducible metric regardless of how many times it is computed; the problem is the feedback loop created when test-set results guide configuration selection, not the metric computation itself.

Why C is correct: When model or hyperparameter choices are made by inspecting test-set scores, information from the test set leaks into the design process. The test set no longer represents genuinely unseen data, so the measured performance is inflated relative to what the model would achieve in deployment. This is test-set contamination or evaluation leakage.

Why D is wrong: Tempting because overfitting to a held-out partition is the correct general concern in evaluation design. Wrong because the scenario describes optimisation against the test set specifically, not the validation set; conflating the two misidentifies which partition is being contaminated and which safeguard is bypassed.

A team compares two open-domain generative QA systems that produce free-form answer sentences rather than extracted spans. Manual inspection shows many answers are factually correct but reworded relative to the reference, yet exact match rates them as wrong. Which automatic metric most directly reduces this systematic under-scoring of correct-but-reworded answers?

• AToken-level exact match computed after lowercasing, article removal, and punctuation stripping during answer normalisation.
• BToken-level F1 over the bag of answer tokens, taking the maximum F1 against each available reference answer.
• CPerplexity of the generated answer under the QA model's own language-model head, averaged across the evaluation set.
• DBERTScore, which compares contextual embeddings of the prediction and reference by cosine similarity rather than surface token overlap.check_circle Correct

Why A is wrong: Normalisation removes superficial formatting differences and helps a little, but it still requires the surviving tokens to match exactly, so a genuinely reworded yet correct answer is still scored as wrong.

Why B is wrong: Token F1 awards partial credit for shared words and is better than exact match, but it still rewards lexical overlap rather than meaning, so a correct paraphrase that reuses few reference words scores low.

Why C is wrong: Perplexity measures how fluent or probable a generation is under the model, which sounds quality-related, but it ignores the reference answer entirely and so cannot tell whether a reworded answer is factually correct.

Why D is correct: BERTScore aligns contextual embeddings and rewards semantic equivalence, so a correct answer phrased differently from the reference still receives a high score; it correlates better with human judgement than surface-match metrics for free-form QA.

A support automation pipeline must route each incoming message to one of three different sub-chains based on a classification produced earlier in the same chain: billing queries go to a billing chain, technical queries to a troubleshooting chain, and everything else to a general chain. The routing decision must be made at runtime from the classification value, and exactly one sub-chain should run per request. Which LangChain construct most directly implements this conditional routing inside an LCEL pipeline?

• ARunnableBranch, which checks each predicate against the classification in order and invokes only the first matching sub-chain, falling back to a default branchcheck_circle Correct
• BRunnableParallel, which evaluates all three sub-chains on the input and then keeps only the result whose branch name matches the earlier classification value
• CRunnableLambda, which is required because conditional routing cannot be expressed declaratively and must always be coded as an imperative if-else inside a wrapped function
• DRunnableRetry, which re-invokes the default chain repeatedly until the classification value stabilises on one of the three recognised category labels

Why A is correct: RunnableBranch takes a list of condition and runnable pairs plus a default, evaluates the predicates in order against the input, and runs only the first matching sub-chain, which exactly implements the single-path conditional routing required.

Why B is wrong: RunnableParallel can hold three named branches, which is the surface appeal, but it runs every branch and merges all outputs, so it neither selects a single chain nor avoids the wasted work the requirement rules out.

Why C is wrong: A RunnableLambda containing if-else could route, which makes this partly true, but the claim that routing cannot be expressed declaratively is false because RunnableBranch exists for exactly this purpose, so the stated necessity is wrong.

Why D is wrong: RunnableRetry is a genuine LangChain feature, which lends it plausibility, but it handles transient failures by re-running a step and has nothing to do with selecting among category-specific sub-chains, so it does not perform routing.

A developer is building a customer support application that must retrieve relevant knowledge-base articles before generating a response. Which LangChain component is the correct abstraction for querying a vector store and returning ranked document chunks to the chain?

• AA Retriever, such as a VectorStoreRetriever, which performs similarity search and returns the most relevant Document objects to feed downstream in the chaincheck_circle Correct
• BA PromptTemplate that embeds the raw knowledge-base text directly into the system message before each call
• CAn OutputParser configured to parse structured JSON from the model's reply and map fields back to knowledge-base identifiers
• DA ConversationBufferMemory instance that stores past exchanges and replays the most recent turn containing the relevant article

Why A is correct: Retrievers are the dedicated LangChain abstraction for fetching relevant documents from a vector store via similarity or MMR search, returning Document objects that subsequent chain steps can use to ground generation.

Why B is wrong: Injecting raw text into a PromptTemplate is tempting because it does pass context to the model, but it bypasses semantic search entirely, cannot scale to large corpora, and fills the context window without relevance ranking.

Why C is wrong: An OutputParser operates on the model's response to extract or structure generated text; it plays no role in fetching documents before generation and cannot query a vector store.

Why D is wrong: ConversationBufferMemory stores dialogue history for maintaining context across turns, not for retrieving external documents by semantic similarity. It would surface only what the model or user previously said, not knowledge-base content.

A team wants to reuse the learned weights of a model pre-trained on English text to initialise a new model for a different but related language, rather than starting from random weights. Which condition most directly determines whether the pre-trained weights can be loaded into the new model at all, before any fine-tuning quality is considered?

• AWhether the target language has more training tokens available than the source language did during its original pre-training run.
• BWhether the new model shares the source model's architecture and tensor shapes, including a compatible embedding and vocabulary layer, so the saved weights can be loaded.check_circle Correct
• CWhether the two languages share a common writing script so that their characters map to overlapping byte sequences in the corpus.
• DWhether the target task uses the same loss function and optimiser settings that were used when the source model was originally pre-trained.

Why A is wrong: Tempting because data volume affects final quality, but it does not govern whether the source weights can be loaded into the new model, since weight reuse is decided before any target data is counted.

Why B is correct: Correct because reusing pre-trained weights requires matching layer structure and tensor shapes; if the embedding or vocabulary dimensions differ, the saved tensors cannot be loaded into the target model regardless of language relatedness.

Why C is wrong: Tempting because a shared script can ease cross-lingual transfer quality, but tokenisation and architecture, not the script alone, determine whether the saved weight tensors can be loaded into the new model.

Why D is wrong: Tempting because matching the training recipe can help convergence, but the loss and optimiser are properties of the training loop, not of the stored weights, so they do not decide whether the weights can be loaded.