Contextual Retrieval for Multimodal RAG on Slide Decks

Build and Persist the Index

We’ll now embed these summarized chunks and store them in a vector store for retrieval. LlamaIndex can persist indices locally or integrate with 40+ external vector databases.

import os
from llama_index.core import (
    StorageContext,
    VectorStoreIndex,
    load_index_from_storage,
)

if not os.path.exists("storage_nodes_iconiq"):
    index = VectorStoreIndex(new_text_nodes, embed_model=embed_model)
    index.set_index_id("vector_index")
    index.storage_context.persist("./storage_nodes_iconiq")
else:
    storage_context = StorageContext.from_defaults(persist_dir="storage_nodes_iconiq")
    index = load_index_from_storage(storage_context, index_id="vector_index")

retriever = index.as_retriever()

Baseline Index (Without Summaries)

We’ll also build a “baseline” index on the original text nodes (without the contextual summaries) to compare the difference in retrieval quality.

if not os.path.exists("storage_nodes_iconiq_base"):
    base_index = VectorStoreIndex(text_nodes, embed_model=embed_model)
    base_index.set_index_id("vector_index")
    base_index.storage_context.persist("./storage_nodes_iconiq_base")
else:
    storage_context = StorageContext.from_defaults(
        persist_dir="storage_nodes_iconiq_base"
    )
    base_index = load_index_from_storage(storage_context, index_id="vector_index")

Build a Multimodal Query Engine

We want a RAG pipeline that:

• Retrieves relevant chunks of text.
• Also loads the page images.
• Sends both text chunks and images to a multimodal LLM (here we illustrate using an OpenAI-like GPT-4 multimodal endpoint, labeled gpt-4o).

import base64
import openai
import os
from typing import Optional, List

from llama_index.core.query_engine import CustomQueryEngine
from llama_index.core.base.response.schema import Response
from llama_index.core.retrievers import BaseRetriever
from llama_index.core.prompts import PromptTemplate
from llama_index.core.schema import NodeWithScore, MetadataMode

QA_PROMPT_TMPL = """\
Below we give parsed text from slides, as well as images.

---------------------
{context_str}
---------------------

Given the context information and no prior knowledge, please answer the query:

Query: {query_str}
Answer:
"""

QA_PROMPT = PromptTemplate(QA_PROMPT_TMPL)

def encode_image(image_path: str) -> str:
    """If you want to inline a local image in base64."""
    with open(image_path, "rb") as f:
        return base64.b64encode(f.read()).decode("utf-8")


class MultimodalQueryEngine(CustomQueryEngine):
    """
    Custom multimodal Query Engine that retrieves text nodes,
    then sends them + image(s) to the new Vision-capable API as documented.
    """

    def __init__(
        self,
        retriever: BaseRetriever,
        model_name: str = "gpt-4o",
        qa_prompt: Optional[PromptTemplate] = None,
    ) -> None:
        super().__init__(qa_prompt=qa_prompt or QA_PROMPT)
        self.retriever = retriever
        self.model_name = model_name

    def custom_query(self, query_str: str) -> Response:
        # 1) Retrieve text nodes
        node_with_scores: List[NodeWithScore] = self.retriever.retrieve(query_str)

        # 2) Build context
        context_str = "\n\n".join(
[nws.node.get_content(metadata_mode=MetadataMode.LLM) for nws in node_with_scores]

        )

        # 3) Format the final prompt
        formatted_prompt_text = self._qa_prompt.format(
            context_str=context_str,
            query_str=query_str,
        )

        # 4) Build the user message with text + images
        user_message_content = [
            {
                "type": "text",
                "text": formatted_prompt_text,
            }
        ]

        for nws in node_with_scores:
            image_path = nws.node.metadata.get("image_path", "")
            if image_path:
                base64_data = encode_image(image_path)
                image_url = f"data:image/jpeg;base64,{base64_data}"
                user_message_content.append(
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": image_url,
                            "detail": "auto"
                        },
                    }
                )

        messages = [
            {
                "role": "user",
                "content": user_message_content,
            }
        ]

        # 5) Call your Vision model
        response = openai.ChatCompletion.create(
            model=self.model_name,
            messages=messages,
            max_tokens=500,
        )

        # 6) Return a Response object
        return Response(
            response=response.choices[0].message.content,
            source_nodes=node_with_scores,
            metadata={},
        )
        
        # 2) Create a query engine
query_engine = MultimodalQueryEngine(
    retriever=index.as_retriever(similarity_top_k=3),
    model_name="gpt-4o",   # or "gpt-4o-mini", "gpt-4-turbo", etc.
)

base_query_engine = MultimodalQueryEngine(
    retriever=base_index.as_retriever(similarity_top_k=3),
    model_name="gpt-4o",
)

Trying Out Queries

Let’s query our new pipeline about AI usage by department.

response = query_engine.query(
    "Which departments use GenAI the most and how are they using it?"
)
print(str(response))

A typical response might look like this:

Based on the parsed markdown text provided, the departments/teams that use 
generative AI the most are:
1. **AI, Machine Learning, and Data Science** with a score of 4.5.
2. **IT** with a score of 4.0.
3. **Engineering / R&D** with a score of 3.9.
These scores are derived from a survey where respondents rated the level of
generative AI usage on a scale of 1-5.
In terms of how these departments are using generative AI:
- **AI, Machine Learning, and Data Science**: While specific use cases for this
department are not detailed in the provided text, it can be inferred that they are
likely using generative AI for advanced data analysis, model development, and
enhancing AI capabilities within the organization.
- **IT**: The IT department is using generative AI for several impactful use cases,
including:
- Ticket management
- Chatbots
- Customer support and troubleshooting
- Knowledge management
- Case summarization
The information about the departments and their use cases comes from the parsed
markdown text. There are no discrepancies between the parsed markdown and the
context provided, as the markdown text clearly outlines both the departments with
the highest usage scores and the specific use cases for the IT department.

Comparatively, if we run the same query on the baseline index:

base_response = base_query_engine.query(
    "Which departments use GenAI the most and how are they using it?"
)
print(str(base_response))

You’ll see the baseline might have fewer details or slightly different retrieval results. Contextual retrieval gives more precise context around the IT usage specifically. The response would look like:

Based on the parsed markdown text provided, the departments that use Generative AI
(GenAI) the most are:
1. **AI, Machine Learning, and Data Science** - This department has the highest 
weighted average score of 4.5 for GenAI usage, indicating significant adoption. The
specific use cases are not detailed in the parsed text, but given the nature of the
department, it is likely involved in developing and refining AI models and
algorithms.
2. **IT** - With a score of 4.0, the IT department is also a leading user of GenAI.
The use cases for IT include internal productivity enhancements and IT operations,
as indicated by the 61% adoption rate for internal productivity and 42% ROI mention
in IT use cases.
3. **Engineering / R&D** - This department has a score of 3.9. While specific use
cases are not detailed in the parsed text, it is reasonable to infer that GenAI is
used for product development and research purposes, as suggested by the 69% 
adoption rate for core product performance enhancements and 50% for natural language
interfaces.
The information is derived from the parsed markdown text, which provides a detailed
breakdown of GenAI usage by department and specific use cases. There are no 
discrepancies between the parsed markdown and the raw text, as the markdown appears
to be a structured representation of the same data. The image was not provided, so
it was not used in forming the answer.

Observing the Benefits of Contextual Retrieval

Here’s another example query, In this next question, the same sources are retrieved with and without contextual retrieval, and the answer is correct for both approaches. This is thanks for LlamaParse Premium’s ability to comprehend graphs.

query = "what are relevant insights from the 'deep dive on infrastructure' section in terms of model preferences, cost, deployment environments?"

response = query_engine.query(query)
print(str(response))

Output

The "Deep Dive on Infrastructure" section from the ICONIQ Growth report provides
insights into the infrastructure aspects necessary for deploying AI solutions. 
However, the parsed markdown text does not explicitly mention model preferences or
costs in this section. Instead, it focuses on infrastructure tooling and deployment
environments.
From the parsed markdown text, we can gather the following insights related to 
deployment environments:
1. **Deployment Environments**: Enterprises are primarily hosting generative AI
workloads on the cloud or using a hybrid approach. The preferred deployment methods
are:
- Cloud: 56%
- Hybrid: 42%
- On-prem: 2%
2. **Cloud Service Providers**: The most utilized cloud service providers for
hosting AI workloads are:
- Amazon Web Services (AWS): 68%
- Microsoft Azure: 61%
- Google Cloud (GCP): 40%
These insights are derived from the parsed markdown text, specifically from the 
sections discussing "Cloud Deployment Method" and "Infrastructure Tooling." There is
no mention of model preferences or cost considerations in the provided text. If
there were any discrepancies or additional details in the image or raw text, they
are not available here, so the answer is based solely on the parsed markdown text
provided.

Now, lets try with the baseline approach:

base_response = base_query_engine.query(query)
print(str(base_response))

Output

The parsed text from the slides does not provide specific insights regarding model preferences, cost, or deployment environments in the 'deep dive on infrastructure' section. The slide titled "Deep Dive on Infrastructure" (page 24) only contains the title, the ICONIQ Growth branding, and confidentiality and copyright notices. There is no detailed information or data presented in the parsed text for this section.

Therefore, based on the parsed markdown text provided, there are no relevant insights available from the ‘deep dive on infrastructure’ section regarding model preferences, cost, or deployment environments. If there were any images associated with this section, they were not provided, and thus no additional insights could be derived from them.

This conclusion is drawn from the parsed markdown text, which lacks any specific information on model preferences, cost, or deployment environments in that section. The image confirms this, as it only shows the title and a graphic without additional details.

If you need insights on these topics, you might want to refer to other sections or slides that specifically address model preferences, costs, or deployment environments.

• Contextual Retrieval might fetch the pages that discuss cloud deployment methods, infrastructure tooling, and cost references, leading to a more thorough response.
• The baseline approach might (in some cases) fail to retrieve the correct chunk or provide less detail.

Comparing both answers helps demonstrate that those short “contextual summaries” in your metadata often lead to more relevant retrieval.

A big thanks to Jerry Liu from LlamaIndex for creating this amazing pipeline.

Conclusion

In this tutorial, we explored the process of parsing a PDF slide deck using LlamaParse to extract both text and images, enriching each text chunk with contextual summaries to enhance retrieval accuracy. We demonstrated how to build a Multimodal RAG pipeline with LlamaIndex, integrating both textual and visual data into a powerful model like GPT-4, showcasing the potential of multimodal LLMs. Finally, we compared results from a baseline index to a contextual index, highlighting the significant improvements in retrieval precision and relevance achieved through the contextual approach. This comprehensive guide equips you with the tools and techniques to build effective multimodal AI solutions.

Key Takeaways

• Contextual retrieval improves chunk matching for queries that might not have a direct keyword overlap.
• Multimodal RAG can incorporate not just text but also images, charts, or diagrams from slides.
• Prompt caching is essential when chunk sizes are large and you’re generating a context summary for each chunk—this can reduce cost significantly.
• If you have web-based content (like store listings, large sets of HTML pages), you can use ScrapeGraphAI to fetch that data, then feed it into the same pipeline.

With these steps, you can adapt the approach to any PDF or external data source—whether it’s a huge enterprise knowledge base, marketing materials, or your company’s internal documentation.

Frequently Asked Questions

This article was published as a part of the Data Science Blogathon.

Hi! I’m Adarsh, a Business Analytics graduate from ISB, currently deep into research and exploring new frontiers. I’m super passionate about data science, AI, and all the innovative ways they can transform industries. Whether it’s building models, working on data pipelines, or diving into machine learning, I love experimenting with the latest tech. AI isn’t just my interest, it’s where I see the future heading, and I’m always excited to be a part of that journey!