Building Business Applications Using SLMs

Businesses today are using AI chatbots to improve customer service and provide instant support. These chatbots powered by artificial intelligence can answer questions and recommend products. Unlike human agents they work 24/7 without breaks making them a valuable tool for companies of all sizes. In this article, we will explore how AI-powered chatbots help businesses in customer service, sales and personalization. 

Learning Objectives 

• Understand how Small Language Models (SLMs) enhance business operations with lower resource consumption.
• Learn how SLMs automate key business tasks like customer support, financial analysis, and document processing.
• Explore the implementation of models like Flan-T5, FinancialBERT, and LayoutLM in business AI applications.
• Analyze the advantages of SLMs over LLMs, including efficiency, adaptability, and industry-specific training.
• Discover real-world use cases of SLMs in AI-driven customer service, finance, and document automation.

This article was published as a part of the Data Science Blogathon.

What are Small Language Models?

Since Large Language Models were too big used too much power and were hard to put on small devices like phones and tablets there was a need for smaller models that could still understand people’s language correctly. Which led to the creation of Small Language Models these are designed to be compact and efficient while providing accurate language understanding. SLMs are specifically made to work well on smaller devices and use less energy. They are also easier to update and maintain. LLMs are trained using massive amounts of computational power and large datasets which means they can easily learn complex patterns and relationships in language.

Their training involves masked language modeling, next sentence prediction, and large-scale pre-training, this allows them to develop a deeper understanding of language. SLMs are trained using more efficient algorithms and smaller datasets, which makes them more compact and efficient. SLMs use knowledge distillation, transfer learning, and efficient pre-training methods, thus getting the same results as larger models while requiring fewer resources.

Difference Between LLMs and SLMs

In the below table we will look into the difference between LLMs and SLMs:

Feature	Large Language Models (LLMs)	Small Language Models (SLMs)
Number of Parameters	Billions to Trillions	Millions to Tens of Millions
Training Data	Massive, diverse datasets	Smaller, more specific datasets
Computational Requirements	Higher (slower, more memory/power)	Lower (faster, less memory/power)
Cost	Higher cost to train and run	Lower cost to train and run
Domain Expertise	More general knowledge across domains	Can be fine-tuned for specific domains
Performance on Simple Tasks	Good to excellent performance	Good performance
Performance on Complex Tasks	Higher capability	Lower capability
Generalization	Strong generalization across tasks/domains	Limited generalization
Transparency/Interpretability	Less transparent	More transparent/interpretable
Example Use Cases	Open-ended dialogue, creative writing, question answering, general NLP	Chatbots, simple text generation, domain-specific NLP
Examples	GPT-3, BERT, T5	ALBERT, DistilBERT, TinyBERT, Phi-3

Advantages

• While LLMs are trained on huge amounts of general data, SLMs can be trained on smaller datasets that are specific to an industry. This makes them really good at understanding the details of language in that industry. 
• SLMs are more transparent and explainable than LLMs. When working with a large language model, it can be hard to understand how it’s making decisions or what it’s basing those decisions on. SLMs are smaller and more straightforward thus makes it easier to understand how they are working. This is really important in industries like healthcare or finance where you need to be able to trust the decisions that the model is making.
• SLMs are more adaptable than LLMs. Because they’re smaller and more specialized we can update or fine-tune them more easily. This makes them really useful for industries where things are changing quickly. For example in the medical field, new research and discoveries are being made all the time. SLMs can be updated quickly to reflect these changes which makes them really useful for medical professionals.

Using SLM in the Business AI

Businesses are increasingly turning to Small Language Models (SLMs) for AI-driven solutions that balance efficiency and cost-effectiveness. With their ability to handle domain-specific tasks while requiring fewer resources, SLMs offer a practical alternative for companies seeking AI-powered automation.

Automating Customer Support with AI Chatbots

Customers expect instant responses to their queries. AI chatbots powered by SLMs enable businesses to provide efficient, round-the-clock support. Key benefits include:

• Automated Customer Support
• Personalized Assistance
• Multilingual Support

Using Google/flan-t5-small for AI Chatbots

Google’s FLAN-T5-Small is a powerful language model that’s part of the T5 (Text-to-Text Transfer Transformer) family. 

Model Architecture:

FLAN-T5-Small is based on the T5 architecture, which is a variant of the Transformer model. It consists of:

• Encoder: Takes in input text and generates a continuous representation.
• Decoder: Generates output text based on the encoded representation.

FLAN-T5-Small Specifics:

This model is a smaller variant of the original T5 model, with approximately 60 million parameters. It’s designed to be more efficient and accessible while still maintaining strong performance.

Training Objectives:

FLAN-T5-Small was trained on a massive corpus of text data using a combination of objectives:

• Masked Language Modeling (MLM): Predicting masked tokens in input text.
• Text-to-Text Generation: Generating output text based on input text.

FLAN (Finetuned Language Net) Adaptation:

The “FLAN” in FLAN-T5-Small refers to a specific adaptation of the T5 model. FLAN involves fine-tuning the model on a diverse set of natural language processing tasks, such as question answering, sentiment analysis, and text classification. This adaptation enables the model to develop a broader understanding of language and improve its performance on various tasks.

Key Features:

• Small size: Approximately 60 million parameters, making it more efficient and accessible.
• Strong performance: Despite its smaller size, FLAN-T5-Small maintains strong performance on various natural language processing tasks.
• Versatility: Can be fine-tuned for specific tasks and adapted to various domains.

Use Cases:

FLAN-T5-Small is suitable for a wide range of natural language processing applications, including:

• Text classification
• Sentiment analysis
• Question answering
• Text generation
• Language translation

Code Example: Using SLMs for AI Chatbot

from transformers import pipeline

# Load the Flan-T5-small model
chatbot = pipeline("text2text-generation", model="google/flan-t5-small")

# Sample customer queries
queries = [
    "What are your business hours?",
    "Do you offer international shipping?",
    "How can I return a product?"
]

# Generate responses
for query in queries:
    response = chatbot(query, max_length=50, do_sample=False)
    print(f"Customer: {query}\nAI: {response[0]['generated_text']}\n")

Input Text

"What are your business hours?",
"Do you offer international shipping?",
"How can I return a product?

Output

Customer: What are your business hours?
AI: 8:00 a.m. - 5:00 p.m.
Customer: Do you offer international shipping?
AI: no
Customer: How can I return a product?
AI: Return the product to the store.

Financial Analysis and Forecasting

SLMs enable businesses to make data-driven financial decisions by analyzing trends and forecasting market conditions. Use cases include:

• Sales Prediction
• Risk Assessment
• Investment Insights

Using FinancialBERT for Market Analysis

Financial BERT is a pre-trained language model specifically designed for financial text analysis. It’s a variant of the popular BERT (Bidirectional Encoder Representations from Transformers) model, fine-tuned for financial applications.

Financial BERT is trained on a large corpus of financial texts, such as:

• Financial news articles
• Company reports
• Financial statements
• Stock market data

This specialized training enables Financial BERT to better understand financial terminology, concepts, and relationships. It’s particularly useful for tasks like:

• Sentiment analysis: Analyzing financial text to determine market sentiment, investor attitudes, or company performance.
• Event extraction: Identifying specific financial events, such as mergers and acquisitions, earnings announcements, or regulatory changes.
• Risk analysis: Assessing financial risk by analyzing text data from financial reports, news articles, or social media.
• Portfolio optimization: Using natural language processing (NLP) to analyze financial text and optimize investment portfolios.

Financial BERT has many applications in finance, including:

• Quantitative trading: Using machine learning models to analyze financial text and make informed trading decisions.
• Risk management: Identifying potential risks and opportunities by analyzing financial text data.
• Investment research: Analyzing financial reports, news articles, and social media to inform investment decisions.

Code Example: Using SLMs for Market Analysis

from transformers import AutoModelForSequenceClassification, AutoTokenizer, pipeline

# Load FinancialBERT model
tokenizer = AutoTokenizer.from_pretrained("yiyanghkust/finbert-tone")
model = AutoModelForSequenceClassification.from_pretrained("yiyanghkust/finbert-tone")

# Create a sentiment analysis pipeline
finance_pipeline = pipeline("text-classification", model=model, tokenizer=tokenizer)

# Sample financial news headlines
headlines = [
    "Tech stocks rally as investors anticipate strong earnings.",
    "Economic downturn leads to market uncertainty.",
    "Central bank announces interest rate hike, impacting stock prices."
]

# Analyze sentiment
for news in headlines:
    result = finance_pipeline(news)
    print(f"News: {news}\nSentiment: {result[0]['label']}\n")

Input Text

"Tech stocks rally as investors anticipate strong earnings.",
"Economic downturn leads to market uncertainty.",
"Central bank announces interest rate hike, impacting stock prices"

Output

News: Tech stocks rally as investors anticipate strong earnings.
Sentiment: Positive
News: Economic downturn leads to market uncertainty.
Sentiment: Negative
News: Central bank announces interest rate hike, impacting stock prices.
Sentiment: Neutral

Enhancing Document Processing with AI

Processing large volumes of business documents manually is inefficient. SLMs can:

• Summarize lengthy reports
• Extract key information
• Ensure compliance

Using LayoutLM for Document Analysis

Microsoft developed LayoutLM-base-uncased as a pre-trained language model. It leverages a transformer-based architecture specifically designed for tasks that require understanding the visual layout of documents.

Key Features:

• Multi-modal input: LayoutLM-base-uncased takes two types of input:
  • Text: The text content of the document.
  • Layout: The visual layout of the document, including the position and size of text, images, and other elements.
• Text embeddings: The model uses a transformer-based architecture to generate text embeddings, which are numerical vectors that represent the meaning of the text.
• Layout embeddings: The model also generates layout embeddings, which are numerical vectors that represent the visual layout of the document.
• Fusion of text and layout embeddings: The model combines the text and layout embeddings to create a joint representation of the document.

How does it work?

Here’s a high-level overview of how LayoutLM-base-uncased works:

• Text and layout input: The model takes in the text content and visual layout of a document.
• Text embedding generation: The model generates text embeddings using a transformer-based architecture.
• Layout embedding generation: The model generates layout embeddings using a separate neural network.
• Fusion of text and layout embeddings: The model combines the text and layout embeddings using a fusion layer.
• Joint representation: The output of the fusion layer is a joint representation of the document, which captures both the text content and visual layout.

Applications

• Document analysis: You can use LayoutLM-base-uncased for tasks such as document classification, entity extraction, and sentiment analysis.
• Form understanding: You can use the model to extract data from forms, including text, checkboxes, and other visual elements.
• Receipt analysis: You can use LayoutLM-base-uncased to extract relevant information from receipts, including items purchased, prices, and totals.

Advantages:

• Improved accuracy: By combining text and layout information, LayoutLM-base-uncased can achieve higher accuracy on tasks that require an understanding of the visual layout of documents.
• Flexibility: The model can be fine-tuned for a variety of tasks and applications.
• Efficient: LayoutLM-base-uncased is a relatively efficient model, requiring less computational resources than some other pre-trained language models.

Code Example: Using SLMs for Market Analysis

from transformers import AutoModelForTokenClassification, AutoTokenizer, pipeline

# Load LayoutLM model
tokenizer = AutoTokenizer.from_pretrained("microsoft/layoutlm-base-uncased")
model = AutoModelForTokenClassification.from_pretrained("microsoft/layoutlm-base-uncased")

# Create a document analysis pipeline
doc_analyzer = pipeline("ner", model=model, tokenizer=tokenizer)

# Sample business document text
business_doc = "Invoice #12345: Total Amount Due: $500. Payment Due Date: 2024-06-30."

# Extract key data
data_extracted = doc_analyzer(business_doc)
print(data_extracted)

Input Text

"Invoice #12345: Total Amount Due: $500. Payment Due Date: 2024-06-30

Output

[{'entity': 'LABEL_0', 'score': 0.5759164, 'index': 1, 'word': 'in', 'start': 0,
'end': 2}, {'entity': 'LABEL_0', 'score': 0.6300008, 'index': 2, 'word': '##vo',
'start': 2, 'end': 4}, {'entity': 'LABEL_0', 'score': 0.6079731, 'index': 3, 
'word': '##ice', 'start': 4, 'end': 7}, {'entity': 'LABEL_0', 'score': 0.6304574,
'index': 4, 'word': '#', 'start': 8, 'end': 9}, {'entity': 'LABEL_0', 'score':
0.6141283, 'index': 5, 'word': '123', 'start': 9, 'end': 12}, {'entity': 'LABEL_0',
'score': 0.5887407, 'index': 6, 'word': '##45', 'start': 12, 'end': 14}, {'entity':
'LABEL_0', 'score': 0.631358, 'index': 7, 'word': ':', 'start': 14, 'end': 15}, 
{'entity': 'LABEL_0', 'score': 0.6065132, 'index': 8, 'word': 'total', 'start': 16, 
'end': 21}, {'entity': 'LABEL_0', 'score': 0.62801933, 'index': 9, 'word': 'amount',
'start': 22, 'end': 28}, {'entity': 'LABEL_0', 'score': 0.60564953, 'index': 10,
'word': 'due', 'start': 29, 'end': 32}, {'entity': 'LABEL_0', 'score': 0.62605065,
'index': 11, 'word': ':', 'start': 32, 'end': 33}, {'entity': 'LABEL_0', 'score':
0.61071014, 'index': 12, 'word': '$', 'start': 34, 'end': 35}, {'entity': 
'LABEL_0', 'score': 0.6122757, 'index': 13, 'word': '500', 'start': 35, 'end': 38},
{'entity': 'LABEL_0', 'score': 0.6424746, 'index': 14, 'word': '.', 'start': 38,
'end': 39}, {'entity': 'LABEL_0', 'score': 0.60535395, 'index': 15, 'word':
'payment', 'start': 40, 'end': 47}, {'entity': 'LABEL_0', 'score': 0.60176647,
'index': 16, 'word': 'due', 'start': 48, 'end': 51}, {'entity': 'LABEL_0', 'score':
0.6392822, 'index': 17, 'word': 'date', 'start': 52, 'end': 56}, {'entity': 
'LABEL_0', 'score': 0.6197982, 'index': 18, 'word': ':', 'start': 56, 'end': 57}, 
{'entity': 'LABEL_0', 'score': 0.6305164, 'index': 19, 'word': '202', 'start': 58,
'end': 61}, {'entity': 'LABEL_0', 'score': 0.5925634, 'index': 20, 'word': '##4',
'start': 61, 'end': 62}, {'entity': 'LABEL_0', 'score': 0.6188032, 'index': 21,
'word': '-', 'start': 62, 'end': 63}, {'entity': 'LABEL_0', 'score': 0.6260454, 
'index': 22, 'word': '06', 'start': 63, 'end': 65}, {'entity': 'LABEL_0', 'score':
0.6231731, 'index': 23, 'word': '-', 'start': 65, 'end': 66}, {'entity': 'LABEL_0',
'score': 0.6299959, 'index': 24, 'word': '30', 'start': 66, 'end': 68}, {'entity':
'LABEL_0', 'score': 0.63334775, 'index': 25, 'word': '.', 'start': 68, 'end': 69}]

Conclusion 

Small Language Models are revolutionizing business AI by offering lightweight and efficient solutions for automation. Whether used in customer support, financial forecasting, or document processing, SLMs provide businesses with scalable AI capabilities while minimizing computational overhead. By leveraging models like Flan-T5, FinancialBERT, and LayoutLM companies can enhance their workflows reduce costs, and improve decision-making.

Link to Notebook.

Key Takeaways 

• SLMs offer efficient, privacy-friendly alternatives to LLMs for various business applications.
• Models like Flan-T5, FinancialBERT, and LayoutLM can automate customer support, financial analysis, and document processing.
• Businesses can enhance AI performance by integrating additional techniques such as NER, OCR, and time-series forecasting.

Frequently Asked Questions

The media shown in this article is not owned by Analytics Vidhya and is used at the Author’s discretion.