App logic

Contains BaseChatApp and large language model integration logic. Implements the core functionality indepent of the UI.

Import statement

from gradiochat.app import *

Explanation of new code

I used the Protocol and runtime_checkable for the first time. Here’s a short explainer.

Python’s Protocol System

Protocols were introduced in Python 3.8 through PEP 544 and are part of the typing module. They provide a way to define interfaces that classes can implement without explicitly inheriting from them - this is called “structural typing” or “duck typing.”

Protocols vs Abstract Base Classes (ABCs)

Abstract Base Classes (Traditional Approach): - Require explicit inheritance (class MyClass(AbstractBaseClass):) - Use the @abstractmethod decorator to mark methods that must be implemented - Check for compatibility based on the class hierarchy (nominal typing) - Enforce implementation at class definition time

Protocols (New Approach): - Don’t require inheritance - classes just need to implement the required methods - Use the Protocol class and @runtime_checkable decorator - Check for compatibility based on method signatures (structural typing) - Can check compatibility at runtime with isinstance() if marked as @runtime_checkable

How It Works

In our code:

@runtime_checkable
class LLMClientProtocol(Protocol):
    def chat_completion(self, messages: List[Message], **kwargs) -> str:
        ...
    
    def chat_completion_stream(self, messages: List[Message], **kwargs) -> Generator[str, None, None]:
        ...

This defines an interface that says “any class with methods named chat_completion and chat_completion_stream with these signatures is considered compatible with LLMClientProtocol.”

The ... in the method bodies is a special syntax that means “this method is required but not implemented here.” It’s similar to pass but specifically for protocol definitions.

Benefits in Our Context

1. Flexibility: We can create any class that implements these methods, and it will be compatible with LLMClientProtocol without explicitly inheriting from it.

2. Easy Testing: We can create mock implementations that automatically satisfy the protocol by just implementing the required methods.

3. Type Checking: Tools like mypy can verify that our classes implement all required methods with the correct signatures.

4. Runtime Checking: With @runtime_checkable, we can use isinstance(obj, LLMClientProtocol) to check if an object implements the protocol.

Example of Use

def process_with_any_llm_client(client: LLMClientProtocol, messages: List[Message]):
    # This function will accept any object that has the required methods,
    # regardless of its class hierarchy
    response = client.chat_completion(messages)
    return response

This would accept our HuggingFaceClient or any other class that implements the required methods, without forcing them to inherit from a common base class.

Define the general LLMClientProtocol structure

Which means it should have the methods defined in LLMClientProtocol.

---

LLMClientProtocol

 LLMClientProtocol (*args, **kwargs)

Protocol defining the interface for LLM clients

Define the LLM Clients

This should at least follow the structure of LLMClientProtocol but can of course be expanded.

HuggingFaceClient

---

HuggingFaceClient

 HuggingFaceClient (model_config:gradiochat.config.ModelConfig)

Client for interacting with HuggingFace models

TogetherAI

---

TogetherAiClient

 TogetherAiClient (model_config:gradiochat.config.ModelConfig)

Client for interacting with models through the TogetherAI API server We use the openai package

Local Ollama client

---

OllamaClient

 OllamaClient (model_config:gradiochat.config.ModelConfig)

Client for interacting with models through a local Ollama API server Uses the official Ollama Python library

Create the LLM client

This function creates the client using the available LLM Client classes. It gets the provider from the model_config. If it finds a LLM Client Class for this provider, it returns that client. If it doesn’t find a LLM Client Class for that provider, it returns a ValueError.

---

create_llm_client

 create_llm_client (model_config:gradiochat.config.ModelConfig)

Factory function to create an LLM client based on the provider.

The internal logic of the chat app

Now the BaseChatApp class is defined. This class is used to instantiate the properties en methods for the internal workings of the chat app. The UI is defined in the ui module.

---

BaseChatApp

 BaseChatApp (config:gradiochat.config.ChatAppConfig)

Base class for creating configurable chat applications with Gradio

Create a HuggingFace test model config

# Eval set to false, because the api key is stored in .env and thus can't be found when
# nbdev_test is run
hf_config = ModelConfig(
    model_name="mistralai/Mistral-7B-Instruct-v0.3", # "Qwen/QwQ-32B" is another possibility, but with vision you need another messages format
    provider="huggingface",
    api_key_env_var="HF_API_KEY",
    api_base_url="https://router.huggingface.co/hf-inference/v1",
    max_completion_tokens=100,
    temperature=0.7
)

# Create the client
client = create_llm_client(hf_config)

Create a Together AI test model config

# Eval set to false, because the api key is stored in .env and thus can't be found when
# nbdev_test is run
ta_config = ModelConfig(
    # model_name="mistralai/Mistral-Nemo-Instruct-2407",
    model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo-Free",
    provider="togetherai",
    api_key_env_var="TG_API_KEY",
)

# Create the client
client = create_llm_client(ta_config)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[11], line 12
      4 ta_config = ModelConfig(
      5     # model_name="mistralai/Mistral-Nemo-Instruct-2407",
      6     model_name="meta-llama/Llama-3.3-70B-Instruct-Turbo-Free",
      7     provider="togetherai",
      8     api_key_env_var="TG_API_KEY",
      9 )
     11 # Create the client
---> 12 client = create_llm_client(ta_config)

Cell In[5], line 9, in create_llm_client(model_config)
      7     return HuggingFaceClient(model_config)
      8 if model_config.provider.lower() == "togetherai":
----> 9     return TogetherAiClient(model_config)
     10 if model_config.provider.lower() == "ollama":
     11     return OllamaClient(model_config)

NameError: name 'TogetherAiClient' is not defined

Create a Ollama test model config

# Eval set to false, because the api key is stored in .env and thus can't be found when
# nbdev_test is run
olla_config = ModelConfig(
    model_name="nchapman/ministral-8b-instruct-2410",
    provider="ollama",
    api_key_env_var="OLLAMA_API_KEY",
)

# Create the client
client = create_llm_client(olla_config)

---------------------------------------------------------------------------
NameError                                 Traceback (most recent call last)
Cell In[10], line 11
      4 olla_config = ModelConfig(
      5     model_name="nchapman/ministral-8b-instruct-2410",
      6     provider="ollama",
      7     api_key_env_var="OLLAMA_API_KEY",
      8 )
     10 # Create the client
---> 11 client = create_llm_client(olla_config)

Cell In[5], line 11, in create_llm_client(model_config)
      9     return TogetherAiClient(model_config)
     10 if model_config.provider.lower() == "ollama":
---> 11     return OllamaClient(model_config)
     12 else:
     13     raise ValueError(f"Unsupported provider: {model_config.provider}")

NameError: name 'OllamaClient' is not defined

test_messages = [
    Message(role="system", content="You are Aurelius Augustinus, helping me to think deeply and be humble and thankfull."),
    Message(role="user", content="Why should I engage with the people around me?")
]
# Test with a simple prompt
try:
    response = client.chat_completion(test_messages)
    print(f"Response received: {response[:100]}...")
except Exception as e:
    print(f"Error: {e}")

# Test with overriden parameters
try:
    print("\nTesting with overridden parameters:")
    response = client.chat_completion(test_messages, max_completion_tokens=50, temperature=0.9)
    print(f"Response: {response[:100]}...")  # Show first 100 chars
except Exception as e:
    print(f"Error: {e}")

Error: Error code: 402 - {'error': 'You have exceeded your monthly included credits for Inference Providers. Subscribe to PRO to get 20x more monthly included credits.'}

Testing with overridden parameters:
Error: Error code: 402 - {'error': 'You have exceeded your monthly included credits for Inference Providers. Subscribe to PRO to get 20x more monthly included credits.'}