Hugging Face Endpoint Deployment

This repository contains the necessary files and configurations for deploying large language models on Hugging Face Endpoints using custom containers.

Project Structure

.
├── .github/
│   └── workflows/
│       └── docker-build-push.yml
├── QVQ-72B-Preview/
│   ├── Dockerfile
│   ├── requirements.txt
│   ├── README.md
│   ├── app/
│   │   ├── __init__.py
│   │   ├── main.py
│   │   ├── models.py
│   │   ├── config.py
│   │   ├── cache.py
│   │   ├── model_manager.py
│   │   ├── middleware.py
│   │   ├── mcp_client.py
│   │   ├── routes.py
│   │   └── utils.py
│   ├── custom_layers/
│   │   └── rotary.py
│   └── qwen_vl_utils/
│       └── __init__.py
└── README.md

Setup Instructions

Prerequisites

• Docker installed with NVIDIA Container Toolkit
• Access to a container registry (Docker Hub or GitHub Container Registry)
• Hugging Face account with Endpoints access
• Python 3.11 or later
• NVIDIA GPU with CUDA 12.6 support
• At least 48GB system RAM recommended

CI/CD Pipeline

The repository includes a GitHub Actions workflow that automatically builds and pushes Docker images to GitHub Container Registry when changes are pushed to the main branch.

Workflow Features:

• Automatic Docker image building
• Model download at runtime
• GPU-optimized builds
• Push to GitHub Container Registry

Deployment

1. Build the Docker Image

docker build -t qvq-72b-preview-server .

2. Test Locally

docker run --gpus all -p 80:80 qvq-72b-preview-server

3. Push to Container Registry

docker tag qvq-72b-preview-server ghcr.io/your-username/qvq-72b-preview-server:latest
docker push ghcr.io/your-username/qvq-72b-preview-server:latest

4. Deploy on Hugging Face Endpoints

1. Go to Hugging Face Endpoints
2. Create a new endpoint
3. Choose "Custom Container"
4. Enter your container registry URI
5. Configure compute resources (GPU recommended)
6. Deploy

API Documentation

The server provides two main endpoints:

1. OpenAI-Compatible Chat Endpoint (/v1/chat/completions)

{
    "model": "QVQ-72B-Preview",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "Hello, how are you?"
        }
    ],
    "temperature": 0.7,
    "max_tokens": 8192
}

2. Multimodal Generation Endpoint (/generate)

{
    "messages": [
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": "System prompt here"
                }
            ]
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image",
                    "image": "image_url_or_base64"
                },
                {
                    "type": "text",
                    "text": "User query here"
                }
            ]
        }
    ]
}

The response will be in the format:

{
    "output": "Generated response text"
}

Server Architecture

The server code is organized into modular components:

• models.py - Pydantic data models for request/response handling
• config.py - Configuration settings and environment variables
• cache.py - Caching functionality for model outputs and weights
• model_manager.py - Model initialization and management
• middleware.py - FastAPI middleware functions
• mcp_client.py - MCP client setup and interactions
• routes.py - API endpoints implementation
• main.py - Application entry point
• utils.py - Utility functions for image/video processing

Contributing

Feel free to submit issues and enhancement requests. When contributing, please:

1. Fork the repository
2. Create a new branch for your feature
3. Submit a pull request with detailed description of changes

License

This project is licensed under the terms specified by the QVQ-72B-Preview model license.