# 🌌 Papers with Code AI Curator (PWC-Curator)

[![License: MIT](https://img.shields.io/badge/License-MIT-purple.svg)](https://opensource.org/licenses/MIT)
[![FastAPI](https://img.shields.io/badge/API-FastAPI-009688.svg?style=flat&logo=fastapi)](https://fastapi.tiangolo.com/)
[![PostgreSQL](https://img.shields.io/badge/Database-PostgreSQL-336791.svg?style=flat&logo=postgresql)](https://www.postgresql.org/)
[![Docker](https://img.shields.io/badge/Orchestration-Docker-2496ED.svg?style=flat&logo=docker)](https://www.docker.com/)
[![Ollama](https://img.shields.io/badge/Inference-Ollama-black.svg?style=flat)](https://ollama.com/)

A production-grade, local-first RAG (Retrieval-Augmented Generation) system and research workspace designed to turn the firehose of new AI research into structured, searchable knowledge.

Unlike generic ArXiv search chatbots, **PWC-Curator** integrates code implementations, evaluation benchmarks, framework tags, and GitHub stars as first-class citizens alongside the paper's text, directly inspired by **paperswithcode.com**.



## 🏗️ Architectural Overview

The system is fully containerized inside a private, high-performance Docker network, isolating your database, backend, and static file routers:

```mermaid
graph TD
    %% Presentation Layer
    subgraph UI [Presentation Layer - Web UI]
        Browser[User Browser]
        SleekDashboard[PWC-Style Dashboard - Papers, Code & SOTA Tables]
        DynamicScraper[Real-Time Curator Input - Paste PWC URL]
        RAGWorkspace[Splitscreen Conversational Workspace]
    end

    %% API Layer
    subgraph API [FastAPI Backend Service]
        FastAPI[FastAPI Router]
        Scheduler[Background Sync & Batch Ingestion Worker]
        SearchEngine[Hybrid Dense-Sparse Search Engine]
        RAGEngine[RAG Prompt & Citation Orchestrator]
    end

    %% Ingestion Pipeline
    subgraph Pipeline [PWC Ingestion & Scraping Pipeline]
        PWCScraper[Dynamic Web Scraper - BeautifulSoup]
        PDFDownloader[Parallel PDF Downloader]
        Parser[Structural PDF Text Extractor]
    end

    %% Storage Layer
    subgraph Data [Storage Layer - Docker Postgres]
        Postgres[(PostgreSQL + pgvector)]
        PaperTable[papers - Rich Metadata, Stars, Tasks, Benchmark Scores]
        ChunkTable[chunks - Structural text & embeddings]
        GINIndex[tsvector - BM25 Keyword Index]
        VectorIndex[HNSW - nomic-embed-text Vector Index]
    end

    %% Local Inference Services
    subgraph InferenceHost [Local or Remote Inference Host]
        LocalOllama[Ollama Daemon - Port 11434]
        Llama3[Llama 3.1 8B LLM]
        NomicEmbed[nomic-embed-text]
    end

    %% Flow connections
    Browser <-->|HTTP / SSE| FastAPI
    SleekDashboard <--> FastAPI
    DynamicScraper -->|POST URL| FastAPI
    RAGWorkspace <--> FastAPI
    
    FastAPI --> PWCScraper
    PWCScraper --> PDFDownloader
    PDFDownloader --> Parser
    Parser --> Postgres
    
    FastAPI <--> Postgres
    FastAPI <--> LocalOllama
    RAGEngine <--> LocalOllama
```

---

## 🚀 Key Technical Core Choices

### 1. Dual Ingestion Engine (PWC-First)
*   **Automatic Seeding**: On initial boot, if the database is empty, the scheduler runs an automated, selective background seeding thread, fetching high-starred papers across primary categories (e.g., Large Language Models, Transformers, Diffusion Models).
*   **On-Demand Real-Time Curation**: Paste any Papers with Code URL (e.g., `https://paperswithcode.com/paper/attention-is-all-you-need`). The backend uses BeautifulSoup to crawl:
    *   **Evaluation benchmarks**: Dataset names, metrics, values, and global rank.
    *   **Repository details**: Official code URL, GitHub star counts, and framework tags (e.g., PyTorch, JAX).
    *   **Domain tags**: Machine learning task badges and dataset labels.

### 2. Structural PDF Parsing (PyMuPDF coordinate tracking)
To prevent the typical lost-in-context issues of raw chunking, we developed a PyMuPDF parser that extracts layout structures:
*   Identifies section boundaries (e.g., *1. Introduction*, *4. Proposed Architecture*) to index paragraphs contextually.
*   Preserves exact page numbers for hyper-precise page citations inside the RAG chatbot.
*   Detects and trims references/bibliographies to keep noise out of the embeddings.

### 3. PostgreSQL + `pgvector` Hybrid Database Schema
All relations, rich metadata lists, and vector embeddings are stored within a unified database, eliminating synchronization lag between document stores and separate vector databases:
*   **`papers` table**: Houses metadata, tasks arrays (`TEXT[]`), and evaluation metrics (`JSONB`).
*   **`chunks` table**: Houses divided paragraph segments (`TEXT`) and their 768-dimensional embeddings (`VECTOR`).
*   Optimized with a fast lexical **GIN full-text search index** on combined title/abstract columns and an **HNSW vector index** with cosine distance operators.

### 4. Dynamic Hybrid Search (Lexical GIN + Dense HNSW)
Our search engine addresses the classic trade-offs between keyword lookup and conceptual matching. A custom linear-comb score fusion is calculated dynamically:

$$\text{Final Score} = w \cdot \text{Dense Score} + (1 - w) \cdot \text{Sparse Score}$$

*   **Lexical Index (BM25 Equivalent)**: Powered by PostgreSQL's `tsvector` and `tsquery` across title, abstract, and text content. Highly accurate for specific terms ("RoPE", "AdamW").
*   **Dense Index (Cosine Vector Similarity)**: Powered by local `nomic-embed-text` embeddings. Outstanding for thematic matching ("scaling laws", "context limits").
*   **Real-time Fusion Slider**: A frontend slider lets you adjust the weight value ($w \in [0, 1]$) in real-time, instantly refreshing results with hybrid scoring!

---

## 🛠️ How to Deploy & Run Locally

### Prerequisites
*   [Docker and Docker Compose](https://docs.docker.com/get-docker/) installed.
*   [Ollama](https://ollama.com/) running on your host system.

### Step 1: Prepare Ollama Models
Make sure Ollama is active on your machine, then pull the recommended model weights:
```bash
# Pull 768-dimensional local text embedding model
ollama pull nomic-embed-text

# Pull primary text generation LLM
ollama pull llama3.1:8b
```

### Step 2: Configure Environment
Create a `.env` file inside the root directory or configure variables directly in `docker-compose.yml`:
```env
DATABASE_URL=postgresql://curator_admin:curator_secure_pass_2026@acr-postgres:5432/pwc_curator
OLLAMA_HOST=http://host.docker.internal:11434
EMBED_MODEL=nomic-embed-text
CHAT_MODEL=llama3.1:8b
```

### Step 3: Spin Up Containers
Launch the stack in the background using Docker Compose:
```bash
docker compose up -d --build
```

Docker will initialize:
1.  `acr-postgres`: Ready with the `pgvector` extension.
2.  `acr-backend`: Ready to parse, index, and query.
3.  `acr-web-proxy`: Unified Nginx serving static assets and reverse proxying backend API endpoints.

### Step 4: Open Workspace
Open your browser and navigate to:
👉 **`http://localhost:8095`**

---

## 📜 License
This project is licensed under the [MIT License](LICENSE).