Qafiyah

Open-source repository of Arabic poetry with database dumps, REST API, and web interface.

Website · API · X Bot · Database Dumps

Qafiyah is an open-source corpus of classical Arabic poetry spanning the major historical eras. It offers full-text Arabic search (Elasticsearch); faceted browsing by era, meter, rhyme letter, theme, and collection; a public REST API with auto-generated OpenAPI docs; and downloadable PostgreSQL dumps. An X bot posts a random poems daily. Built for readers, researchers, and developers working with classical Arabic literature.

One request, no auth, returns a random classical Arabic poem as plain text.

curl https://api.qafiyah.com/v1/poems/random?option=lines

Full schema and interactive playground: api.qafiyah.com/v1/docs.

• Qafiyah
  • About
  • Try it
  • Table of Contents
  • Tech Stack
    • Core
    • Web (apps/web)
    • API (apps/api)
    • Bot (apps/bot)
    • Data Layer
    • Search
    • Tooling
  • Architecture
  • Database
  • Getting Started
    • Prerequisites
    • Installation
    • Development
  • Scripts
  • Rate Limits and Terms of Use
  • Documentation
  • Sponsor
  • License

Bun + Turborepo monorepo: four apps and five shared packages.

qafiyah/
├── apps/
│   ├── web/          Astro on-demand SSR (Bun + @astrojs/node) behind nginx proxy_cache; renders each route per request by fetching the API via oRPC
│   ├── api/          Hono REST API (browsing + /v1/search), Bun server (Docker container)
│   ├── worker/       Postgres→Elasticsearch sync (boot reindex + weekly reconcile), Bun server (Docker container)
│   └── bot/          X/Twitter bot; posts 4× daily via GitHub Actions cron
└── packages/
    ├── db/           Drizzle ORM schema, queries, and Postgres client factory
    ├── search/       Elasticsearch client, Arabic analyzers/mappings, query + reindex/reconcile
    ├── contracts/    Shared oRPC contract definitions
    ├── constants/    Shared brand, URLs, and dev-port constants
    └── typescript/   Shared TypeScript configs (base, astro, bun)

Package dependencies, who imports whom at compile time:

graph TD
  subgraph APPS
    WEB["apps/web\nAstro · React islands"]
    API["apps/api\nHono · Bun server (Docker)"]
    WORKER["apps/worker\nPG→ES sync · Bun (Docker)"]
    BOT["apps/bot\nGitHub Actions cron"]
  end
  subgraph PACKAGES
    DB["packages/db\nDrizzle ORM · queries"]
    SEARCH["packages/search\nElasticsearch · Arabic analyzers"]
    CONTRACTS["packages/contracts\noRPC · Valibot schemas"]
    CONSTANTS["packages/constants"]
  end
  WEB --> CONTRACTS & CONSTANTS
  API --> DB & SEARCH & CONTRACTS & CONSTANTS
  WORKER --> DB & SEARCH & CONSTANTS
  BOT --> CONTRACTS & CONSTANTS
  DB --> SEARCH & CONTRACTS & CONSTANTS
  SEARCH --> CONSTANTS
  CONTRACTS --> CONSTANTS

packages/typescript ships shared tsconfig presets (base, astro, bun) consumed via extends, not code imports; omitted from the graph above.

Two architectural constraints. packages/db is consumed only by apps/api and apps/worker; no Drizzle or Postgres imports exist under apps/web or apps/bot. apps/web has no DB access: it renders each route on demand (SSR), querying the API per request through server-only oRPC accessors in src/lib/server/ (pointed at INTERNAL_API_URL), while browser islands fetch the public API via src/lib/api/ (rpc.ts, client.ts, orpc.ts).

Runtime data flow, how requests move once deployed:

graph LR
  BROWSER["Browser"]
  BOT["apps/bot"]
  GHA(["GitHub Actions\ncron 4×/day"])
  TW(["X / Twitter"])

  subgraph VPS["Docker on VPS · docker compose"]
    subgraph WEBIMG["web image"]
      NGINX["nginx\nproxy_cache · static assets"]
      WEB["Astro SSR · Bun\napps/web"]
    end
    API["apps/api\nHono · Bun"]
    WORKER["apps/worker\nPG→ES sync"]
    PG[("PostgreSQL")]
    ES[("Elasticsearch")]
  end

  BROWSER -->|"page loads"| NGINX
  NGINX -->|"proxy 127.0.0.1:4321"| WEB
  BROWSER -->|"island data · search, random"| API
  WEB -->|"per-request SSR oRPC · INTERNAL_API_URL"| API
  API -->|"browsing · @qafiyah/db · SQL"| PG
  API -->|"/v1/search · @qafiyah/search"| ES
  WORKER -->|"read · @qafiyah/db"| PG
  WORKER -->|"reindex / reconcile"| ES
  GHA --> BOT
  BOT -->|"GET /poems/random"| API
  BOT -->|"post tweet"| TW

Everything inside Docker on VPS ships from docker-compose.yml (docker compose up -d --build or bun run deploy): Postgres, Elasticsearch, api, worker, and the web image. The web image bundles nginx (proxy_cache + static assets) in front of the Astro SSR server; the SSR server reaches api over the internal network (INTERNAL_API_URL). The api reads Postgres for browsing and Elasticsearch for /v1/search; the worker keeps Elasticsearch in sync with Postgres (boot reindex + weekly reconcile). Browser islands call the public API directly. The bot runs on GitHub Actions, outside the VPS.

The corpus is modeled as poems and verses, organized by poet, era, meter, rhyme letter, theme, and collection. Live counts come from the data, not this README.

PostgreSQL custom-format dumps are published in dumps/ and refreshed periodically. They are provided for research and integration as an alternative to scraping the API; see the restore instructions.

• Bun (version pinned via the root packageManager field)
• Docker (runs Postgres, Elasticsearch, and the apps)
• PostgreSQL with pg_restore (only needed for restoring dumps directly; the Dockerized workflow handles this automatically)

git clone https://github.com/alwalxed/qafiyah.git
cd qafiyah
bun install

bun run dev        # seeded Postgres + Elasticsearch + worker (Docker) + hot-reloading web & API

dev brings up the database and Elasticsearch containers (DB auto-seeded on first boot; the worker reindexes Elasticsearch if its alias is empty), writes the local .env files, then starts the dev servers. For a full containerized run, use bun run up.

Dev and build

Quality

Boundary checks

Aggregate and utilities

The API is free, requires no authentication, and is provided on a best-effort basis with no SLA.

• Fair use. Per-IP throttling is enforced at the server level. For bulk access, prefer the PostgreSQL dumps over paginating the API.
• Caching. Responses are cacheable; cache them client-side when possible to reduce load.
• Stability. v1 endpoints are stable. Breaking changes ship behind a new major version.
• Attribution. Not required, but appreciated if you publish work that relies on the corpus.

• Search Implementation
• Deployment
• Contributing Guidelines
• Code of Conduct
• Security Policy

If Qafiyah is useful to you, support the project on GitHub Sponsors. Sponsorship funds dataset upkeep, API hosting, and ongoing maintenance.

Released under the MIT License.