Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 15 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# Lingput - AI-Powered Comprehensible Input for Language Learning

[![Tests](https://github.com/mark-mdev/lingput/actions/workflows/pr-tests.yml/badge.svg)](https://github.com/mark-mdev/lingput/actions/workflows/pr-tests.yml)
[![Deploy](https://github.com/mark-mdev/lingput/actions/workflows/deploy.yml/badge.svg)](https://github.com/mark-mdev/lingput/actions/workflows/deploy.yml)
[![Tests](https://github.com/markmdev/lingput/actions/workflows/pr-tests.yml/badge.svg)](https://github.com/markmdev/lingput/actions/workflows/pr-tests.yml)
[![Deploy](https://github.com/markmdev/lingput/actions/workflows/deploy.yml/badge.svg)](https://github.com/markmdev/lingput/actions/workflows/deploy.yml)

**Demo:**
https://lingput.dev/
Expand All @@ -18,15 +18,15 @@ Unlike generic flashcard apps, Lingput adapts to your vocabulary and provides **

## Architectural & Technical Highlights

This project was built to production-grade standards, demonstrating expertise in full-stack development, system design, and scalability. Here are the key technical features:

- **Scalable Background Processing:** Utilizes a robust **Job Queue System (BullMQ & Redis)** to handle complex, long-running AI tasks (story and audio generation) asynchronously. This ensures the API remains fast and responsive, providing a seamless user experience with real-time progress updates on the frontend.
- **Clean Backend Architecture:** The Express.js backend is built on a **testable, multi-layered architecture** (Controller, Service, Repository) with **Dependency Injection** for loose coupling and maintainability.
- **Robust Caching Strategy:** Leverages **Redis** for caching frequently accessed data (like stories and word lists), significantly reducing database load and improving API response times.
- **Robust Caching Strategy:** Leverages **Redis** for caching frequently accessed data (like stories and word lists), significantly **reducing database load** and improving API response times.
- **Secure Authentication:** Implements a secure, modern authentication system using **HTTP-only cookies** with access and refresh tokens to protect against XSS attacks.
- **Advanced Frontend State Management:** The Next.js frontend features a **custom React hook (`handleJob`)** to intelligently manage the lifecycle of background jobs, abstracting away the complexity of polling and providing optimistic UI updates.
- **Containerized for Production:** The entire application is containerized using **Docker and Docker Compose**, ensuring consistent, reproducible deployments for all services (backend, frontend, workers, NGINX).

Full tech stack: [Tech Stack](#tech-stack)

---

## CI/CD
Expand All @@ -42,7 +42,7 @@ This repo ships with a simple, reliable pipeline built around **Docker**, **GitH

On every **Pull Request** and on **pushes to `main`**, GitHub Actions runs:

- **ESLint** for the codebase.
- **ESLint**.
- **Unit/Integration tests**.
- Dependency caching to keep CI fast.

Expand Down Expand Up @@ -71,7 +71,6 @@ On every **Pull Request** and on **pushes to `main`**, GitHub Actions runs:
- [Features](#features)
- [Tech Stack](#tech-stack)
- [Quickstart](#quickstart)
- [Production Deploy](#production-deploy)
- [Roadmap](#roadmap)
- [Contributing](#contributing)
- [License](#license)
Expand All @@ -88,14 +87,14 @@ On every **Pull Request** and on **pushes to `main`**, GitHub Actions runs:

## Features

- 🔐 **Auth with secure cookies** - register/login with HTTP-only tokens, refresh flow included.
- 📊 **Vocabulary assessment** - quick test estimates your vocab size using a frequency list.
- 📚 **Personalized story generation** - AI generates stories with your known words (plus a few new).
- 🌍 **Chunked translation** - story is split into chunks with translations for easier comprehension.
- 🎧 **Audio generation** - full audio track (story + translations with pauses), stored in Supabase.
- 📝 **Smart word tracking** - The app doesn't just show translations, it saves words with examples and helps you track your progress.
- **Background jobs** - BullMQ workers handle long-running tasks with progress updates.
- 🚀 **Caching** - Redis caches stories and word lists for fast responses.
- **Auth with secure cookies** - register/login with HTTP-only tokens, refresh flow included.
- **Vocabulary assessment** - quick test estimates your vocab size using a frequency list.
- **Personalized story generation** - AI generates stories with your known words (plus a few new).
- **Chunked translation** - story is split into chunks with translations for easier comprehension.
- **Audio generation** - full audio track (story + translations with pauses), stored in Supabase.
- **Smart word tracking** - The app doesn't just show translations, it saves words with examples and helps you track your progress.
- **Background jobs** - BullMQ workers handle long-running tasks with progress updates.
- **Caching** - Redis caches stories and word lists for fast responses.

---

Expand Down Expand Up @@ -151,7 +150,7 @@ On every **Pull Request** and on **pushes to `main`**, GitHub Actions runs:

```bash
# Clone the repository
git clone https://github.com/mark-mdev/lingput
git clone https://github.com/markmdev/lingput
```

Create `.env` files for backend and frontend:
Expand Down Expand Up @@ -185,25 +184,6 @@ App: [http://localhost:3050](http://localhost:3050)

---

## Production Deploy

Use `docker-compose.yml` with prebuilt images:

- `markmdev/lingput-backend`
- `markmdev/lingput-worker`
- `markmdev/lingput-frontend`
- `markmdev/lingput-lemma`
- `markmdev/lingput-nginx`

Steps:

1. Build & push images to your registry.
2. Update image names in `docker-compose.yml`.
3. Provide production env vars (`OPENAI_API_KEY`, `SUPABASE_URL`, `SUPABASE_SERVICE_API_KEY`, `DATABASE_URL`, `REDIS_HOST`, `REDIS_PORT`, `JWT_SECRET`, etc).
4. Expose NGINX (`80` by default).

---

## Contributing

Contributions welcome!
Expand Down
2 changes: 1 addition & 1 deletion apps/backend/.env.defaults
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,4 @@ REDIS_HOST=redis
REDIS_PORT=6379
DATABASE_URL=postgresql://lingput:cmpinputpswd@postgres:5432/lingput?schema=public
APP_PORT=4000
LEMMA_SERVICE_URL=http://lemma:8000
LEMMA_SERVICE_URL=http://lemmas:8000
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ export class LemmatizationService {
let response: OpenAIResponse;
try {
response = await this.openai.responses.create({
model: "gpt-5-mini",
reasoning: { effort: "low" },
model: "gpt-5-nano",
reasoning: { effort: "minimal" },
input: [
{
role: "system",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,8 +13,8 @@ export class StoryGeneratorService {
let response: OpenAIResponse;
try {
response = await this.openai.responses.create({
model: "gpt-5-mini",
reasoning: { effort: "low" },
model: "gpt-5",
reasoning: { effort: "minimal" },
input: [
{
role: "system",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ export class TranslationService {
let response: OpenAIResponse;
try {
response = await this.openai.responses.create({
model: "gpt-5-mini",
reasoning: { effort: "low" },
model: "gpt-5",
reasoning: { effort: "minimal" },
input: [
{
role: "system",
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,13 @@ const answerSchema = z.object({
export class VocabAssessmentController {
constructor(private vocabAssessmentService: VocabAssessmentService) {}

skip = async (req: AuthedRequest, res: Response) => {
const user = req.user;

await this.vocabAssessmentService.skipAssessment(user.userId, "en", "de");
res.status(200).json(formatResponse({ success: true }));
};

start = async (req: AuthedRequest, res: Response) => {
const user = req.user;

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,7 @@ export function buildVocabAssessmentRouter(
const router = Router();

router.get("/start", authMiddleware, asyncHandler(controller.start));
router.post("/skip", authMiddleware, asyncHandler(controller.skip));
router.post("/answer", authMiddleware, asyncHandler(controller.answer));

return router;
Expand Down
58 changes: 41 additions & 17 deletions apps/backend/src/modules/vocabAssessment/vocabAssessmentService.ts
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,21 @@ export class VocabAssessmentService {
private redisWordsCache: RedisWordsCache,
) {}

async skipAssessment(
userId: number,
sourceLanguage: string,
targetLanguage: string,
) {
const words = await this.getWordRanking(sourceLanguage, targetLanguage);
const knownVocabulary = words.slice(0, 50);
const vocabularyDTO: UserVocabularyDTO[] = knownVocabulary.map((word) => ({
word: word.word,
translation: word.translation,
article: null,
}));
await this.vocabularyService.saveManyWords(vocabularyDTO, userId);
}

async startAssessment(
userId: number,
sourceLanguage: string,
Expand Down Expand Up @@ -113,23 +128,7 @@ export class VocabAssessmentService {
};
}

let words: WordRanking[] | null;
words = await this.redisWordsCache.getWords(sourceLanguage, targetLanguage);
if (!words) {
words = await this.vocabAssessmentRepository.getWords(
sourceLanguage,
targetLanguage,
);
try {
await this.redisWordsCache.saveWords(
sourceLanguage,
targetLanguage,
words,
);
} catch (error) {
logger.error("[cache] Failed to save words in Redis", error);
}
}
const words = await this.getWordRanking(sourceLanguage, targetLanguage);

const wordsToReview = state.wordsToReview;
const result = this.checkAnswer(answer, wordsToReview);
Expand Down Expand Up @@ -191,6 +190,31 @@ export class VocabAssessmentService {
};
}

private async getWordRanking(
sourceLanguage: string,
targetLanguage: string,
): Promise<WordRanking[]> {
let words: WordRanking[] | null;
words = await this.redisWordsCache.getWords(sourceLanguage, targetLanguage);
if (!words) {
words = await this.vocabAssessmentRepository.getWords(
sourceLanguage,
targetLanguage,
);
try {
await this.redisWordsCache.saveWords(
sourceLanguage,
targetLanguage,
words,
);
} catch (error) {
logger.error("[cache] Failed to save words in Redis", error);
}
}

return words;
}

private checkAnswer(
answer: Record<string, boolean>,
wordsToReview: WordRanking[],
Expand Down
3 changes: 2 additions & 1 deletion apps/frontend/src/app/(protected)/dashboard/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@ import useAuthRedirect from "@/features/auth/hooks/useAuthRedirect";
export default function DashboardPage() {
const { viewMode, setViewMode, chosenStoryId } = useViewMode();
const { stories, error, isLoading, mutateStories } = useStories();
const { wordsCount } = useWordsCount();
const { wordsCount, isWordsCountLoading } = useWordsCount();
const chosenStory = stories?.find((s) => String(s.id) === chosenStoryId) ?? null;
const { setWordStatus } = useWordStatus(chosenStory, mutateStories);

Expand All @@ -38,6 +38,7 @@ export default function DashboardPage() {
<OnboardingProvider wordsCount={wordsCount} useCoachmarkSteps={useCoachmarkSteps}>
<Dashboard
wordsCount={wordsCount}
isWordsCountLoading={isWordsCountLoading}
viewMode={viewMode}
handleChangeToNewStoryViewMode={() => setViewMode("newStory")}
handleChangeToAllStoriesViewMode={() => setViewMode("allStories")}
Expand Down
24 changes: 14 additions & 10 deletions apps/frontend/src/app/(protected)/vocab-assessment/page.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export default function VocabAssessmentPage() {
sessionUUID ? "loading" : "ready"
);

const { wordsCount, isLoading } = useWordsCount();
const { wordsCount, isWordsCountLoading } = useWordsCount();
const [apiResponse, setApiResponse] = useState<AssessmentResponse | null>(null);
const [answer, setAnswer] = useState<Record<number, boolean>>({});

Expand All @@ -32,14 +32,18 @@ export default function VocabAssessmentPage() {
useEffect(() => {
const fetch = async () => {
if (sessionUUID && apiResponse === null) {
const clientApi = new ClientApi();
const vocabAssessmentApi = new VocabAssessmentApi(clientApi);
const result = await vocabAssessmentApi.continue(sessionUUID);
setApiResponse(result);
if (result.status === "active") {
setStatus("started");
} else if (result.status === "completed") {
setStatus("completed");
try {
const clientApi = new ClientApi();
const vocabAssessmentApi = new VocabAssessmentApi(clientApi);
const result = await vocabAssessmentApi.continue(sessionUUID);
setApiResponse(result);
if (result.status === "active") {
setStatus("started");
} else if (result.status === "completed") {
setStatus("completed");
}
} catch {
throw new Error("Unknown server error");
}
}
};
Expand Down Expand Up @@ -98,7 +102,7 @@ export default function VocabAssessmentPage() {
return (
<VocabAssessment
status={status}
isLoading={isLoading}
isLoading={isWordsCountLoading}
wordsCount={wordsCount}
apiResponse={apiResponse}
answer={answer}
Expand Down
29 changes: 29 additions & 0 deletions apps/frontend/src/app/(protected)/vocab-assessment/skip/page.tsx
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@
"use client";

import useWordsCount from "@/features/vocab/hooks/useWordsCount";
import { VocabAssessmentApi } from "@/features/vocabAssessment/api";
import { ClientApi } from "@/lib/ClientApi";
import { useRouter } from "next/navigation";
import { useEffect, useRef } from "react";

export default function SkipVocabAssessmentPage() {
const { wordsCountUpdate } = useWordsCount();
const clientApi = new ClientApi();
const vocabAssessmentApi = new VocabAssessmentApi(clientApi);
const router = useRouter();

const called = useRef(false);
useEffect(() => {
if (called.current) return;
called.current = true;
const skip = async () => {
await vocabAssessmentApi.skip();
wordsCountUpdate();
router.replace("/dashboard");
};

skip();
}, [router]);

return <p>Loading...</p>;
}
1 change: 0 additions & 1 deletion apps/frontend/src/app/error.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,6 @@ export default function Error({
reset: () => void;
}) {
useEffect(() => {
console.error(error);
toast.error(error.message);
}, [error]);

Expand Down
27 changes: 20 additions & 7 deletions apps/frontend/src/components/AssessmentRequiredOverlay.tsx
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import Link from "next/link";

export default function AssessmentRequiredOverlay({ wordsCount }: { wordsCount: number }) {
export default function AssessmentRequiredOverlay({
wordsCount,
}: {
wordsCount: number | undefined;
}) {
if (wordsCount !== 0) return null;

return (
Expand All @@ -11,12 +15,21 @@ export default function AssessmentRequiredOverlay({ wordsCount }: { wordsCount:
Please complete the <span className="font-semibold">vocab-assessment</span> first so we
can prepare personalized stories for you.
</p>
<Link
href="/vocab-assessment"
className="inline-block bg-indigo-600 text-white px-6 py-2.5 rounded-lg font-semibold hover:bg-indigo-700 transition-colors"
>
Take assessment
</Link>
<p className="text-green-400 font-semibold">You can skip it for the demo purposes.</p>
<div className="flex flex-col gap-2">
<Link
href="/vocab-assessment/skip"
className="inline-block bg-green-600 text-white px-6 py-2.5 rounded-lg font-semibold hover:bg-green-700 transition-colors"
>
Skip and Continue
</Link>
<Link
href="/vocab-assessment"
className="inline-block bg-indigo-600 text-white px-6 py-2.5 rounded-lg font-semibold hover:bg-indigo-700 transition-colors"
>
Take assessment
</Link>
</div>
</div>
</div>
);
Expand Down
Loading