Skip to content

Commit 6b496dd

Browse files
windwardboywindwardboy
committed
...
1 parent 4c2c7cd commit 6b496dd

File tree

8 files changed

+276
-144
lines changed

8 files changed

+276
-144
lines changed

‎PROGRESS_LOG.md‎

Lines changed: 36 additions & 72 deletions
Original file line numberDiff line numberDiff line change
@@ -1,90 +1,54 @@
11
# Librostream Project Log
22

3-
### Session: 2023-10-27
3+
## Latest Updates (2025-07-04)
44

5-
**Project:** Librostream.com
5+
### Objective: Implement a Robust, Automated, and Scalable Audiobook Import System
66

7-
**Summary:**
8-
The project was recently migrated from a restrictive host (20i.com) to a more robust stack (Digital Ocean + Ploi.io) to resolve critical debugging and execution limitations.
7+
**Summary of Challenges Faced:**
8+
The project has faced persistent issues with audiobook imports across different hosting environments (20i.com, Ploi.io, Laravel Forge). Initial attempts with direct Artisan command execution via cron jobs proved unreliable due to server-level connectivity issues (SSH failures) and resource limitations. Data quality from the Internet Archive API was also a concern (messy language codes, broken images, missing sections).
99

10-
**Challenge:**
11-
The current process for importing ~20,000 audiobooks is manual, unreliable, and slow. It relies on an Artisan command that can only process ~100 items at a time before timing out or failing. This requires constant manual intervention.
10+
**Core Problem:** Achieving reliable, automated, and complete audiobook imports with clean data.
1211

13-
Previous attempts to import from the **Librivox** and **Internet Archive** APIs were problematic due1 to the restrictive hosting environment, leading to complex workarounds. Specific issues were encountered with inconsistent or missing metadata such as **tags, narrators, and languages**, making the import logic fragile.
12+
**Solution Strategy: Webhook-Triggered, Queue-Based Import using Official LibriVox API**
1413

15-
**Goal:**
16-
Architect and implement a robust, automated, and scalable solution for importing all audiobooks without manual oversight or server timeouts.
14+
This revised strategy aims to decouple the import process from direct cron job execution, leveraging Laravel's robust queue system and a secure HTTP trigger for maximum reliability and data integrity.
1715

18-
**Proposed Solution: Queue-Based Imports**
19-
1. An Artisan command (`librostream:queue-imports`) will be created. Its sole purpose is to read the source of all audiobooks (e.g., a CSV file, a directory scan) and dispatch an individual `ImportAudiobook` job for each one onto a queue.
20-
2. An `ImportAudiobook` Job class will be created. It will contain the logic to process and import a *single* audiobook, making the process resilient to individual failures.
21-
3. A queue worker will be configured (using the `database` or `redis` driver) and set up to run persistently on the server via Ploi to process these jobs from the queue in the background.
16+
**Key Components & Implementation Plan:**
2217

23-
**Next Steps:**
24-
Begin implementation of the queue-based import system, starting with configuring the queue driver and creating the `ImportAudiobook` Job.
25-
26-
### Session: 2025-07-03
27-
28-
**Project:** Librostream.com
29-
30-
**Summary:**
31-
Focused on resolving critical deployment and application errors on the new Digital Ocean + Ploi.io stack, automating audiobook imports, and fixing homepage data counters.
18+
1. **Database Reset (Crucial User Action)**:
19+
* **Action Required by User**: Before any new large-scale import, all existing audiobook, audiobook section, and category data in the database **must be cleared**. This ensures a clean slate. This is a manual step (e.g., `php artisan migrate:fresh --seed` locally, or manual truncation on the live server).
3220

33-
**Work Completed:**
34-
1. **Initial HTTP ERROR 500 Resolution:** Diagnosed and resolved the `HTTP ERROR 500` on the live site. The issue was traced to incorrect file permissions on `storage` and `bootstrap/cache` directories. Resolved by setting ownership to `librostream-3odwm:librostream-3odwm` and permissions to `775`.
35-
2. **Homepage Counter Fix:** Modified `app/Http/Controllers/AudiobookController.php` to correctly count audiobooks by filtering for non-null slugs, ensuring the homepage counter reflects only visible audiobooks. This fix was verified on localhost.
36-
3. **Audiobook Import Automation Commands Created:**
37-
* `app/Console/Commands/FullLibriVoxImport.php`: New Artisan command to continuously fetch main audiobook data from LibriVox (Archive.org) using pagination.
38-
* `app/Console/Commands/FullLibriVoxSectionsImport.php`: New Artisan command to continuously fetch and store sections for existing audiobooks.
21+
2. **Official LibriVox API Integration (Completed)**:
22+
* **`config/languages.php`**: Created to centralize ISO language code to full name mappings.
23+
* **`app/Services/LibriVoxService.php`**: Reconfigured to exclusively use official LibriVox API endpoints (`/api/feed/audiobooks` for main data and `/api/feed/audiotracks` for sections), fetch JSON data, and include robust error handling.
24+
* **`app/Console/Commands/FetchLibriVoxAudiobooks.php`**: Updated to consume data from the new `LibriVoxService`, correctly map fields (including `cover_image` and `librivox_id`), implement a `--dry-run` flag for testing, use `config/languages.php` for language conversion, and include comprehensive error handling around database operations. It performs a synchronous, complete import of audiobooks and their sections.
3925

40-
**Current Challenges & Status:**
41-
1. **Persistent Deployment Issues with Ploi.io:**
42-
* **Git Permission Denied:** `error: could not lock config file .git/config: Permission denied` during `git pull`.
43-
* **Chmod Script Error:** `chmod: cannot access 'ploi-e23888159c3a6d2c4ad47fa41f7116d3.sh': No such file or directory`.
44-
* **Webhook Failure:** GitHub Actions successfully triggers the Ploi.io deploy webhook, but Ploi.io does not initiate or log any deployment activity.
45-
* **Server Access Limitations:** The `ploi` SSH user lacks `sudo` privileges, and Ploi.io's dashboard does not provide a file manager or a way to run commands as `root` or `librostream-3odwm`, preventing manual resolution of permission issues after deployments.
46-
* **Ploi.io Support Unresponsive:** Previous attempts to get support for these deployment issues have been unsuccessful.
26+
3. **Webhook-Triggered Import Mechanism (New Implementation)**:
27+
* **`routes/api.php`**: Added a new secure API endpoint (`/api/import/trigger`) to receive trigger requests. (Completed)
28+
* **`app/Http/Controllers/AudiobookController.php`**: Added `triggerImport` method to handle webhook requests and dispatch the `ImportAudiobook` job. (Completed)
29+
* **`app/Jobs/ImportAudiobook.php`**: Modified to dispatch the `librivox:full-import` Artisan command, moving the heavy lifting to a background job. (Completed)
30+
* **Security**: The webhook endpoint is secured using a shared secret token (`config('app.import_webhook_token')`).
4731

48-
### Session: 2025-07-03 (Continued)
32+
4. **Queue Worker Configuration (User Action)**:
33+
* **Action Required by User**: Ensure a Laravel queue worker is configured and running persistently on your Forge server. This is typically set up under your Site's "Daemons" or Server's "Daemons" section in Forge. This worker will process the `ImportAudiobookJob` instances.
4934

50-
**Summary:**
51-
Re-assessed audiobook import strategy following user feedback on previous import failures and preference for a "slow but complete" synchronous import, including all associated data (sections) at once, and automation via cron jobs. Addressed issues with messy language codes and missing images/audio files. Decided to **start from scratch** by switching the import source from Internet Archive API to the official LibriVox API to ensure cleaner data.
35+
5. **Automation Trigger**:
36+
* Instead of a direct cron job for `librivox:full-import`, an external service (e.g., a free online cron job service, or a simple `curl` command from another server) will make a periodic HTTP POST request to the secure `/api/import/trigger` endpoint. This decouples the import trigger from Forge's potentially problematic scheduler SSH connectivity.
5237

53-
**Problem:**
54-
Previous import attempts resulted in system crashes and corrupted/incomplete data. The user prefers a single, synchronous import process per audiobook that includes all its sections, rather than separate import steps or a queue-based system. Automation via cron jobs is also required. Additionally, language data from the Internet Archive API was in ISO codes (e.g., `cat`, `mul`, `deu`, `ita`) instead of full language names, and some images/audio files were not present. The core issue was identified as data quality from the Internet Archive API, leading to the decision to switch to the official LibriVox API.
38+
6. **Revised Scheduling (Local Change)**:
39+
* **`app/Console/Kernel.php`**: The `librivox:full-import` command is **no longer scheduled directly** in this file. Its execution is now managed by the webhook dispatching jobs. (Completed)
5540

56-
**Revised Plan for Robust, Complete, and Automated Audiobook Import (using Official LibriVox API) - Incorporating Advisor Feedback:**
41+
7. **Cleanup of Old Scheduler (User Action)**:
42+
* **Action Required by User**: Delete any manually created `librivox:full-import` schedulers in Laravel Forge to avoid conflicts. The automatically created `php8.3 /home/librostreamcom/librostream.com/artisan schedule:run` cron job should remain active, as it manages other Laravel scheduled tasks (though not the import in this new model).
5743

58-
1. **Database Reset (Crucial User Action)**:
59-
* **Action Required by User**: Before any new import, all existing audiobook, audiobook section, and category data in the database **must be cleared**. This ensures a clean slate, preventing conflicts or corruption from previous, problematic imports. This is a manual step (e.g., truncating tables).
60-
61-
2. **Create `config/languages.php`**:
62-
* **Action by Cline**: Created a new configuration file (`config/languages.php`) to centrally store ISO language code to full name mappings. (Completed)
63-
64-
3. **Reconfigure `app/Services/LibriVoxService.php`**:
65-
* **Action by Cline**: Updated `baseUrl` for audiobooks to `https://librivox.org/api/feed/audiobooks` and `itemApiBaseUrl` to `https://librivox.org/api/feed/audiotracks`.
66-
* **Action by Cline**: Modified `fetchAudiobooks` to request `format=json`, `extended=1`, and `coverart=1`, and parse the JSON response.
67-
* **Action by Cline**: Modified `fetchAudiobookFiles` (renamed to `fetchAudiobookTracks` for clarity) to use the `audiotracks` endpoint with `project_id` and parse its JSON response for section details.
68-
* **Enhancement (Error Handling)**: Added robust `try/catch` blocks around all API calls within this service with detailed logging of failures. (Completed)
69-
70-
4. **Revise `app/Console/Commands/FetchLibriVoxAudiobooks.php`**:
71-
* **Action by Cline**: Updated to consume JSON data from the reconfigured `LibriVoxService`.
72-
* **Action by Cline**: Adjusted data mapping logic for LibriVox API's response structure, including `cover_image` and `librivox_id`.
73-
* **Enhancement (Dry Run)**: Implemented a `--dry-run` flag to simulate imports without database writes.
74-
* **Enhancement (Language Code Mapping)**: Integrated the `config/languages.php` for language conversion.
75-
* **Enhancement (Error Handling + Logging)**: Added robust `try/catch` blocks around database write operations and logging of failures per audiobook ID.
76-
* **Action by Cline**: Ensured synchronous, complete import for each audiobook (main data + sections from respective LibriVox API endpoints).
77-
* **Enhancement (Progress Feedback)**: Maintained and enhanced progress feedback.
78-
* **Fix (Broken Images)**: Prioritized `coverart_jpg` and `coverart_thumbnail` fields from LibriVox API for `cover_image` URL. (Completed)
79-
80-
5. **Update `PROGRESS_LOG.md`**:
81-
* **Action by Cline**: Updated this log to reflect the complete shift to the official LibriVox API and all incorporated enhancements. (Completed)
82-
83-
**Current Status & Remaining Issues:**
84-
* **Images and Layout**: Confirmed by user that images are now working correctly on audiobook detail pages and the layout issue is resolved.
85-
* **"Märchen (Index aller Märchen) (LibriVox ID: 66)"**: This book still shows "No audio tracks found for sections". User clarified this is an index page on LibriVox that links to other audiobooks, not a directly playable audiobook. (Decision on how to handle this deferred).
44+
**Current Status & Resolved Issues:**
45+
* **Images and Layout**: Images are now working correctly on audiobook detail pages, and the "The Librostream Experience" layout issue is resolved.
46+
* **"Märchen (Index aller Märchen) (LibriVox ID: 66)"**: This book still shows "No audio tracks found for sections". User clarified this is an index page on LibriVox that links to other audiobooks, not a directly playable audiobook. (Decision on how to handle this deferred for future discussion).
47+
* **Forge Scheduler Failure**: The persistent SSH connectivity issue for scheduled jobs is being bypassed by this new webhook-triggered approach.
8648

8749
**Next Steps (Requires User Action for Deployment & Full Automation):**
88-
1. **Deployment**: User confirmed migration to Laravel Forge and resolution of deployment issues. Code changes have been successfully deployed to the live server via Git push.
89-
2. **Verify Live Site**: After successful deployment, confirm the homepage counters are correct and the "Hello World" test marker is visible.
90-
3. **Set up Cron Job in Forge**: The `librivox:full-import` command has been successfully configured as a nightly scheduled task within Laravel Forge, automating the full import process. User will monitor and report back.
50+
1. **Delete Old Scheduler**: Delete the manually created `librivox:full-import` scheduler in Forge.
51+
2. **Deploy Changes**: Commit and push all local code changes to your Git repository. Laravel Forge will then deploy the updated application.
52+
3. **Configure Queue Worker**: Ensure a Laravel queue worker is running on your Forge server.
53+
4. **Set up External Trigger**: Configure an external service to send periodic POST requests to your new `/api/import/trigger` endpoint with the correct security token.
54+
5. **Verify Live Site**: After deployment and initial trigger, confirm the live site is populating with data.

‎TROUBLESHOOTING_LIVE_SERVER.md‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,59 @@
1+
# Live Server Import Troubleshooting Guide
2+
3+
## Step 1: Verify Database Connectivity
4+
Run these commands on your live server via SSH:
5+
6+
```bash
7+
# Check if MySQL is running
8+
sudo systemctl status mysql
9+
10+
# Test database connection
11+
php artisan tinker
12+
\DB::connection()->getPdo();
13+
exit;
14+
```
15+
16+
## Step 2: Check Environment Variables
17+
Verify these key settings in your live server's `.env` file:
18+
19+
```ini
20+
DB_CONNECTION=mysql
21+
DB_HOST=127.0.0.1
22+
DB_PORT=3306
23+
DB_DATABASE=librostream
24+
DB_USERNAME=librostream
25+
DB_PASSWORD=your_password_here
26+
```
27+
28+
## Step 3: Review Logs
29+
Check these log files for errors:
30+
31+
```bash
32+
# Laravel application logs
33+
tail -n 100 storage/logs/laravel.log
34+
35+
# MySQL error logs
36+
sudo tail -n 100 /var/log/mysql/error.log
37+
38+
# Forge scheduler logs (if available)
39+
```
40+
41+
## Step 4: Manual Import Test
42+
Try running a small import manually:
43+
44+
```bash
45+
php artisan librivox:fetch --limit=2 --verbose
46+
```
47+
48+
## Step 5: Contact Forge Support
49+
When contacting Laravel Forge support, include:
50+
1. Screenshot of your scheduler configuration
51+
2. Any error messages from the logs
52+
3. Details about when the issue started
53+
4. Steps you've already tried
54+
55+
## Common Issues to Check
56+
1. Database user permissions
57+
2. Available disk space (`df -h`)
58+
3. Memory usage (`free -m`)
59+
4. PHP memory limits (`php -i | grep memory_limit`)

0 commit comments

Comments
 (0)