Refactor project structure, enhance logic, update configurations, and improve code quality (coderamp-labs#85)

* Refactor project structure, enhance logic, update configurations, and improve code quality

Refactoring and Logic Improvements

- Refactored the `_scan_directory` function in `src/gitingest/ingest_from_query.py` by extracting loop logic into the new `_process_item` function, and further separating functionality into `_process_symlink` and `_process_file`
- Replaced multiple return statements with error raising and catching, introducing custom exceptions (`MaxFilesReachedError`, `MaxFileSizeReachedError`, `AlreadyVisitedError`) in the `_process_item` and `_scan_directory` functions
- Enhanced the logic in the `process_query` function in `src/process_query.py` for better flow and maintainability
- Improved the logic in `_generate_token_string` in `src/gitingest/ingest_from_query.py`
- Refined the `download_ingest` function in `src/routers/download.py` for better clarity and functionality

Exception Handling Enhancements

- Replaced broad `Exception` handling with specific `OSError` in the `_read_file_content` function in `src/gitingest/ingest_from_query.py`
- Refined exception handling throughout the codebase, including removing redundant try-except-raise blocks, e.g., in `clone_repo` function in `src/gitingest/clone.py`
- Added custom exceptions to `src/gitingest/exceptions.py`: `MaxFilesReachedError`, `MaxFileSizeReachedError`, and `AlreadyVisitedError`
- Included explicit re-raising of exceptions in various functions for improved error propagation

Test Suite Refactoring

- Cleaned up and reorganized test files:
  - Moved tests from `src/gitingest/tests/` to `tests/`
  - Consolidated fixtures from `tests/test_ingest.py` into `tests/conftest.py`
  - Removed redundant content from `tests/conftest.py`
- Migrated configuration from `pytest.ini` to `pyproject.toml`, deleted `pytest.ini`, and updated `.dockerignore`

Documentation Improvements

- Added `darglint` for enforcing `numpy` docstring style in `.pre-commit-config.yaml` for `src/` files
- Updated docstrings throughout the codebase, including adding module docstrings where needed
- Updated `README.md`:
  - Added "GitHub stars" badge
  - Moved the "Discord" badge to its own line
  - Replaced occurrences of "Gitingest" with "GitIngest" for consistency and clarity

Linting and Code Quality

- Integrated `pylint` into `.pre-commit-config.yaml` for both `src/` and `tests/` directories
- Created `tests/.pylintrc` for linting configuration specific to test files

Code Clean-up

- Removed the redundant `src/__init__.py` file

Naming Conventions and Code Style

- Renamed `logSliderToSize` to `log_slider_to_size` in `src/server_utils.py` for consistency with Python's naming conventions
- Added explicit encoding specification in multiple instances of `open` throughout the code

Author: filipchristiansen

.dockerignore

@@ -37,5 +37,4 @@ docs/
 tests/
 *.md
 LICENSE
-pytest.ini
 setup.py

.pre-commit-config.yaml

@@ -83,3 +83,51 @@ repos:
       - id: markdownlint
         description: "Lint markdown files."
         args: ["--disable=line-length"]
+
+  - repo: https://github.com/terrencepreilly/darglint
+    rev: v1.8.1
+    hooks:
+      - id: darglint
+        name: darglint for source
+        args: [--docstring-style=numpy]
+        files: ^src/
+
+  - repo: https://github.com/pycqa/pylint
+    rev: v3.3.3
+    hooks:
+      - id: pylint
+        name: pylint for source
+        files: ^src/
+        additional_dependencies:
+          [
+            click,
+            fastapi-analytics,
+            pytest-asyncio,
+            python-dotenv,
+            slowapi,
+            starlette,
+            tiktoken,
+            uvicorn,
+          ]
+      - id: pylint
+        name: pylint for tests
+        files: ^tests/
+        args:
+          - --rcfile=tests/.pylintrc
+        additional_dependencies:
+          [
+            click,
+            fastapi-analytics,
+            pytest,
+            pytest-asyncio,
+            python-dotenv,
+            slowapi,
+            starlette,
+            tiktoken,
+            uvicorn,
+          ]
+
+  - repo: meta
+    hooks:
+      - id: check-hooks-apply
+      - id: check-useless-excludes

README.md

@@ -4,9 +4,11 @@
 
 [![License](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/cyclotruc/gitingest/blob/main/LICENSE)
 [![PyPI version](https://badge.fury.io/py/gitingest.svg)](https://badge.fury.io/py/gitingest)
+[![GitHub stars](https://img.shields.io/github/stars/cyclotruc/gitingest?style=social.svg)](https://github.com/cyclotruc/gitingest)
 [![Downloads](https://pepy.tech/badge/gitingest)](https://pepy.tech/project/gitingest)
 [![GitHub issues](https://img.shields.io/github/issues/cyclotruc/gitingest)](https://github.com/cyclotruc/gitingest/issues)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+
 [![Discord](https://dcbadge.limes.pink/api/server/https://discord.com/invite/zerRaGK9EC)](https://discord.com/invite/zerRaGK9EC)
 
 Turn any Git repository into a prompt-friendly text ingest for LLMs.

pyproject.toml

@@ -1,6 +1,60 @@
+[project]
+name = "gitingest"
+version = "0.1.2"
+description="CLI tool to analyze and create text dumps of codebases for LLMs"
+readme = {file = "README.md", content-type = "text/markdown" }
+requires-python = ">= 3.10"
+dependencies = [
+    "click>=8.0.0",
+    "fastapi-analytics",
+    "fastapi[standard]",
+    "python-dotenv",
+    "slowapi",
+    "starlette",
+    "tiktoken",
+    "uvicorn",
+]
+license = {file = "LICENSE"}
+authors = [{name = "Romain Courtois", email = "romain@coderamp.io"}]
+classifiers=[
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+]
+
+[project.scripts]
+gitingest = "gitingest.cli:main"
+
+[project.urls]
+homepage = "https://gitingest.com"
+github = "https://github.com/cyclotruc/gitingest"
+
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[tool.setuptools]
+packages = {find = {where = ["src"]}}
+include-package-data = true
+
+# Linting configuration
 [tool.pylint.format]
 max-line-length = 119
 
+[tool.pylint.'MESSAGES CONTROL']
+disable = [
+    "too-many-arguments",
+    "too-many-positional-arguments",
+    "too-many-locals",
+    "too-few-public-methods",
+    "broad-exception-caught",
+    "duplicate-code",
+]
+
 [tool.pycln]
 all = true
 
@@ -14,3 +68,12 @@ filter_files = true
 
 [tool.black]
 line-length = 119
+
+# Test configuration
+[tool.pytest.ini_options]
+pythonpath = ["src"]
+testpaths = ["tests/"]
+python_files = "test_*.py"
+asyncio_mode = "auto"
+python_classes = "Test*"
+python_functions = "test_*"

pytest.ini

@@ -1,9 +1,11 @@
+""" Configuration file for the project. """
+
 MAX_DISPLAY_SIZE: int = 300_000
 TMP_BASE_PATH: str = "/tmp/gitingest"
 DELETE_REPO_AFTER: int = 60 * 60  # In seconds
 
 EXAMPLE_REPOS: list[dict[str, str]] = [
-    {"name": "Gitingest", "url": "https://github.com/cyclotruc/gitingest"},
+    {"name": "GitIngest", "url": "https://github.com/cyclotruc/gitingest"},
     {"name": "FastAPI", "url": "https://github.com/tiangolo/fastapi"},
     {"name": "Flask", "url": "https://github.com/pallets/flask"},
     {"name": "Tldraw", "url": "https://github.com/tldraw/tldraw"},

src/config.py

@@ -1,3 +1,5 @@
+""" gitingest: A package for ingesting data from git repositories. """
+
 from gitingest.clone import clone_repo
 from gitingest.ingest import ingest
 from gitingest.ingest_from_query import ingest_from_query

src/gitingest/__init__.py

@@ -1,3 +1,7 @@
+""" Command-line interface for the GitIngest package. """
+
+# pylint: disable=no-value-for-parameter
+
 import click
 
 from gitingest.ingest import ingest
@@ -40,7 +44,7 @@ def main(
 
     Raises
     ------
-    click.Abort
+    Abort
         If there is an error during the execution of the command, this exception is raised to abort the process.
     """
     try:

src/gitingest/cli.py

@@ -1,7 +1,8 @@
+""" This module contains functions for cloning a Git repository to a local path. """
+
 import asyncio
 from dataclasses import dataclass
 
-from gitingest.exceptions import AsyncTimeoutError
 from gitingest.utils import async_timeout
 
 CLONE_TIMEOUT: int = 20
@@ -59,11 +60,7 @@ async def clone_repo(config: CloneConfig) -> tuple[bytes, bytes]:
     Raises
     ------
     ValueError
-        If the repository does not exist or if required query parameters are missing.
-    RuntimeError
-        If any git command fails during execution.
-    AsyncTimeoutError
-        If the cloning process exceeds the specified timeout.
+        If the 'url' or 'local_path' parameters are missing, or if the repository is not found.
     """
     # Extract and validate query parameters
     url: str = config.url
@@ -81,29 +78,25 @@ async def clone_repo(config: CloneConfig) -> tuple[bytes, bytes]:
     if not await _check_repo_exists(url):
         raise ValueError("Repository not found, make sure it is public")
 
-    try:
-        if commit:
-            # Scenario 1: Clone and checkout a specific commit
-            # Clone the repository without depth to ensure full history for checkout
-            clone_cmd = ["git", "clone", "--single-branch", url, local_path]
-            await _run_git_command(*clone_cmd)
-
-            # Checkout the specific commit
-            checkout_cmd = ["git", "-C", local_path, "checkout", commit]
-            return await _run_git_command(*checkout_cmd)
+    if commit:
+        # Scenario 1: Clone and checkout a specific commit
+        # Clone the repository without depth to ensure full history for checkout
+        clone_cmd = ["git", "clone", "--single-branch", url, local_path]
+        await _run_git_command(*clone_cmd)
 
-        if branch and branch.lower() not in ("main", "master"):
+        # Checkout the specific commit
+        checkout_cmd = ["git", "-C", local_path, "checkout", commit]
+        return await _run_git_command(*checkout_cmd)
 
-            # Scenario 2: Clone a specific branch with shallow depth
-            clone_cmd = ["git", "clone", "--depth=1", "--single-branch", "--branch", branch, url, local_path]
-            return await _run_git_command(*clone_cmd)
+    if branch and branch.lower() not in ("main", "master"):
 
-        # Scenario 3: Clone the default branch with shallow depth
-        clone_cmd = ["git", "clone", "--depth=1", "--single-branch", url, local_path]
+        # Scenario 2: Clone a specific branch with shallow depth
+        clone_cmd = ["git", "clone", "--depth=1", "--single-branch", "--branch", branch, url, local_path]
         return await _run_git_command(*clone_cmd)
 
-    except (RuntimeError, asyncio.TimeoutError, AsyncTimeoutError):
-        raise  # Re-raise the exception
+    # Scenario 3: Clone the default branch with shallow depth
+    clone_cmd = ["git", "clone", "--depth=1", "--single-branch", url, local_path]
+    return await _run_git_command(*clone_cmd)
 
 
 async def _check_repo_exists(url: str) -> bool:

src/gitingest/clone.py

@@ -1,12 +1,13 @@
+""" Custom exceptions for the GitIngest package. """
+
+
 class InvalidPatternError(ValueError):
     """
     Exception raised when a pattern contains invalid characters.
-
     This exception is used to signal that a pattern provided for some operation
     contains characters that are not allowed. The valid characters for the pattern
     include alphanumeric characters, dash (-), underscore (_), dot (.), forward slash (/),
     plus (+), and asterisk (*).
-
     Parameters
     ----------
     pattern : str
@@ -27,3 +28,24 @@ class AsyncTimeoutError(Exception):
     This exception is used by the `async_timeout` decorator to signal that the wrapped
     asynchronous function has exceeded the specified time limit for execution.
     """
+
+
+class MaxFilesReachedError(Exception):
+    """Exception raised when the maximum number of files is reached."""
+
+    def __init__(self, max_files: int) -> None:
+        super().__init__(f"Maximum number of files ({max_files}) reached.")
+
+
+class MaxFileSizeReachedError(Exception):
+    """Raised when the maximum file size is reached."""
+
+    def __init__(self, max_size: int):
+        super().__init__(f"Maximum file size limit ({max_size/1024/1024:.1f}MB) reached.")
+
+
+class AlreadyVisitedError(Exception):
+    """Exception raised when a symlink target has already been visited."""
+
+    def __init__(self, path: str) -> None:
+        super().__init__(f"Symlink target already visited: {path}")