Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
40ffdb8
bump version to 0.1.3
cyclotruc Feb 13, 2025
c3cea34
fix publish.yml
cyclotruc Feb 13, 2025
3534128
Update pyproject.toml
cyclotruc Feb 13, 2025
c761be2
Improved error message for private repositories
cyclotruc Feb 14, 2025
8327001
fix cleanup for hosted version
cyclotruc Feb 14, 2025
d592e2e
add submodules to clone
cyclotruc Feb 15, 2025
5a311bc
add submodules to tests
cyclotruc Feb 15, 2025
3787b98
feat: Add Python 3.7 Support and Restore Compatibility with Older Syn…
filipchristiansen Feb 17, 2025
b2e9da4
update readme for jupyter
cyclotruc Feb 17, 2025
3bfd391
Remove redundant path resolution and enable rmtree error handling (#183)
filipchristiansen Feb 17, 2025
eac7674
feat: add gist.github.com support and fix ingest_async bug (#184)
filipchristiansen Feb 17, 2025
737fe9e
Capitalize a couple of words (#180)
ninadsachania Feb 17, 2025
9ea759a
remove unused gitingest.com from hosts
cyclotruc Feb 17, 2025
f605b16
fix test_query_parser with gist.github.com
cyclotruc Feb 17, 2025
1a9a518
increase timeout to 60 seconds
cyclotruc Feb 17, 2025
8307f17
increase timeout to 60 seconds
cyclotruc Feb 17, 2025
1ac97de
Fix typos (#190)
ninadsachania Feb 18, 2025
cca9c78
feat(cli):Add support for .gitingest file processing in query ingesti…
AbhiRam162105 Feb 19, 2025
90edef6
feat: partial cloning (#188)
filipchristiansen Feb 19, 2025
e3450b4
remove unused setup.py (#192)
cyclotruc Feb 19, 2025
42e4e26
fix: correct title attribute for Edge Add-ons link (#199)
CharlesCNorton Feb 24, 2025
aa4d007
Refactor/ingestion (#209)
cyclotruc Mar 4, 2025
dca7cdf
Create scorecard.yml (#211)
cyclotruc Mar 4, 2025
0d8f14a
Update scorecard.yml (#212)
cyclotruc Mar 4, 2025
4b86fdf
Update scorecard.yml (#213)
cyclotruc Mar 4, 2025
e6094f8
Normalize Windows paths and handle directory paths for include patter…
filipchristiansen Mar 6, 2025
c0e8db4
Fix/dirtree (#218)
cyclotruc Mar 6, 2025
4becb01
remove subpath hack (#219)
cyclotruc Mar 6, 2025
821f580
Refactor ingestion logic to unify single-file and directory output, r…
filipchristiansen Mar 7, 2025
d71faff
add comments
cyclotruc Mar 7, 2025
1763aea
Refactor/pydantic(#226)
cyclotruc Mar 10, 2025
206f0e0
chore: bump dependencies to address security vulnerabilities (#227)
filipchristiansen Mar 12, 2025
832a5b1
[StepSecurity] ci: Harden GitHub Actions (#229)
step-security-bot Mar 13, 2025
c31a23b
Remove unused pattern_type parameter from IngestionQuery fixture (#228)
filipchristiansen Mar 13, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -6,14 +6,17 @@ on:
pull_request:
branches: [ main ]

permissions:
contents: read

jobs:
test:
runs-on: ${{ matrix.os }}
strategy:
fail-fast: true
matrix:
os: [ubuntu-latest, macos-latest, windows-latest]
python-version: ["3.10", "3.11", "3.12", "3.13"]
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12", "3.13"]

steps:
- uses: actions/checkout@v4
Expand Down
91 changes: 17 additions & 74 deletions .github/workflows/publish.yml
Original file line number Diff line number Diff line change
@@ -1,96 +1,39 @@
name: Publish to PyPI
name: "Publish to PyPI"

on:
release:
types: [created] # Trigger only when a release is created
workflow_dispatch: # Allows manual triggering of the workflow
types: [created]
workflow_dispatch:

permissions:
contents: read

jobs:
release-build:
runs-on: ubuntu-latest

steps:
# Step 1: Check out the code
- name: Checkout code
uses: actions/checkout@v4

# Step 2: Set up Python
- name: Set up Python
uses: actions/setup-python@v5
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: "3.13"

# Verify version matches the release tag
- name: Verify version
if: github.event_name == 'release'
run: |
PROJECT_VERSION=$(grep "^version = " pyproject.toml | cut -d'"' -f2)
TAG_VERSION=${GITHUB_REF#refs/tags/}
TAG_VERSION=${TAG_VERSION#v}

if [ "$PROJECT_VERSION" != "$TAG_VERSION" ]; then
echo "Version mismatch: pyproject.toml version ($PROJECT_VERSION) doesn't match release tag ($TAG_VERSION)"
exit 1
fi

# Step 3: Build the package
- name: Build release distributions
- name: Build package
run: |
python -m pip install --upgrade pip
python -m pip install build
pip install build
python -m build

# Step 4: Verify the built package
- name: Verify package
run: |
python -m pip install twine
if [ ! -f "dist/*.whl" ] || [ ! -f "dist/*.tar.gz" ]; then
echo "Expected wheel and source distribution files not found in dist/"
exit 1
fi
python -m twine check dist/*

# Step 5: Upload release artifacts
- name: Upload release artifacts
uses: actions/upload-artifact@v4
- uses: actions/upload-artifact@v4
with:
name: release-dists
name: dist
path: dist/

pypi-publish:
runs-on: ubuntu-latest
needs: [release-build]
runs-on: ubuntu-latest
environment: pypi
permissions:
id-token: write # Required for PyPI trusted publishing

id-token: write
steps:
# Step 1: Retrieve release distributions
- name: Retrieve release distributions
uses: actions/download-artifact@v4
- uses: actions/download-artifact@v4
with:
name: release-dists
name: dist
path: dist/

# Step 2: Publish to PyPI using OIDC
- name: Publish package distributions to PyPI
uses: pypa/gh-action-pypi-publish@release/v1
with:
verbose: true
print-hash: true

# Step 3: Verify package is available on PyPI
- name: Verify PyPI upload
run: |
# Wait a bit for PyPI to process the upload
sleep 30

# Extract package name from the wheel file (assuming it exists)
WHEEL_FILE=$(ls dist/*.whl | head -n 1)
PACKAGE_NAME=$(basename "$WHEEL_FILE" | cut -d'-' -f1)
PACKAGE_VERSION=$(basename "$WHEEL_FILE" | cut -d'-' -f2)

if ! pip install $PACKAGE_NAME==$PACKAGE_VERSION --no-deps --dry-run; then
echo "Failed to verify package on PyPI"
exit 1
fi
- uses: pypa/gh-action-pypi-publish@release/v1
52 changes: 52 additions & 0 deletions .github/workflows/scorecard.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
name: OSSF Scorecard
on:
# For Branch-Protection check. Only the default branch is supported. See
# https://github.com/ossf/scorecard/blob/main/docs/checks.md#branch-protection
branch_protection_rule:
# To guarantee Maintained check is occasionally updated. See
# https://github.com/ossf/scorecard/blob/main/docs/checks.md#maintained
schedule:
- cron: '33 11 * * 2'
push:
branches: [ "main" ]

# Declare default permissions as read only.
permissions: read-all

jobs:
analysis:
name: Scorecard analysis
runs-on: ubuntu-latest
permissions:
# Needed to upload the results to code-scanning dashboard.
security-events: write
# Needed to publish results and get a badge (see publish_results below).
id-token: write

steps:
- name: "Checkout code"
uses: actions/checkout@b4ffde65f46336ab88eb53be808477a3936bae11 # v4.1.1
with:
persist-credentials: false

- name: "Run analysis"
uses: ossf/scorecard-action@0864cf19026789058feabb7e87baa5f140aac736 # v2.3.1
with:
results_file: results.sarif
results_format: sarif

# Public repositories:
# - Publish results to OpenSSF REST API for easy access by consumers
# - Allows the repository to include the Scorecard badge.
# - See https://github.com/ossf/scorecard-action#publishing-results.
publish_results: true

# Upload the results as artifacts (optional). Commenting out will disable uploads of run results in SARIF
# format to the repository Actions tab.

# Upload the results to GitHub's code scanning dashboard (optional).
# Commenting out will disable upload of results to your repo's Code Scanning dashboard
- name: "Upload to code-scanning"
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: results.sarif
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -173,3 +173,6 @@ Caddyfile

# ignore default output directory
tmp/*

# Gitingest
digest.txt
6 changes: 5 additions & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@ repos:
hooks:
- id: pyupgrade
description: "Automatically upgrade syntax for newer versions."
args: [--py3-plus, --py36-plus, --py38-plus, --py39-plus, --py310-plus]
args: [--py3-plus, --py36-plus]

- repo: https://github.com/pre-commit/pygrep-hooks
rev: v1.10.0
Expand Down Expand Up @@ -95,13 +95,15 @@ repos:
files: ^src/
additional_dependencies:
[
chardet,
click,
fastapi-analytics,
pytest-asyncio,
python-dotenv,
slowapi,
starlette,
tiktoken,
tomli,
uvicorn,
]
- id: pylint
Expand All @@ -111,13 +113,15 @@ repos:
- --rcfile=tests/.pylintrc
additional_dependencies:
[
chardet,
click,
fastapi-analytics,
pytest,
pytest-asyncio,
python-dotenv,
slowapi,
starlette,
tomli,
tiktoken,
uvicorn,
]
Expand Down
2 changes: 1 addition & 1 deletion CONTRIBUTING.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Contributing to Gitingest

Thanks for your interest in contributing to Gitingest! 🚀 Gitingest aims to be friendly for first time contributors, with a simple python and html codebase. We would love your help to make it even better. If you need any help while working with the code, please reach out to us on [Discord](https://discord.com/invite/zerRaGK9EC).
Thanks for your interest in contributing to Gitingest! 🚀 Gitingest aims to be friendly for first time contributors, with a simple Python and HTML codebase. We would love your help to make it even better. If you need any help while working with the code, please reach out to us on [Discord](https://discord.com/invite/zerRaGK9EC).

## How to Contribute (non-technical)

Expand Down
36 changes: 27 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@

Turn any Git repository into a prompt-friendly text ingest for LLMs.

You can also replace `hub` with `ingest` in any GitHub URL to access the coresponding digest.
You can also replace `hub` with `ingest` in any GitHub URL to access the corresponding digest.

[gitingest.com](https://gitingest.com) · [Chrome Extension](https://chromewebstore.google.com/detail/adfjahbijlkjfoicpjkhjicpjpjfaood) · [Firefox Add-on](https://addons.mozilla.org/firefox/addon/gitingest)

Expand All @@ -26,6 +26,10 @@ You can also replace `hub` with `ingest` in any GitHub URL to access the corespo
- **CLI tool**: Run it as a shell command
- **Python package**: Import it in your code

## 📚 Requirements

- Python 3.7+

## 📦 Installation

``` bash
Expand All @@ -37,7 +41,7 @@ pip install gitingest
<!-- markdownlint-disable MD033 -->
<a href="https://chromewebstore.google.com/detail/adfjahbijlkjfoicpjkhjicpjpjfaood" target="_blank" title="Get Gitingest Extension from Chrome Web Store"><img height="48" src="https://github.com/user-attachments/assets/20a6e44b-fd46-4e6c-8ea6-aad436035753" alt="Available in the Chrome Web Store" /></a>
<a href="https://addons.mozilla.org/firefox/addon/gitingest" target="_blank" title="Get Gitingest Extension from Firefox Add-ons"><img height="48" src="https://github.com/user-attachments/assets/c0e99e6b-97cf-4af2-9737-099db7d3538b" alt="Get The Add-on for Firefox" /></a>
<a href="https://microsoftedge.microsoft.com/addons/detail/nfobhllgcekbmpifkjlopfdfdmljmipf" target="_blank" title="Get Gitingest Extension from Firefox Add-ons"><img height="48" src="https://github.com/user-attachments/assets/204157eb-4cae-4c0e-b2cb-db514419fd9e" alt="Get from the Edge Add-ons" /></a>
<a href="https://microsoftedge.microsoft.com/addons/detail/nfobhllgcekbmpifkjlopfdfdmljmipf" target="_blank" title="Get Gitingest Extension from Microsoft Edge Add-ons"><img height="48" src="https://github.com/user-attachments/assets/204157eb-4cae-4c0e-b2cb-db514419fd9e" alt="Get from the Edge Add-ons" /></a>
<!-- markdownlint-enable MD033 -->

The extension is open source at [lcandy2/gitingest-extension](https://github.com/lcandy2/gitingest-extension).
Expand All @@ -61,7 +65,7 @@ gitingest --help

This will write the digest in a text file (default `digest.txt`) in your current working directory.

## 🐛 Python package usage
## 🐍 Python package usage

```python
# Synchronous usage
Expand All @@ -71,17 +75,31 @@ summary, tree, content = ingest("path/to/directory")

# or from URL
summary, tree, content = ingest("https://github.com/cyclotruc/gitingest")
```

By default, this won't write a file but can be enabled with the `output` argument.

```python
# Asynchronous usage
from gitingest import ingest_async
import asyncio

result = asyncio.run(ingest_async("path/to/directory"))
```

By default, this won't write a file but can be enabled with the `output` argument.
### Jupyter notebook usage

```python
from gitingest import ingest_async

# Use await directly in Jupyter
summary, tree, content = await ingest_async("path/to/directory")

```

This is because Jupyter notebooks are asynchronous by default.

## 🌐 Self-host
## 🐳 Self-host

1. Build the image:

Expand All @@ -104,7 +122,7 @@ If you are hosting it on a domain, you can specify the allowed hostnames via env
ALLOWED_HOSTS="example.com, localhost, 127.0.0.1"
```

## ✔️ Contributing to Gitingest
## 🤝 Contributing

### Non-technical ways to contribute

Expand All @@ -114,7 +132,7 @@ If you are hosting it on a domain, you can specify the allowed hostnames via env

### Technical ways to contribute

Gitingest aims to be friendly for first time contributors, with a simple python and html codebase. If you need any help while working with the code, reach out to us on [Discord](https://discord.com/invite/zerRaGK9EC). For detailed instructions on how to make a pull request, see [CONTRIBUTING.md](./CONTRIBUTING.md).
Gitingest aims to be friendly for first time contributors, with a simple Python and HTML codebase. If you need any help while working with the code, reach out to us on [Discord](https://discord.com/invite/zerRaGK9EC). For detailed instructions on how to make a pull request, see [CONTRIBUTING.md](./CONTRIBUTING.md).

## 🛠️ Stack

Expand All @@ -124,10 +142,10 @@ Gitingest aims to be friendly for first time contributors, with a simple python
- [tiktoken](https://github.com/openai/tiktoken) - Token estimation
- [posthog](https://github.com/PostHog/posthog) - Amazing analytics

### Looking for a JavaScript/Node package?
### Looking for a JavaScript/FileSystemNode package?

Check out the NPM alternative 📦 Repomix: <https://github.com/yamadashy/repomix>

## Project Growth
## 🚀 Project Growth

[![Star History Chart](https://api.star-history.com/svg?repos=cyclotruc/gitingest&type=Date)](https://star-history.com/#cyclotruc/gitingest&Date)
18 changes: 10 additions & 8 deletions pyproject.toml
Original file line number Diff line number Diff line change
@@ -1,25 +1,25 @@
[project]
name = "gitingest"
version = "0.1.2"
version = "0.1.4"
description="CLI tool to analyze and create text dumps of codebases for LLMs"
readme = {file = "README.md", content-type = "text/markdown" }
requires-python = ">= 3.10"
requires-python = ">= 3.8"
dependencies = [
"click>=8.0.0",
"fastapi-analytics",
"fastapi[standard]",
"python-dotenv",
"slowapi",
"starlette",
"tiktoken",
"uvicorn",
"tomli",
"typing_extensions; python_version < '3.10'",
]

license = {file = "LICENSE"}
authors = [{name = "Romain Courtois", email = "romain@coderamp.io"}]
classifiers=[
"Development Status :: 3 - Alpha",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3.7",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
Expand Down Expand Up @@ -53,6 +53,7 @@ disable = [
"too-few-public-methods",
"broad-exception-caught",
"duplicate-code",
"fixme",
]

[tool.pycln]
Expand All @@ -75,5 +76,6 @@ pythonpath = ["src"]
testpaths = ["tests/"]
python_files = "test_*.py"
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "function"
python_classes = "Test*"
python_functions = "test_*"
Loading
Loading