feat: add support for improved handling of jupyter notebooks

[filipchristiansen]

This PR introduces the process_notebook function to process .ipynb files and return them as Python scripts, converting markdown and raw cells into multi-line string literals. It also refactors the function name ingest_from_query to run_ingest_query in ingest_from_query.py to avoid naming conflicts with the module, ensuring clearer code organization.

Changes include:

• Adding the process_notebook function to handle Jupyter notebooks.
• Renaming the ingest_from_query function to run_ingest_query to avoid naming conflicts with the module.
• Updating _read_file_content to invoke process_notebook for .ipynb files.
• Adding unit tests in test_notebook_utils.py for the notebook processing logic.
• Adding integration tests in test_ingest.py to verify that .ipynb files trigger process_notebook.

These changes integrate Jupyter notebook processing into the file ingestion workflow, while also improving code clarity and test coverage.

[cyclotruc]

So I tested this with:
https://raw.githubusercontent.com/cyclotruc/test/refs/heads/main/Exploration%20of%20Airline%20On-Time%20Performance.ipynb and got:

 File "/workspaces/gitingest/src/gitingest/notebook_utils.py", line 30, in process_notebook
    for cell in notebook["cells"]:
                ~~~~~~~~^^^^^^^^^
KeyError: 'cells'

It seems to fail because the "cells" field is nested

[filipchristiansen]

@cyclotruc

The use of worksheets is deprecated. (However, I have now still added support for it as requested.)

The worksheets field is a list, but we have no UI to support multiple worksheets. Our design has since shifted to heading-cell based structure, so we never intend to support the multiple worksheet model. The worksheets list of lists shall be replacedwith a single list, called cells."
https://github.com/ipython/ipython/wiki/IPEP-17:-Notebook-Format-4#remove-multiple-worksheets

[filipchristiansen]

@cyclotruc Tests added for the notebook_utils module – ready for review.

[IsNoobgrammer]

@filipchristiansen liked your PR , can you create a new PR such that ,cell number, cell type are commented above the source and if cells[output][-1]['text'] are commented below , also we can make such that that results always init_s with "### Jupyter-Notebook"
or some filter to identify it as a notebook , thanks!! for this PR , too

If you are busy , do mention , I would create a PR in that case

[filipchristiansen]

@filipchristiansen liked your PR , can you create a new PR such that ,cell number, cell type are commented above the source and if cells[output][-1]['text'] are commented below , also we can make such that that results always init_s with "### Jupyter-Notebook" or some filter to identify it as a notebook , thanks!! for this PR , too

If you are busy , do mention , I would create a PR in that case

@IsNoobgrammer

What do you mean by output here? cells is a list, so I assume output is an int? but then cells[output][-1] would be the last character in the cell, and cells[output][-1]['text'] would be trying to index the last character with a ['text'] key. Maybe I am misunderstanding something obvious here. Please let me know.

For the second point, may I ask your use case for this? You would still identify that it is a notebook based on the .ipynb file extension.

[filipchristiansen]

@filipchristiansen liked your PR , can you create a new PR such that ,cell number, cell type are commented above the source and if cells[output][-1]['text'] are commented below , also we can make such that that results always init_s with "### Jupyter-Notebook" or some filter to identify it as a notebook , thanks!! for this PR , too

If you are busy , do mention , I would create a PR in that case

What do you (@cyclotruc) say about the suggestion to start each notebook with ### Jupyter-Notebook directly after the following, but before its content?

================================================
File: /notebooks/foo/bar.ipynb
================================================

[IsNoobgrammer]

@filipchristiansen liked your PR , can you create a new PR such that ,cell number, cell type are commented above the source and if cells[output][-1]['text'] are commented below , also we can make such that that results always init_s with "### Jupyter-Notebook" or some filter to identify it as a notebook , thanks!! for this PR , too
If you are busy , do mention , I would create a PR in that case

@IsNoobgrammer

What do you mean by output here? cells is a list, so I assume output is an int? but then cells[output][-1] would be the last character in the cell, and cells[output][-1]['text'] would be trying to index the last character with a ['text'] key. Maybe I am misunderstanding something obvious here. Please let me know.

For the second point, may I ask your use case for this? You would still identify that it is a notebook based on the .ipynb file extension.

1st part
meant for each cell
so for each code-type cell the outputs is [] if no output ;

else-if it ran and has output then

"outputs":[{"name":"stderr","output_type":"stream","text":"/usr/local/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n\n  from .autonotebook import tqdm as notebook_tqdm\n"}]

for nth-cell in cell:
, so in my part

outputs=nth-cell.get("outputs","") ; if outputs: output = outputs[-1]["text"] ; del outputs # [-1] always gets last ran output

2nd part
Oh! My bad , its redundant
However
data: List[str] =ingested_data.split("================================================")
so we can pass that whole document to llm or other source , without mentioning it is a python code file or jupyter code snippets and it might also help in sending data without specifying any private metadata

[filipchristiansen]

@filipchristiansen liked your PR , can you create a new PR such that ,cell number, cell type are commented above the source and if cells[output][-1]['text'] are commented below , also we can make such that that results always init_s with "### Jupyter-Notebook" or some filter to identify it as a notebook , thanks!! for this PR , too
If you are busy , do mention , I would create a PR in that case

@IsNoobgrammer
What do you mean by output here? cells is a list, so I assume output is an int? but then cells[output][-1] would be the last character in the cell, and cells[output][-1]['text'] would be trying to index the last character with a ['text'] key. Maybe I am misunderstanding something obvious here. Please let me know.
For the second point, may I ask your use case for this? You would still identify that it is a notebook based on the .ipynb file extension.

1st part meant for each cell so for each code-type cell the outputs is [] if no output ;

else-if it ran and has output then

"outputs":[{"name":"stderr","output_type":"stream","text":"/usr/local/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n\n  from .autonotebook import tqdm as notebook_tqdm\n"}]

for nth-cell in cell: , so in my part

outputs=nth-cell.get("outputs","") ; if outputs: output = outputs[-1]["text"] ; del outputs # [-1] always gets last ran output

2nd part Oh! My bad , its redundant However data: List[str] =ingested_data.split("================================================") so we can pass that whole document to llm or other source , without mentioning it is a python code file or jupyter code snippets and it might also help in sending data without specifying any private metadata

1. Regarding Including Cell Outputs:

   I think I now understand what you meant. For each code-type cell, if outputs is an empty list (i.e., no output), we would do nothing. However, if the cell has output, you want to comment below the source code what the last output was. I'm not sure if we should include the output by default.

   We could consider adding a command-line flag for the terminal client that alters this behavior to include outputs when desired. Similarly, for the web client, this might be an option under "Advanced Settings" or a similar section.

   I created Issue #119 for this feature request. If you believe such a feature would be useful, please comment on that issue. Your feedback will help us consider and prioritize this request formally.

2. Regarding Identification of Notebook Content:

   I'm not entirely sure what you mean here. Could you please elaborate?