Embedder rework #160

andhreljaKern · 2025-07-25T13:37:36Z

Main PR: #160

Related PRs:

Note

New repository to be used as a "single-run" migration container: refinery-embedder-migration. Migration steps are available in the PR descriptions:

Local Testing

Clone the new migration repository:
- git clone https://github.com/code-kern-ai/refinery-embedder-migration
[optional] Stop all docker compose services in dev-setup:
- bash stop
In your terminal, navigate to the cloned refinery-embedder-migration:
- run the script with bash start dry-run qdrant-cleanup (the qdrant-cleanup argument is used to perform a Qdrant cleanup. Run dry by adding a dry-run argument)
- ensure the embedders are in fact migrated by inspecting the directory ~/repos/dev-setup/inference. It should contain a backup folder with old embedders. Each <project-id>/<embedding-id> directory should contain embedder.json or reducer.pkl or both
Inside dev-setup, run bash start -a -b embedder-rework
- validate migration by inspecting the embeddings in refinery
Delete the folder ~/repos/dev-setup/inference/backup

Affected Areas

refinery-ui
- remove Cohere and Python support
- project export
refinery-embedder:
- moved embedders library into refinery-embedder
- embedders models dump/load
- remove Cohere and Python support
refinery-submodule-model
- remove Cohere and Python support
refinery-gateway:
- embedding creation/deletion
- remove Cohere and Python support
refinery-neural-search
- remove Python embedder support
- distances & similarity threshold
cognition-gateway:
- embedding validation in src/controller/strategy_steps (and src/controller/pipeline/steps/strategy_steps/neural_search.py)
cognition-task-master
- remove Cohere and Python support

… into dev

src/embedders/__init__.py

src/embedders/classification/contextual.py

src/embedders/classification/reduce.py

src/embedders/util.py

start

JWittmeyer · 2025-07-30T08:47:18Z

fresh start => loading Global guard [references] dev project => embeddings fail

Error log

2025-07-30 10:43:51 INFO:     172.18.0.9:43380 - "DELETE /delete/e4e92e6a-2d5e-4900-9fb3-1f2385acac1f/57daad7a-d073-4461-b476-8a5454dd4fa7 HTTP/1.1" 500 Internal Server Error
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 78, in app
2025-07-30 10:43:51     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-07-30 10:43:51     |     raise exc
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-07-30 10:43:51     |     await app(scope, receive, sender)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
2025-07-30 10:43:51     |     response = await f(request)
2025-07-30 10:43:51     |                ^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 302, in app
2025-07-30 10:43:51     |     raw_response = await run_endpoint_function(
2025-07-30 10:43:51     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 215, in run_endpoint_function
2025-07-30 10:43:51     |     return await run_in_threadpool(dependant.call, **values)
2025-07-30 10:43:51     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
2025-07-30 10:43:51     |     return await anyio.to_thread.run_sync(func)
2025-07-30 10:43:51     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2025-07-30 10:43:51     |     return await get_async_backend().run_sync_in_worker_thread(
2025-07-30 10:43:51     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
2025-07-30 10:43:51     |     return await future
2025-07-30 10:43:51     |            ^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
2025-07-30 10:43:51     |     result = context.run(func, *args)
2025-07-30 10:43:51     |              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/program/app.py", line 155, in delete_embedding
2025-07-30 10:43:51     |     status_code = controller.delete_embedding(project_id, embedding_id)
2025-07-30 10:43:51     |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/program/controller.py", line 493, in delete_embedding
2025-07-30 10:43:51     |     shutil.rmtree(json_path.parent)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/shutil.py", line 742, in rmtree
2025-07-30 10:43:51     |     onerror(os.lstat, path, sys.exc_info())
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/shutil.py", line 740, in rmtree
2025-07-30 10:43:51     |     orig_st = os.lstat(path, dir_fd=dir_fd)
2025-07-30 10:43:51     |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     | FileNotFoundError: [Errno 2] No such file or directory: '/inference/e4e92e6a-2d5e-4900-9fb3-1f2385acac1f/57daad7a-d073-4461-b476-8a5454dd4fa7'
2025-07-30 10:43:51     +------------------------------------
2025-07-30 10:43:51 
2025-07-30 10:43:51 During handling of the above exception, another exception occurred:
2025-07-30 10:43:51 
2025-07-30 10:43:51 Traceback (most recent call last):
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
2025-07-30 10:43:51     result = await app(  # type: ignore[func-returns-value]
2025-07-30 10:43:51              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2025-07-30 10:43:51     return await self.app(scope, receive, send)
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2025-07-30 10:43:51     await super().__call__(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
2025-07-30 10:43:51     await self.middleware_stack(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
2025-07-30 10:43:51     await self.app(scope, receive, _send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__
2025-07-30 10:43:51     with recv_stream, send_stream, collapse_excgroups():
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
2025-07-30 10:43:51     self.gen.throw(typ, value, traceback)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__
2025-07-30 10:43:51     response = await self.dispatch_func(request, call_next)
2025-07-30 10:43:51                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/program/app.py", line 34, in handle_db_session
2025-07-30 10:43:51     response = await call_next(request)
2025-07-30 10:43:51                ^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 159, in call_next
2025-07-30 10:43:51     raise app_exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
2025-07-30 10:43:51     await self.app(scope, receive_or_disconnect, send_no_error)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
2025-07-30 10:43:51     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-07-30 10:43:51     await app(scope, receive, sender)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__
2025-07-30 10:43:51     await self.middleware_stack(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app
2025-07-30 10:43:51     await route.handle(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 290, in handle
2025-07-30 10:43:51     await self.app(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 78, in app
2025-07-30 10:43:51     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-07-30 10:43:51     await app(scope, receive, sender)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
2025-07-30 10:43:51     response = await f(request)
2025-07-30 10:43:51                ^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 302, in app
2025-07-30 10:43:51     raw_response = await run_endpoint_function(
2025-07-30 10:43:51                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 215, in run_endpoint_function
2025-07-30 10:43:51     return await run_in_threadpool(dependant.call, **values)
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
2025-07-30 10:43:51     return await anyio.to_thread.run_sync(func)
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2025-07-30 10:43:51     return await get_async_backend().run_sync_in_worker_thread(
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
2025-07-30 10:43:51     return await future
2025-07-30 10:43:51            ^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
2025-07-30 10:43:51     result = context.run(func, *args)
2025-07-30 10:43:51              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/program/app.py", line 155, in delete_embedding
2025-07-30 10:43:51     status_code = controller.delete_embedding(project_id, embedding_id)
2025-07-30 10:43:51                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/program/controller.py", line 493, in delete_embedding
2025-07-30 10:43:51     shutil.rmtree(json_path.parent)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/shutil.py", line 742, in rmtree
2025-07-30 10:43:51     onerror(os.lstat, path, sys.exc_info())
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/shutil.py", line 740, in rmtree
2025-07-30 10:43:51     orig_st = os.lstat(path, dir_fd=dir_fd)
2025-07-30 10:43:51               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51 FileNotFoundError: [Errno 2] No such file or directory: '/inference/e4e92e6a-2d5e-4900-9fb3-1f2385acac1f/57daad7a-d073-4461-b476-8a5454dd4fa7'
2025-07-30 10:43:51 ERROR:    Exception in ASGI application
2025-07-30 10:43:51   + Exception Group Traceback (most recent call last):
2025-07-30 10:43:51   |   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 77, in collapse_excgroups
2025-07-30 10:43:51   |     yield
2025-07-30 10:43:51   |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 183, in __call__
2025-07-30 10:43:51   |     async with anyio.create_task_group() as task_group:
2025-07-30 10:43:51   |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 772, in __aexit__
2025-07-30 10:43:51   |     raise BaseExceptionGroup(
2025-07-30 10:43:51   | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
2025-07-30 10:43:51   +-+---------------- 1 ----------------
2025-07-30 10:43:51     | Traceback (most recent call last):
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
2025-07-30 10:43:51     |     result = await app(  # type: ignore[func-returns-value]
2025-07-30 10:43:51     |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2025-07-30 10:43:51     |     return await self.app(scope, receive, send)
2025-07-30 10:43:51     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2025-07-30 10:43:51     |     await super().__call__(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
2025-07-30 10:43:51     |     await self.middleware_stack(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
2025-07-30 10:43:51     |     raise exc
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
2025-07-30 10:43:51     |     await self.app(scope, receive, _send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__
2025-07-30 10:43:51     |     with recv_stream, send_stream, collapse_excgroups():
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
2025-07-30 10:43:51     |     self.gen.throw(typ, value, traceback)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
2025-07-30 10:43:51     |     raise exc
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__
2025-07-30 10:43:51     |     response = await self.dispatch_func(request, call_next)
2025-07-30 10:43:51     |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/program/app.py", line 34, in handle_db_session
2025-07-30 10:43:51     |     response = await call_next(request)
2025-07-30 10:43:51     |                ^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 159, in call_next
2025-07-30 10:43:51     |     raise app_exc
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
2025-07-30 10:43:51     |     await self.app(scope, receive_or_disconnect, send_no_error)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
2025-07-30 10:43:51     |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-07-30 10:43:51     |     raise exc
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-07-30 10:43:51     |     await app(scope, receive, sender)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__
2025-07-30 10:43:51     |     await self.middleware_stack(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app
2025-07-30 10:43:51     |     await route.handle(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 290, in handle
2025-07-30 10:43:51     |     await self.app(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 78, in app
2025-07-30 10:43:51     |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-07-30 10:43:51     |     raise exc
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-07-30 10:43:51 INFO:     172.18.0.9:43386 - "DELETE /delete/e4e92e6a-2d5e-4900-9fb3-1f2385acac1f/a3d93b8b-6c8a-4e4f-a299-0db54bc2f833 HTTP/1.1" 500 Internal Server Error
2025-07-30 10:43:51     |     await app(scope, receive, sender)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
2025-07-30 10:43:51     |     response = await f(request)
2025-07-30 10:43:51     |                ^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 302, in app
2025-07-30 10:43:51     |     raw_response = await run_endpoint_function(
2025-07-30 10:43:51     |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 215, in run_endpoint_function
2025-07-30 10:43:51     |     return await run_in_threadpool(dependant.call, **values)
2025-07-30 10:43:51     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
2025-07-30 10:43:51     |     return await anyio.to_thread.run_sync(func)
2025-07-30 10:43:51     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2025-07-30 10:43:51     |     return await get_async_backend().run_sync_in_worker_thread(
2025-07-30 10:43:51     |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
2025-07-30 10:43:51     |     return await future
2025-07-30 10:43:51     |            ^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
2025-07-30 10:43:51     |     result = context.run(func, *args)
2025-07-30 10:43:51     |              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/program/app.py", line 155, in delete_embedding
2025-07-30 10:43:51     |     status_code = controller.delete_embedding(project_id, embedding_id)
2025-07-30 10:43:51     |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     |   File "/program/controller.py", line 493, in delete_embedding
2025-07-30 10:43:51     |     shutil.rmtree(json_path.parent)
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/shutil.py", line 742, in rmtree
2025-07-30 10:43:51     |     onerror(os.lstat, path, sys.exc_info())
2025-07-30 10:43:51     |   File "/usr/local/lib/python3.11/shutil.py", line 740, in rmtree
2025-07-30 10:43:51     |     orig_st = os.lstat(path, dir_fd=dir_fd)
2025-07-30 10:43:51     |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51     | FileNotFoundError: [Errno 2] No such file or directory: '/inference/e4e92e6a-2d5e-4900-9fb3-1f2385acac1f/a3d93b8b-6c8a-4e4f-a299-0db54bc2f833'
2025-07-30 10:43:51     +------------------------------------
2025-07-30 10:43:51 
2025-07-30 10:43:51 During handling of the above exception, another exception occurred:
2025-07-30 10:43:51 
2025-07-30 10:43:51 Traceback (most recent call last):
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
2025-07-30 10:43:51     result = await app(  # type: ignore[func-returns-value]
2025-07-30 10:43:51              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
2025-07-30 10:43:51     return await self.app(scope, receive, send)
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
2025-07-30 10:43:51     await super().__call__(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
2025-07-30 10:43:51     await self.middleware_stack(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
2025-07-30 10:43:51     await self.app(scope, receive, _send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 182, in __call__
2025-07-30 10:43:51     with recv_stream, send_stream, collapse_excgroups():
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/contextlib.py", line 158, in __exit__
2025-07-30 10:43:51     self.gen.throw(typ, value, traceback)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_utils.py", line 83, in collapse_excgroups
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 184, in __call__
2025-07-30 10:43:51     response = await self.dispatch_func(request, call_next)
2025-07-30 10:43:51                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/program/app.py", line 34, in handle_db_session
2025-07-30 10:43:51     response = await call_next(request)
2025-07-30 10:43:51                ^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 159, in call_next
2025-07-30 10:43:51     raise app_exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/base.py", line 144, in coro
2025-07-30 10:43:51     await self.app(scope, receive_or_disconnect, send_no_error)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
2025-07-30 10:43:51     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-07-30 10:43:51     await app(scope, receive, sender)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 716, in __call__
2025-07-30 10:43:51     await self.middleware_stack(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 736, in app
2025-07-30 10:43:51     await route.handle(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 290, in handle
2025-07-30 10:43:51     await self.app(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 78, in app
2025-07-30 10:43:51     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
2025-07-30 10:43:51     raise exc
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
2025-07-30 10:43:51     await app(scope, receive, sender)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/routing.py", line 75, in app
2025-07-30 10:43:51     response = await f(request)
2025-07-30 10:43:51                ^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 302, in app
2025-07-30 10:43:51     raw_response = await run_endpoint_function(
2025-07-30 10:43:51                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/fastapi/routing.py", line 215, in run_endpoint_function
2025-07-30 10:43:51     return await run_in_threadpool(dependant.call, **values)
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/starlette/concurrency.py", line 38, in run_in_threadpool
2025-07-30 10:43:51     return await anyio.to_thread.run_sync(func)
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/anyio/to_thread.py", line 56, in run_sync
2025-07-30 10:43:51     return await get_async_backend().run_sync_in_worker_thread(
2025-07-30 10:43:51            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 2470, in run_sync_in_worker_thread
2025-07-30 10:43:51     return await future
2025-07-30 10:43:51            ^^^^^^^^^^^^
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 967, in run
2025-07-30 10:43:51     result = context.run(func, *args)
2025-07-30 10:43:51              ^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/program/app.py", line 155, in delete_embedding
2025-07-30 10:43:51     status_code = controller.delete_embedding(project_id, embedding_id)
2025-07-30 10:43:51                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51   File "/program/controller.py", line 493, in delete_embedding
2025-07-30 10:43:51     shutil.rmtree(json_path.parent)
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/shutil.py", line 742, in rmtree
2025-07-30 10:43:51     onerror(os.lstat, path, sys.exc_info())
2025-07-30 10:43:51   File "/usr/local/lib/python3.11/shutil.py", line 740, in rmtree
2025-07-30 10:43:51     orig_st = os.lstat(path, dir_fd=dir_fd)
2025-07-30 10:43:51               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-07-30 10:43:51 FileNotFoundError: [Errno 2] No such file or directory: '/inference/e4e92e6a-2d5e-4900-9fb3-1f2385acac1f/a3d93b8b-6c8a-4e4f-a299-0db54bc2f833'

resolved

JWittmeyer · 2025-07-30T08:48:36Z

i think we can remove token embeddings as option for the embeddings since they are not being used

resolved
discussed

JWittmeyer · 2025-07-30T11:37:38Z

I dont think the migration dry run works as intendet. The files are already moved to a backup folder and the log says nothing was changed even though it should have one project with three embedings to be changed

resolved
discussed

perf: use pickle.dumps

andhreljaKern added 3 commits July 22, 2025 19:24

fix: missing embedders module

e958924

Merge branch 'dev' of https://github.com/code-kern-ai/refinery-embedder…

b64e832

… into dev

feat: embedder rework

865cada

This was referenced Jul 25, 2025

Embedder rework code-kern-ai/refinery-gateway#338

Merged

Embedder rework code-kern-ai/refinery-submodule-model#180

Merged

andhreljaKern added 3 commits July 25, 2025 15:59

perf: remove cohere requirements

27e17b7

perf: remove cohere + python embedder support

96f6a28

chore: update submodules

e3879fa

andhreljaKern mentioned this pull request Jul 25, 2025

Embedder rework code-kern-ai/refinery-ui#79

Merged

perf: support openai > 1.0.0

ccbef0d

andhreljaKern mentioned this pull request Jul 28, 2025

Embedder rework code-kern-ai/refinery-neural-search#102

Merged

andhreljaKern added 4 commits July 28, 2025 23:53

perf: openai alignment

71840fe

perf: openai alignment

bba274c

perf: hf caching + openai alignment

d3e4234

feat: remove count-based embedders

5524dc3

andhreljaKern marked this pull request as ready for review July 30, 2025 07:55