perf(ingestion): make max event and batch size configurable

[hassiebp]

Important

Make event and batch size limits configurable via environment variables in ingestion_consumer.py, with default values and a test update in test_langchain.py.

• Configuration:
  • Introduced LANGFUSE_MAX_EVENT_SIZE_BYTES and LANGFUSE_MAX_BATCH_SIZE_BYTES environment variables in ingestion_consumer.py.
  • Defaults set to 1MB and 2.5MB respectively.
• Testing:
  • Updated test_langchain.py to increase overhead threshold from 50ms to 100ms.
• Potential Improvements:
  • Consider adding error handling for invalid environment variable values to prevent ValueError exceptions.

^{This description was created by for 2184df1. It will automatically update as commits are pushed.}

---

Greptile Summary

Disclaimer: Experimental PR review

Makes event and batch size limits configurable via environment variables in the Python SDK, improving flexibility for handling different payload sizes in production environments.

• Added LANGFUSE_MAX_EVENT_SIZE_BYTES environment variable with 1MB default in /langfuse/_task_manager/ingestion_consumer.py
• Added LANGFUSE_MAX_BATCH_SIZE_BYTES environment variable with 2.5MB default in /langfuse/_task_manager/ingestion_consumer.py
• Consider adding error handling for invalid environment variable values to prevent ValueError exceptions

[greptile-apps]

_{1 file(s) reviewed, 1 comment(s)
Edit PR Review Bot Settings | Greptile}

[greptile-apps]

logic: No error handling for invalid integer values in environment variables. Could crash on startup if LANGFUSE_MAX_EVENT_SIZE_BYTES or LANGFUSE_MAX_BATCH_SIZE_BYTES contain non-integer values.

Suggested change

	MAX_EVENT_SIZE_BYTES = int(os.environ.get("LANGFUSE_MAX_EVENT_SIZE_BYTES", 1_000_000))
	MAX_BATCH_SIZE_BYTES = int(os.environ.get("LANGFUSE_MAX_BATCH_SIZE_BYTES", 2_500_000))
	def safe_int_env(key: str, default: int) -> int:
	try:
	return int(os.environ.get(key, default))
	except ValueError:
	logging.warning(f"Invalid integer value for {key}, using default {default}")
	return default
	MAX_EVENT_SIZE_BYTES = safe_int_env("LANGFUSE_MAX_EVENT_SIZE_BYTES", 1_000_000)
	MAX_BATCH_SIZE_BYTES = safe_int_env("LANGFUSE_MAX_BATCH_SIZE_BYTES", 2_500_000)