fix: Skip redundant S3 upload when file already exists after rollback

[dimitri-yatsenko]

Summary

• Fixes upload_filepath re-uploads files after transaction rollback — should check S3 before uploading #1397 — upload_filepath re-uploads multi-GB files after transaction rollback
• Before uploading, check S3 via a single stat_object call for an existing object with matching size and contents_hash metadata
• If the file is already on S3 (from a prior rolled-back attempt), skip the upload and just re-insert the DB tracking entry
• Adds s3.Folder.stat() method; refactors exists() to use it

Root cause

S3 uploads are not transactional. When a transaction rolls back after a successful upload (e.g., DB connection timeout during a long make()), the file remains on S3 but the tracking entry is lost. On retry, upload_filepath checks only the DB, finds no entry, and re-uploads the entire file — creating an infinite retry loop for large files.

What changed

datajoint/s3.py: New stat() method returns the full stat_object result (size, metadata) or None — single HTTP HEAD request. exists() refactored to use it.

datajoint/external.py: In upload_filepath's else branch (no DB entry), before calling _upload_file:

1. Call s3.stat() on the expected S3 path
2. If object exists with matching size and contents_hash from metadata → skip upload, log info
3. If skip_checksum mode, match on size only
4. Always insert the DB tracking entry regardless

Test plan

• Verify existing external storage tests pass
• Manual test: upload large filepath, kill connection before commit, verify retry skips re-upload

🤖 Generated with Claude Code