Skip to content

Add User deplicate handling option; overhaul Instagram module to include reels#71

Merged
dale-wahl merged 9 commits intomasterfrom
insta_reels
Feb 17, 2026
Merged

Add User deplicate handling option; overhaul Instagram module to include reels#71
dale-wahl merged 9 commits intomasterfrom
insta_reels

Conversation

@dale-wahl
Copy link
Member

Added reels to the Instagram module

  • verified all the various pages in the module
  • Added reel pages (single reels, reels feed)
  • Added some new post pages (e.g. saved, reposts)
  • Included reels in explorer capture
  • Pass "view" on as _zs_instagram_view attribute to determine source page (and infer reel vs post)
    Note: A user's "user_saved" page has mixed reels and posts in the same JSON list so they are not obviously distinguishable
  • Currently appending partial_ to ids as well as adding _zs_partial attribute in order to all users to both capture all a user's posts/reels as well as load each individually in order to capture the media URL and caption.
    User pages load partial post/reels that may not include media URLs (only including thumbnails).
    • We can try to address this more systematically in the future with a validation function to upsert a recorded post or something similar

Highlight reels are not included as they contain virtually nothing but a link to the actual reel. Thus to capture, a user must click the "See full reel" link.

@dale-wahl
Copy link
Member Author

Will address #8 and #43 as the reel information is much more robust now (even partial is more robust than previously).

@dale-wahl
Copy link
Member Author

Alright. Added an optional decide_action function allowing modules to decide between skip, insert, update, and merge.

  • Instagram is using insert currently on full items if the previous was flagged as partial (missing video link); this maintains collection order and which videos were visited individually.
  • added a module_id/source_platform check to existing records; probably never was an issue, but it seems possible that an ID used in one module could cause a fail in another (rarity would depend on how nav_index ids are set by browser)

@dale-wahl dale-wahl requested a review from stijn-uva February 10, 2026 10:45
@dale-wahl dale-wahl changed the title Overhaul Instagram module to include reels Add User deplicate handling option; overhaul Instagram module to include reels Feb 11, 2026
@dale-wahl
Copy link
Member Author

Doubled down. Added new feature allowing users to decide how to handle duplicates.

  • Feature locks when items have been captured
    • this is not necessary, but I think it will help avoid confusion; from the backend's perspective it means we do not have to interpret which item to update (we would probably want to index collected timestamp and get the most recent item)
  • Combines with the decide_action module function which I renamed overwrite_partial. This now returns True/False depending on whether the module decides a new item should replace an old.
    • On same nav, this overwrite_partial is all that is considered (so same page duplicate items are skipped unless this function is called and decides otherwise)
    • With global duplicate check, overwrite_partial is used to determine if an item should actually be replaced (full items overwriting partial, but not vice versa).

One oddity: reloading a page does not trigger an update to nav_index, so you cannot reload a page in order to recapture (with "keep duplicates"). I am not sure if that is desired.

@dale-wahl dale-wahl merged commit c978a92 into master Feb 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants