Feature/276 harvesting metadata from a provided repository url #278

Aidajafarbigloo · 2024-10-29T08:48:18Z

Added the URL to the hermes harvest command. Now, the command hermes harvest harvest the metadata from the local repository, and hermes harvest --url <URL> allows harvesting metadata from the provided URL, with support for GitHub and GitLab repositories.

(e.g., hermes harvest --url https://github.com/NFDI4Energy/SMECS)

…ting

Aidajafarbigloo · 2024-10-29T08:56:36Z

@sferenz
Could you please take a look at this pull request and share your feedback?

…a via path

Aidajafarbigloo · 2025-01-31T10:23:40Z

Harvesting metadata from the provided URL (GitHub/GitLab). Command: hermes harvest --path <URL>

Aidajafarbigloo · 2025-01-31T10:24:00Z

@sferenz
Could you please take a look at this pull request and share your feedback?

sferenz

Thanks for the nice code! Please have a look at the comments :)

sferenz · 2025-01-31T16:43:28Z

src/hermes/commands/base.py

        """Load settings from the configuration file (passed in from command line)."""

-        toml_data = toml.load(args.path / args.config)
+        toml_data = toml.load("." / args.config)


Does this still work if a regular path is given to HERMES?

Yes, specifying a directory path containing CFF or CodeMeta files is also acceptable. For example, the following command works:

hermes harvest --path C:\path\to\your\directory

sferenz · 2025-01-31T16:43:50Z

src/hermes/commands/harvest/base.py



-class HarvestSettings(BaseModel):
+class _HarvestSettings(BaseModel):


Why did you rename this?

I didn’t intend to change the class name. There was an issue with incorrectly pulling the original code for base.py from an updated version of it. This occurred due to a recent update in the settings classes, where all were made private in the develop branch of HERMES (commit a6c1a5e).

sferenz · 2025-01-31T16:45:57Z

src/hermes/commands/harvest/util/remote_harvesting.py

+        return None
+
+
+def _download_to_tempfile(url: str, filename: str) -> pathlib.Path:


Do you delete the tempfiles later?

In the current code the temp files for CFF and CodeMeta are stored separately in C:\Temp on the local machine.

Will these files be deleted after the extraction process?

The files won't be deleted after harvesting, however, I can modify the code to delete temp files after extraction. Do you agree with this change?

Yes, I think the temp files should be deleted at the end of the process.

I updated the code to remove the temp files after the harvesting process. Could you please have a look on the changes?

Aidajafarbigloo · 2025-02-03T10:48:22Z

Thanks for the nice code! Please have a look at the comments :)

@sferenz Thank you for the comments.

Add functionality to remove temp files generated during remote harvesting.

Remove temp files after harvesting CFF metadata

Remove temp files after harvesting CodeMeta metadata

…rovided-repository-URL' to incorporate the recent updates

To support repository URL as a path

sferenz · 2025-04-14T14:20:32Z

@sdruskat This pull request is ready to merge, can you please assign us a reviewer?

zyzzyxdonta

Thanks for your work!

I had a first look and I would like to suggest a slightly different approach. I think it would be beneficial to have the --url argument that you had (as indicated by your PR description). This would allow us to do the following:

Create a temporary directory
download the remote repository given by --url to this directory
overwrite args.path with the temporary directory path
run the normal harvesting step
delete the temporary directory

In this case there is no need to change anything in any of the plugins (I think). Only the base harvest command needs to worry about downloading and then deleting the files.

What do you think?

zyzzyxdonta · 2025-04-25T09:21:47Z

src/hermes/commands/harvest/util/remote_harvesting.py

+        return None
+
+
+def remove_temp_file(file_path: pathlib.Path, temp_dir: pathlib.Path = pathlib.Path("C:/Temp")):


C:/Temp is windows-specific. You could use tempfile.TemporaryDirectory and place the files in there. Then, instead of deleting the files one by one, you can use .cleanup() on the TemporaryDirectory object.

zyzzyxdonta · 2025-04-25T09:26:22Z

src/hermes/commands/harvest/util/remote_harvesting.py

+    return corrected_url.replace("https:/", "https://")
+
+
+def fetch_metadata_from_repo(repo_url: str, filename: str) -> t.Optional[pathlib.Path]:


This method makes multiple HTTP requests. I think it would be nice to use the hermes user agent, just to let the services know who we are. You can do something like:

from hermes.utils import hermes_user_agent session = requests.Session() session.headers.update({"User-Agent": hermes_user_agent})

then use the session to make the requests:

session.get(api_url)

Aidajafarbigloo · 2025-10-28T12:40:24Z

Thanks for your work!

I had a first look and I would like to suggest a slightly different approach. I think it would be beneficial to have the --url argument that you had (as indicated by your PR description). This would allow us to do the following:

Create a temporary directory

download the remote repository given by --url to this directory

overwrite args.path with the temporary directory path

run the normal harvesting step

delete the temporary directory

In this case there is no need to change anything in any of the plugins (I think). Only the base harvest command needs to worry about downloading and then deleting the files.

What do you think?

Thanks! I’ve implemented this approach and am testing it with a few different repositories.
Quick note: Cloning large repositories takes too long, it would be good to replace full clones with a shallow clone to check only for CITATION.cff or codemeta.json later.

Aidajafarbigloo and others added 3 commits October 10, 2024 13:21

Issue softwarepub#276 - Add a new argument to accept a URL for harves…

453198b

…ting

Issue softwarepub#276 - Harvest metadata from the provided URL

3a7c9ad

Issue softwarepub#276 - Store harvested data from URL

153e676

Aidajafarbigloo marked this pull request as draft October 29, 2024 08:49

Aidajafarbigloo added 5 commits January 30, 2025 16:15

Issue softwarepub#276 - Harvest metadata from CFF via path

16cba5d

Issue softwarepub#276 - Harvest metadata from CodeMeta via path

afb8189

Issue softwarepub#276 - Refactor functions for harvesting CFF/CodeMet…

09401ed

…a via path

Issue softwarepub#276 - Revert to original base.py

f193cc9

Issue softwarepub#276 - Update base.py

1bab2c7

sferenz suggested changes Jan 31, 2025

View reviewed changes

Aidajafarbigloo added 3 commits February 7, 2025 14:24

Add functionality to remove temp files

25eec31

Add functionality to remove temp files generated during remote harvesting.

Remove temp files

98814f4

Remove temp files after harvesting CFF metadata

Remove temp files

dd56827

Remove temp files after harvesting CodeMeta metadata

sferenz approved these changes Feb 7, 2025

View reviewed changes

Aidajafarbigloo and others added 4 commits February 12, 2025 13:40

Merge branch 'develop' into 'feature/276-harvesting-metadata-from-a-p…

88ad304

…rovided-repository-URL' to incorporate the recent updates

Issue softwarepub#276 - Add SPDX headers

5f75ad1

softwarepub#276 - Merge latest changes from develop into feature

b9e5523

Update base.py

4d901fc

To support repository URL as a path

Aidajafarbigloo marked this pull request as ready for review April 14, 2025 13:55

zyzzyxdonta self-requested a review April 25, 2025 08:12

zyzzyxdonta requested changes Apr 25, 2025

View reviewed changes

Aidajafarbigloo added 3 commits May 14, 2025 08:59

Fix issues: HERMES user agent and temporary files

3aa06a0

Fix hermes clean command

1bd4d1f

Small fix

3918954

Aidajafarbigloo added 3 commits June 6, 2025 09:19

Load token from toml file

f170481

Use token

a894259

Small fix

14fc040



		class HarvestSettings(BaseModel):
		class _HarvestSettings(BaseModel):

		return None


		def _download_to_tempfile(url: str, filename: str) -> pathlib.Path:

		return None


		def remove_temp_file(file_path: pathlib.Path, temp_dir: pathlib.Path = pathlib.Path("C:/Temp")):

		return corrected_url.replace("https:/", "https://")


		def fetch_metadata_from_repo(repo_url: str, filename: str) -> t.Optional[pathlib.Path]:

Feature/276 harvesting metadata from a provided repository url #278

Are you sure you want to change the base?

Feature/276 harvesting metadata from a provided repository url #278

Uh oh!

Conversation

Aidajafarbigloo commented Oct 29, 2024

Uh oh!

Aidajafarbigloo commented Oct 29, 2024

Uh oh!

Aidajafarbigloo commented Jan 31, 2025

Uh oh!

Aidajafarbigloo commented Jan 31, 2025

Uh oh!

sferenz left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aidajafarbigloo commented Feb 3, 2025

Uh oh!

sferenz commented Apr 14, 2025

Uh oh!

zyzzyxdonta left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Aidajafarbigloo commented Oct 28, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

zyzzyxdonta left a comment •

edited

Loading