From f4e6368e24c189f1b1530ab40cfbb1839b7564b2 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 04:52:14 +0100 Subject: [PATCH 01/14] Improving documentation --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 1264c3a..c9261dd 100644 --- a/README.md +++ b/README.md @@ -53,9 +53,9 @@ If you want the script to process VM vdisks - ### Additional User-Defined Datasets: -This is where you can add other datasets (non appdata ot vm ones) to be processed by the script: +This is where you can add other datasets (non appdata or vm ones) to be processed by the script: -- `source_datasets_array`: Specify custom paths in the format pool/dataset, e.g., "tank/mydata". +- `source_datasets_array`: Specify custom paths in the form of `"pool/dataset"`, e.g., `"tank/mydata"`. You can add whole ZFS disks to be watched for folders, e.g., `"disk1`" for the first disk in your array. ## Running the Script From 52c3814e7abf318f39061dec3d27e98e6247b148 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 09:26:24 +0100 Subject: [PATCH 02/14] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c9261dd..14d26c6 100644 --- a/README.md +++ b/README.md @@ -19,7 +19,8 @@ The script will do the following Before using the script, ensure the following: -- Unraid server (version 6.12 or higher) with ZFS support. +- You have installed Unraid server (version 6.12 or higher) +- Your Unraid server has ZFS-formatted drives in your array or pool(s) ready to be used. - [User Scripts](https://forums.unraid.net/topic/48286-plugin-user-scripts/) plugin is installed. - (Optional) [ZFS Master plugin](https://forums.unraid.net/topic/122261-plugin-zfs-master/) plugin is installed for enhanced ZFS functionality. - Plugins are installed via Unraid's Community Apps From e4f43294bdba9d6b0b6075b96b2f17816ca2ec58 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 09:29:31 +0100 Subject: [PATCH 03/14] Update README.md --- README.md | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 14d26c6..fae1014 100644 --- a/README.md +++ b/README.md @@ -2,12 +2,15 @@ This script is for converting directories into ZFS datasets on an Unraid server and runs in the User Scripts plugin. -It's proficient in processing appdata from Docker Containers, vdisks from VMs and various other locations within a single run. For directories storing appdata or VM vdisk data, the script is able to detect active containers or VMs that are using these folders. It will automatically stop these containers or VMs prior to initiating the conversion. -Set to operate on a schedule via Unraid user scripts, this tool then can continue to monitor datasets, making certain that their associated child folders remain as datasets. This is especially valuable when, for instance, installing a new container: its appdata will be converted automatically. Such functionality is invaluable for users keen on snapshotting individual containers, VMs, or various data structures. +It's proficient in processing appdata from Docker Containers, vdisks from VMs and various other locations within a single run. + +For directories storing appdata or VM vdisk data, the script is able to detect active containers or VMs that are using these folders. It will automatically stop these containers or VMs prior to initiating the conversion. + +Set to operate on a schedule via Unraid user scripts, this tool then can continue to monitor datasets, making certain that their associated child folders remain as datasets. When, for example, creating a new Docker container, its appdata will be converted automatically when this script runs. Such functionality is invaluable for users keen on snapshotting individual containers, VMs, or various data structures. ## Overview -The script will do the following +The script will do the following: - Evaluate whether a directory qualifies for conversion from a folder to a ZFS dataset. - Intelligently stop relevant Docker containers or VMs that are tied to directories earmarked for ZFS dataset conversion. From cee7126b2badfc579a994e7ba69700268666e154 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 09:32:02 +0100 Subject: [PATCH 04/14] Update README.md --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index fae1014..8dfb93e 100644 --- a/README.md +++ b/README.md @@ -1,7 +1,7 @@ # Unraid Auto Dataset Watcher & Converter v2 -This script is for converting directories into ZFS datasets on an Unraid server and runs in the User Scripts plugin. +This script is for converting directories into ZFS datasets on an Unraid server and runs in the User Scripts plugin. It's proficient in processing appdata from Docker Containers, vdisks from VMs and various other locations within a single run. For directories storing appdata or VM vdisk data, the script is able to detect active containers or VMs that are using these folders. It will automatically stop these containers or VMs prior to initiating the conversion. @@ -50,7 +50,7 @@ If you want the script to process Docker appdata - If you want the script to process VM vdisks - -- `should_process_vms`: Set to "yes" this tells the script the location contains vdisks so it can safely deal with it. +- `should_process_vms`: Set to "yes" this tells the script the location contains vdisks, allowing it to safely handle this specific usage scenario. - `source_pool_where_vm_domains_are`: Specify the source pool containing the VM domains. - `source_dataset_where_vm_domains_are`: Specify the source dataset for VM domains. - `vm_forceshutdown_wait`: Duration (in seconds) to wait before force stopping a VM if it doesn't shut down gracefully. @@ -67,7 +67,7 @@ This is where you can add other datasets (non appdata or vm ones) to be processe After you have configured the script, follow these steps: 1. Save any changes you've made to the script. -2. Run the script using the User Scripts plugin. For the initial run, if there are a significant number of folders requiring conversion, click the 'Run in Background' button. This ensures that you won't have to keep the browser window open, as closing it would otherwise terminate the script. +2. Run the script using the User Scripts plugin. For the initial run, if there are a significant number of folders requiring conversion, click the 'Run in Background' button. This ensures that you won't have to keep the browser window open, as closing it would otherwise terminate the script. 3. Configure the script to operate on a schedule that suits your needs, ensuring automated and timely conversions. ------------------------------------------------------------------ @@ -127,7 +127,7 @@ _Key Concepts_: - **rsync**: A fast, versatile utility for copying files and directories. It's often used for mirroring and backups. Keeps timestamps and permissions etc -**How script Works**: +**How the Script Works**: 1. The script first checks whether it should process Docker containers or VMs based on the user's settings. 2. For Docker containers, the script examines their bind mounts. If any bind mount's true location resides inside a regular folder (and not a ZFS dataset) in the designated source path for appdata, that container is stopped. From 912eedc6e93b82a148e39851a6d875838b308435 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 09:38:42 +0100 Subject: [PATCH 05/14] Update README.md --- README.md | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/README.md b/README.md index 8dfb93e..1ba2dad 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,5 @@ # Unraid Auto Dataset Watcher & Converter v2 - This script is for converting directories into ZFS datasets on an Unraid server and runs in the User Scripts plugin. It's proficient in processing appdata from Docker Containers, vdisks from VMs and various other locations within a single run. @@ -8,6 +7,12 @@ For directories storing appdata or VM vdisk data, the script is able to detect a Set to operate on a schedule via Unraid user scripts, this tool then can continue to monitor datasets, making certain that their associated child folders remain as datasets. When, for example, creating a new Docker container, its appdata will be converted automatically when this script runs. Such functionality is invaluable for users keen on snapshotting individual containers, VMs, or various data structures. + +> [!WARNING] +> ## Disclaimer +>While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes. +> Please use it wisely and responsibly. Remember: a single dataset and snapshots tha only exist on the same disk are not a backup. + ## Overview The script will do the following: @@ -73,7 +78,7 @@ After you have configured the script, follow these steps: ------------------------------------------------------------------ ------------------------------------------------------------------ -**Simplified Working Principle:** +## Simplified Working Principle 1. **Initialization**: @@ -118,7 +123,7 @@ After you have configured the script, follow these steps: - The script logs all actions taken, from the initial dataset path checks to the stopping and restarting of containers and VMs. -_Key Concepts_: +## Key Concepts - **Bind Mount**: A type of mount where a source directory or file is superimposed onto a destination, making its contents accessible from the destination. Used heavily in Unraid Docker templates. @@ -127,7 +132,7 @@ _Key Concepts_: - **rsync**: A fast, versatile utility for copying files and directories. It's often used for mirroring and backups. Keeps timestamps and permissions etc -**How the Script Works**: +## How the Script Works 1. The script first checks whether it should process Docker containers or VMs based on the user's settings. 2. For Docker containers, the script examines their bind mounts. If any bind mount's true location resides inside a regular folder (and not a ZFS dataset) in the designated source path for appdata, that container is stopped. @@ -136,15 +141,8 @@ _Key Concepts_: 5. Once the conversion process is done, the script restarts the containers and VMs it had stopped. 6. Prints results -**CONTRIBUTE TO THE PROJECT** +## Contribute to the Project Your insights and expertise can make a difference! If you've identified improvements or have suggestions for the script, I'd truly appreciate your contributions. Help me make this tool even better. I'm open to feedback, code enhancements, or new ideas. - - -**DISCLAIMER** - -While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes. - -Use wisely and responsibly!!! From 35eb540c57f8270deb6a545372f60e25cab0df8d Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 09:53:29 +0100 Subject: [PATCH 06/14] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 1ba2dad..1314a62 100644 --- a/README.md +++ b/README.md @@ -64,7 +64,7 @@ If you want the script to process VM vdisks - This is where you can add other datasets (non appdata or vm ones) to be processed by the script: -- `source_datasets_array`: Specify custom paths in the form of `"pool/dataset"`, e.g., `"tank/mydata"`. You can add whole ZFS disks to be watched for folders, e.g., `"disk1`" for the first disk in your array. +- `source_datasets_array`: Specify your paths in the form of `"pool/dataset"`, e.g., `"tank/mydata"`. You can add whole ZFS formatted disks to be watched for folders, e.g., `"disk1"` for the first disk in your array. If in doubt, check the GUI of the ZFS Master plugin in `Main` on your Unraid dashboard, and the correct name will be visible in the `ZFS Master` section of the page. ## Running the Script From 6df091e18dd91aacbbccd2c2192368e31d34f7ad Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:05:01 +0100 Subject: [PATCH 07/14] Update README.md --- README.md | 14 +++++++++++--- 1 file changed, 11 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 1314a62..bcbfaf1 100644 --- a/README.md +++ b/README.md @@ -125,12 +125,20 @@ After you have configured the script, follow these steps: ## Key Concepts +Whilst the below are very much huge simplifications, allowing less experienced users to grasp the fundamental concepts of ZFS is helpful for troubleshooting and avoiding disaster. + - **Bind Mount**: A type of mount where a source directory or file is superimposed onto a destination, making its contents accessible from the destination. Used heavily in Unraid Docker templates. -- **ZFS Dataset**: A ZFS dataset can be thought of as a sort of advanced folder with features like compression, quota, and snapshot capabilities. - -- **rsync**: A fast, versatile utility for copying files and directories. It's often used for mirroring and backups. Keeps timestamps and permissions etc +- **ZFS Dataset**: A ZFS dataset could be likened to an advanced folder with features like compression, quota, and snapshot capabilities. Though more experienced users may find likening them to the concept of a subvolume within a disk or pool to be more helpful. +- **rsync**: A fast, versatile utility for copying files and directories. It's often used for mirroring and backups. Keeps timestamps and permissions etc. + +- **Snaphots**: Snapshots are information that exists outside of the user filesystem and allows the ZFS filesystem to log changes between snapshots, meaning you can undo bad changes and file deletions. + +> [!CAUTION] +> Whilst you can roll back a snapshot, you can never revert to a snapshot that was taken at a later time. Ergo, you cannot return to the present point prior to rolling back the snapshot. The dataset has changed permanently. This highlights why a single dataset without replication of the data and snapshots is not a backup solution. This is the reason you would want this script, so that you can have more than one copy of your data and snapshots. Rolling back a snapshot is similar to 'undo', but there is no 'redo' on a sole dataset. +> If you need a few files or dirs, rather than rolling back a whole dataset, it is instead recommended to `cd .zfs` in the top level path of a dataset. The `.zfs` is not a user-visible path within the tree, but you can `cd` in and copy your file(s) out without needing to roll back a snapshot. You cannot move files or change files within the snapshot itself, you can only copy data out of it. +> An alternative solution to using `cd .zfs` is in duplicating to a different target dataset and experimenting with rolling back to different snapshots on a new dataset instead of your current one. ## How the Script Works From acb0bb9d6451fb57d66d85daebf7cb50c231d3ac Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:06:55 +0100 Subject: [PATCH 08/14] Update README.md --- README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index bcbfaf1..174ce12 100644 --- a/README.md +++ b/README.md @@ -136,8 +136,12 @@ Whilst the below are very much huge simplifications, allowing less experienced u - **Snaphots**: Snapshots are information that exists outside of the user filesystem and allows the ZFS filesystem to log changes between snapshots, meaning you can undo bad changes and file deletions. > [!CAUTION] -> Whilst you can roll back a snapshot, you can never revert to a snapshot that was taken at a later time. Ergo, you cannot return to the present point prior to rolling back the snapshot. The dataset has changed permanently. This highlights why a single dataset without replication of the data and snapshots is not a backup solution. This is the reason you would want this script, so that you can have more than one copy of your data and snapshots. Rolling back a snapshot is similar to 'undo', but there is no 'redo' on a sole dataset. +> Whilst you can roll back a snapshot, you can never revert to a snapshot that was taken at a later time. Ergo, you cannot return to the present point prior to rolling back the snapshot. The dataset has changed permanently. +> +>This highlights why a single dataset without replication of the data and snapshots is not a backup solution. This is the reason you would want this script, so that you can have more than one copy of your data and snapshots. Rolling back a snapshot is similar to 'undo', but there is no 'redo' on a sole dataset. +> > If you need a few files or dirs, rather than rolling back a whole dataset, it is instead recommended to `cd .zfs` in the top level path of a dataset. The `.zfs` is not a user-visible path within the tree, but you can `cd` in and copy your file(s) out without needing to roll back a snapshot. You cannot move files or change files within the snapshot itself, you can only copy data out of it. +> > An alternative solution to using `cd .zfs` is in duplicating to a different target dataset and experimenting with rolling back to different snapshots on a new dataset instead of your current one. ## How the Script Works From f4ee624e32af008703c52a90e0d451d04ea160b1 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:07:49 +0100 Subject: [PATCH 09/14] Update README.md --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 174ce12..5ab8161 100644 --- a/README.md +++ b/README.md @@ -11,6 +11,7 @@ Set to operate on a schedule via Unraid user scripts, this tool then can continu > [!WARNING] > ## Disclaimer >While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes. +> > Please use it wisely and responsibly. Remember: a single dataset and snapshots tha only exist on the same disk are not a backup. ## Overview From 255b60eca5c5f9c8d7ec4748118d3c31d331532a Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:08:20 +0100 Subject: [PATCH 10/14] Update README.md --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5ab8161..a6b26bd 100644 --- a/README.md +++ b/README.md @@ -12,7 +12,9 @@ Set to operate on a schedule via Unraid user scripts, this tool then can continu > ## Disclaimer >While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes. > -> Please use it wisely and responsibly. Remember: a single dataset and snapshots tha only exist on the same disk are not a backup. +> Please use it wisely and responsibly. +> +> Remember: a single dataset and snapshots tha only exist on the same disk are not a backup. ## Overview From b0c568ad6c2429d02a1c2c6c343de6d2e0c1f746 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:08:53 +0100 Subject: [PATCH 11/14] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index a6b26bd..080cec1 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ Set to operate on a schedule via Unraid user scripts, this tool then can continu > [!WARNING] > ## Disclaimer ->While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes. +> While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes. > > Please use it wisely and responsibly. > From 50065f46fbef24081fbeda2d492eddaf86bba86d Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:09:39 +0100 Subject: [PATCH 12/14] Update README.md --- README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 080cec1..660d956 100644 --- a/README.md +++ b/README.md @@ -5,8 +5,9 @@ It's proficient in processing appdata from Docker Containers, vdisks from VMs an For directories storing appdata or VM vdisk data, the script is able to detect active containers or VMs that are using these folders. It will automatically stop these containers or VMs prior to initiating the conversion. -Set to operate on a schedule via Unraid user scripts, this tool then can continue to monitor datasets, making certain that their associated child folders remain as datasets. When, for example, creating a new Docker container, its appdata will be converted automatically when this script runs. Such functionality is invaluable for users keen on snapshotting individual containers, VMs, or various data structures. +Set to operate on a schedule via Unraid user scripts, this tool then can continue to monitor datasets, making certain that their associated child folders remain as datasets. When, for example, creating a new Docker container, its appdata will be converted automatically when this script runs. +Such functionality is invaluable for users keen on snapshotting individual containers, VMs, or various data structures. > [!WARNING] > ## Disclaimer From adf8fbe9021c82229b24d913553b300001f7c48a Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:13:20 +0100 Subject: [PATCH 13/14] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 660d956..ec1f3aa 100644 --- a/README.md +++ b/README.md @@ -147,6 +147,10 @@ Whilst the below are very much huge simplifications, allowing less experienced u > If you need a few files or dirs, rather than rolling back a whole dataset, it is instead recommended to `cd .zfs` in the top level path of a dataset. The `.zfs` is not a user-visible path within the tree, but you can `cd` in and copy your file(s) out without needing to roll back a snapshot. You cannot move files or change files within the snapshot itself, you can only copy data out of it. > > An alternative solution to using `cd .zfs` is in duplicating to a different target dataset and experimenting with rolling back to different snapshots on a new dataset instead of your current one. +> +> It is always recommended to `hold` at least one snapshot, meaning the snapshot can never be destroyed automatically until the hold is released. The snapshot you hold should be the best and sometimes most recent version when you transfer your data into a dataset, or convert a folder/dir into a dataset. This can enable you to undo future potential problems. +> +> For the people at the back: ZFS snapshots on a single disk are not a backup. ## How the Script Works From a325b819785402816c3294160b2c25821480d185 Mon Sep 17 00:00:00 2001 From: Hammy Havoc Date: Wed, 4 Jun 2025 10:13:47 +0100 Subject: [PATCH 14/14] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index ec1f3aa..5dea4a5 100644 --- a/README.md +++ b/README.md @@ -142,7 +142,7 @@ Whilst the below are very much huge simplifications, allowing less experienced u > [!CAUTION] > Whilst you can roll back a snapshot, you can never revert to a snapshot that was taken at a later time. Ergo, you cannot return to the present point prior to rolling back the snapshot. The dataset has changed permanently. > ->This highlights why a single dataset without replication of the data and snapshots is not a backup solution. This is the reason you would want this script, so that you can have more than one copy of your data and snapshots. Rolling back a snapshot is similar to 'undo', but there is no 'redo' on a sole dataset. +> This highlights why a single dataset without replication of the data and snapshots is not a backup solution. This is the reason you would want this script, so that you can have more than one copy of your data and snapshots. Rolling back a snapshot is similar to 'undo', but there is no 'redo' on a sole dataset. > > If you need a few files or dirs, rather than rolling back a whole dataset, it is instead recommended to `cd .zfs` in the top level path of a dataset. The `.zfs` is not a user-visible path within the tree, but you can `cd` in and copy your file(s) out without needing to roll back a snapshot. You cannot move files or change files within the snapshot itself, you can only copy data out of it. >