Skip to content
70 changes: 46 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,25 @@
# Unraid Auto Dataset Watcher & Converter v2

This script is for converting directories into ZFS datasets on an Unraid server and runs in the User Scripts plugin.
It's proficient in processing appdata from Docker Containers, vdisks from VMs and various other locations within a single run.

This script is for converting directories into ZFS datasets on an Unraid server and runs in the User Scripts plugin.
It's proficient in processing appdata from Docker Containers, vdisks from VMs and various other locations within a single run. For directories storing appdata or VM vdisk data, the script is able to detect active containers or VMs that are using these folders. It will automatically stop these containers or VMs prior to initiating the conversion.
Set to operate on a schedule via Unraid user scripts, this tool then can continue to monitor datasets, making certain that their associated child folders remain as datasets. This is especially valuable when, for instance, installing a new container: its appdata will be converted automatically. Such functionality is invaluable for users keen on snapshotting individual containers, VMs, or various data structures.
For directories storing appdata or VM vdisk data, the script is able to detect active containers or VMs that are using these folders. It will automatically stop these containers or VMs prior to initiating the conversion.

Set to operate on a schedule via Unraid user scripts, this tool then can continue to monitor datasets, making certain that their associated child folders remain as datasets. When, for example, creating a new Docker container, its appdata will be converted automatically when this script runs.

Such functionality is invaluable for users keen on snapshotting individual containers, VMs, or various data structures.

> [!WARNING]
> ## Disclaimer
> While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes.
>
> Please use it wisely and responsibly.
>
> Remember: a single dataset and snapshots tha only exist on the same disk are not a backup.

## Overview

The script will do the following
The script will do the following:

- Evaluate whether a directory qualifies for conversion from a folder to a ZFS dataset.
- Intelligently stop relevant Docker containers or VMs that are tied to directories earmarked for ZFS dataset conversion.
Expand All @@ -19,7 +31,8 @@ The script will do the following

Before using the script, ensure the following:

- Unraid server (version 6.12 or higher) with ZFS support.
- You have installed Unraid server (version 6.12 or higher)
- Your Unraid server has ZFS-formatted drives in your array or pool(s) ready to be used.
- [User Scripts](https://forums.unraid.net/topic/48286-plugin-user-scripts/) plugin is installed.
- (Optional) [ZFS Master plugin](https://forums.unraid.net/topic/122261-plugin-zfs-master/) plugin is installed for enhanced ZFS functionality.
- Plugins are installed via Unraid's Community Apps
Expand All @@ -46,30 +59,30 @@ If you want the script to process Docker appdata -

If you want the script to process VM vdisks -

- `should_process_vms`: Set to "yes" this tells the script the location contains vdisks so it can safely deal with it.
- `should_process_vms`: Set to "yes" this tells the script the location contains vdisks, allowing it to safely handle this specific usage scenario.
- `source_pool_where_vm_domains_are`: Specify the source pool containing the VM domains.
- `source_dataset_where_vm_domains_are`: Specify the source dataset for VM domains.
- `vm_forceshutdown_wait`: Duration (in seconds) to wait before force stopping a VM if it doesn't shut down gracefully.

### Additional User-Defined Datasets:

This is where you can add other datasets (non appdata ot vm ones) to be processed by the script:
This is where you can add other datasets (non appdata or vm ones) to be processed by the script:

- `source_datasets_array`: Specify custom paths in the format pool/dataset, e.g., "tank/mydata".
- `source_datasets_array`: Specify your paths in the form of `"pool/dataset"`, e.g., `"tank/mydata"`. You can add whole ZFS formatted disks to be watched for folders, e.g., `"disk1"` for the first disk in your array. If in doubt, check the GUI of the ZFS Master plugin in `Main` on your Unraid dashboard, and the correct name will be visible in the `ZFS Master` section of the page.


## Running the Script

After you have configured the script, follow these steps:

1. Save any changes you've made to the script.
2. Run the script using the User Scripts plugin. For the initial run, if there are a significant number of folders requiring conversion, click the 'Run in Background' button. This ensures that you won't have to keep the browser window open, as closing it would otherwise terminate the script.
2. Run the script using the User Scripts plugin. For the initial run, if there are a significant number of folders requiring conversion, click the 'Run in Background' button. This ensures that you won't have to keep the browser window open, as closing it would otherwise terminate the script.
3. Configure the script to operate on a schedule that suits your needs, ensuring automated and timely conversions.

------------------------------------------------------------------
------------------------------------------------------------------

**Simplified Working Principle:**
## Simplified Working Principle

1. **Initialization**:

Expand Down Expand Up @@ -114,16 +127,32 @@ After you have configured the script, follow these steps:

- The script logs all actions taken, from the initial dataset path checks to the stopping and restarting of containers and VMs.

_Key Concepts_:
## Key Concepts

Whilst the below are very much huge simplifications, allowing less experienced users to grasp the fundamental concepts of ZFS is helpful for troubleshooting and avoiding disaster.

- **Bind Mount**: A type of mount where a source directory or file is superimposed onto a destination, making its contents accessible from the destination. Used heavily in Unraid Docker templates.

- **ZFS Dataset**: A ZFS dataset can be thought of as a sort of advanced folder with features like compression, quota, and snapshot capabilities.

- **rsync**: A fast, versatile utility for copying files and directories. It's often used for mirroring and backups. Keeps timestamps and permissions etc
- **ZFS Dataset**: A ZFS dataset could be likened to an advanced folder with features like compression, quota, and snapshot capabilities. Though more experienced users may find likening them to the concept of a subvolume within a disk or pool to be more helpful.


**How script Works**:
- **rsync**: A fast, versatile utility for copying files and directories. It's often used for mirroring and backups. Keeps timestamps and permissions etc.

- **Snaphots**: Snapshots are information that exists outside of the user filesystem and allows the ZFS filesystem to log changes between snapshots, meaning you can undo bad changes and file deletions.

> [!CAUTION]
> Whilst you can roll back a snapshot, you can never revert to a snapshot that was taken at a later time. Ergo, you cannot return to the present point prior to rolling back the snapshot. The dataset has changed permanently.
>
> This highlights why a single dataset without replication of the data and snapshots is not a backup solution. This is the reason you would want this script, so that you can have more than one copy of your data and snapshots. Rolling back a snapshot is similar to 'undo', but there is no 'redo' on a sole dataset.
>
> If you need a few files or dirs, rather than rolling back a whole dataset, it is instead recommended to `cd .zfs` in the top level path of a dataset. The `.zfs` is not a user-visible path within the tree, but you can `cd` in and copy your file(s) out without needing to roll back a snapshot. You cannot move files or change files within the snapshot itself, you can only copy data out of it.
>
> An alternative solution to using `cd .zfs` is in duplicating to a different target dataset and experimenting with rolling back to different snapshots on a new dataset instead of your current one.
>
> It is always recommended to `hold` at least one snapshot, meaning the snapshot can never be destroyed automatically until the hold is released. The snapshot you hold should be the best and sometimes most recent version when you transfer your data into a dataset, or convert a folder/dir into a dataset. This can enable you to undo future potential problems.
>
> For the people at the back: ZFS snapshots on a single disk are not a backup.

## How the Script Works

1. The script first checks whether it should process Docker containers or VMs based on the user's settings.
2. For Docker containers, the script examines their bind mounts. If any bind mount's true location resides inside a regular folder (and not a ZFS dataset) in the designated source path for appdata, that container is stopped.
Expand All @@ -132,15 +161,8 @@ _Key Concepts_:
5. Once the conversion process is done, the script restarts the containers and VMs it had stopped.
6. Prints results

**CONTRIBUTE TO THE PROJECT**
## Contribute to the Project

Your insights and expertise can make a difference! If you've identified improvements or have suggestions for the script, I'd truly appreciate your contributions. Help me make this tool even better.

I'm open to feedback, code enhancements, or new ideas.


**DISCLAIMER**

While this script has been thoroughly tested and is believed to be reliable, unforeseen edge cases may arise. By using this software, you acknowledge potential risks and agree to use it at your own discretion. The author assumes no responsibility for any unintended outcomes.

Use wisely and responsibly!!!