Key Takeaways:
- Utilizing command line arguments and setting up symbolic links can efficiently manage and optimize the storage of models and assets, significantly reducing disk space usage.
- The choice of storage medium (HDD, SSD, NVMe m.2 drives) directly impacts the performance of loading and running Stable Diffusion models.
- External drives, NAS, and cloud storage for completed projects or older model versions can help maintain an organized and efficient workflow, though each comes with its own set of considerations regarding access speed and convenience.
A downside when working with diffusion models locally is they can quickly consume disk space. This becomes a problem if you are using many different applications for individual use cases or projects.
What’s Taking Up Space
Here’s a list of the biggest source of storage hogs that you’ll find taking up space on your machine:
| Model/Asset | Size Estimate |
|---|---|
| Stable Diffusion 1.5 | 2GB – 4GB |
| Stable Diffusion 2.1 | 5GB |
| Stable Diffusion XL | 7GB |
| Stable Diffusion Cascade | 3.5GB – 20GB |
| ControlNet | 1.5GB |
| LoRAs | < 1GB (total) |
| CLIP | ~1GB |
| Other Assets (VAEs, Upscalers, etc.) | 0.3GB – 1GB |
Here’s an accompanying video covering the key points within this article.
Native Solutions for Asset Management
ComfyUI, WebUI Forge, Automatic1111 WebUi, Fooocus, and SD.Next all have built-in files and/or command line arguments that allow you to set one directory “of truth” that contains all the source assets you use when working with Stable Diffusion models.
Before Proceeding: Take a minute to decide whether you want one application as the ‘record of truth’ for your models, or if you want a directory outside of all applications (recommended) to be the place where all your assets are stored.
Comfy UI
ComfyUI has the ‘extra_model_paths.yaml.example‘ file, of which you can set the path of your preferred directory for models, LoRAs, Clip, VAE, and other critical assets. Or if you have Automatic1111 WebUI previously installed, you can set the base_path of Automatic1111 WebUI and pull from there.
When modifying this file, make sure you remove the ‘.example’ from the file extension then restart ComfyUI.
Automatic 1111 Stable Diffusion Web UI
Automatic1111 Stable Diffusion WebUI has a full list of many different directory-related command line arguments that can be added to the webui-user file to set the location of your models and such. Be sure to not only add the command line argument, but the file location as well.
Example:
--ckpt-dir /path/to/your/models --vae-dir /path/to/your/vaesNote: The –EXAMPLE-path command line arguments only define a single file. You’ll likely always want to use the –EXAMPLE-dir arguments instead.
Fooocus
After starting Fooocus for the first time, a config.txt file will be generated in the root directory. Once generated, you can update where on your machine your models, LoRAs, and other assets are located:
Just update each of these lines with the new location.
SD.Next & Web UI Forge
Forge and SD.Next are essentially forks (a modified copy of) Automatic1111 Web UI, and support many of the same command line arguments.
If you are using SD.Next, check their command line arguments here for defining directory locations.
As for WebUI Forge, they recently added the --forge-ref-a1111-home command line argument which references all the files from Automatic1111 WebUI so you don’t have to copy everything over.
Within WebUI Forge, you can still use all the directory-related command line arguments if you wish.
The Power of Symbolic Links
Beyond developer-supported configurations, symbolic links are an alternative. These are effectively aliases or shortcuts that you can set that reference the file or folder location on your system.
Creating a symbolic link is straightforward:
- For Mac and Linux: Open a terminal window and use the
ln -s <source> <target>command, where<source>is the model’s storage location, and<target>is the intended shortcut location. You can learn more about symlinks here. - For Windows: Utilize the
mklinkcommand. When linking a directory, ensure the/Dflag is included to designate a directory reference. Unlike Mac and Linux, the order of the source and target is reversed, with the source being the second argument. Here’s the official Microsoft documentation on mklink.
Symbolic links can significantly simplify model management without duplicating files across different locations.
They may be a bit more advanced than the native solutions, but a very viable option nevertheless.
Data Deduplication Programs
Data deduplication tools like fdupes, rmlint, and dupeGuru are programs specifically designed for handling redundant files on a system.
Only consider these if you are comfortable with the command line and understand the risks of removing duplicate files.
Choosing the Right Storage Location for Diffusion Models
Before offloading models to a slower or larger drive, its important to understand the impact that read speed may have on your productivity.
Typical read speed of a HDD is 80MB/s whereas a SSD read speed is more than double at 200MB/s. Lastly, modern computers may support NVMe m.2 drives, which provide a read speed of 5000MB/s (source).
External drives and Network Attached Storage solutions should only be used to store completed projects or previous model versions as transfer speeds are even slower.
Cloud or off-site storage could be considered, especially when sharing assets with other team members.
Lastly, re-downloading the models may work, but carries significant risk as files may no longer be published, hosting provider shuts down, etc. and should be used as a last resort.