Obsidian-iCloud sync

Keep your notes in sync

Obsidian-iCloud sync header image

For a few years now I’ve been using Obsidian as my second brain. The magic of Obsidian is that it’s only a bunch of markdown files and a clever editor on top of it. This means that any file-syncing solution should be good enough. But the truth is a little bit more complicated than that.

The use case

I maintain three different vaults. One of them is for my personal things, notes, contacts, appointments, and chores. The second one is for work, where I keep meeting my activity journal, notes, project details, plans, and general troubleshooting procedures. Finally, the third one is the one that serves my digital garden publicly.

I currently use iCloud for file syncing and some sort of backup. Yes, I know it is not strictly a backup mechanism: if you delete a file in iCloud it’s removed from everywhere, but if your computer expires tomorrow, you’ll be happy you’ve been saving your stuff under the iCloud drive.

Also, I need access to my information when on the go. Sometimes I’m using another computer and even if I’m not working, I need to check something. Sometimes I’m out and I only have access to my phone and I need to add something to my pool of knowledge.

However, a very common problem with it is that, as your vault grows, iCloud will keep de-prioritizing that folder to be kept on disk. It will only keep references to them and download them on demand, which sounds transparent enough but doesn’t work quite well with Obsidian. It makes it slow, and it prevents search and some plugins from operating properly.

Deactivating “Optimize my Mac” (which drives this behaviour) might not be an option if you don’t want your full iCloud to be stored locally.

iCloud for Windows has a Pin Folder functionality that will ensure a selected folder stays locally even though the rest of them might now. Why this is not available to MacOS, it’s a mystery to most of the internet. ¯\_(ツ)_/¯

Failed solutions

Here are some approaches I’ve tried and why they didn’t work for me.

Keeping a local copy and just copying it to iCloud every once in a while. While this approach looks simple and easy to do, it comes with a problem: what do you do about moved/renamed/deleted files? You would have to keep track of all the changes and apply them to the other vaults. At this point, copying is not enough, you’re talking about synchronization.

Synchronization then, using rsync. I tried this too – but what happens when you make changes in another computer or on mobile? At this point, your rsync process becomes destructive, because it’s only one way.

Two-way synchronization, using unison. Unison is a really cool tool: you set it up with multiple folders and with some really complex settings (but really smart defaults) and it will listen to changes in them and sync changes from any of those to all the others. However, it doesn’t recognize iCloud files as real files, so it will detect everything as being deleted once iCloud decides to offload any of the files.

I was really close to testing multi-computer synchronization out of iCloud using tools like SyncThing, but I stopped myself right before that: if I need to sidestep the tools I use (iCloud for synchronization), then I’m only complicating my own ecosystem.

At this point, we have cornered the problem enough to understand what we need to attack: iCloud needs to keep the files locally.

The solution

The brctl command (bird service control center) will allow us to download files and folders from iCloud, and it nudges it in the direction of keeping the folder downloaded for a bit.

#!/bin/bash
download_icloud_folder() {
    # Retrieve the first parameter as the folder to download
    local folder="$1"

    # Check that the parameter is not empty and that the folder exists
    if [ -z "$folder" ] || [ ! -d "$folder" ]; then
        echo "The folder \"$folder\" does not exist"
        return 1
    fi

    find "$folder" -type f -exec brctl download "{}" \;
}

# Check if the script was invoked directly
if [ "${BASH_SOURCE[0]}" == "${0}" ]; then
    download_icloud_folder "$@"
fi

(Source: AlphaGit/dotfiles - download-icloud-folder.sh)

What I’ve done is create a cronjob that every hour will just ask brctl to keep that folder downloaded, and it has worked wonders for me.

To install this into my regular cronjob, I have added the following to my install.sh script:

if [ ! -d "$HOME" ]; then
    HOME=$(eval echo ~)
fi
ESCAPED_HOME=$(echo $HOME | sed 's/\//\\\//g')

# Install cron jobs
crontab -l | cat - $SCRIPT_DIR/.alpha_crontab | sed -e "s/\\\$HOME/$ESCAPED_HOME/g" | sort -r | uniq | crontab -

(Source: AlphaGit/dotfiles - install.sh)

This is very simple: it gets or detects the $HOME location to use, it escapes the slashes in it (replacing / with \/), and then it adds the existing contents of the crontab file to the .alpha_crontab file, replacing the $HOME text with the contents detected before and storing it in crontab again.

What’s the content of the .alpha_crontab file, you ask? Very simple:

TZ=America/Toronto
2 * * * * $HOME/functions/download-icloud-folder.sh $HOME/Library/Mobile\ Documents/iCloud~md~obsidian

(Source: AlphaGit/dotfiles - .alpha_crontab)

As a last note, don’t forget to give /usr/sbin/cron full disk access in Privacy settings, or otherwise it will complain it cannot execute the find operation on the iCloud folder. I haven’t quite found an elegant way to automate this step. To do it manually, you can just drag the executable file from Finder into the Settings app.

Design choices

There are a few choices that I had to make that are not quite intuitive.

It runs periodically: As I mentioned, iCloud does not comply with your decision to keep a folder downloaded, but it will comply with downloading it once you specify it. brctl works better for downloading than the UI features provided by iCloud and Finder.

The timeframe can be adjusted to your liking, I’ve found that doing this every hour has been a sweet spot for me. It depends on how frequently you return to Obsidian and how frequently you use the space on your disk (which would prompt iCloud to offload some of the folders out).

It needs to use absolute paths: As I have been working on my dotfiles repository for a while, most of what I can make relative is better, as it will adapt to different setups and even different people. However, cron is not able to properly execute the commands on relative paths, or even use ~, so I used

It needs to iterate on files: While brctl claims to have support on folders, my practical tests showed that operating on files directly will trigger the download right away. I have no explanation for that one.

Also, I could not find a way to properly pass multiple files at once (not that it would be feasible anyway, unless it accepted glob parameters which I highly doubt). The download command is undocumented at the moment.

Alpha's Manifesto

Navigation

My Other Sites

Obsidian-iCloud sync

The use case

Failed solutions

The solution

Design choices

Other solutions

Obsidian Sync

Dropbox

Git