What is a method for syncing a large amount of data, and splitting the load between multiple hosts?
- Start by getting a sorted list of directories and their size (in KB)
du --max-depth=1 /mnt/source | sort -n > ~/source-du
- Copy the ~/source-du file to your local machine
- Open the file in a text editor that understands tabs, like Notepad++ or MacVim (generally a text editor running in a terminal emulator will convery tabs to spaces, which won't work for us)
- Copy the contents of the file into a spreadsheet editor, like Microsoft Excel, or Google Sheets