Setting up Kobo sync with Calibre Web
There I was, minding my own business downloading ebooks from my Calibre server to my Kobo Libra 2 via the web browser when I stumbled...
A video from Novaspirit Tech inspired this project, which I tweaked to my particular needs. Credit goes to Don for showing me how to get started on this, and that it was even possible. Basically, I have the ability to route all of my Docker traffic through a virtual private network (VPN). That means everything, including my Deluge BitTorrent client.
I considered the Transmission-OpenVPN container, but I wanted to have the flexibility to route other containers through my VPN should I choose to in the future. Thus, I opted for the Gluetun container. It allows me to use the VPN provider I want, which is Proton VPN.
I keep several of my devices connected to Proton VPN at any given time, usually through a master device such as a router. I even built a travel version using a Raspberry Pi running OpenWRT that simply needs to connect to a host network. I then pair up my devices to it and boom, tunneled traffic. I'll share that story another time since I need to rebuild the project after my SD card was damaged.
One of my own needs with this project was that I didn't want to fill up my virutal machine's storage. While I allocated quite a bit to the headless Debian installation that's running all of my Docker containers, I have a Network Attached Storage (NAS) server for a reason. So I wanted Deluge to route through a VPN and download to my NAS via a Samba (SMB) share to save myself the time of copying files from the virtual machine (VM) to my NAS.
This added significant complexity with permissions, which, as anyone who cosplays as a Linux sysadmin knows, can be a huge pain. I ran into countless issues with Deluge failing to see the SMB share, but this BitTorrent client proved to be, by far, an easier task to get running than my attempts to use Transmission.
I've used Deluge before, but my go-to BitTorrent clients have typically been either Transmission or qBitTorrent. (I used µTorrent for a while, too.) So when considering this project, a Transmission container was my end goal. I tested the setup with Deluge initially since that's what Don did in his video. It worked until I added the NAS component, and then it would either refuse to see the storage or downloads would slow to a crawl and eventually stop. So I nuked the container and spun up a Transmission one.
From the get-go, I ran into permissions issues. Typically when I'm working with Docker — and I still consider myself a newbie — I create the directories first before mapping the volumes for the containers. But I didn't do that for Transmission, so Docker (or Portainer) did it for me, which I suspect introduced a host of permissions problems.
I tried to resolve this as best as I could by tweaking things with chmod
, chown
, and chgrp
to switch things between my local account, root, and the docker group to no avail. Transmission couldn't access its resume file, so even if I did get a torrent going, I couldn't pause or lose connection. Some downloads would complete just fine, but if the download stopped for any reason, I had to delete the torrent and start over.
"Welp," I figured, "that was progress," so I mounted my NAS' downloads directory to my Debian server and attempted to download my test torrent again. Transmission threw an error at me immediately. I was stumped, because it was a similar situation to Deluge earlier.
When organizing my filesystem structure, I like to be as clean and organized as possible. I had created a /nas
folder under the /media
directory, which is where I pointed the SMB share in my /etc/fstab
file. After messing around with permissions and ownership, I moved the SMB share to a /downloads
folder in my /home
directory. Same problems. I spent hours on this process with no luck, so I nuked the container and started over.
I tried moving the SMB mount location to somewhere inside the container's config folder. I checked the corresponding share folder on the NAS and Transmission had created three folders there: download, complete, and incomplete. Progress! Except perhaps not. Downloads would still happen, but they would vanish somewhere I couldn't find. I checked DSM — the OS for my Synology NAS — and saw nothing in the downloads folder I had set up. My test torrent download was just floating somewhere, or the Transmission UI was lying to me when it said it had completed the job.
I decided to give up after some six hours of messing with this and came back the next day... and I never was successful. So I gave up. I learned a lot about Linux permissions and adjusting them in the command line during this process, so I choose to look at this positively instead of getting frustrated at the wasted hours I could have used doing something else.
I decided to return to Deluge out of sheer stubbornness. I set up the container, routed its traffic through Gluetun, and tried connecting my NAS to the container over SMB again. To be perfectly frank, I don't know what worked this time, because I repeated the same steps I did before, and yet I was successful.
In my /etc/fstab
file, I pointed my NAS to ~/downloads
(with the full file path). Since my user was part of the Docker group, I figured that I shouldn't have permissions issues with the container downloading to the folder.
When I loaded up the Deluge web GUI, I confirmed that my IP address pointed to a Proton server, not my home IP. The available storage also showed in the terabytes, confirming that Deluge's download system was connected to my NAS. I grabbed a test torrent magnet link, brought it over to Deluge, and started the download. It worked.
The down speed started to ramp up until it hit multiple megabytes per second. Deluge never paused or slowed down until the process finished. I popped over to my NAS to check if the file had downloaded properly and, sure enough, there it was.
Overall, this container routing through another container setup works rather well. However, if you use Watchtower — a container that monitors your other containers for updates — like I do, then you'll have to check in on Deluge every so often. If Gluetun gets an update, and it does quite often, you'll need to reconnect Deluge to it. I recommend doing so in Portainer for ease of use.
Of course, you might think that you're limited to the web GUI for the container, but you can turn the desktop Deluge apps into thin clients. Simply input the IP address for the server running the Deluge container and leave the port as default. Start the daemon on the client machine (important!), input the username and password, then you'll be able to control the Deluge container from your local machine.
I could establish some elaborate file structure with my NAS and the Docker container to ensure what I download goes to the proper folder on the former. Right now, I go into the Downloads folder on the NAS and rename things to match my naming schema, moving the files to their proper places afterward. This whole process is not automated, but it's a lot easier than downloading a torrent to my PC, then uploading it to my NAS. (I also seed many of my torrents to contribute to the community, such as different FOSS projects.)
With this project, I found that rare joy you get when you accomplish something, especially if you've overcome obstacles along that road. Just like many other things in life, this process reminds me of something Marcus Aurelius said, "The mind adapts and converts to its own purposes the obstacle to our acting. The impediment to action advances action. What stands in the way becomes the way."
So what did I learn? I can list a few things.
Firstly, to not give up when something doesn't work the first time. If you meet failure, try again until you exhaust all available options. What would have happened if I nuked the Deluge container the first time, retried my fstab entry, and spun everything up again?
Like I said before, I learned so much about Linux permissions. Considering I didn't have a GUI to serve as a crutch, I had to rely on the terminal to get things done. While I did ultimately didn't make Transmission work like I wanted, the process taught me priceless information about chown
, chmod
, and chgrp
.
Even if you're not using BitTorrent technology for nefarious reasons, you should absolutely mask the traffic behind a VPN. My ISP, Fios, doesn't seem to care about torrent traffic on its network, but yours might. It's equally important to find a VPN provider that you trust — most are not trustworthy. Check the logging policies, privacy statements, and, perhaps most importantly, the countries of origin. You could also make your own VPN via a VPS, but that's another topic.
This is also the project where I started getting serious about my documentation best practices. Not only does it benefit me in case I need to redo this or face a similar issue in the future, but it might help some of you replicate this for yourself.
Playing with Docker is a lot of fun and my Deluge/Gluetun combination is just a small piece of the pie that I have going on one of my Debian installations. My virtualization resources are limited to my mini PCs' hardware for the time being, but when I build my own servers in the near future, I plan to have a lot more services running.
The best projects are the ones that teach you something along the way. After all, that's part of the point of having a homelab.
You can view the documentation for this project on my wiki.