Backblaze B2 and Go

30 November 2015

I’ve been implementing a Go client library for the Backblaze B2 cloud storage service: go-backblaze. It works extremely well, but is a bit slow to get files back.

Update 12th December: I’ve spoken to Backblaze, who have been working to improve performance. I have performed some new tests and written them up.

I’ve implemented a parallel download feature which you can use to download several files at the same time - this doesn’t seem to affect the speed of individual downlaods very much, so I assume that the downloads are limited for one of two reasons

  1. There is a download speed cap in place
  2. The downloads each come from separate parts of the cluster file system, and so don’t affect each other.

Downloading 5 copies of the same file in parallel doesn’t seem to affect the download speed, nor do 5 sequential downloads. Whatever you do, each download seems to run at about 200KiB/sec.

Note - the b2 command used here is my own implementation of a b2 client, rather than the python b2 client provided by Backblaze

Uploading 221MiB (sequentially): 1.65MiB/sec

$ time b2 put -v VirtualBox-5.0.10-104061-OSX.dmg N1.dmg Bitcasa_1.5.1615.dmg  Wireshark\ 2.0.0\ Intel\ 64.dmg
VirtualBox-5.0.10-104061-OSX.dmg                     90044871 [==================] 1973.19 KB/s 100%
N1.dmg                                               76457433 [==================] 2142.65 KB/s 100%
Bitcasa_1.5.1615.dmg                                 33741540 [==================] 2380.39 KB/s 100%
Wireshark 2.0.0 Intel 64.dmg                         31522805 [==================] 1838.25 KB/s 100%

real	2m14.334s
user	0m8.893s
sys	0m7.747s

Downloading sequentially: very slow!

$ time b2 get -v -j 1 VirtualBox-5.0.10-104061-OSX.dmg N1.dmg Bitcasa_1.5.1615.dmg  Wireshark\ 2.0.0\ Intel\ 64.dmg
VirtualBox-5.0.10-104061-OSX.dmg                     90044871 [========>---------]  270.50 KB/s  54%
^C
real	3m1.831s
user	0m7.825s
sys	0m9.693s

Downloading in parallel: better - 0.56MiB/sec average

$ time b2 get -v -j 5 VirtualBox-5.0.10-104061-OSX.dmg N1.dmg Bitcasa_1.5.1615.dmg  Wireshark\ 2.0.0\ Intel\ 64.dmg
VirtualBox-5.0.10-104061-OSX.dmg                     90044871 [==================]  224.20 KB/s 100%
N1.dmg                                               76457433 [==================]  236.36 KB/s 100%
Bitcasa_1.5.1615.dmg                                 33741540 [==================]  212.05 KB/s 100%
Wireshark+2.0.0+Intel+64.dmg                         31522805 [==================]  213.37 KB/s 100%

real	6m35.563s
user	0m30.205s
sys	0m27.586s