Kothar Labs Contact Me

Streaming multipart HTTP requests in Go
Sat, 23 Feb 2019 / blog / go

Having seen Peter Bourgon's post on multipart http responses come up in the Golang Weekly newsletter, I thought I would write some notes on a related problem I was looking at recently: how to stream a multipart upload to a remote HTTP server.

My specific use case is to receive a stream from AWS S3, and pipe it into the upload API for Asana. It's probably ending up back in S3, but that's neither here nor there.

The multipart writer provided with Go's standard library makes serving multipart responses really easy, but sending an HTTP request needs an io.Reader to provide a streaming request body, which doesn't fit nicely with the io.Writer based support.

I found a couple of answers solving this problem, either by manually encoding the multipart headers or by running a separate goroutine to write into a pipe which can then be read by the HTTP client.

I settled on something in between: writing the headers to a byte buffer, then slicing the buffer and composing an io.MultiReader from the parts and the body reader from the S3 getObject response.

// Error checking omitted
output, _ := r.s3.GetObject(&s3.GetObjectInput{
    Bucket: aws.String(bucket),
    Key:    aws.String(key),
})

var contentType string
if output.ContentType != nil {
    contentType = *output.ContentType
} else {
    contentType = "application/octet-stream"
}

// Write header
buffer := &bytes.Buffer{}
partWriter := multipart.NewWriter(buffer)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
    fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
        escapeQuotes(field), escapeQuotes(filename)))
h.Set("Content-Type", contentType)

partWriter.CreatePart(h)
headerSize := buffer.Len()

// Write footer
partWriter.Close()

// Create request
request, _ := http.NewRequest(http.MethodPost, c.getURL(path), io.MultiReader(
    bytes.NewReader(buffer.Bytes()[:headerSize]),
    r,
    bytes.NewReader(buffer.Bytes()[headerSize:])))

request.Header.Add("Content-Type", partWriter.FormDataContentType())
resp, _ := httpClient.Do(request)

This worked well, and avoids needing to read the whole object into memory or caching it to disk. A cleaner solution might be to implement a reader-based equivalent in the multipart package.

Comments

Generic Adapters in Go
Sun, 13 Aug 2017 / blog / go / generics

While playing about with ideas for generic types in Go, I have come up with what I'm calling a 'Generic Adapter'.

The approach I've taken here is essentialy that described by Kevin Gillette in a 2013 Post to golang-nuts. It implements 'erasure-style' generics by wrapping the generic implementation of each method with an adapter function which handles type assertions for the caller.

type Container struct {
    T SomeType

    Set func(value SomeType) SomeType
}

type Int64Container struct {
    T int64

    Set func(value int64) int64
}

func myProgram() {
    c := &Int64Container{}
    generic.Factory(Container{}).Create(c)
}

Source code implementing a proof of concept can be found on Bitbucket. I can't really say I recommend anyone using it, but it's satisfying to show that it works!

Comments | Read more...

Backblaze B2 Performance Part 2
Sun, 13 Dec 2015 / projects / go / b2

After my earlier experiments with B2, I had an extremely interesting call with Backblaze about B2 features and performance.

Firstly, they have recently added a caching layer to speed up serving repeatedly requested files. This reduces the delay as the file is reassembled from Reed-Solomon slices. They also suggested that I do some new tests, as they thought I should be seeing faster speeds, even for first-access.

Comments | Read more...

Backblaze B2 and Go
Mon, 30 Nov 2015 / projects / go / b2

I've been implementing a Go client library for the Backblaze B2 cloud storage service: go-backblaze. It works extremely well, but is a bit slow to get files back.

Update 12th December: I've spoken to Backblaze, who have been working to improve performance. I have performed some new tests and written them up.

I've implemented a parallel download feature which you can use to download several files at the same time - this doesn't seem to affect the speed of individual downlaods very much, so I assume that the downloads are limited for one of two reasons

  1. There is a download speed cap in place
  2. The downloads each come from separate parts of the cluster file system, and so don't affect each other.

Downloading 5 copies of the same file in parallel doesn't seem to affect the download speed, nor do 5 sequential downloads. Whatever you do, each download seems to run at about 200KiB/sec.

Comments | Read more...