Streaming multipart HTTP requests in Go

Having seen Peter Bourgon’s post on multipart http responses come up in the Golang Weekly newsletter, I thought I would write some notes on a related problem I was looking at recently: how to stream a multipart upload to a remote HTTP server.

My specific use case is to receive a stream from AWS S3, and pipe it into the upload API for Asana. It’s probably ending up back in S3, but that’s neither here nor there.

The multipart writer provided with Go’s standard library makes serving multipart responses really easy, but sending an HTTP request needs an io.Reader to provide a streaming request body, which doesn’t fit nicely with the io.Writer based support.

I found a couple of answers solving this problem, either by manually encoding the multipart headers or by running a separate goroutine to write into a pipe which can then be read by the HTTP client.

I settled on something in between: writing the headers to a byte buffer, then slicing the buffer and composing an io.MultiReader from the parts and the body reader from the S3 getObject response.

---

// Error checking omitted
output, _ := r.s3.GetObject(&s3.GetObjectInput{
	Bucket: aws.String(bucket),
	Key:    aws.String(key),
})

var contentType string
if output.ContentType != nil {
	contentType = *output.ContentType
} else {
	contentType = "application/octet-stream"
}

// Write header
buffer := &bytes.Buffer{}
partWriter := multipart.NewWriter(buffer)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
	fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
		escapeQuotes(field), escapeQuotes(filename)))
h.Set("Content-Type", contentType)

// Writes part header to buffer, then
// we capture the size for later
partWriter.CreatePart(h)
headerSize := buffer.Len()

// Write footer (without body)
partWriter.Close()

// Create request
request, _ := http.NewRequest(http.MethodPost, c.getURL(path), io.MultiReader(
	bytes.NewReader(buffer.Bytes()[:headerSize]),
	r,
	bytes.NewReader(buffer.Bytes()[headerSize:])))

request.Header.Add("Content-Type", partWriter.FormDataContentType())
resp, _ := httpClient.Do(request)

This worked well, and avoids needing to read the whole object into memory or caching it to disk. A cleaner solution might be to implement a reader-based equivalent in the multipart package.