Kothar Labs

Having seen Peter Bourgon's post on multipart http responses come up in the Golang Weekly newsletter, I thought I would write some notes on a related problem I was looking at recently: how to stream a multipart upload to a remote HTTP server.

My specific use case is to receive a stream from AWS S3, and pipe it into the upload API for Asana. It's probably ending up back in S3, but that's neither here nor there.

The multipart writer provided with Go's standard library makes serving multipart responses really easy, but sending an HTTP request needs an io.Reader to provide a streaming request body, which doesn't fit nicely with the io.Writer based support.

I found a couple of answers solving this problem, either by manually encoding the multipart headers or by running a separate goroutine to write into a pipe which can then be read by the HTTP client.

I settled on something in between: writing the headers to a byte buffer, then slicing the buffer and composing an io.MultiReader from the parts and the body reader from the S3 getObject response.

// Error checking omitted
output, _ := r.s3.GetObject(&s3.GetObjectInput{
    Bucket: aws.String(bucket),
    Key:    aws.String(key),
})

var contentType string
if output.ContentType != nil {
    contentType = *output.ContentType
} else {
    contentType = "application/octet-stream"
}

// Write header
buffer := &bytes.Buffer{}
partWriter := multipart.NewWriter(buffer)
h := make(textproto.MIMEHeader)
h.Set("Content-Disposition",
    fmt.Sprintf(`form-data; name="%s"; filename="%s"`,
        escapeQuotes(field), escapeQuotes(filename)))
h.Set("Content-Type", contentType)

partWriter.CreatePart(h)
headerSize := buffer.Len()

// Write footer
partWriter.Close()

// Create request
request, _ := http.NewRequest(http.MethodPost, c.getURL(path), io.MultiReader(
    bytes.NewReader(buffer.Bytes()[:headerSize]),
    r,
    bytes.NewReader(buffer.Bytes()[headerSize:])))

request.Header.Add("Content-Type", partWriter.FormDataContentType())
resp, _ := httpClient.Do(request)

This worked well, and avoids needing to read the whole object into memory or caching it to disk. A cleaner solution might be to implement a reader-based equivalent in the multipart package.

While playing about with ideas for generic types in Go, I have come up with what I'm calling a 'Generic Adapter'.

The approach I've taken here is essentialy that described by Kevin Gillette in a 2013 Post to golang-nuts. It implements 'erasure-style' generics by wrapping the generic implementation of each method with an adapter function which handles type assertions for the caller.

type Container struct {
    T SomeType

    Set func(value SomeType) SomeType
}

type Int64Container struct {
    T int64

    Set func(value int64) int64
}

func myProgram() {
    c := &Int64Container{}
    generic.Factory(Container{}).Create(c)
}

Source code implementing a proof of concept can be found on Bitbucket. I can't really say I recommend anyone using it, but it's satisfying to show that it works!

I've been trying out a number of file sync clients recently, finding that many are trying to solve the same problem and meeting with varying success. Probably the best-known service in this category is Dropbox, but nearly every big cloud services company, and a multitude of startups which base their services on top of them, are providing a 'cloud drive' of some description.

I've been experimenting with the Plex Media Server, and decided to have a go at securing remote access to the 'Web Client' interface using SSL (by default it runs over an unencrypted HTTP connection).

This post is essentially a gathering of instructions from where I found them on the internet.

I'm a co-author on a paper which was presented at RTNS this year. Sadly I didn't have time to attend the conference, but it's nice to have been able to contribute!

Re-Sampling for Statistical Timing Analysis of Real-Time Systems

I needed to use Disqus from a GWT app, so I needed to solve the problem of loading threads on demand as virtual pages in the app were changed.

A small amount of reverse engineering and experimentation led me to construct a utility class (below).

I also posted this answer on Stack Overflow.

Now that Spotify has discontinued the free account, and restricted me to an 'open' account with only 10 hours of music a month, I've ended up paying for an 'unlimited' account.

While I'm a little disappointed with this turn of events, it does mean I am now able to use the Linux client. It seems pretty decent so far, but I wanted to map some keys to do things like play/pause and next.

I was doing some reading a while back into lock-free algorithms for multi-threaded communication using buffers. The general idea is to use a circular buffer in shared memory to allow one process (or thread) to write data into a buffer so that another process can read it. The algorithm relies on atomic compare-and-swap instructions, which are part of the x86 instruction set and also present in several other architectures which have features designed to assist with concurrency.

I use Java a lot, and the common practice of naming java packages with long URI-derived namespaces often leads to my bash prompt becoming extremely long. Here's a script to display an abbreviated path on the bash prompt.

I've been experimenting with PolyORB at work, and the documentation, while fine for configuring and compiling your application, is sorely lacking on any help with implementing your CORBA servants.

Here I'll attempt to provide some brief hints on how to do things that don't seem to be described in the documentation. The PolyORB mailing list is a good place to find more information.