David's Blog

After a very long time (with much starting and stopping of work) I finally got media uploads working on my micropub server!

In celebration, here's a picture of the bunny that hangs out in my yard:

With that out of the way, I thought I'd describe a bit about how this works and some of the challenges I hit along the way (including in writing this post!).

First, I had to implement the media upload portion of the micropub spec. This wasn't too bad but I needed to be able to store a few things:

• The media file itself needed to be stored somewhere.
• I had to then store some metadata about the file so I could later fetch the file data.

This was broken out into two parts. For storing file data itself, I created a separate server from my micropub server. I called this rustyblobjectstore (terrible name but whatever). It's a thin blob store API meant to abstract over any particular file storage details. The commit I linked at the start of the post goes into more detail but the idea is that I can easily change file storage backends through this service without the micropub server needing to know anything about that.

For now, as an experiment in how this works, I'm just using a sqlite database to store the file data.

For metadata about the media, I created an additional table in the micropub server's database. This is used to store content type, a content hash, as well as created/modified dates. The content hash is used in the media URLs on the micropub server as well as the key for the object store API so it's kind of content-addressed storage (kind of).

Implementing all of that was fairly straightforward and I had it working months ago. One of the first things I hit, though, once I had that all working was that the server library I was using sent headers as lower case ("location") whereas the client I was using to test uploads checked only for camel case header names ("Location"). This meant that the client I used to test never saw that the image was uploaded properly. It additionally had the fallback behavior of encoding the image data as base64 and including it in the body content of the post if the upload "failed" so this made observing this failure pretty hard without actually doing a publish and ensuring the body of the new post looked right (didn't contain a giant based64 encoded image blob).

I tried working around this a few ways, specifying specifically an uppercase header name in application logic (didn't matter) as well as trying to rewrite the header with Nginx config (also didn't work although I never figured out why on this one). I also found an open pull request on the client to fix this behavior but that did not get merged during this time.

I considered writing/configuring a totally separate proxy server with the sole purpose of doing that but that seemed silly and toilsome. So that's where this implementation sat for months. At some point I became aware that the underlying libraries for the web "framework" I use now supported the camel case header output but I still let things sit. I didn't see an easy way to wire that into my server without making a modification to the framework.

Finally I realized I could configure the server separately from the handlers and managed to enable the config option I needed.

I was then finally able to test the image upload end to end and see it working!

Some things I realized when testing though:

• I can post an image through the "editor" but not by posting a "note" through the same client. The "note" sends a "photo" property in the micropub post body rather than including an image tag in the content of the note and so my server (currently) does not display the image despite the upload happening properly.
  
  • There was a related issue where uploading multiple images in the same "note" actually won't post the note at all due to decoding issues of that photo property.
• Finally, when writing this post and adding the bunny picture above at first it seemingly wasn't uploading properly (I was worried it was falling back to that base64 blob behavior). This seems to be due to differing behavior of the upload image file size? Uploading a smaller version of the same photo seems to have worked just fine... There is more debugging to do here because that's not a great limitation.

Anyway, this allowed me to merge a long-running branch and desired feature into my micropub server. Maybe I'll post more images now. Maybe it'll just unblock me from implementing more desired stuff on the server itself since I won't have to deal with maintaining that long running feature branch.