tokio::task::spawn_blocking
) doesn't work very well.wrk2
and the web server were running on the same 6-core (12 hyper thread) machine. Without these two thread offloading changes, my server handled 1000 rps with a mean latency of 3ms, p90 of 4ms, and max of 6ms. At 5000 rps it had a mean latency of 8ms, a p90 of 22ms, and max of 58ms. However, once my two commits were in to move my db calls + markdown-to-html logic off of the Tokio runtime, the server was starting to fall apart at 1000 rps, sometimes not even completing that workload.tokio::task::spawn
at places I want concurrency. Seems a little silly because I'd have to wrap the whole thing in an async block to make a future. It'd potentially help for the two queries I can do concurrently. On the whole, I'm not actually sure if the overhead of shifting these sqlite queries over threads is more than just running the queries synchronously in line but that was why I was experimenting a bit. Another approach might be to keep using spawn_blocking
but introduce some concurrency control + load shedding on the calls before scheduling the blocking tasks. I ended up trying out the rayon approach.rayon
for a CPU-bound thread pool + a tokio::sync::oneshot
to communicate the result back into a Tokio task. Before gluing together these bits, I figured I'd see if someone had already done something similar and found tokio-rayon. It provides the convenience function I needed to submit tasks to the rayon global executor (by default, sized at the number of logical cores in your system). It actually has both LIFO and FIFO versions of spawn. The downside was, this wasn't sufficient for solving my server going non-responsive under load when offloading these particular calls to other threads.Error(None)
coming from within my application. For whatever reason the logs weren't providing a module or file here (or I missed it). I had already been experimenting with moving some stuff over from the log
crate to the tracing
crate and log messages can be converted to tracing Events just by changing the library I used, no other code changes required. This pointed me at the module and function at least and this is when I started suspecting my database connection pool. A few edits to the log message I suspected confirmed that suspicion.diesel::r2d2
). The default connection pool size was 10 and the default timeout for acquiring a connection was 30s. This meant if the connection pool was overwhelmed by connection requests, a bunch of those would stack up and all fail after 30s. This partly explains why my changes made the server go unresponsive! I was requesting potentially more connections than before my code changes.I haven’t done one of these in a few years but feel like this one warrants it.
Unsurprisingly, 2020-2022 were pretty quiet years for my running. 2018 was a strong year for me but unfortunately ended in injury. 2019 was a year of recovery and rebuilding, and then 2020-2022 turned into much the same. I was lucky to be healthy throughout that time and while I wasn’t training for very many races, I was running pretty consistently. Not a lot, but consistently. A silver lining was that it helped me rebuild habit and consistently get out there even during a time of low motivation. Sure, I put in a decent training block for CIM in 2021, but otherwise I was mostly “just running” without much in the way of big weekly mileage or even too many workouts. 2023 was supposed to be different, I planned a return bigger training blocks and goal races with PRs in mind. In many ways it was a big year and I have a lot to be thankful for that motivates me for the coming year.
Unfortunately, I didn’t run CIM after all of that work. In a trip to SF in early Nov I injured my knee. The day after running my 10 mile PR I flew to SF and then in the evening I stubbornly pedaled an e-bike with a dead battery to my hotel while carrying my luggage. This combination of things proved to be too much for my right knee and I had a lot of pain in it for a few weeks. After a few weeks of relative rest I had to accept that CIM wasn’t happening (and also that I wouldn’t make a quick enough recovery for any other early winter races).
I’m ending the year having run just over 2200 miles (2206.7 as of this writing, which will be the total unless I decide to risk running on fresh snow with my knee that is still not totally healed). This is far and away the most I’ve run in a year since at least 2013. I’ve had recent years in the 1700 or 1800 miles range but haven’t crossed 2000 miles for the year since at least 2012 or 2013.
I’m currently able to do ~3 mile runs more or less pain free but I still don’t have a full idea of what a recovery timeline looks like in order to begin training again so it’s hard to set goals for next year. During marathon training I was thinking that next year would be good to get back to shorter races (5k to 10k) and work on speed as that was a particularly weak point in my training this past cycle (strides and even 10k pace felt *hard*). But I also now feel like I have unfinished business with the marathon given that I had essentially my best ever marathon training cycle but didn’t get the chance to fully express that fitness. I’m also coming away with the lesson that even though I don’t feel like I over trained (not ending my cycle with an over use injury from running is a big win for me), I still need to be cautious with other things I’m doing when I’m putting my body under lots of stress for big training cycles. I’m sure that cumulative fatigue contributed to my injury during that fateful bike ride.
So, I’m thankful for a 2023 that showed me I can still put in big mileage and recover after many many years of being unable to put together this type of training. I’m also thankful for a 10 mile PR.
In 2024 my priorities are:
1. Recover. Be patient and get back to consistent 5-6 days a week of running, no matter how conservative I have to be to get there. It’s not worth rushing it and ending up in another injury cycle.
2. Be healthy for GLR. It’s headed back to the UP next year and a brand new day 3. If I can’t be healthy enough by July, something very serious is wrong.
3. Re-evaluate other race goals as health and fitness allow but I’m still holding out hope to be able to put together a strong Fall 2024 training cycle of some kind.
While I started or made progress on reading many books in 2022, unfortunately this year was thin in books that I completed. Still, there were a few that stood out for me. I'm going to focus on just a few fiction books that I enjoyed this year.
This was the conclusion to her A Chorus Of Dragons series. I'd mentioned another book in this series in my 2019 year in books and haven't written another similar post since. I thoroughly enjoyed the entire series. The characters were interesting and the world building was unique. It had quite a broad set of places and people and, if anything, was slightly too large for me to keep in my head in the span between book releases so I imagine it would be easier to follow along with now that all of the books have been written and released.
The other day when I was at the book store, I stumbled on another book by Lyons, The Raven Tower. It was an easy purchase even though I don't know what it's about!
The Poppy War is a story of generations of recurring conflict between two countries with hints of the supernatural early on until really expanding on that aspect of this world in the mid and late parts of the book. Overall I enjoyed the story and mythology that were built up in this book. Since it was a book with war, I was prepared for it to be violent but it was quite graphic and gruesome at points and didn't know that going in. Knowing that now, I still would read it but it's probably worth knowing more than I did before starting. I'd classify it as even more violent and brutal than Game of Thrones was (both the books and TV show).
This is, I think, the 2nd to last book in the Cradle series. I think the last few books have really hit their stride and I'm looking forward to the next one. Both this series and the next are easy reading (I get through the books fast, too fast) but also very satisfying. Just the kind of fun fantasy book I like to read for entertainment, good pace and doesn't feel like it's high effort to consume.
I'm somewhat sad the series is coming to an end next year but I hear that there's other series by this author - perhaps I'll check those out.
This, series has a mix of fantasy book + video game magic system feel that is pretty interesting. I know that is what the author is going for and it's not a style I'd encountered before his books. They're quite fun and I like what is happening with multiple series (another being The War of Broken Mirrors) occurring in the same world, on differing timelines and with different writing styles.
Testing media uploads on the rewritten site with an upload of the cutest captcha ever
I've been doing the first few days of this year's advent of code with awk (GNU awk to be precise). The first day I learned some new things about arrays with awk.
The first was that you can set a global variable called PROCINFO["sorted_in"]
to determine the order that arrays (which are really associative arrays not "vectors", I believe) are iterated over for for-loops. There's a bunch of options for sorting array values by string or numerically or by type and the options have ascending and descending variants. One caveat is that since this is controlled by a global variable, you will change the default for later loops unless you preserve the current value and then restore it. The docs are very good.
The second piece was that the array sorting functions (asort
and asorti
) can also be provided with the same options to adjust sort ordering (e.g. to treat array values as numbers instead of strings).
That second bit was what I needed for my AOC solution on day 1 but both of these are good to know.
My site now supports stripping Exif tags (and others) on media when uploaded via the Micropub media endpoint I've implemented. This was important to me to ensure that, when possible, identifying information like location is removed from any images I upload. This is doable before uploading the images but integrating it into the web endpoint makes it much more likely that it'll happen (especially if I ever get the micro pub implementation up to the point where something like a mobile upload can happen).
The code itself to do so within the upload handler was pretty straightforward so the more interesting bit is how I got to the point of being able to write that code and also packaging it up once it compiled.
I did a fair amount of looking around for solutions to this problem. Since my server is implemented in Rust, I was looking for Rust bindings to strip tags as reliably as possible (and didn't want to have to go down the rabbit hole of implementing it myself, for now).
There were some pure-Rust image crates but none seemed to have support for what I wanted to do. There was also `magick-rust` which provides Rust bindings for the ImageMagick MagickWand library. I would have preferred bindings to other portions of the ImageMagick API as the MagickWand stripping bindings seem to do a bit more other stuff than I really wanted but I still wasn't quite willing to build bindings to the rest of the API. The goal was stripping tags without implementing a whole lot of stuff in Rust in order to do so.
One problem was that the strip functionality provided in this API wasn't yet surfaced in the `magick-rust` crate. This seemed like something I would be able to add without going down a huge rabbit hole so I proceeded with this approach (the alternative at this point, given no other Rust options, was to shell out to ImageMagick itself so this was preferable). It also gave me a chance to learn about `rust-bindgen` as that is how this crate was built. A small PR later, the crate would support stripping tags! Excitingly, that PR was quickly accepted and merged.
Along the way, I also learned that default `rust-bindgen` cargo features involve building `clap`, a CLI parsing crate that's somewhat large (but also quite handy). This was surprising to me and stood out as my server doesn't currently have that crate as a dependency, so I dug in there. I learned that the default build of rust-bindgen also builds it as a CLI which requires that crate. I haven't made a PR out of this change yet but it seemed fairly easy to turn off that extra functionality, so I did so in a branch.
The other issue I ran into with this "small" bit of functionality was that I needed to be able to have the proper version of ImageMagick available at runtime so that my server could actually do the stripping (this dependency is not statically linked). This lead to a host problems.
First, the apt-based repositories the existing image build had access to were still on a 6.x version of ImageMagick. The `magick-rust` crate requires a build in the 7.0.x range or later. At first, I tried to work around this by building ImageMagick from source during the image build. This didn't seem too bad initially, I didn't have a whole lot of trouble getting the source downloaded, built, and installed within my Dockerfile. The trouble was, even once that was happening, the server was erroring when the image tag stripping code was called.
Looking into this some more, it turned out that a bunch of "optional" ImageMagick dependencies were not available (and had been mentioned as such during the `configure` script) and this meant that it didn't know how to read a bunch of common image formats. I had a really rough time trying to make all those deps available to the build. I tried to list them all out in an `apt-get` command, did things like installing the 6.x version to make sure its deps were available, and more, all to no avail.
Separately, I've been experimenting with Nix package manager (and NixOS) recently and had no problems building and running the server from within a `nix-shell` environment. Nix had a new enough ImageMagick and making the symbols available to the server at runtime wasn't really much of a problem. I knew that there was a way to build Docker images with nix and so, failing all of my more traditional attempts at packaging my server into a container image with its new ImageMagick dependency, I decided to look at packaging the server in a Docker image with nix.
This first involved being able to build my micropub server via `nix-build`. I'd written some other package derivations at this point so it seemed feasible but I hadn't done any Rust packages yet. It turned out not to be too bad, I ended up primarily following the `naersk` readme (and transitively, `niv`'s too). That paired with some additional suggestions from the NixOS wiki and Xe's blog (which has also been quite helpful with Nix in general) got me going. Once I had the project being built via `nix-build`, adding the Docker bits was relatively straightforward! (I leaned on essentially the same resources there, with the addition of the official manual).
Once all that was said and done, I had a minimal Docker image that could run my site and (finally) strip Exif tags from images that I upload!
I have more thoughts about Nix package management that I'm working to write up. It appeals to me in quite a few ways but some of its requirements also make me question whether I want to stick with it. In a lot of ways, the problems are endemic to package management or configuration management tools in general and so I might just have to decide to live with this one given all the benefits it brings. More on that soon, I hope.
Hennepin Ave is a major street here in Minneapolis and due for reconstruction. The city has been through multiple rounds of design and public comment and has arrived at a single proposed design that is currently open for public comment and is due to be up for review by the city council in the next few months.
Tonight I submitted a public comment on the design and figured I'd share it here as well. Below is an unedited form of that public comment, with a few hyperlinks added. It could be edited better but my emphasis was on getting the comment in to the city over making the comment perfect. For those who want more information about the specifics of what is changing, all of the project materials are available here.
--------------------------------------------------------------------------------------------------------------------
I strongly support the 24/7 bus lanes as well as the addition of a raised bike lane in the project area. There is good evidence and policy reason to move forward with these elements. First, a large percentage of people using Hennepin are on buses at rush hour (upwards of 40 to 50%) at all hours of the day. It is respectful of those people's time to prioritize buses that move more people with less space over individuals in personal vehicles. Second, the city's Transportation Action Plan states that Hennepin is designated to be part of the all ages and abilities bicycle network by 2030. This reconstruction is the time to follow that policy and add protected bike lanes. Finally, I really like that the north side the project will connect up with the Bryant Ave bike bridge to connect with the rest of the city.
Improved pedestrian facilities are also very important for Hennepin Ave. Crossing 4+ lanes of traffic can be dangerous when drivers are not paying attention and the traffic noise makes it unpleasant to walk down Hennepin. Strategies employed in the proposed plan including shifting some of those lanes to other uses (like buses) and decreasing the distance to cross Hennepin should improve that situation for pedestrians and overall lower injury rates on the street. Per the crash analysis done, pedestrian and bicycle crashes are at a especially high rate compared with the rest of the city. As someone who frequently traverses Hennepin on foot or by Bike, I am very concerned about improving this situation.
On parking, personally I am not concerned by the minor removal of parking along Hennepin. I usually walk and/or bike to business on Hennepin. However, in the cases I do feel the need to drive I can say that, as someone who lives within a few blocks of Hennepin within the Wedge, parking on Hennepin not something I would ever want to do even if a spot were available. This is due to the narrow lanes on Hennepin paired with the speeds that drivers employ along the street - it just does not feel like a safe or convenient place to park. I can also say that my personal experience matches the parking study in that there is often plenty of parking available within the a block or two on each side of Hennepin, so removal of a few parking spaces should not significantly affect accessibility to businesses for those that choose to drive.
One aspect of the plan that I do not like, and question whether it will hinder the design's effectiveness towards meeting city goals, is in maintaining the two left turn lanes onto northbound Hennepin from Lake Street. The dedicated bus lane does not start until two blocks north of that intersection, at the Uptown Metro Transit station. It seems to me that congestion at the Lake and Hennepin intersection will hinder the goals of the city and Metro Transit of providing reliable transit capabilities with BRT . Since there is no dedicated bus lane this intersection could delay service and make it more unreliable, which seems counter to the stated goals.
I understand the intent is to leave the two lanes due to volume of traffic making that turn, however it seems to me that reducing driving lanes just two blocks after the turn will still encounter the same congestion problems meant to be avoided by leaving the two left turn lanes, just on Hennepin instead of Lake St. I'd like to see more information on why planners believe this part of the proposed design will not hinder buses given the priority to be given to them. Has a traffic study been done on this option vs the reducing the left turn to one lane? It seems like the current design would result in more congestion and/or personal vehicles driving in the bus lane due to last minute merging than reducing the number of turn lanes from Lake on to Hennepin, thereby adding back pressure in high traffic times.
Overall, I think that this plan is a well balanced one meeting the city's adopted policies and many of the city's stated goals. It seems to have taken into account much community feedback on all aspects and I am excited for a more multi-modal Hennepin that is safer and more pleasant! The Metropolitan Council's interactive maps of Census data show that the reconstruction area touches or is near the 3 neighborhoods of highest population density in the city. These areas deserve a safe, multi-modal, enjoyable corridor for every day activities and this proposed design does a lot to get us to that goal.
After a very long time (with much starting and stopping of work) I finally got media uploads working on my micropub server!
In celebration, here's a picture of the bunny that hangs out in my yard:
With that out of the way, I thought I'd describe a bit about how this works and some of the challenges I hit along the way (including in writing this post!).
First, I had to implement the media upload portion of the micropub spec. This wasn't too bad but I needed to be able to store a few things:
This was broken out into two parts. For storing file data itself, I created a separate server from my micropub server. I called this rustyblobjectstore (terrible name but whatever). It's a thin blob store API meant to abstract over any particular file storage details. The commit I linked at the start of the post goes into more detail but the idea is that I can easily change file storage backends through this service without the micropub server needing to know anything about that.
For now, as an experiment in how this works, I'm just using a sqlite database to store the file data.
For metadata about the media, I created an additional table in the micropub server's database. This is used to store content type, a content hash, as well as created/modified dates. The content hash is used in the media URLs on the micropub server as well as the key for the object store API so it's kind of content-addressed storage (kind of).
Implementing all of that was fairly straightforward and I had it working months ago. One of the first things I hit, though, once I had that all working was that the server library I was using sent headers as lower case ("location") whereas the client I was using to test uploads checked only for camel case header names ("Location"). This meant that the client I used to test never saw that the image was uploaded properly. It additionally had the fallback behavior of encoding the image data as base64 and including it in the body content of the post if the upload "failed" so this made observing this failure pretty hard without actually doing a publish and ensuring the body of the new post looked right (didn't contain a giant based64 encoded image blob).
I tried working around this a few ways, specifying specifically an uppercase header name in application logic (didn't matter) as well as trying to rewrite the header with Nginx config (also didn't work although I never figured out why on this one). I also found an open pull request on the client to fix this behavior but that did not get merged during this time.
I considered writing/configuring a totally separate proxy server with the sole purpose of doing that but that seemed silly and toilsome. So that's where this implementation sat for months. At some point I became aware that the underlying libraries for the web "framework" I use now supported the camel case header output but I still let things sit. I didn't see an easy way to wire that into my server without making a modification to the framework.
Finally I realized I could configure the server separately from the handlers and managed to enable the config option I needed.
I was then finally able to test the image upload end to end and see it working!
Some things I realized when testing though:
Anyway, this allowed me to merge a long-running branch and desired feature into my micropub server. Maybe I'll post more images now. Maybe it'll just unblock me from implementing more desired stuff on the server itself since I won't have to deal with maintaining that long running feature branch.
Today I learned about a builtin macro in Rust that does this without adding additional lines or requiring formatting a string with the contents of some variable(s): The dbg macro will accept any expression, print it, and also return the result:
Prints and returns the value of a given expression for quick and dirty debugging.
let a = 2; let b = dbg!(a * 2) + 1; // ^-- prints: [src/main.rs:2] a * 2 = 4 assert_eq!(b, 5);
If you're writing a Rails app and use ActiveRecord, this is a potential pitfall to be aware of:
Let's say you have some model, MyModel, that you want to do a bulk lookup of based on a set of IDs that come from elsewhere. You'd do something like:
models = MyModel.where(id: some_array_of_ids)
This will work just fine. However, there is a bit of a lurking performance cliff with this approach. It turns out that, surprisingly, ActiveRecord doesn't de-duplicate the array of IDs before constructing the SQL query text. That is, if you happen to have an array with 5 IDs but they're all the same ID then you'll end up with an underlying query like:
SELECT * FROM my_models WHERE id IN (7, 7, 7, 7, 7);
On an array of just 5 elements this isn't too much of a big deal and doesn't hurt correctness in any way. You'll still get back a either 0 or 1 results depending on if ID 7 exists in your table. However, once you're potentially looking up tens, hundreds, or thousands of IDs at once then you might hit some larger performance problems.
At least on the database I'm most familiar with running in production, MySQL, it seemingly doesn't optimize this query in the way you'd expect and it could take much longer than just looking up the single ID of 7 (or if your array is much larger and you have multiple IDs in it, all duplicated, this can get quite bad). Even if your database does optimize this query, the query you're sending over the wire is still larger than it should be.
So the safer, more performant way to query for a list of IDs if you don't already know that they are unique is to first ensure they are before your query:
models = MyModel.where(id: some_array_of_ids.uniq)
The other week, a new teammate asked about my approach to resolving bugs. This was in the context of receiving a bug report (e.g. in the form of a ticket, task, etc) and so it might be slightly different than debugging in a broader sense. Still, I realized I had never formally written out (or discussed) debugging steps before and so thought it might be worth writing down more broadly. Writing it down made me think about and articulate what exactly I would do where before I might have had a mental model or internally intuitive process to follow but not in a well-defined way.
This might seem like it's very straightforward but in thinking about it more, I think it's a form of tacit knowledge in that you build it up over time but don't often read about it, discuss it, or otherwise think about it.
I listed out some steps from receiving the bug report through to resolution:
1. Read the bug report and understand the incorrect behavior as well as the expected or desired behavior. Check whether there were steps provided for reproducing the bug.
2. Try to reproduce the bug on your own. This could be in your local development environment (ideally) or in a staging/production environment if need be. If there were reproduction steps included with the bug, use those steps. Otherwise, try to recreate the conditions for the bug on your own. If you can't reproduce (or can't do so reliably) it then debugging and fixing it will be difficult or impossible.
3. Once you can reproduce it, you know the bug still exists. The next step is to start determining what code relates to the behavior you were just seeing and to read the code. If it's a small codebase or one you know well, finding and reviewing the code might be straightforward. If it's a particularly large project then there may be some searching required (either locally with tools like git-grep, ripgrep, editor search tools, or web-based tools like Github code search and livegrep).
4. Once you have a rough idea of the code related to the bug, the next step is trying to understand what behavior is triggering the bug. This could involve reading the code to understand some logic, figuring out why incorrect data is being passed in or produced, writing a test case to trigger the bug in code automatically, or just tweaking code and running it to experiment and see what happens (perhaps while looking at or adding logging or other diagnostics). This process could be quite involved depending on the complexity of the bug!
5. If you've found where the bug is in code and understand it, now you can try to implement a fix! Sometimes this is something like fixing off-by-one logic errors or similar minor mistakes but other times it can be much more involved. Verify that your changes fix the bug by running through the reproduction steps and seeing the bug is now gone. Further, and where possible/reasonable, implement a test case to automatically verify the bug is fixed and to prevent regressions. I'd be remiss if I didn't also mention that you should spend time trying to understand why the code was written in the way that it was in the first place. It could have been an intentional decision and understanding if that decision still makes sense is part of (attempting to) resolve the bug. Tools like viewing commit history or old tasks/tickets can often help with determining this context if you are not sure of it.
6. Create one or more commits with your bug fix (and ideally your new test case(s) as well). If the project you are working on has a code review process, follow the workflow for creating a new code review.
If you can get through all of these steps, chances are you've fixed the bug! Following these steps isn't always as straightforward as this short description might sound. Reproducing a bug, finding the relevant code, and figuring out why that code is behaving the way that it does are all worthy of longer posts of their own. Lately Julia Evans has been talking about debugging a lot on Twitter lately, sharing both some really good tips for debugging as well as describing why it can be a difficult process. Here's a blog post about that that I thought was very good.
Even once you've done some debugging and understand a bug, you might not easily be able to fix the bug! Sometimes there will be extensive refactoring needed to address the problem in a clean way or there could be architectural limitations that make fully addressing the bug difficult. That won't always be the case though! I've found that debugging is a great way to learn about a system, likely because it forces you to read lots of code!
The other weekend I decided to try out Nelson Elhage's llama project which is a tool for offloading computation to AWS Lambda. This post is going to be a collection of my thoughts about the project and its potential.
A while back I had read a couple of the blog posts / newsletter entries that described llama and was intrigued by the tool but at the time the focus seemed to have been distributed compilation of C++ projects. I'm not doing so much C++ any more and so didn't have a strong use for that at the moment and set it aside but the idea stuck in the back of my mind. At some point this came back to mean and I started wondering if there was something like GNU Parallel for distributed computation but on AWS Lambda where the machines didn't have to be waiting for you 24/7? What I'd not remembered was that llama is a general tool that can do just that!
A hobby project of mine lately has involved fetching a number of CSV files daily and doing some light processing on them (sort of git scraping). This was a project tracking the data my state publishes around COVID-19 vaccination rates across multiple dimensions. While it didn't involve a ton of CPU bound work, fetching CSV files in parallel and doing the small amount of processing I do seemed like a great way to poke at llama once I realized it had the ability to distribute arbitrary work.
With something like GNU Parallel you need remote machines sitting around ready to do compute though. With lambda, it's on demand and can scale to very high parallelism for most purposes. Having such a tool could facilitate every day compute tasks that are expensive (slow locally on an older laptop, or drain battery life and create heat, or some combination of those) being done remotely. Assuming a fast enough internet connection, that is. Of course, things that take less time to compute locally than uploading and downloading the input/output to and from S3 might make less sense to do remotely. Overall, I'm excited about this concept - it provides a way to scale compute (or other things, like fast network access) without being tied to a powerful machine. After spending time with it, I think llama is a good early implementation of this idea.
I'm going to share some notes now about getting set up and my initial attempts to use llama. Overall, the README is sufficient to get started.
I did run into a few issues building/installing the command line tool on my Mac machine (tried both Go 1.15.x and 1.16.x)
commands.
Setting up the AWS Credentials and CloudFormation stack was very seamless once I had the tool built and installed!
seemed like a nice proof of concept to start learning the tool - if every file is fetched by a different lambda host, then I shouldn't hit these problems.
call and had the mapping file listed as an input file. Then, in the llama invocation, I'd use the mapping file to look up the URL I needed to curl while using the input line as the output file
was a bit easier to use for one-off single file use cases as it has the local:remote input/output file mapping.
in mind and try to integrate it into workflows where it makes sense to enable offloading of computation. Llama seems like a good tool to have in the toolbox for both regularly run jobs that can be broken down as well as one-off shell pipelines that might take a bit more processing power.
Use 'git show <rev>:<path>' to output a single file's contents from a particular revision. This can be redirected into a file or piped into other programs, etc.
Something I have occasionally wondered is whether git provides a way to read a single file from a particular commit/revision without checking out that commit (and thus changing the state of the whole working copy). It seemed like something git should support but in brief searches and man page readings, never found a way to do so.
Today, I finally read through the gitrevisions(7) manpage (referenced by git-show's) and saw the <rev>:<path> syntax:
<rev>:<path>, e.g. HEAD:README, master:./README
A suffix : followed by a path names the blob or tree at the given path in the tree-ish object named by the part before the colon. A path starting with ./ or ../ is relative to the current working directory. The given path
will be converted to be relative to the working tree’s root directory. This is most useful to address a blob or tree from a commit or tree that has the same tree structure as the working tree.
This seems to be exactly what I needed and it turns out git show supports this syntax for outputting a particular file!
Yesterday I read Dealing with Non-ASCII Characters and the talk about curly quotes copied and pasted from Google Docs or Slack reminded me of a "fun" problem that I used to have to deal with occasionally that was along similar lines. It had to do with copying and pasting a full command line invocation complete with flags and arguments out of a tool that was commonly used on a project I was working on. Maybe you’d paste in a fully formed command as an example into a design or a set of FAQs or something – however when someone copied and pasted it back out a flag that might’ve been --debug
turned into —debug
. In other words, the double hyphen was getting turned into an em-dash.
Alone, this isn’t really a huge problem but might not work as expected. If you use some certain command line parsing libraries/frameworks that make your life easier in other ways, then this situation might result in failures or maybe even an unhandled exception and a stack trace being spewed out pointing at parsing code that’s inside of the framework where it’s reading argv
.
What to do?
Depending on the library being used and exactly how it parses the parameters, there may not be an easy way to handle exceptions like this from within the framework. You might have to wrap your call that "enters" the framework in your main
function with some exception handling in order to be able to avoid these unhandled exceptions. Then you could provide a specialized "are you sure you used valid ASCII characters in your command?" error message. However, as Julia Evans points out: unicode is valid in command line arguments! Unfortunately I don’t really have any fix to offer for this situation if your framework doesn’t catch it for you. This post is more a word of warning in case you or someone you know comes to you asking what’s going on with a command that was copied and pasted from elsewhere – I had to debug this for people a number of times in the past. Probably the easiest "fix", if you can change the place the command was copied from, is to ensure double dashes weren’t "helpfully" turned into an em-dash.
Jonathan provided a cool regex at the end of his post though. It will help with detecting non-ASCII characters and describes a trick for remembering the visible range of the ASCII table. It won’t help with the exact situation I describe above but if you’re trying to figure out why some JSON you copied and pasted is now invalid or something, it’s definitely worth keeping in mind.
The other day I was working on a small web project (more on this soon) and was using Warp. I wanted to impl
some From<T>
conversions for certain error types into my custom type to make error handling a little less clunky. I ran into some problems with doing that and I must not have realized it at the time but I believe I was on Rust 1.40. I saw this morning that Rust 1.41 relaxed some of those restrictions so that you can implement traits on foreign types as long as the trait is local to your crate. While this didn’t help my particular case, I might’ve been able to use it if I refactoring things around.
I also noticed that Rust 1.42 had already been released as well (March 12th). It seems to have brought some nice improvements to pattern matching including allowing subslices in a slice pattern and the matches!
macro. The former would probably be useful in the type of byte matching I was working with recently in my photo sorting tool (see part 1, part 2 coming soon).
I guess it’s time for me to subscribe to the [Rust blog](https://blog.rust-lang.org/) in my Newsblur account!
As an update to my previous post, I wanted to say that I hooked up my keyboard and mouse to a KVM that I had lying around in a box. This makes switching away from the work computer at the end of the day much easier (no more crawling under my desk).
Friday marked the end of three weeks of 100% working from home and about a week and a half of Bay Area shelter-in-place being in effect. The weekend started to feel like staying at home was “the new normal”. Overall, I think that I have adjusted to the working remote part of this experience rather well but less so to the other parts of shelter-in-place (nor do I hope to ever fully adjust to those…). I even like certain aspects of working from home more than being in the office: easy impromptu Zoom meetings w/o booking a room, more control over my space/desk, a more relaxing and less urgent morning overall. I even purchased a webcam to make meetings from my desk easier!
Of course, I miss seeing people in the office daily and sharing things like lunch or tea with my coworkers. It would certainly be nice to have a dedicated space for working as well. One side effect of not having that dedicated space (as well as more time at home overall) is that I’ve noticed it’s much easier to let work happen after working hours. For example I needlessly jumped into debugging a minor incident at work when others were already on it and I was not paged (though I am on call this week for other systems).
The little I’ve been out in visiting businesses, it has also seemed that people are adjusting and taking reasonable precautions: Grocery stores like Luke’s Local and Safeway have instituted “one in, one out” rules to limit the number of visitors in their stores and there are lines outside. My favorite coffee shop in the city, Bernie’s, had chairs blocking direct access to their bar area to create more space between those working and customers. People broadly seem to be more respectful of the “6 feet” rule now as well.
For working days, I have been trying to remain in a decent morning routine: make tea, go for a walk, have breakfast, and start work either after breakfast or while eating it. I do need to get better about "after work" routines though. I’ve also been either going for a run or a second walk when I sign off of work for the day. One problem I have been encountering is that it is too easy to leave my work computer plugged into my monitor and keyboard at my desk – partially because I need to crawl behind my desk to swap it back to my other computer) and it is also too easy to leave Slack up and get sucked in to work if something comes up. I’m trying to improve the after work routine this week.
All of that said, I worry this pandemic will continue to get worse in the coming days and weeks – both in California and in the country as a whole. I think this has affected my overall productivity in work and everyday life. On the one hand I know that I am incredibly fortunate and privileged to have a job that I enjoy and can continue doing in the face of all these societal changes. Especially considering that 3 million+ people applied for unemployment insurance in the US just last week. On the other hand, things being less productive adds to my personal worry a small amount ("I’m not working like the normal me"). That is OK. I try to remind myself that things aren’t supposed to be normal right now and am fortunate that work is supportive of adjusted expectations in this time.
The bright spot is that early indications are that shelter in place is working for the Bay Area. I’m happy that we moved on that early. I’ve been following the John’s Hopkins daily situation report and, more recently, the SF Chronicle’s case tracking graphs for CA and the Bay Area. The Chronicle uses a combination of state/local health department data as well as that from the COVID Tracking project. Early indications aside (fingers crossed that they are true), California’s 60K pending tests are a big unknown.Here’s to the next two to four weeks seeing marked improvement in the Bay Area, followed by California and the country at large...
This is the initial post in what is intended to be a series where I write about what I learn while reverse engineering photo file formats with the goal of reading metadata (e.g. Exif). It isn’t meant to be a definitive guide to the format but rather documenting the thought process of taking the format apart by only looking at file data rather than reading specifications, library implementations, or other more qualified information sources. Why? Because sometimes using hexdump
and hacking together a custom tool is just more fun.
Towards the beginning of the year I was interested in writing a tool that would sort my photos into a directory tree that followed the year/month/date/<file>
format. I have heard of such tools myself but wanted a little project I could work on for myself. To make it more fun, I wanted to see whether I could do this without any library support for reading Exif data out of photos and instead parse the needed information myself. It seemed to be the sort of thing that would be perfect for Rust (or any memory safe language, for that matter): reading a set of file data with unknown formats from usually trustworthy but sometimes potentially untrusted sources. I figured that I could start by implementing my sorting program but if I finished that and still had motivation to learn more about these formats that I could continue and write a generalized library. Whether I have that motivation is still TBD. I’ll also note that I wanted to rely on the date embedded into the file rather than the file metadata provided by the filesystem (such as ctime
) as that can be unreliable in certain circumstances.
Before I got started I did some basic searching and learned that the CR2 files that my Cannon cameras output in raw mode are only one of multiple "raw formats" and that they are not a standardized format. That’s good to know, there’s no spec to speak of anyhow! I did learn that there’s a program called `dcraw` which is a C program (not library!) maintained by Dave Coffin that supports most raw formats. This is more focused on processing the photos rather than extracting metadata. While I found that interesting, I didn’t use it as a base for my program. I was open to researching further if I got stuck but so far have not needed to. With all that out of the way, let’s dig in!
First off, I leaned pretty heavily on standard Unix tools for extracting the info that I wanted. hexdump
and strings
are your friends for this type of work. In this post I mostly focus on the output of hexdump
but any hex editor would also do. The following is a hexdump
of the first few hundred bytes of an image I took, the header does go on for a fair bit longer and I see things like the lens that I used and the aperture setting in that later output but for now I’m most concerned with extracting the date/time of the photo which is near the start of the file.
$ hexdump -C -n 320 photo.CR2
00000000 49 49 2a 00 10 00 00 00 43 52 02 00 4a b2 00 00 |II*.....CR..J...|
00000010 12 00 00 01 03 00 01 00 00 00 70 17 00 00 01 01 |..........p.....|
00000020 03 00 01 00 00 00 a0 0f 00 00 02 01 03 00 03 00 |................|
00000030 00 00 ee 00 00 00 03 01 03 00 01 00 00 00 06 00 |................|
00000040 00 00 0f 01 02 00 06 00 00 00 f4 00 00 00 10 01 |................|
00000050 02 00 0e 00 00 00 fa 00 00 00 11 01 04 00 01 00 |................|
00000060 00 00 7c 18 01 00 12 01 03 00 01 00 00 00 01 00 |..|.............|
00000070 00 00 17 01 04 00 01 00 00 00 34 61 1e 00 1a 01 |..........4a....|
00000080 05 00 01 00 00 00 1a 01 00 00 1b 01 05 00 01 00 |................|
00000090 00 00 22 01 00 00 28 01 03 00 01 00 00 00 02 00 |.."...(.........|
000000a0 00 00 32 01 02 00 14 00 00 00 2a 01 00 00 3b 01 |..2.......*...;.|
000000b0 02 00 01 00 00 00 00 00 00 00 bc 02 01 00 00 20 |............... |
000000c0 00 00 c4 b2 00 00 98 82 02 00 01 00 00 00 00 00 |................|
000000d0 00 00 69 87 04 00 01 00 00 00 be 01 00 00 25 88 |..i...........%.|
000000e0 04 00 01 00 00 00 72 aa 00 00 74 b1 00 00 08 00 |......r...t.....|
000000f0 08 00 08 00 43 61 6e 6f 6e 00 43 61 6e 6f 6e 20 |....Canon.Canon |
00000100 45 4f 53 20 37 37 44 00 00 00 00 00 00 00 00 00 |EOS 77D.........|
00000110 00 00 00 00 00 00 00 00 00 00 48 00 00 00 01 00 |..........H.....|
00000120 00 00 48 00 00 00 01 00 00 00 32 30 32 30 3a 30 |..H.......2020:0|
00000130 32 3a 30 31 20 31 34 3a 33 32 3a 31 34 00 00 00 |2:01 14:32:14...|
00000140 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
Let’s note down some relevant metadata that we can see here. I don’t particularly need this right now for sorting but noting it for later: 245 bytes in we can see the the brand, a null byte, and the model name of my camera. Canon
and Canon EOS 77D
. It sort of seems like the manufacturer of the camera (Cannon) is delimited by a null after which the model is included. There are quite a few null bytes following this so it’s the model is a fixed-width string where the nulls are used to pad out to particular offsets or there’s some delimiter patten that I haven’t recognized yet.
Now for the stuff I need to do directory sorting: We can see that 298 bytes in we have the a bunch of null bytes (and a random ’H’...) and the start of our date string: 2020:0
. The next line of our hexdump output gives us the second half of the date, a space, and the time: 2:01 14:32:14
. Three null bytes trail that. Putting that all together we get our date/time of 2020:02:01 14:32:14
!
Now, I’m guessing that the metadata fields are not aligned around 16 byte lines like our hexdump output is here. It isn’t immediately clear what, if any, line oriented alignment there might be on this data. There certainly appear to be a lot of null bytes surrounding the useful data. Perhaps it’s just a lot of data packed in at various offsets and my particular camera doesn’t populate most of the fields. 🤷
For a very brittle first implementation of date all I had to do was hard code this offset into my sorting program, read the next 10 bytes and parse that to get my directory structures. You can see the parsing logic in the first commit of my photo sorting tool. That commit only parsed and printed dates, didn’t quite sort all of the photos just yet. The second commit took the set of bytes that represented the date and wired up logic to do file renaming. If you read through that, you’ll note there was also a mode to rename using git
rather than straight filesystem renames. That is because some of my photos are managed by `git-annex` and provides some extra safety on these operations so that anything can be undone so that I don’t lose any data.
It turned out that this very unreliable implementation works! But only for photos taken with this particular camera in this particular format. Some older CR2 files I have from a different Cannon camera have a really similar structure but different offset for the date. Additionally, JPG file written by both of these cameras have JPG magic number stuff prefixing all this metadata so we’ll have to build something more reliable. We’ll start covering all this in more detail in part 2.
I read over 25 new books this year and re-read a bunch of old favorites as well. I wanted to cover some of my favorites here as well as those that I’ve been dragging my feet on finishing even though they’re on topics I’m interested in or stories I enjoy.
Thanks to Jonathan Palardy for inspiring me to read 10 pages a day, I formed a new habit to get me reading more than ever. I’ve long loved reading new books but also long felt too exhausted to do so in the evening. By re-arranging my schedule and making time to read in the mornings, I’ve managed to make consistent progress on books I’ve long wanted to read. Since Thanksgiving I’ve slightly fallen out of my routine due to holiday season hectic schedules (and probably also some laziness) but am excited to recommit to this goal in the new year.
Here are some of the books I read in 2019 that stood out for me, in no particular order.
This was a rather interesting book about the origins of trail systems, the history of certain trails and of the activity of hiking. It covers animals and their trail systems as well as various cultures’ relationship with trails and hiking. I really enjoyed reading this one and was rather sad when I left my copy on an airplane when I was only part way through the book – the first time I’ve ever lost something on an airplane! Thankfully I managed to make it through the book before losing my second copy.
A history of San Francisco in the late 60s and early 70s up through the early 90s. When I started reading at least 10 pages a day, this was the book that I was struggling to get through. For non-fiction books I find the motivation I have towards the end of the day to engage and learn is basically gone after a day of working. That said, I learned a ton of interesting tidbits about people and places from this book!
A look through Adam Savage’s career and life as well as some methods he’s picked up over the years for keeping life organized. A fun, quick read.
I discovered Will Larson’s blogsometime last year. The first piece on his blog that really stuck with me was about systems thinking and the book that he recommended as an introduction to the topic by Donella Meadows was one of my favorite books last year so I was really excited to pick this book up.
It is beautifully designed inside and out. While it is mostly about engineering management in the tech industry, as a non-manager I found it to be helpful as well. I enjoy Will’s thoughtful discussion of various components of life on a team of people building a technology product and particularly where the traditional approach is questioned or approached in a different light. As a reader of his blog, I found some of the content to be pulled from there but there was also enough new content that expanded on it as well.
This book introduces multiple ideas in software design and explores the trade offs between some of these ideas. While I don’t agree with all of the ideas, the book effectively demonstrates that different situations call for different approaches to software design and highlights that simplicity has value.
I had somehow not read this one before. A quick read and still very applicable to today despite being published in the 50s.
I actually enjoyed this one more than volume 1 in this new series in the same world (but different time) as His Dark Materials. I always enjoyed the world of Lyra and the daemons and this book was no different.
This book, along with The Name of All Things, makes Jenn Lyons is one of my favorite new authors. As a first published book (as far as I could tell), I thought she was amazingly good at world building. The storytelling style of having a sort of book within a book and the narrator being a character recounting events was interesting and a nice, different approach as well. I was happy to see that was carried over to The Name of All Things. The timeline was sometimes a bit confusing but I appreciated how everything felt like it came together at in the end. I also always love footnotes and this book has lots of footnotes [1].
Neal Stephenson’s new book is largely a fictional fantasy book about the (maybe) not-too-distant future with AIs, robotic "presence" bodies, quantum computing, full brain scans being used to simulate people after death, and more. It also happens to raise interesting questions about the nature of the self and of death: consciousness, personality, memory, and soul vs brain.
I tore through those books that had already been published and spent a lot of this year excitedly waiting for the next installments. I love the concept of the world in these books and it is one of my favorite new series.
Here’s a short list of books that I didn’t manage to finish this year despite them being in progress for some time. These are not books I consciously decided not to finish but rather that I moved on to other books that found my interest at the time. I hope to finish these soon:
[1]: I have a longstanding TODO item to make this blog support footnotes in a better way...
There are many good options for tools that provide insights into your running programs for Linux including newer tools like writing eBPF filters against the kernel and older ones like strace
, perf
, and `gdb`. Today I want to talk a little bit about a feature of many Unix kernels that is often used by other tools but can also be useful on its own for resolving little questions or problems you might have. That feature is ProcFS.
I wanted to write about a couple of use cases that others may find useful in certain cases. The first is the ability to dump the command line and environment of a given process. There are certainly other ways of obtaining the command line of a process but it isn’t always easy to inspect the environment variables present at runtime. This can be especially useful when trying to debug or verify some behavior when the program doesn’t otherwise expose its configuration at runtime (e.g. via logging).
Let’s pretend that we have a process that we want to inspect. Perhaps it’s a Python process that is doing some web serving. Let’s use ProcFS to determine the command line arguments and the environment variables set within this process:
$ pgrep python
2961
$ cat /proc/2961/cmdline
python3.6-mhttp.server3000
$ xxd /proc/2961/cmdline
00000000: 7079 7468 6f6e 332e 3600 2d6d 0068 7474 python3.6.-m.htt
00000010: 702e 7365 7276 6572 0033 3030 3000 p.server.3000.
$ cat /proc/2961/environ
MYENV=ohhaiSECRET=blah
$ xxd /proc/2961/environ
00000000: 4d59 454e 563d 6f68 6861 6900 5345 4352 MYENV=ohhai.SECR
00000010: 4554 3d62 6c61 6800 ET=blah.
First, we determine the PID for our process to inspect. From there we dump the contents of /proc/<pid>/cmdline
and /proc/<pid>/environ
(I do so both in ascii and hex so that you can see the exact format). Note that each command line argument and environment variable is separated by a null byte. Here you can see that we have a web server listening on port 3000 using Python’s built in http.server
tool. We’re also able to see that there are two environment variables that in the server process’ environment and what their names values are.
I’ve personally found this environment variable dumping ability quite useful for debugging a program at runtime. Maybe you have access to the source and are observing some behavior in logs which is not trivial to reproduce and are trying to correlate that behavior back to the source. If the program derives behavior from the environment, sometimes being able to dump the exact state of the environment can be helpful in your debugging.
Another possibility is to inspect the state of a given file descriptor within a process. If you know a given file descriptor ID, you can use /proc/<pid>/fdinfo/<fd>
to dump the state of the file descriptor (including the current position of a file, the mode used to open the descriptor, socket or event state, and more).
One last handy thing that comes up occasionally in debugging a hung process (especially one in uninterruptible sleep, the dreaded D
state) that you can’t get relevant logs for or hooked up to a debugger is to get the stack trace of the kernel syscall that the process is waiting on, if any. Especially for processes in uninterruptible sleep, where the hang is usually I/O related, this can be helpful for getting at least some small indication of what type of thing is blocking the progress from making progress. Here’s an example from a web server that is running normally:
root@server:~# cat /proc/2411/stack
[<0>] ep_poll+0x29c/0x3a0
[<0>] SyS_epoll_wait+0xc6/0xe0
[<0>] do_syscall_64+0x73/0x130
[<0>] entry_SYSCALL_64_after_hwframe+0x3d/0xa2
[<0>] 0xffffffffffffffff
We can see that the process is in epoll wait, which should not be surprising for a web server waiting on socket events. Examples of things that could have been shown would be things like reading/writing to a socket, a file on disk, or some remote filesystem.
It would be well worth your time to read some sections of man 5 proc
for more details of various Procfs features. You’re almost certainly guaranteed to learn something and it might surprise you to learn about all the various things that are able to be introspected via a file-like interface. Everything is a file, indeed.
Last week I read an OpEd in the NYTimes with the above title that I found to miss the mark a fair bit. It may even result in discouraging people from setting up this important protection on their most critical accounts, which would be a most unfortunate outcome. You should enable two-factor authentication on any account that you can. The end of the article concludes with a call for more data about the effectiveness of two-factor against various forms of account compromise which I can entirely agree with; however, this is buried deeply beneath paragraphs casting doubt on the effectiveness of two-factor auth.
The primary two complaints seem to be that two-factor authentication cannot protect against sophisticated phishing attacks and that we have not enough information about its effectiveness, given the perceived inconvenience. Admittedly, it does state that these facts alone are not enough reason not to enable it, but that is after seven paragraphs describing phishing attacks against various types of account but then equivocates by introducing the complaint about lack of data for its effectiveness.
My primary issue with this is that it seems to conflate phishing - one type of account compromise attack - with the broader class of account compromise attacks. There is not a single mention of the protection that two-factor authentication can provide against a compromised primary credential (i.e., a password). The protection provided against credential dumps from breached services cannot be ignored. Even if you follow other best practices and do not reuse any passwords across websites, there is often a gap of months between the time a breach happens and the time it is known publicly. Without two-factor authentication, an account on a service that has had a password breach is even more vulnerable than it would be without the second factor. Assuming no password reuse, I would go so far as to say that two-factor authentication vastly mitigates primary credential theft since it would require a secondary attack (e.g., phishing, phone compromise, etc.) to bypass the second factor.
Is the author right that some two-factor authentication methods are vulnerable to phishing? No doubt about it, yes they are. However, even the weakest forms (e.g., SMS as a second factor) can mitigate primary credential compromise as described above. There are also two-factor auth methods that are much stronger against phishing attacks than SMS or one-time passcode generator devices/apps. U2F devices are one such method. They prevent phishing as there is no passcode to phish - they work via a challenge-response mechanism that fails on a phishing website even if the human doing authenticating is fooled. Another lesser protection is a push-based mechanism like Duo Push where you have a chance to inspect the login request. Chances are that it’s still possible to be fooled by a phishing attempt with a push-based second factor as mentioned in the original article, but, given a location for the login attempt, it is at least possible to notice that the attempt didn’t come from where you expected. A mismatch in the location of the phishing server and the authenticating user is likely to be the case in many less sophisticated phishing attacks. (Full disclosure: I previously worked for Duo.)
Lastly, we have the call for more data from larger organizations about the effectiveness of 2FA against phishing attacks. I’m on board with this and wish larger organizations like Google or Facebook’s security team were more transparent about results here. That said, the existence of an attack against specific 2FA methods does not equate to proof that 2FA doesn’t work at all. I worry about people not well educated in computer security reading critical articles like this and concluding 2FA isn’t worth the hassle. I’m not arguing that we shouldn’t question our current best practices but that when it comes to security discussions consumed by the general public, we have to be well balanced because there is a real risk.
Frequently when processing data, it’s useful to slice it by a certain dimension. It can quickly answer all sorts of questions and help
Some examples of the types of questions it might answer are:
...and endless others. It’s one of the most frequent tools I reach for whether I’m debugging, refactoring, designing a new feature, or architecting a new system. Being able to look at the same data from multiple angles is invaluable. Programmers aggregate data like this all the time in database queries and on the command line.
An infrequently mentioned function called groupby
in the itertools
module does just that for iterables in Python. One frustration with the standard library’s particular implementation that might not be obvious is that the data likely needs to be sorted first. Quoting the docs:
The operation of groupby() is similar to the uniq filter in Unix. It generates a break or new group every time the value of the key function changes (which is why it is usually necessary to have sorted the data using the same key function). That behavior differs from SQL’s GROUP BY which aggregates common elements regardless of their input order.
This isn’t always convenient and doesn’t seem like it should be required in most cases. Fortunately, with the help of collections.defaultdict
we can build a grouping function that doesn’t have this requirement:
import collections
def group_by(iterable, key_fn):
"""
>>> def is_odd(num):
... return num % 2 != 0
...
>>> group_by(range(10), is_odd)
defaultdict(<class ’list’>, {False: [0, 2, 4, 6, 8], True: [1, 3, 5, 7, 9]})
"""
result = collections.defaultdict(list)
for item in iterable:
result[key_fn(item)].append(item)
return result
The primary tradeoff of this version when compared to the standard library is that the input iterable has to be finite. The benefit of being able to assume pre-sorted data is that you can build an iterator over intermediate results, by each key type. Most often I already have the complete set of data so this isn’t a concern for me but it is a consideration.
This isn’t always the tool you want. In particular, your datastore is probably faster at grouping things before returning results than the Python VM is. However, data isn’t always already in a datastore - it can also come from files, input streams, the network, whatever. Go forth and group things!
A few weeks ago I had occasion to chunk a potentially large list of items into smaller lists for processing in batches in a Python codebase. While I remembered that itertools
didn’t have such functionality built-in, it did have a recipe to do so, called `grouper`. For my use case, this recipe worked fine so I used it but it got me wondering about why Python didn’t provide what seems to be a fairly common operation (in another codebase I’d had occasion to do this a lot and we had a helper function for it). This led me down two paths.
One was to gut check that I wasn’t crazy for thinking this seemed to be a valid standard library function. I spot checked a few "batteries included" languages that I have some familiarity with (Ruby and Rust) and each have at least one built-in solution for this. Even C++ is considering including a solution in the standard via the Ranges-V3 proposal but that isn’t a surprise given the breadth of language features and library functions (particularly around containers) that C++ has.
The other path was to check out the history for why this hadn’t been included in Python. That lead me to a thread on the Python mailing lists](https://mail.python.org/pipermail/python-dev/2012-June/120781.html)) and a Python issue). While it would be easy to summarize my (outside) view of the reason for not including a standard function as "bikeshedding", the discussions did bring up a number of regarding the interface and implementation.
Some of these tradeoffs:
At the end of the day, I filed this under one of the oddities of Python and moved on with the task at hand but thought it would be interesting to share. There simply isn’t consensus over what constitutes the right chunking helper function for the Python standard library, nor does there appear to be any interest in figuring out the sticking points. FWIW one of the biggest sticking points seem to be around what to do with an off-size chunk at the end but I find myself in agreement with the original mailing list poster that this isn’t a huge deal and callers can handle this quite easily if it were a concern.
For kicks, I threw together a few additional chunking functions, each with a slightly different implementation:
import itertools
def chunk(iterable, n=1):
"""
Returns an iterable of lists of at most length n. Does not add filler
values if not enough input values for a given chunk.
>>> list(chunk(range(6), n=2))
[[0, 1], [2, 3], [4, 5]]
>>> list(chunk(range(6), n=4))
[[0, 1, 2, 3], [4, 5]]
"""
it = iter(iterable)
current = 0
next_chunk = []
while True:
while current < n:
try:
next_chunk.append(next(it))
current += 1
except StopIteration:
if next_chunk:
yield next_chunk
raise StopIteration()
yield next_chunk
current = 0
next_chunk = []
def chunk2(lst, n=1):
"""
Chunks an input list by size n using slices.
>>> list(chunk2(list(range(6)), n=2))
[[0, 1], [2, 3], [4, 5]]
>>> list(chunk2(list(range(6)), n=4))
[[0, 1, 2, 3], [4, 5]]
>>> list(chunk2(list(range(4)), n=5))
[[0, 1, 2, 3]]
"""
last_used = 0
while last_used < len(lst) - 1:
chunk = lst[last_used:last_used + n]
last_used += n
yield chunk
raise StopIteration()
def chunk3(iterable, n=1):
"""
Chunks any iterable by size n using islice.
>>> list(chunk3(range(6), n=2))
[[0, 1], [2, 3], [4, 5]]
>>> list(chunk3(range(6), n=4))
[[0, 1, 2, 3], [4, 5]]
>>> list(chunk3(range(4), n=5))
[[0, 1, 2, 3]]
"""
last_used = 0
while True:
chunk = itertools.islice(iterable, last_used, last_used + n)
last_used += n
if chunk:
yield chunk
if len(chunk) != n:
raise StopIteration()
Inspired by a tweet from @munificentbob) I recently read this paper about Lua. I took some notes while reading and though I’d dump them here. Some of the implementation decisions and data structures were pretty interesting to me. These notes are pretty raw and only a little bit cleaned up from my handwritten shorthand.
Many things that were very interesting to me in the paper but these were some of the highlights (I have some notes on all of these below):
I don’t have very much experience with building interpreters or compilers - an undergrad compilers class and a fair bit of reading about CPython implementation details - but still found this paper very readable. It touches on concepts at a high level and seems to have fairly good references for deeper reading if you want to learn more about a given topic. With all that said, here are the notes I took:
Lua does not have arrays, only tables (hash tables)
- Lua 5 can recognize tables used as arrays and back these w/ an array for efficiency
Efficiency: fast compilation and execution of Lua programs
- fast, smart, one-pass compiler and fast VM
The compiler is 30% of the size of Lua core
- It is possible to embed Lua without the compiler and provide pre-compiled programs
The scanner and parsers are hand written
- smaller and more portable than yacc-generated code
- used yacc until 3.0
Compiler uses no IR
- still performs some optimization (although the paper didn’t cover this afaik)
For portability, cannot use Direct Threaded code. See refs [8, 16]
- Uses while-switch dispatch loop
- Complicated implementation sometimes for portability
Eight types: nil, boolean, number, string, table, function, user data, thread
- Numbers are doubles
- Strings are arrays of bytes with explicit sizes
- Tables are associative arrays
- Userdata: blocks of memory (pointers)
- - light (memory managed by user) and heavy (garbage collection)
- threads are coroutines
Tagged union used to represent types
- copying is expensive (3-4 words) because of this
Needed for portability, can’t use tricks that some languages (e.g. smalltalk) use to embed type info in spare bits.
- (because byte alignment differences on various platforms)
Tables are backed by both a hash table and by arrays
- keys like strings end up in hash table
- Numeric keys from 1 onward are not stored, values stored in the array
The backing array has a size limit N
- goal for at least 1/2 of N to be used
- "largest N such that at least half of the slots between 1 and N are in use and at least one used slot between N/2 + 1 and N"
- Access to the array backed keys is faster because no hashing and takes half the memory (due to not storing keys).
- Hash part is chained scatter table w/ Brent’s Variation (Ref [3])
- Another paper for me to read
A closure has a reference to its Prototype, environment (e.g. global vars), and an array of references to Upvalues
- Upvalues used to access outer local variables
Function parameters in Lua are local variables
When the variable goes out of scope, it migrates into a slot in the up value itself
- (So a copy is only incurred when needed)
When a new closure is created, the runtime goes through all outer locals and sees if the variable is already an Upvalue in the linked list.
- If found, reuse, otherwise create a new Upvalue.
List search typically probes only a few nodes because the list contains at most one entry for each local variable that is used by a nested function.
- Question: Why a linked list? Is the mutation and traversal here infrequent enough to avoid allocation and pointer indirection overhead? Why not normal array or some other structure?
create
function receives a main function and creates a new coroutine with that function. Returns a value of type thread
that represents that coroutine.resume
(re)starts execution, calling the main function that was provided.yield
suspends execution and returns control to the call that resumed that coroutine.resume
/yield
correspond to recursive calls/returns of the Lua interpreter function, using the C stack to track the Lua stack for coroutines.
- This implies the interpreter is able to run multiple times, recursively, within the same process without issue.
- The closure implementation helps here by avoiding issues with locals going out of scope.
Two common problems with register-based machines are code size and decoding overhead:
- Instructions have to specify operands so most instructions are larger than instructions on a stack based machine because of implicit operands.
- The paper says instructions are typically 4 bytes vs 1-2 bytes of previous stack-based implementation but that many fewer instructions are emitted for common operations so code size isn’t significantly larger.
- There is overhead in decoding operands from register machine instructions compared to implicit operands of stack machines.
- Due to machine alignment and efficient use of logical operations for decoding, this is still fairly cheap.
Instructions are 32 bits divided into three or four fields.
- Operations are 6 bits, providing 64 possible opcodes
- Field A is always 8 bits.
- Fields B and C take 9 bits each or can be combined into an 18 bit field (Bx is unsigned or SBx is signed).
- Most instructions use three address format, A is the register for the result, B and C point to the operands.
- Operands are either a register or a constant.
- This makes typical Lua operations (like attribute access) take only one instruction.
Branching post a problem so the VM uses two instructions:
- The branch instruction itself, and alum that should be done if the test succeeds.
- The jump is skipped if it fails.
- This jump is fetched and executed within the same dispatch cycle as the branch so it is something of a special case within the VM.
Lua uses two parallel stacks for function calls:
- One stack has one entry for each active function and stores the function, return address, and base index for the activation record.
- The other is a large array of Lua values that keeps the activation records.
- each activation record has all temporary values (params, locals)
I wrote my own URL shortener.
That statement might seem surprising at first but I built it for a very specific purpose: I keep a notebook as a cross between a planner, todo list, and scratch paper to keep notes. I use a modified version of the Bullet Journal system and wanted a way to link notes on paper to the webpages I needed to refer to for notes, ideas, and todo items. This was the core of the technical requirements but there were philosophical reasons for building my own URL shortener as well.
I wanted a URL shortener that provided as short of a URL as possible so that I could write it down on paper in my notebooks. Publicly available shortener services and open source projects generally produce fairly lengthy URLs when you’re considering hand writing them and so I came up with my own key space based on my expected usage. I’ll get to that in a moment. The philosophic reasons alluded to before were around data longevity and ownership. I go through approximately two notebooks each year and have been using this method of organization for my life and work for a few years already and don’t see that changing any time soon.
I was concerned with being able to export all URLs I’ve shortened with their keys in the case that whatever service I chose went offline. This is something I looked into before embarking on implementing my own shortener but couldn’t find anything that quelled my concerns so I spent a couple hours one weekend to get the basics implemented.
As mentioned before, I wanted to make the short URLs as easy to write down as possible. I considered the simple approach of incrementing an integer key but quickly moved on. At first it would work very nicely but it would rather quickly reach four-digit keys, even with a low rate of use of 1-3x per day.
Instead, I considered a key space similar to that of base 64 encoding. I didn’t want to deal with non-alphanumeric characters though so I used [a-zA-Z0-9]
which ends up being 62 characters. Next I considered how long each key would have to be so that all URLs could be of the same length. Two character keys (62^2
or 3,844) might have lasted a few years but would have eventually run out. Three character keys (62^3
or 238,328) seemed likely to cover me for as long as I wanted since my usage pattern would be manually generated short URLs on demand and fairly infrequently at that.
There are three main components of my URL shortener: The input endpoint that shortens URLs, the lookup endpoint that turns a short URL into the real deal, and the short code generator.
Let me start with that last one. I could have done something fancy to create my short codes like having an incrementing primary key that mapped to my three character sequences (e.g. 1=aaa, 2=aab, 3=aac, etc
) or some other mechanism to reliably generate these codes. Instead, I took the simple approach of randomly generating three characters within my 62 character range and assuming it would be fine. I then rely on the database to enforce uniqueness. I haven’t had any issues with this approach to date. Sure, in theory, I could get collisions and either fail or retry. Randomly generating codes until you get an unused one is not guaranteed to ever terminate but I’ve created probably a few hundred entries so far with no issues. It’s not like this being 100% reliable is the most critical thing in the world.
The lookup portion consists of a simple SQL query followed by a 302 redirect. That’s it!
Lastly, we have the input to create a short URL. I started with a simple web form that consisted only of an input box and submit button. This worked well: read the input, validate it, generate a short URL DB entry, and finally output the short code. Eventually, I supported a non-form interface for an integration with Alfred as well. With that, I have a working URL shortener! It’s deployable on Heroku as a small flask app that works fine with their free web and Postgres tiers for one person. I’ll note that this has been the first time I’ve used Postgres for any sort of app (I’ve always used MySQL for relational database storage before). There were a few differences but it has worked well for me thus far!
At first, I would load up the web form for my shortener, put the URL I wanted to shorten into the input box, and get back the short code. From there, I developed a shorthand to write the short URL into my paper notes: I use s/{shortcode}
to denote a URL I might need. For example, I’d write s/xyz
in my notes.This means that on paper I can denote any URL with five characters. That’s pretty great!
Once I developed this usage pattern and shorthand, I got to thinking about how I use the shortener. I decided that since I’m already a heavy Alfred user, I should try building a workflow for creating and consuming short URLs. I developed an API endpoint for shortening URLs that was safe from bots polluting my key namespace and a way to create/view short URLs.
Now that I have that done, my usage of the URL shortener is very simple: Open Alfred, type “s <URL>” to create a short URL. This will copy the short code to my clipboard and displays it in large text on my screen. Now, I can copy it down into whatever I was writing a note about in my shorthand (e.g. s/xyz
). In order to look up that short code, I simply type s/xyz
into Alfred and hit enter. It will open a browser to the fully qualified short URL and the shortener service will do the rest!
It isn’t a big project but it fits into this category of “personal infrastructure” applications that I’ve been thinking about and working on over time. Maybe I’ll write more about that topic in the future.
The initial version of this service took a short time one afternoon. This was followed by the little bit of time to get the Alfred workflow going. After that, I’ve really not spent much (if any) time on this project. That said, I do have a small wish list that maybe I’ll get to one day:
I’ll be honest: with my level of Python and web experience, this project wasn’t very complicated. It was enjoyable though! For one, I found a way to add simplicity and utility to my life with a minimal time investment. I also got to do a small amount of web development again during a time that I was doing totally different things at work. That was fun for me!
Lastly, it further supported this idea of personal infrastructure that I’ve had bouncing around my head for a while (and have also sort of discussed before with my friend Alex). At a very high level, this concept is that very small custom apps can make my life easier/better in a number of ways. Since I’m a programmer, they are not a big investment to prototype out. I’ve thought about how this can improve the lives of non-technical people too but I have nothing super enlightening to report there at the moment.
Do you have any “personal infrastructure” projects? I’d love to hear about them!
At the end of last year, I moved to a new team at work.
I had spent a little over three years on my previous team where I worked on Facebook’s container daemon, blob distribution mechanism, service orchestration system, and, mostly, tooling for developers that ties all these systems together so they are able to deploy services.
A good portion of my time on the team was spent on debugging. Debugging production issues in our components. Debugging other people’s services when they needed help. Debugging the interaction between the orchestration system and other people’s services or deployment automation. Even debugging standard Linux utilities like mount
. This meant that I got really comfortable diving into foreign code bases. I still believe this is maybe the best skill I developed in those three years.
That said, over the last few months I’ve learned the difference between spelunking through an unfamiliar codebase with a specific purpose and having a strong mental model of the codebase’s structure, nuances, and quirks. Tracing an execution path backward to figure out why an error happens or figuring out what an undocumented flag does are sort of bottom-up exploration of a codebase. These seem pretty different from day to day work on modules and classes. At first, I was a little bit discouraged by this unproductive wall that I’d hit. I was still able to make contributions to my new codebase and had meetings with various new teammates where they explained various parts of the architecture. Even after all this, I felt that something was missing.
There’s a difference between knowing how a system works at the architectural level and how it works at an implementation level. You might know that requests flow in and hit certain components in a specific order or understand the data model. That doesn’t mean you know what is happening when communicating with your data store or as requests flow through your web app. That level of understanding comes from reading and working with the code regularly.
Upon further reflection on my feeling I realized that over the last couple years the things I had prided myself on were (partially) derived from my deep knowledge of the code and systems I worked on every day. The ability to recognize flaws and proactively work towards resolution, to know what needed doing and prioritize across all of those options - these were skills that had become second nature, that I took deep satisfaction from. I then reminded myself that it hadn’t always been that way. It took me between six months and a year to feel truly productive on my previous team. These abilities arose from long-term work on the codebase. Ramping up on a new project doesn’t happen as if by magic just because you’ve successfully done so on a previous project. It takes work and, if I may theorize, it’s a skill just like any other in that it probably gets easier with repetition. It’s not a skill that we practice very often because we often stay working within a codebase for years at a time and ramping up is time-consuming.
So far, I’ve contributed where possible by identifying pain points that prevented me from more easily ramping up: Adding Type Annotations in more confusing portions of the codebase, refactored some of the more complex methods, and started addressing common oncall issues. While this is all helpful for developer productivity, it (mostly) doesn’t impact the people using our product. I’ve also investigated a few reported bugs which have helped me gain a better end to end understanding of the system. I just have to continue to remind myself that productivity on a larger scale will come with time and effort.
What do you do to learn a new production codebase that is complex?
A brief update on a seemingly defunct blog: Over the weekend I finished up the first iteration of porting my blog from WordPress over to a static site generator. For now I’m using Pelican and the [Blue Penguin](https://github.com/jody-frankowski/blue-penguin) theme. I wanted to move to a static site generator in the hopes to simplify the hosting and writing processes to incentivize myself to write a bit more. I selected Pelican because it had an importer from WordPress and seems to work well enough for now.
The first new post I hope to get up some time in the next week will be about what I’m learning in the process of getting getting up to speed in a new codebase.
I’ve been somewhat preparing for the [Bayshore marathon](http://www.bayshoremarathon.org/) through the winter and into the spring. Realistically, my workouts have been more suited for 5k to half-marathon training but since I was coming off of injury, I think any workouts were better than no workouts.
As a result, I’ve had to scale back my goal expectations. Various calculators like the Daniels VDot and McMillan estimate I should be able to run under 2:50 even off of some of my more average 5k and 10k times, and something like 2:55 based off of last year’s Bayshore half time. I know I haven’t done equivalent marathon training to last year’s 5k and half marathon training so I’m hoping that I can run sub-3:05 to qualify for Boston, and would like to run 2:59 or so. I tell myself that if I qualify for Boston, that will be my first marathon on a ’real’ training cycle.
I’m still not sure sub-3 is realistic. My long runs haven’t been long enough or frequent enough and I haven’t done enough marathon pace work or tempo runs. My [longest run was 16 miles](https://openrunlog.org/u/david/run/536e7410d1ec593175b27d27) this past weekend. Tomorrow, I’ll do my last tempo run of 5-6 miles, probably around half marathon pace or just under it. After that there is nothing left to do. We’ll just have to find out what happens.
I could take the easy route out and just run this thing for fun but I’ve already done that for too many races this year. My fitness and weekly mileage may not be where I think they should be but that isn’t an excuse to shy away from running what I’m capable of for where I’m at now.
I’m going to shoot for a sub 3 hour time anyway, it could be a hilariously bad experience or it could go really well. If I don’t try, I’ll never know.
Hopefully this is the last catch-up post I have to write\!
I left off the last post talking about how I was recovering from injury. While this is true, I wasn’t able to quite make it back into full shape in time for indoor or outdoor track season.
I was able to start doing some track workouts and long runs again after the beginning of 2014 with minor soreness in my foot resulting from them. I got more diligent than ever in stretching and rehab exercises and continued to make slow but distinct progress in the healing of my plantar fasciitis. I was extremely out of shape when I started doing workouts but they soon got at least a little better. I began thinking that maybe I could run a decent mile on the track before refocusing my efforts on training for the Bayshore full marathon that I happened to sign up for early in the year.
While I never quite ran what I would call a decent mile, I was able to run a 5:11 at an indoor meet at MSU. This isn’t close to my PR at all but at least I was able to start racing again\! I also raced a 3k that went very poorly at Illinois - 10:49 for that. I later ran 18:10 for 5k at the NIRCA outdoor track nationals meet without pushing all out during the race. That’s one recent race that I can be happy with. Still not near my PR but it’s beginning to show some fitness - especially given the limited mileage I’ve been able to run.
Somewhere between signing up for Bayshore and the end of the track season, I got the crazy idea to sign up for a [50k trail race](http://trailmarathon.com/). Over distance training for the marathon, right?\!
A little over a week ago I completed this race. I ran 5:15 for the 50k, it was a trail race so the distance wasn’t exact. I enjoyed this race a lot - the trails were great\! It was very painful after 20 miles - for context, I crossed 20 miles at just under 3 hours. I’d love to do this race again - but better trained. I knew going in that it would be rough given my little winter training. My longest run was something like 14 miles and I hadn’t done many trail runs because of the harsh Michigan winter. Despite that I still gave it a go\! I ended up 30th overall which wasn’t quite what I was looking for but was still good enough to win a cool clear-glass mug\!![Photo: 26 miles down, 5 to go](https://scontent-b.xx.fbcdn.net/hphotos-frc3/v/t1.0-9/p526x296/10268494101520488769273746179096581971912398_n.jpg?oh=44c37529663c74b1ed7310415b369144&oe=53D1FF4E)Me,
And another photo of the medal and the mug:![Photo: So this morning I ran 31+ miles through the woods and when I got done some random people handed me this cup. Strange.](https://scontent-b.xx.fbcdn.net/hphotos-prn1/t1.0-9/p403x403/603640102036079631806926952410282844026144_n.jpg)
My racing for the spring season doesn’t end at that 50k. Just 4 weeks after my first ultra, I’m still scheduled to run the Bayshore full\! I’ve been focusing on trying to recover quickly enough from the 50k to get one more long run in before it’s time to taper for the marathon. That’s a tight schedule and I’m hoping it gives enough time.
My goal is to run a sub-3 hour time and qualify for Boston. We’ll see how that works but I’m telling myself that 20 miles in 3 hours on the Poto trails last weekend were pretty darn close to 26.2 in 3 hours on a flat road course. Only time will.
After Bayshore I think I’ll take a little time away from long distance running and focus on building mileage again with some 5-10k races. And GLR, can’t forget that\!
Then I’ll be moving to SF soon after that for a new job\! Exciting\! Maybe I’ll return to the trails when I get back to California.
*This post has been a long time coming... and has morphed into a recap of all running I did between summer and the end of the year in December*
Summer 2013: In which I learn about trail running, run more miles per week and per day than ever before, have some crazy trail running adventures, and become injured.
I’ll start with my summer training. I picked my training back up after taking a little time easy after the [Bayshore Half](http://davidwilemski.com/blog/2013/05/bayshore-half-marathon-race-report/ "Bayshore Half Marathon Race Report") and began to ramp up the mileage again. It was a little more difficult to do this than expected: I had a few nagging aches and pains despite the time off and moving to San Francisco for the summer on a new work schedule disrupted my running a bit, but overall, I was pleased with the ramp up period.
I was able to start hitting 40-60 mile weeks again fairly quickly. My goal for the summer was to ramp up to consistently higher weekly mileage than I had done before - I wanted to run 80 mile weeks. By the end of July, I had run my first 80 mile week. That was exciting\! More on that week in a bit.
Training was going well. I was enjoying my internship and enjoying my running. I was hittings the mileage hard to have a great last NIRCA cross country season. What happened?
At the beginning of the summer training cycle I started using new shoes for some of my runs. The Saucony Mirage 2, I actually really love that shoe and it took a beating this summer and was still in great shape.
There were a few problems though, mostly my fault. The first was that I probably ramped up the amount of mileage I was doing in the new shoe too quickly, especially since it’s way less of a shoe than my normal super supportive Asics. Next, while I wasn’t doing a lot of running on the San Francisco hills, I WAS walking up and down an extremely steep hill to get to the bus each day. One of the steepest in the city, I’m told. While this isn’t really a big deal, it probably didn’t help my already too-tight calves. Lastly, I started playing weekly games of Ultimate after work with some other interns. While that was a lot of fun, I played barefoot much of the time and I think that also contributed to my situation.
In the end, it all culminated into an injury. It was relatively minor at first but progressed to the point where I couldn’t put any weight on my foot in the morning before I finally accepted that I needed to stop running. Plantar Fasciitis. Fun times. I’ve learned my lesson as I’m still dealing with it.
Over the course of my summer in California I met some great people and had some fantastic adventures, all thanks to running\!
I was convinced to go race [Table Rock](http://insidetrail.com/ai1ec_event/la-sportiva-table-rock/) with new friends who were into trail running. I did the 10k while they ran the 27k. This just happened to be at the end of my first 80 mile week.
It was a great race and I had a lot of fun. Going in I knew that there was a crazy hill at the beginning of the race but I was confident that I could compete for the win - and the possibility of a course record. For context, nobody has gone under 60 minutes on the 10k course. The winner was able to set a new record of 60:07 which was a few minutes faster than the previous record. I didn’t really know what I was getting into but still managed to eek out a 68:26 which was good enough for 4th place after I took two wrong turns. What an introduction to trail racing that was\!
As far as racing goes, that was the only serious race of the summer for me - and turned out to be one of the few races I ran seriously for months after.
I did manage to have a few other running related adventures. On a whim, I agreed to pace and crew for my friend Lucas for part of his first 100 miler since one of his other pacers had to back out last minute. This was an interesting experience for me and Lucas ran a great race (and won)\! I had a lot of fun and this was my first introduction to ultras.
I could write much more about that day but I’ll just say that running on the trails in the Marin headlands was simply fantastic and running through those hills with no light but a headlamp was both eery and peaceful at the same time. one of the fantastic views during the SF 100 miler
Later in the summer that same group of us went to Alaska and did some running. Through the woods and in the mountains and even up [Exit Glacier](http://www.nps.gov/kefj/planyourvisit/exit-glacier.htm).
Exit Glacier\!
This trip was essentially the end of my summer running adventures. After this point I started trying to rehab my foot a bit because I realized that I would have not even a small part of my cross country season if I didn’t attempt to take care of it.
I started taking days off and biking instead. I started stretching more. However when I got back to Ann Arbor I got caught up in getting in shape for cross country again. I began running higher mileage again, did a trail half marathon for fun, and did the first track workout of the year. The next day I couldn’t put weight on my foot - I had to accept that it was time to take more drastic measures.
I eventually started going to physical therapy. Through I wasn’t told I had to take off completely, I took about 3 weeks off and then began running maybe 20 miles a week again. This didn’t completely heal my foot but I was able to run. It wouldn’t hurt while running but still got rather tight after runs. Uneven ground such as trails and grass was even worse than flat pavement.
In the end, my lesson was that I didn’t listen to my body when it was hurting and then was too stubborn to do what was right. As a result I missed out on both cross country and track. I ran some races but was doing them non-competitively. Thankfully, now I’m well on my way to recovery.
All said, by the end of 2013 I had run 2246.9 miles. To reflect a bit on
Some of my secondary goals were a sub-36 10k, 3000 miles for the year, and to finish a marathon. None of these were completed either. I was on pace for 3000 miles until I had to reduce training volume in the late summer. Hopefully in 2014 I can run a few more total miles for the year than in 2013\!
This week when I was at Sweetwaters (a common occurrence), I tried some Sencha tea as it was their tea of the day. I liked it. I looked up more information about it and apparently the taste varies based on the temperature of water used to steep it which is really amazing to me.
P.S. I’ve been really busy with school but I have some running related posts from my adventures this summer queued up to go out soon\!
This weekend was finally the day I’d been training almost six months for: the Bayshore half in Traverse City\!
I began base building at the end of December after taking time off from Fall cross country / road races. I was consistently hitting 50 and 60 mile weeks with a weekly tempo run. By mid February, I was consistently hitting 70 mile weeks with down weeks in the mid 60 range usually. At that point I mostly transitioned from tempos to weekly indoor track workouts with MRun because running workouts with people is better. I maintained that mileage through outdoor track season which I competed in, and then just a couple more quality weeks before getting to Bayshore. The week leading up to Bayshore was probably among the worst of my training weeks ever - I never felt fresh or healthy and the small nagging injuries didn’t seem so small anymore.
[You can view my training long for more details.](http://openrunlog.org/u/david)
We woke up early. Very early. The shuttle from our campsite to the race’s finish area left at 5am so I was up at around 4:30. I ate a poptart when I woke up and then about an hour before the race had a granola bar. When we got to the race’s packet-pickup/staging area we then had to take another shuttle out along the bay to the half marathon’s starting point (it’s a point to point race). It was a pretty cold morning and we got to the start line too early to start warming up so it was kind of cold standing around. When it was FINALLY time to start warming up I just did some light jogging for about 10 minutes and then changed into my flats and did just a few strides to make sure they were tied tight enough. When I got to the start line I finally saw the rest of my friends on MRun\!
The start was a bit weird since they had to synchronize with the marathon that started 15 minutes earlier and then we were off\! The first mile I was left wondering if I was really on pace since it felt so easy - I was right on pace but almost none of my training miles felt quite so easy as that.
For about the first half of the race I was right on target of staying between 6:00 and 6:10ish with splits of 6:05, 6:08, 6:06, 6:11, 6:14 through mile 5.
About 5.5 miles in I started slowing down a bit and it also happened to be about the same time that I got stuck in no-man’s land with nobody to run with. It wasn’t that I was dead or even felt that tired but my legs were heavy and I didn’t know how I should be feeling and was worried about crashing later on at mile 10 or 11 so I just tried to keep my effort up. The number of "Go Blue\!’s" that I got from wearing my MRun singlet from both spectators and marathoners once we crossed paths with them was astounding and helped me keep going. I even got a "Go Blue\!" from the 3rd place marathoner\!
Splits for miles 6-12: 6:31, 6:29, 6:29, 6:43, 6:46, 6:42, 6:41
I wouldn’t say that I ever felt good during this time but I never felt exhausted either. I guess I’d say I felt pleasantly exerted. After how I felt for the last 1.1 miles and the day after, I now know that I could have probably pushed these miles much more.
Right at the 12 mile mark my friend Caspar caught up with me. We picked it up for the last mile and I told him we were going to catch the guy that was up in front of us (who had passed me earlier). My 13th mile was my fastest mile with a split of 6:04. After I picked it up for that mile I knew that I had been going to slow for the second half of the race - I felt like I could have hung on at that pace for another few miles at the finish.
I ended up finishing with a time of 1:23:57. It wasn’t quite in my A-B goal range of 1:18-1:20 but I’m not disappointed with it either. My fallback goal was faster than 1:25 and I did just that. In any case, it left me wanting more.
I’d say that I ran the first half of the race very well. I went out conservatively and felt pretty good because of it. I now know that I could have hung on at that pace for the rest of the race but this one was a learning experience. A number of friends told me they dropped about 5 minutes between their first and second time racing a half. That’s exactly what I intend to do next time I do one - even though I don’t quite know when that might be. Some people are trying to convince me to do the Detroit half in the Fall but I may just want to focus on cross country then instead since it’s my last chance to race cross country with MRun.
I ended up finishing in 28th place overall and 3rd in my age group. I’m reasonably happy with my result but not completely satisfied. I just keep reminding myself that I’m still less than a year in to training competitively again and that I don’t have that aerobic base from years of running high mileage in anymore. I’ll improve with time and patience.
After the race we hung around downtown Traverse City for the day and even went to the beach\! It was a fun day involving pie and other delicious food. The day after the race we packed up camp and headed home. When I got home I did what I planned to be an easy 5 miles for recovery around argo (which just so happens to be my last run in Ann Arbor until I get back there from San Francisco in the Fall). It started off nice and easy but I ended up picking up the pace for every mile with the last two miles in the 7:00 to 6:50 range. I guess I felt pretty good by the end\! Afterwords I felt well enough to go play a couple hours worth of volleyball and frisbee with some friends (and even biked there). It was fun but was also yet another sign that I could have run harder during the race. I just never got that sore after the race - in fact I felt better immediately after the race than I did during most weeks of training towards the end of the cycle.
I’m now going to take a week off of running (maybe try to squeeze in a long bike ride or two before I fly to California on Friday) and then the next week will just be easy running every other day. After that, I’ll jump right back into training\! It will be time to start thinking about cross country season and I intend on taking full advantage of summer training\! I’m hoping to run more than I ever have with a peak of high 80s or low 90 mpw to try and have the best season possible.
Now that we’re leaving the cold months and entering the final stretch of my training for the Bayshore half, I thought I’d take a look at my winter training.
\[caption id="attachment\_425" align="alignnone" width="698"\]<http://davidwilemski.com/blog/wp-content/uploads/2013/04/Open-Run-Log-Dashboard-1.png> My weekly mileage since late Fall. I’m very pleased with my ramp-up after some rest. That most recent week is still in progress and should be in the 55-60 mile range...\[/caption\]
Early in the winter through early March I got some very solid base miles in with a few encouraging early tempo runs. I’ve managed to keep my weekly milage above 50 for every week of this year with the average closer to 60 miles per week. So far I have only taken one day of for the year. I also managed to hit a new weekly high of 75 miles a few weeks ago.[
](http://davidwilemski.com/blog/wp-content/uploads/2013/04/Open-Run-Log-Dashboard-1.png)
I had been trying to avoid racing for a while but sometimes it’s just too hard to resist. I ran the [Ice Cube 10k](http://davidwilemski.com/blog/2013/02/ice-cube-10k-race-report/ "Ice Cube 10k Race Report") in February and then the Chili-Heart 5k later that month. It was early in the year still and I mostly used those races as tempo efforts. I ended up coming in 1st place for both of those races and won some cool trophies:\[gallery ids="426,415"\]
Finally, I have been putting in some good workouts with MRun through the cold weather on the indoor track. We recently moved back to the outdoor track and I could not be happier - as nice as it was to do speedwork all winter, it was rough on my hips and legs.
I also was fortunate enough to be able to participate in Race Across Michigan with MRun which was a 24 hour relay that we held that started in Detroit and ended in South Haven that we used as a fundraiser for the special olympics. It was an incredible experience and worth the lack of sleep and many miles run - I ran 18 miles across 3 runs that day including a 13 miler. Some others ran even more than that breaking 20 or
The past two weekends I raced some track meets with MRun racing the 5k. Neither of those races went particularly well but they were my first two times racing a 5k on the track. I ran 17:56 at our meet with MSU and then 17:49 this past weekend in Indiana at NIRCA track nationals. I also ran a 4:52 1500m after my 5k at NIRCA but that was more for fun and I didn’t even bother wearing my flats for the race. I went out in a 70 second 400 and just decided to hang on and see what I could do after going out hard (on purpose).
What’s next? The next two weekends, I will race twice more. This coming weekend I’ll race the Martian 10k which was a race I also did last year when I was barely in shape - this year I hope to go about 36:30s or even sub 36 if things go well. I’ll use it as an evaluation for my half marathon fitness since that will be just under half marathon pace - if I can’t hold that pace now then there are some serious issues with my half marathon goal. The following weekend I’ll race another 5000m on the track at EMU with MRun. It’s a varsity track meet but I’ve been told that it’s still possible that we’re competitive. Either way, it should be a cool experience and I hope to finally break that 17 minute barrier in my 5k.
After those two races, I’ll refocus on training for my half. I’ll have 5 weeks until Bayshore and will be getting rid of much of the speedwork and transitioning back into tempo runs and workouts at half marathon goal pace. I’d like to do a few more 4-6 mile tempos at pace and one time do a 3x20 minute workout at goal pace (6 minute pace). If I do well in that workout then I’ll be very confident in my goal for Bayshore and if not then maybe I’ll start out the race with a pace 5-10 seconds slower than that and try to hold that and maybe work down to a faster pace if I feel good later in the race...
That was my winter in a nutshell\! I’ll be passing 1000 miles for the year in another week or two which puts me in a very good place to hit my goal of 3000 miles for the year\!
Today I drove up to and ran the Ice Cube 10k with some fellow a2runners (they actually did the half). The highlights are that I took first overall with a time of 38:12. Also, I got a really awesome race award. It was handmade and now I have a collection of unique race awards (including the one from [my first win](http://davidwilemski.com/blog/2012/10/summerfall-recap/ "Summer/Fall Recap"))\!
This is my second ever race win and my first race of the year - looks like things are off to a great start\!
If you want to read more details, I’ve posted a more in-depth report here: http://openrunlog.org/u/david/run/51201295f69d202ebd0e09c3
\[gallery ids="415,418,417,416,414"\]
Since getting some extra time over winter break I was able to dig up a project from earlier this year. I’ve posted a few previews of my progress on Twitter over the past few days - but today I’m ready to open up the site for others to test and use.
Open Run Log is a website that I built to use as a training log. I’ve built in some auto-generated graphs and have ideas for other things in the future like goal setting and tracking, public profile pages and sharing, custom graphs, or other fun things. It is at a point where I felt comfortable switching to it for tracking my runs and so today I imported my old training log into the site so that I didn’t lose that data. It is only going to improve from here on out and I’m very excited by the possibilities\! (Yes, I’m a data nerd when it comes to my running log.)
![year](http://openrunlog.org)
So why not use one of the many other run tracking websites? Because I didn’t want to miss the chance to mix my two passions (running and coding). Also I was excited about all the different ways in which I could analyze run data in the future so having control over the code and data was important to me - and I wanted to give that freedom to others as well which is why [I made the source code available](https://github.com/davidwilemski/openrunlog). I think that projects like WordPress and ThinkUp provide great software and have a great model of making both the software and code available to anyone interested and I wanted to do the same.
If you give it a try, be sure to let me know what you think\!
I wanted to take some time to reflect on my year of racing, it was prompted by me finding the data for this tweet:
\[tweet https://twitter.com/davidwilemski/status/275343866285142016\]
(and, yes, pun partially intended)
This is mostly going to focus on late Summer through Fall since that’s when I was the most dedicated and began to start reaching for goals I’ve had for a long time. Who doesn’t love data?
First, let’s look at those 18 races: <span style="color: #ff6600;">Color=PR</span>
Date Race Time Pace
1/5/2012 Super 5k 21:44:00 7:00:38
<span style="color: #ff6600;">2/29/2012 Leap Year 4 Miler 26:51:00 6:42:45</span>
3/11/2012 Shamrocks & Shenanigans 5k 20:20:00 6:33:32
3/24/2012 Passion for life 5k 20:54:00 6:44:30
<span style="color: #ff6600;">4/14/2012 Martian 10k 42:53:00 6:55:00</span>
<span style="color: #ff6600;">5/26/2012 Bayshore 10k 42:06:00 6:47:25</span>
<span style="color: #ff6600;">8/11/2012 Run Thru Hell 10 miler 1:15:20 7:32</span>
8/26/2012 Run For the Corn 5k 19:28:00 6:16:46
9/16/2012 Run Wild for the Detroit Zoo 5k 18:18:00 5:54:11
9/22/2012 Michigan Fall Classic 5k 18:54:00 6:05:48
<span style="color: #ff6600;">9/29/2012 Sean Earl Lakefront Invitational 8k 31:05:00 6:15:15</span>
10/6/2012 Run Scream Run 5k 17:57:00 5:47:25
10/14/2012 MSU Spartan Grand Classic 5k 18:47:00 6:03:32
10/27/2012 headless horseman 5k 17:19:00 5:35:09
11/10/2012 A2 Turkey Trot 5k 18:23:00 5:55:48
<span style="color: #ff6600;">11/13/2012 Wayne lightfest 8k 30:15:00 6:05:11</span>
<span style="color: #ff6600;">11/22/2012 Detroit Turkey Trot 10k 37:34:00 6:03:32</span>
<span style="color: #ff6600;">12/1/2012 Holiday Hustle 5k 17:12:00 5:32:54</span>
That last PR is the one I’m most proud of both because it has been a goal to improve my 5k since high school and because it compares most favorably against my other PRs. (It’s my "best" race)
Additionally, I ran a number of new race distances this year. I ran my first 4 mile race - but was just getting back in shape then and that distance is not too common. I ran my first three 10k races (and improved on each one - arriving at a PR I’m pleased with for the time being). I also ran my first two 8k races, one of them being during cross country season with MRun. Lastly, I ran a 10 miler. That race, of all of these races, was by far the most difficult. I clearly was not prepared for a longer distance race at the time and the soreness I had for days afterwards reminded me of that.
I only really got back into training at (what I would consider) a high level in the end of August. Noting that, I can be nothing but pleased with what I was able to accomplish this year. Additionally, it makes me extremely excited for what I can accomplish with a full year of quality training during this coming year.
Now comes the part where I set goals for next year. I have 3 main goals and an additional set of milestones that I think will happen if I’m able to accomplish my goals.
Goals:
Other things I think are achievable on my way to meeting these goals:
The goal that will make me happiest is the sub-17 5k but I think that the combination of all of these things are possible on my way there. I feel that I am finally improving as a runner again, picking up were I left off 4 years ago. I want to see how far I can take this.
Yesterday I ran the [Holiday Hustle](http://holiday5k.runningfitsites.com/) in Dexter.
I’m just going to say that i think the mile markers were placed a bit off but am pretty certain the course was still 3.1 based on a few different measurements. Given that, I think that my mile splits were something like this: 5:35 or maybe a little faster, 5:45ish, 5:30. I wish I knew exactly what they were.
This course was deceptively hilly and tough. There were lots of rolling hills and there was a pretty long hill right before mile 2 that was very rough. I’ll admit that for a little while around that 2 mile marker I was thinking that this PR wasn’t going to happen. I just kept telling myself to stick with the people near me and not let them get to far. To just to hang on. It Worked.
I finished at 17:12 which is 5 seconds faster than my previous best set
[Full results here.](http://www.timing.runningfitsites.com/raceresults.php?RaceID=284)
I’m now done racing hard for the rest of this year. I plan on taking about a week off entirely after this morning’s long run. After that I’ll start training again for indoor and outdoor track with MRun this winter and spring. I also signed up for the [Bayshore Half-Marathon](http://www.bayshoremarathon.org/half-marathon) yesterday morning (it sold out in less than 2 hours)\! This will be my first half and the longest race I’ve ever done. The distance doesn’t scare me too much because I’ve done 12-14 mile long runs regularly but racing the distance is a whole other deal. I’ve got time to train for that thankfully.
I’m 3/3 in PRs in my last three races and that’s awesome\! I’m finally faster than high school me from four years ago\!
Last Thursday on Thanksgiving I raced a 10k at the Detroit Turkey Trot. This was a big day for me because it was my first 10k encounter since Bayshore in the summer. I knew I had a chance for a major PR because I am in much better shape now that I was then.
Going into the race, my goal was to hit 6 minute pace for as long as possible and then just see if I could go faster for the last mile or so. It turns out most of that worked out. I missed all of the mile markers until mile 5 so for most of the race I had no idea how fast I was going but was just focusing on keeping things fast but smooth and not pushing too hard. After the 1st mile or so where I had moved up though the pack I found a group doing about my pace and just sort of tucked in. I crossed the 5 mile mark at 29:55 and was extremely happy to see that everything was going according to plan.
The last 1.2 was a bit rough it turns out but I still finished in 37:34 which is a huge PR of almost four and a half minutes\! Awesome.
Then it was time to go chow down on some delicious Thanksgiving day food. The Turkey Trot was a great start to the day though, even if I did have to wake up at 4:30am.
Back in mid-November I did the Lightfest 8k. This wasn’t \really\ a race since it wasn’t timed and also the course was longer than 8k, which I’ll get to in a second. Additionally, we had trouble finding the right place to park so we were late to the start and didn’t warm up.
Despite all that I decided to run fastish and see what I could do. The first 4 miles were Between 6:05 and 6:20 but I missed most of the mile markers so I don’t know exactly what they were. I was okay with this - I just enjoyed the course (it was fantastic to see the light displays) and also kind of had fun weaving through people playing catchup to the runners.
My last mile (yes, full mile. In an 8k.) was 5:10. It turned out that after 4 miles I was warmed up and realized that if I ran fast I could PR my 8k despite all the setbacks I mentioned above. It turns out the course passed the 8k mark, had a 5 mile mark, and then finished just past there. Based on extrapolation for 5mile-\>8k time I put myself at 30:15 which is a 50 second PR in my 8k\! Win\!
In recent posts I’ve hinted at a new project and I’m ready to show off the first version:
[![coursegrab screenshot](http://coursegrab.com/static/screen1.png "coursegrab screenshot")](http://coursegrab.com)Coursegrab is a little project that I’ve been wanting to build for a while now. The idea is simple: My university doesn’t provide waitlists for all classes so that students can automatically be enrolled in a class when space opens up which means students are always checking the website for class status.
This service will send a text message when a class you want opens up. No more checking whether you can get into that class you want to take\!
Freedom\! I finally have some time where I don’t have to work on class projects for a little bit. This is fantastic because I’ve been pretty stressed under the project load lately - this means more time for running, work, and side projects\!
Blargh. This race didn’t turn out to be what I had hoped. I’m not pleased by this but the only thing to do is put this one behind me.
Mile 1: I went out with the chase pack, we hit just below 5:20 and I was pleased with that, right on target instead of blasting the first mile like I did last race. But the first mile is always easy.
My thoughts the rest of the race: "Why am I getting passed? ... Just keep going ... Ouch, why was that 2 mile so slow?"
Mile 2 came by in right around 11:15 or 11:20. I ran a 6 flat 2nd mile and that right there was the race for me. The target of 11 or under wasn’t even close. Mile 3 was even worse. I finished in 18:23.
Since today doesn’t qualify as a good race, I’m now searching for a way to fix that. I was planning on running the Lightfest 8k in Livonia real easy just for fun this coming Tuesday night, now maybe I’ll use it as a tempo run and PR my 8k - should be do able since the only 8k I’ve run was on a cross country course that was hilly.
I thought doing easy running to maintain fitness for the next few weeks was a good idea but now I’m leaning the other way. 2-3 more weeks of hard training while I get through the Detroit Turkey Trot and Holiday Hustle seems like a good plan.
My hosting company had to fully reset my web server this week due to a breach on their side. Thankfully, I had this blog fully backed up thanks to the wonderful PressBackup plugin, but it was annoying to have to set everything else back up again - and I may have lost some other minor stuff.
It just serves as a good reminder to backup your data regularly\!
I’d like to build something using only Redis for persistence. Its new scripting interface seems really cool. I have ideas, just need to get out of this busy project/exam filled time... maybe Thanksgiving break will give me enough time.
UMich Students: Stay tuned.
TL;DR: I ran a 17:19 5k last night - just 2 seconds from my PR\!
Well I haven’t posted in a while but I’ve been running really well. I’m going to try to start posting more again but to do that I first need to write up everything since Bayshore... here we go - this could be a doozie.
After Bayshore, I took some time off of. Something like 10 days of absolutely no running. While that was difficult, it turned out to be worth it because I was able to come back with some good hard training.
\[caption id="attachment\_343" align="alignnone" width="603" caption="Weekly mileage since Bayshore - Races marked in blue"\]<http://davidwilemski.com/blog/wp-content/uploads/2012/10/weeklymileagepostbayshore.png>\[/caption\]
That graph basically sums up my training for the past few months. I spent some time after Bayshore not racing at all, just slowly building up my mileage again into the 40MPW range.
I ran the Run Thru Hell 10 miler - not because I was ready for it but mostly just for fun. I had run the shorter race there before so wanted to do the 10 miler too. I went with a group of friends and it was a great morning. It was a really tough course and I was really sore afterwards due to all the hills. Hills, as we’ll find out later, are my downfall in races.
Towards the end of summer I had trouble keeping decent mileage due to vacation and other trips but still managed to get some okay running in (this means average runs and usually a weekly longer run of 8ish miles). After Run Thru Hell, I didn’t plan on racing until September at the Detroit Zoo. The dip after the 10 miler was due to a hiking trip I took for a week but I’ll save that for another post. The day after I came back, I ended up doing a pretty hard workout with a friend and found out that even though I had done almost no running in the week leading up to that, it turns out hiking 40 miles in a week keeps you in decent shape. That workout is essentially the start of a really REALLY rapid increase in mileage that was both risky and rewarding.
The only reason this is notable is because it is my first time ever winning a race. I was back home for a weekend and ended up going to this race and I was planning on treating it as a tempo run. This was still just days after getting back from hiking. I showed up to this race and realized it was going to be small - that’s fine it was still just a tempo. There ended up being maybe only 15 people at the race - that’s the smallest 5k race I’ve ever seen by far but I really enjoyed the course. I took off at a decent but not very fast pace and was immediately all alone. It was just me and the lead golf cart the entire time which was a pretty cool experience. I ended up running a 19:28 for the win and got this cool wooden cut out thing.
<http://davidwilemski.com/blog/wp-content/uploads/2012/10/2012-08-26-09.33.53small.jpeg>
So that was pretty cool. I have this hanging on my wall now.
I’m not going to give much information about the other races in the interest of keeping this readable and interesting. In September I started running with MRun, Michigan’s running club, a couple of times a week - mostly on Tuesday nights for their workouts. I also have gone to a few cross country races with them - which have been fun but not fast for me. (This is where the hills come in). I also upped my mileage from about 40 a week to 70 a week in about a month - that came back to bite me but I’m also convinced it helped a lot.
So that brings me through all the fall races up until yesterday and by this point I’m at 70+ miles a week. All of my runs began feeling sluggish and my legs were always tired, never recovering. Last Thursday was basically when I realized this cannot continue. I ran one of the most difficult workouts of my life - 16x400m intervals at 82 seconds each, 200m rest - and was already exhausted going into it. The early repeats were never easy even though 82 seconds isn’t exactly a fast 400m for me right now (I’m pretty sure I could go sub 5 in a mile right now). I did hit that pace for each of the 16 repeats but boy was it hard. I ended up really down about running for most of last week. I had decided to take the week realllllly easy and see if I couldn’t recover. Maybe 50 miles at most. Even still, most of my runs early this week felt terrible and my legs were just burnt out. Slow runs felt fast and I never felt like I could go faster. I ended up taking this Thursday off and only running an easy 4 miles on Friday. This brings me to last night’s race.
I felt okayish for the first time in over a week after Friday’s run. I was beginning to feel confident that I could run fast again. I had looked at previous results for this race and saw that in the past 3 years the winning times were between 16:50ish and 18 flat (last year was 18:00). I began to get it in my head that I could win this race.
I ended up planning on going out with whoever was in the lead for the first mile and see what they were going to do. My plan was to run to win and not worry about PRing because I didn’t think I had quite fully recovered yet (turns out I had).
This race was won in 15:59 which is 5:09/mile pace. Guess what my first mile was? Yup. All I remember thinking after that first mile was that it had to be short, no way I just ran 5:09. Turns out I did. So the winner ended up out of sight pretty quickly and it was just me and someone else in 2 and 3, with another runner not too far behind in 4th and no one else even close. I was trying to keep the pace up but ended up getting passed right before the 2 mile mark. So now I was in 3rd trying to keep 2nd in view and I went through the 2nd mile in about 5:45. At this point my only concern was not dying completely because I knew that I had a shot at a really good time for myself. I struggled to keep the pace up in the last mile but knowing I had finished the first 2 miles in just under 11 minutes kept me going - I knew I had a shot at PRing\! I ended up getting passed again right before the final turn and I just had no response at this point - 4th place for me it was. I was just trying to get to the finish. There was a nice long straight portion right before the finish which was really great. I could see the clock from a long way off and it was still under 17 minutes. I put in everything I had but still watched the clock tick up. I was feet away from the line and saw it tick past my PR of 17:17, it hurts that I was so close but now I am 100% convinced that I have that PR in me somewhere. I finished in 4th at 17:19 with 2nd and 3rd at 17:14 and 17:15 I believe.
[The race results are online.](http://www.everalracemgt.com/results/full.php?2012/headless-horseman-5k-4.html)
I’ve already signed up for the Detroit Turkey Trot 10k on Thanksgiving and I’m planning on doing another 5k the weekend after that - where I’ll hopefully finally PR.  Until those races, I think I’ll go back to maybe mid-60 miles a week and just play it by ear. I don’t want to burn myself out again. After those races are over I’m going to call it quits on this cycle of training. I’ll take some time off for some much needed rest and then just base build for the winter and not worry about speedwork for a while. I’d like to see what a 100 mile week is like so I may try building up to that very slowly over the winter.
I spent most of this weekend camping up in Traverse City with a group of friends. We were there for the [Bayshore races](http://www.bayshoremarathon.org/). Most of the people I was with had spent months preparing for their race this weekend - and I was sort of a last minute addition. Everyone had an awesome race and PR’d.
I was there to run the 10k and going in I honestly wasn’t very confident because I’ve pretty much been in some state of injured since my [last race](http://davidwilemski.com/blog/2012/04/race-report-martian-10k-2012/ "Race Report: Martian 10K 2012"). I took about 10 days off towards the beginning of May and had only been running again for about 2 weeks when we got up there and most of the running since Martian has been pretty easy miles, nothing hard. Additionally the only speedwork I’ve done was crammed into the past 2 weeks with the group and neither workouts went amazingly well.
My plan going into the race was to run 7 min pace for the first 5k and then see how I was feeling and maybe pick it up for the second 5k. My thinking was that if I had any chance of running a good race I would need to push hard in the second half. Things didn’t go quite as planned
The first mile I went out focusing on running easy and keeping a smooth pace. I ended up going through the mile marker in 6:22. Needless to say that was a bit of a surprise. I’m used to going out fast but I made an extra effort to go out easy. I focused on slowing down the next few miles. Miles 2 and 3 were at 6:45 and 6:58 respectively.
Somewhere in between mile 3 and 4 I just sort of zoned out and cruised - that ended in a mile pace of 7:19. It was definitely a bit slow but I didn’t end up too worried because I knew I was still averaging under 7 minute pace because of the fast first 2 miles. The 5th mile I focused on getting back in stride - 7 flat.
For all of mile 6 I focused on going as fast as possible, to really push it. I kept telling myself I just had a mile left. I was reeling some people in and that helped me stay on pace.
I finished in 42:06 which was a pretty good PR over Martian. I was extremely surprised because like I said, I feel I was way more prepared for that race than this one. I somehow kept everything together and had a solid race here. It definitely excites me that I’m improving. [Full race results here](http://www.timing.runningfitsites.com/raceresults.php?RaceID=97).
I’m completely focused on getting fast again at this point. I’ll do anything I can to get closer to that goal. The first step in that process is to take two weeks off in an attempt to shake off  the nagging injuries in my foot. I’ll reevaluate at that point but as long as everything goes well I’ll start building a nice base. I want to take things nice and slow and not build up too quickly and end up with injuries like this cycle.
The end goal is to start racing again in the Fall and hopefully start on the long road to a 5k PR.
Saturday morning I finally ran my first 10K\! I was out the door by 6:30 to get to Dearborn for the race that started at 8. It was an early morning to be sure but that’s always fine if it’s for running. I was pretty conflicted over whether to eat anything when I woke at 6 because it was right in the span of time where I may or may not regret but also I was unsure of if I would need it because this race is longer than I’m used to (normally I wouldn’t need to eat for a 5k). I didn’t end up eating and we got to the race in time for me to get ready with a bathroom visit and a bit of a warmup (only a mile, would have liked to do a bit more but it worked out nicely).
A few minutes before the race I lined up at what I thought was not too far back but it turned out I got boxed in after the start. This happens too much - other runners really don’t consider where they should be based on their starting time. I was doing a slow jog for probably almost a minute before I could get through the crowd at all and then really pushed for the first mile to make up time.
I went into the race planning on running just under 7 minute pace for each mile and had decided if I could do that then I’d be at least satisfied with what I ran. The first mile turned out being around 6:45 between the adrenaline of the start and me being angry and running hard to make up time from the start.
The next few miles went by easily. The pace felt pretty relaxed for me compared to my usual constant exhaustion for a 5k. The second and third miles I ran in 6:53 and 6:55 respectively - right on pace\! Additionally I really didn’t feel too fatigued after this which was part of the plan. My breathing was easy and my legs felt good. It turned out that miles four and five were a little more difficult but not so much that I noticed. Either I completely missed the 4 mile mark or it wasn’t up yet but I caught the 5 mile mark. They were actually just setting up that mile marker as I passed it and I was no where near the front of the pack so this was pretty surprising to me. I averaged 7:01 for those two miles and went into the last mile ready to push a bit harder.
Now the last mile was actually more difficult but still wasn’t bad - I guess this means I could have raced faster but I was just sticking to the plan. There were a few good sized up hill portions towards the end of the race and those were rough but otherwise the last 1.2 miles felt good because I knew I was almost done. The last turn was made right around the 6 mile mark which gave a straightaway of about 200m. This I really liked because it gave you time to see the finish coming and really push. The last 1.2 miles I finished in about 8:22 which is under
I finished in a total of 42:53 and I’m happy with that. It is definitely something I can improve on but it’s a good starting place I think. Overall it was a good day to race. It was a bit rainy early on but that cleared pretty quickly and the course was pretty nice. Like I said, there were a few rough uphills but otherwise I think the course was pretty fast. It was winding and there was enough room on the course (granted I was only around 3 or 4 other runners for most of the race).
Full results can be found here.
The rest of Saturday I was limping around and my right leg was bothering me quite a bit when I put weight on it. Sunday I felt a bit better and even went for an easy 5 mile run with some friends and it didn’t bother me too much but I could definitely still feel that my leg wasn’t 100%. Monday was worse again and I even cut my run short (4 miles instead of 5) because I didn’t want to aggravate it any more than I already had.
I’m going to take today and tomorrow off and ice a bit before I reevaluate what to do but I’m really hoping things get better. It kills me not to be able to run.
I was asked to speak at a WordPress workshop this week hosted by Lyndsay of MeadowFete. I think the workshop went really nicely but it doesn’t matter what I think - it matters what the attendes think. So hopefully they got a lot out of it.
Anyhow, I’ve made the [slides available here.](http://davidwilemski.com/blog/wp-content/uploads/2012/02/CreateYourWPWorkshopSecurityTalk.pdf)
Get in touch if you have questions or comments - I’m happy to help however I can\!
The other week I sort of decided to run this 5k at the last minute. The course was almost entirely flat and overall organized pretty well. The downside was that I didn’t improve on my recent 5k times at all - I guess things weren’t as smooth as I thought last post. Anyhow, I finished with a 20:54 which somehow got me the age group win as well. I was actually very disappointed with my time and really felt like I shouldn’t have won the age group either - especially since the course was so flat. Full results can be found here:Â <http://runmichigan.com/r_view.php?id=15130>
Since then it has been training as usual. Last Friday I actually ran the first hard track workout that I’ve done in a while with some friends. We did 4x5:00 hard with 3:00 rest in between and then 3x200m hard as well. This workout was actually very encouraging for me since all of my laps on the track were very even and I was right on pace for 6 min miles all the way though. So maybe in the past that would have been nothing for me, for now I’ll take it\!
I was very sore over the weekend following that workout (I forgot how much speedwork hurts\!) but managed to get over it by Monday and then yesterday I was able to run a 12 mile long run which turned out to be right at 8 min pace average. This was another hugely comforting workout and it turned out today I felt pretty good on my [group run](http://www.meetup.com/A2Runners/events/58450522/).
I’m getting excited for the Martian 10K - it’s just about a week and a half away\!
I’ve had an inkling of this thought a few times this week so I thought I’d spend some time fleshing it out.
I’m starting to actually feel like a runner again. I’m finally hitting some pretty decent mileage this week and feeling good while doing it. I’m not waking up every morning feeling like I ran into a wall out of fatigue from the previous day’s run. Even in this odd March heat we’ve been having I’ve been putting some good (longer) runs in.
I’m planning on racing tomorrow and I’m pretty excited for that. I’m going to hit over 40 miles this week (probably more like 45) for the first time probably since high school. I’m building my mileage base for this summer’s plans. Things are going well\!
I know the ’hit a wall’ feeling will come back once I’ve actually upped my milage to 60 or 70+ MPW but all in good time. Also I know my average mile pace for training runs could be up and that’s something to focus on soon too. For now I’m just gonna enjoy the fact that I’m feeling great about my running. This is definitely good motivation to keep my training going and not let it fizzle out like in the past.
I’ve had reports that the CAPTCHA wasn’t working - therefore I disabled it. I really dislike filling them in and know others do too, so maybe this will help discussion?
This morning was the Shamrocks & Shenanigans 5k hosted by Running Fit. It was really a fun morning and they did a great job hosting this race. This really was a big event - I heard estimates that about 3000 people participated. (UPDATE: There were 2113 finishers)Â The final results haven’t been posted yet for me to get an actual count but I’ll update this post when they are (This also means I don’t have my official time yet but I’ve got my watch time).
Before the race I met up with some friends from the Running Fit Monday run group. We hung around the race area a bit afterwards as there was a lot going on and it got really nice outside after the race. The weather today is beautiful and I almost want to go for another run\!
I have mixed feeling about the actual race but this is more an indication that my training is lacking than anything else. My goal was to go under 20 mins today and I think that was within reach. It didn’t quite happen but I am showing progress in terms of average pace from the last 2 races. Â You gotta focus on the positives, right?
I mostly succeeded in going out at the right pace. The first mile was a loop around downtown and then sent us down Main St. My first mile was 6:17 which I was pleased with at the time. The second mile really hit me hard it seems. We continued out on Main and turned around right at Stadium and came back down Main. There are a few rolling hills on this portion and I started slowing way down. My second mile was right at 7 flat and now I wish I would have done more to push harder but I know that during the race I was doing all I could to stick with the pack I was in. The last mile I was able to pull it together again get back into the race. This part brought us the rest of the way back down Main and for a set of loops around downtown to the finish. There was a hill immediately before the finish that I got destroyed by - but otherwise this last stretch was good. For the third mile I had a time of 6:19 and the last tenth I ran 0:43 (like I said, hill destroyed me). This got me a finishing time of 20:20. It wasn’t exactly what I was looking for but it IS an improvement from the past few months...
I really need to figure out what happened during that middle mile. If I could have held on a little better in the middle of this race then my goal time could have actually been met since I was able to actually have pretty consistent 1st and 3rd mile splits. I didn’t do any speedwork this week since I was already upping my mileage (I hit 40 miles this week) and I’ve already been worried about injury so I didn’t push it.
As I said after last race, I think mile repeats are going to be essential for me to smooth out the difference between these splits. I just need to find a good day to do them. I’m thinking that Tuesdays are going to be the day. This would be right between a probable long mileage day on Monday with the Running Fit group and the Wednesday A2Runners group that is generally an easy short run that could be good for recovery.
Overall, this was a great day and I will definitely be racing this again\! There was lots of fun to be had and the course was pretty nice - except for the hill/turn right at the finish.
Update:[Results are posted.](http://www.rftiming.net/index.php?option=com_content&view=article&id=36&Itemid=47)
Yesterday I \[grudgingly\] took a day off of running. After waking up this morning I’m not sure that it really made a difference but we shall see tonight when I run. This was the first day I took off in the month and a half+ since I began training again. The hope is that I’ll be fresh enough this week to up my mileage after having 2 weeks of slow, bad runs and feeling on the edge of injury.
I have a number of goals for the coming few months which I have sort of referred to previously. This is my attempt at formalizing these goals and outlining roughly how I plan to get there.
In the immediate term (meaning the next 2-3 months) there are a number of races I’d like to  do and a certain level of training I’d like to be at by the end of this time. I don’t think any of the following is too unreasonable but I am definitely open to suggestions.
This week I plan on increasing my weekly mileage to 40 miles a week. Three weeks ago I hit 38 miles (maybe earlier than I planned on) and the previous two weeks have been 33 and most recently 29 miles. 40 miles, while not nearly what I hope to top out at, is a good place to maintain training as I get back into racing and more frequently do workouts. Additionally I plan on racing the [Shamrocks and Shenanigans 5k](http://www.runshamrocks.com/index.php?option=com_content&task=view&id=23&Itemid=42)this Sunday.
Where am I going with all of this?
Essentially, there are a number of races I’d like to be in between now and the end of April and I’d also like to begin increasing my milage again around the end of April. It is my thinking that while 40 miles isn’t \that\** much, it should be enough to have a solid running base in me and begin my journey into regaining some real speed.
I’ve said before that my two big running goals currently are running a sub-17 min 5k and just racing a full marathon. My target marathon is the first ever Ann Arbor Marathon in the middle of June. In order to reach both of these goals I need to increase training volume by June and increase intensity so I can build speed.
I think that it is doable (by no means optimal) to race this marathon in June if I am able to increase my mileage to 50 mpw by the end of April and maybe even hit 60 before June. I’ll be spending my time between now and the end of April focusing on doing more workouts and definitely more races (almost one every 2 weeks between now and then):
After that time I think I’ll plan just 1, maybe 2, races in May and then focus entirely on the two big races I have planned in June. I’ll be focusing on increasing my training volume from 40 mpw to maybe 60 by the first weeks of June before the Marathon.
I’ll be racing the Dexter-Ann Arbor Half-Marathon (I [ran the 5k](http://davidwilemski.com/blog/2011/06/race-report-dexter-ann-arbor-5k/ "Race Report: Dexter-Ann Arbor 5K") last year). I see this as a chance to see where I’m at two weeks before the Marathon. It will be my first half and should be a good gauge for my marathon training. I don’t really plan on racing it all out but I will say that I’m hoping for somewhere in 1:30 - 1:35 by that time. After this race I’ll be just holding steady until racing my first marathon just a few weeks later. The main goal is to finish but the competitive part of me also wants to do well. I’m thinking that maybe a target of around 3:30 is reasonable, but we shall see. If I can do better than that, great - otherwise I’ll just focus on finishing. I’m really looking forward to this. I don’t really plan on focusing on marathons after this, it’s just a goal to see how far I can push myself. Maybe I’ll try and qualify for Boston one day but I’m not too concerned about it.
Assuming I can get through all of this without any major issues and no injuries (something I’m being very cautious about), I’ll have completed one of my two big running goals\! How will I complete the other? Well, I won’t plan specifics now but in general I’ll spend the rest of June recovering and building mileage back to whatever I was at before the marathon and will hopefully be racing again around the end of June.
I’ll reevaluate when the time comes and decide on a more concrete and specific training and race schedule after what I’ve just outlined. I plan on topping out at 70 miles a week and hopefully with that and some intense speed work I’ll be putting up some fast 5k times once again by this fall\!
Now back to the daily training\!
Yesterday I ran the [leap year 4 miler](http://www.everalracemgt.com/events/full.php?2012/leap-year-4-miler.html). It was in Willow Metro Park and was different from many of the races that I have run in that it was in the evening. The park had no streetlights on the course and it was quite foggy so it was actually somewhat difficult to see towards the end of the race but overall it was quite fun. The course was extremely flat (fast\!) and the little bit of rain didn’t really bug me all that much.
My goal was to run under 27 minutes, which I was able to do. My splits were quite embarrassing though - just really inconsistent. I was looking for about 6:30-6:40 pace during the race but that really isn’t what happened. I went out feeling pretty good and thought I was right on pace; It turned out I ran a 6:08 for the first mile which was really quite surprising for how little I thought I was exerting myself and the shape I consider myself in. It turned out that was not sustainable at all though. I tried slowing down a bit but that mile caught up with me quickly.
The next two miles were 6:55 and 7:15 and were quite tough to keep the pace up. There was also a bit of rain but that actually felt kind of good. There was a turn around right before the 3 mile marker and that is where I decided to really start pushing again. I was able to hold a really good pace without feeling like I was about to fall apart for that entire mile. I even had a decent kick at the end which means I probably had more in me for the middle two miles - I just couldn’t find it.
I finished the last mile in 6:34 for a finishing time of 26:51. Overall sometime I’m happy with because I beat my goal. I do think that if I had been a bit more even in my splits that it could have been even better but it is no use worrying about that now. Full results for the race are here.
Some takeaways from this race:
In order to help me keep a more steady pace as I train for some upcoming races I plan on doing more mile repeat workouts. I know in the past they’ve really helped my pacing and improve my speed over the course of a race.
CreateYourWPWebsite.com WP Security talk \ [pdf \]
Earlier this week I was getting a lot of pain in the top of my foot when I was running from what felt like the shoe laces digging into my foot. I don’t believe that I’ve been tying my shoes differently but I was unable to adjust my shoe in any way to stop the pain. I’ve tied my shoes the same way for as long as I’ve been running and never had an issue.
I decided to try tying my shoes like a ’normal’ shoe - without the top corners laced - over the past few runs and things seem to be better. I find it really interesting that this could make such a difference and it’s something I’ll definitely keep an eye on.
These Nike shoes (same shoe, different pair) have given me more problems than any shoe before. I’ll definitely be going back to good o’l Asics for the next pair.
So. Last weekend I ran the Super 5k. This was my first race since October and after only really running for about a month at ~30 miles/week. Between then and now I was only running a day or two a week for 3-4 miles - not very much.
Needless to say, this was a pretty rough race for me because I was so out of shape. Anyhow, [I finished in 21:44](http://www.rftiming.net/results/superbowl/superbowl20125k_overall.htm) which I guess I’ll take for being so out of shape. The course wasn’t bad; it was an out and back and had a slight hill before the turn around. The finish was not straight which is not ideal but I wasn’t exactly going all that fast...
Now a quick update on training: Since New Years I’ve been running with a few groups which I’m hoping keeps me motivated to keep running the rest of the week. This has been going well so far. I’ve also tentatively begun some speed workouts. Most recently some mile repeats at around 6:45 pace. This was actually incredible difficult for me which is extremely frustrating for me since I know that in the past I’ve cranked out long runs at that pace.
That brings me to my running goals for this year:
I’ve been getting some good base mileage in the past month or so and should probably come up with a more thought out training plan for at least into late Spring.
My old host took all of their servers offline so there was a bit of downtime while I fixed the site. Everything should be back to normal now.
I promised to upload the slides from my WordCamp Detroit talk so that people could use them as a reference.
I’ve uploaded them to slideshare and made a PDF of them available as well:
Slideshare:[
http://www.slideshare.net/davidwilemski/word-camp2011-introwordpresssecurity](http://www.slideshare.net/davidwilemski/word-camp2011-introwordpresssecurity)
PDF:[
](http://www.slideshare.net/davidwilemski/word-camp2011-introwordpresssecurity)WordCamp2011-IntroWordPressSecurity[
](http://www.slideshare.net/davidwilemski/word-camp2011-introwordpresssecurity)
Last Sunday I ran the Big House Big Heart 5k. The race started outside of Michigan stadium, looped around part of Central Campus and then finished by going through the tunnel entrance to the Big House and down to the middle of the field. Overall, I loved the course; It had only one somewhat large hill near the beginning of the race and then was completely flat or downhill the entire race.
There were, literally, tons of people that ran the race. Based on the results, there were over 6000 finishers\! This is probably one of the biggest races I’ve even been in. I think it’s great that many people wanted to run - the problem was that I made the mistake of not starting right up front and I got boxed in for a large portion of the race. It took almost a minute for me to cross the starting line after the race started - that didn’t affect my time as it was chip timed but it did affect the number of people that I had to weave in an out of to pass.
The first mile (which was around 6:50) was constant passing and weaving and speeding up and slowing down. The same for most of the 2nd mile as well. This really affected my racing a lot. I’d say that the ’race’ turned into more of a ’run’ for me since I wasn’t able to get to race pace at any point.
That said, it was a lot of fun and a beautiful day for a race. I learned my lesson and I won’t make the same mistakes again. I ended up finishing in 20:22 which was a top 100 finish. I’m not thrilled with my time but I’m glad I ran all the same\!
The full results are here:Â <http://www.rftiming.net/results/bhbh/bhbh20115k_overall.htm>
This past Sunday I ran a 5K at the Detroit Zoo. It was my first in a while, but after training all summer I’ve been wanting to start racing again. The course was a loop around the outside of the zoo, most of which was a neighborhood. It was rather flat with only one slight hill just after the 2 mile mark.
I ended up running in 19:47 which is my best time in a while but still no where near the speed I used to be. It was also fast enough to win my age group. I finally broke 20 minutes - something I’ve been trying to do for most of the spring and summer. My splits were about 6:08, 12:35, 19:47. I really didn’t feel like I went out too fast and actually felt rather good the entire race. Next time I race I’m going to try going out at about 6:15 and hold that pace through the race to have more even mile times.
I run the Big House Big Heart race on October 9th but hopefully will have a chance to race once more before then. Full results for the Run Wild race are hosted at raceservices.com
In my [last post](http://oromis.davidwilemski.com/blog/219/mcommunity-xss/ "MCommunity XSS"), I briefly mentioned that I thought MCommunity was sitting on a pretty nice JSON API. Today, I’ll be covering some of the endpoints that I mentioned and what the data looks like when it comes back. I’m writing this not because it was particularly difficult to figure out - anyone with a web browser can do it quickly with the developer tools - but because I enjoyed getting a glimpse into how the service was structured.
When I first looked at it, MCommunity had an endpoint that the Javascript was POSTing to to get details about who was logged in.
Doing a GET to https://mcommunity.umich.edu/mcPeopleService/private/people/getAuthorization/dummy will give a few details about the user who is currently signed in. For instance, when I am signed in, mine looks like:
{
"authcheck": {
"errors": "",
"authStatus": true,
"displayName": "David Thomas Wilemski Jr",
"dn": "uid=dtwilems,ou=people,dc=umich,dc=edu",
"elevatedStatus": false,
"uniqname": "dtwilems"
}
}
Note: They’ve since stopped using the "dummy" signifier in place of some sort of numerical ID; My guess is that it was an attempt to change that endpoint per session so it wouldn’t be so easy to call. However, "dummy" still works as does any integer - and most strings too. Maybe it was changed for some other reason, I guess I’ll never know.
As you can see, this is very useful information when exploiting XSS. It tells you exactly which user is currently authenticated. While the XSS vulnerability was still present, I could have written a script to notify me when someone viewed my profile and tell me exactly who was viewing it. Furthermore, because the script would have been executing in the context of the other user, it could have used the next endpoint that I’ll be discussing to ask for all of a user’s personal information that is stored in MCommunity and send that to a malicious attacker. That’s some serious analytics.
The next endpoint gives information for a uniqname based on who the user asking for it is. For instance, the URL:
https://mcommunity.umich.edu/mcPeopleService/private/people/dtwilems
gives info on me\! Here is what I see when I view the page:
{
"person": {
"distinguishedName": "uid=dtwilems,ou=people,dc=umich,dc=edu",
"errors": "",
"aboutMeView": 2,
"acl": [
"3#entry#ou=People,dc=umich,dc=edu#drink",
"3#entry#ou=People,dc=umich,dc=edu#umichAltPhone",
"3#entry#ou=People,dc=umich,dc=edu#umichAltAddress"
],
"affiliation": [
"College of Engineering - Faculty and Staff",
"CoE-IT - Faculty and Staff",
"Undergraduate Engineering - Student",
"Alumni"
],
"aliases": [
"David Thomas Wilemski",
"David Wilemski Jr",
"David Wilemski",
"David T Wilemski",
"David Thomas Wilemski Jr"
],
"altAddressView": 1,
"altPhoneView": 1,
"associatedDomain": "engin.umich.edu",
"displayName": "David Thomas Wilemski Jr",
"drink": "Cranberry Juice",
"email": "dtwilems@umich.edu",
"emailForwarding": [
"dtwilems@mail.umich.edu",
"dtwilems.umich@gmail.com",
"dtwwtd@gmail.com"
],
"faxPhoneView": 2,
"ferpa": "N",
"homeAddressView": 2,
"homePhoneView": 2,
"imView": 2,
"mobilePhone": "------MY CELL NUMBER-------",
"mobilePhoneView": 2,
"noticeView": 2,
"pagerPhoneView": 2,
"permanentAddress": "-------PLACE WHERE I LIVE--------",
"spamFilter": "TRUE",
"title": "Student, Undergraduate Engineering",
"uniqname": "dtwilems",
"urlView": 2,
"vacationView": 2
}
}
However, if I call the API for someone else’s info then it only returns information that they have marked as public. Well, actually, you can mark things as public, or just viewable to logged in users (auth), or private (self). They actually return information structured pretty well, and as we’ll see, I’ve since discovered more API endpoints like these - but first I’d like to talk about the last endpoint that I used while digging around with the XSS exploit.
This is the one that really made things go. POSTing to this with the correct variables and a valid cookie will update a profile:
https://mcommunity.umich.edu/mcPeopleService/private/people/updateContact
This is the one that could have helped to cause the [MySpace worm](http://namb.la/popular/) of MCommunity - if I was evil. Instead, I reported it like any upstanding Michigan community member would have. :D
So, like I said, the MCommunity site has a nice API. As a matter of fact, Bryan and I have just discovered some various WADL files that describe additional endpoints. I’ll list the ones we know about out now for some of you curious folk:
What would I like to see MCommunity do with its API? Open it up\! I mean, we’re a top computer science school for crying out loud\! I think that the students and alumni here could do some truly amazing things if this were to be opened up and documented a little better. Actually, this is something I think that the entire University could do better. Initiatives like the Mobile Center are great but the applications that they put out are still purely proprietary. In fact, their APIs page has had basically a "coming soon" message on it for over a year now.
As it is, any student who wants to build a mobile or web application related to the University is usually forced to revert to lame attempts at scraping a web page for information and other gross, hackish things in order to get functionality out of their application. I’ve experienced this first hand with both [Umich Dining](http://davidwilemski.com/umichdining) and MSchedule. We actually found computer readable formats for both of those applications but not without some exploring first. The data feeds aren’t advertised in any way and I think that they need to be. I’ve heard very similar stories over and over from other students who have tried to build some Michigan specific application at a hackathon or for other purposes and, honestly, it gets frustrating.
I will say that it seems that the University is getting better at trying to support development here on campus - we’re just not quite there yet. If anyone who works for the University who can help move this along is reading, just know that you’re moving in the right direction - we just want you to get there faster\!
Also, us MSchedule developers would like to be able to register users for classes so as to create a much better user experience. Pipe dreams, we know...
The University of Michigan [recently launched](http://michigandaily.com/news/u-launches-new-directory-program-mcommunity) a new directory site called MCommunity that is to be used to search for information on people related to the University.
A few days after the site launched my friend [Bryan Kendall](http://bryankendall.com) found a persistent XSS vulnerability in MCommunity within a couple minutes of looking at the site. He reported the problem to the Michigan security team.
Days passed and I got curious about how MCommunity itself worked as it obviously used a lot of AJAX to load information on pages. So I took a look at what those calls looked like using Chrome’s developer tools. It looks as though MCommunity is sitting on top of a pretty nice JSON API\! They included endpoints for gathering info on an account, the current user’s auth info, and various other endpoints including the one for updating a profile.
So what did I do? I wanted to update my own profile via XSS of course - just to see how it worked. I embedded a Javascript snippet in the alt address field that Bryan found was vulnerable that loaded an [external script](http://davidwilemski.com/static/js/mcommunityxss/mcommunityprofileedit.js) hosted on my server.
\[caption id="attachment\_220" align="alignleft" width="391" caption="Screenshot of the snippet in the form before I submitted it"\][![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2011/08/mcommunity.png "mcommunity")](http://oromis.davidwilemski.com/blog/219/mcommunity-xss/mcommunity/)\[/caption\]
That external script simply made some AJAX calls to various endpoints with the purpose of updating the user who was viewing my profile’s Alt. Phone number to "I have you now". This was accomplished by grabbing the user’s authentication information and posting to the proper place; because the Javascript was being executed in the context of a logged in user it posted their existing auth cookie along with the post and acted on behalf of the user but without their permission.
\[caption id="attachment\_221" align="alignleft" width="205" caption="Screenshot of the modified field on my profile after being XSS’d by myself"\][![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2011/08/mcommunity-1.png "mcommunity-1")](http://oromis.davidwilemski.com/blog/219/mcommunity-xss/mcommunity-1/)\[/caption\]
After testing this to see if it worked with a couple of friends (letting them know what would happen before hand), I stopped loading that script into my profile so as not to actually modify anyone else’s data. I replaced it with a simple alert that said that MCommunity was vulnerable and that concerned users should notify the University.
Had I been malicious, I could have modified others’ profiles so that when they were viewed, they too would attack other people’s profiles. This worm could have spread out of control and caused much grief. There are also any number of other things that an attacker could do with an open XSS bug. This is why testing for these things in development is so important.
I promptly reported the vulnerability as well because I didn’t want others doing this to me\! I quickly got a response from the security team saying they had passed it on to the MCommunity team. About a week later, on August 2nd, Â I noticed that the vulnerability had been fixed and no longer worked. They must have patched it in one of [these releases](http://www.itcs.umich.edu/mcommunity/releasenotes/directory/July2011.php)Â - despite not mentioning it in those notes.
Overall the response from the University was very good. I do hope that they are on the lookout for further vulnerabilities in the application. It houses so much information on the entire Michigan community that it would be disappointing to see that be abused by a malicious attacker.
This morning I ran the Firecracker 5k which is my second race in two weeks time. Going in I felt pretty tired from the mileage I’ve been running and not being used to that. However, my first mile was under 6:15 despite feeling like I was running well slower than 7 minute pace. I tried evening out my pace for the second mile but pretty much blew up before the end of that mile. I did the best I could through the rest of the race, but ended up letting a lot of people pass me. I finished in 20:51, so not much worse than my last two races, but still not the sub 20 I’ve been looking for. I’m going to have to work on my race pacing some, I’ve forgotten what that’s like. Full race results are here.
Welp, this past weekend was the Artful Dodger 5K in St. Clair. I went back home to run it and get another race under my belt for the summer. My goal going in (as with the [last race](http://oromis.davidwilemski.com/blog/206/race-report-dexter-ann-arbor-5k/ "Race Report: Dexter-Ann Arbor 5K")) was to be under 20 mins, but I ended up with 20:16. So, faster than my previous race but still not where I wanted to be. For the record my splits were about 6:24, 6:44, 7:08 and I finished 15th overall (14 male,
The 4th of July will be the Firecracker 5K here in Ann Arbor, I’m REALLY hoping I can finally go sub 20.
Well, last weekend was the race I talked about in my last post. There were way more people there than I thought there would be.
Parking ended up being a nightmare so I didn’t have time to warm up before the race. This meant I took the first mile easier than I wanted to minimize injury risk and ran the next two at a hard pace instead of at full race pace. The first was still possibly my fastest mile because there was a bunch of down hill running. The next two miles I ran with a steady pace (I think, I don’t have splits). I ended up running 20:45; so, slightly slower than I wanted to but something I can be pleased with with the small amount of training I’ve been doing. Overall, I loved the course. It had the right mix of hills and flat running, I’ll definitely be doing this one again - but maybe a longer distance next year? We shall see.
I took this past week pretty easy, only a few hard runs and nothing over
I’ve been running off and on again for most of the past year struggling with injuries (pulled quad, shin splints). I think I’m close to being able to say that that stuff is finally past me. I’ve been running ~20 miles a week for the past few weeks and so far so good (a few twinges in my legs the past few days that I’m gonna keep an eye on).
[![New Shoes\!](http://oromis.davidwilemski.com/blog/wp-content/uploads/2011/05/2422872127175379995126425392632619092892302_o-300x225.jpg "New Shoes!")](http://oromis.davidwilemski.com/blog/202/running-update-no-more-injuries/2422872127175379995126425392632619092892302_o/)Last night I was heading out for my run and got this crazy idea that I wanted to do a mile time trial.Yes, despite it being the first day on new shoes, being super wet out, and my legs acting up a little (I did say it was a crazy idea\!). I’ll be running a 5k next weekend and wanted to see what of hard, but not all out, mile time I could get so I know how fast to go out in this race.
So I ran a mile and a half warm up to the track. Did I mention it’s probably been a little over 2 years since I’ve been on a track? I stretched a little then did my mile. It was pretty humid still despite it being midnight but forgot about that quickly as I started. My first lap felt easy - not too fast, not too slow - at about 1:30. The second lap went by in another 1:30 split, but I was really feeling it by the end of that lap. The third lap’s split was 1:32 and the final lap’s was 1:33 despite me feeling like I picked it up - I had forgotten how that works. So I ended with a 6:05 mile after not training hard for months. Something to improve on, but I’m content for now. It really surprised me how well I actually kept my splits - that was something I always struggled with before.
Next weekend I’m going to shoot for 6:30 pace in my 5k and see what happens (assuming the legs are fine this week). Hopefully I’m not being too ambitious. I’d really like to go sub 20 mins as a baseline for getting back into racing.
I’ll post something about my results next weekend after the race, and I plan on doing more running posts to try and keep myself accountable for my running. Sometimes the hardest part is convincing myself to get out the door :)
Last October I wrote about my [first experience with Google App Engine](http://oromis.davidwilemski.com/blog/85/building-a-quick-and-dirty-web-service-on-google-app-engine-webpastr/) which took the form of a reallllly simple service that I called WebPastr.
It has been something that I built once and sort of forgot about. I’ve used it off and on since building it but no one else has - which is about what I expect. One thing that was unexpected, however, was the fact that shortly after writing that first blog post about it, it got [posted to reddit’s /r/AppEngine](http://www.reddit.com/r/AppEngine/comments/drlmx/buildingaquickanddirtywebserviceongoogle/). As it happens, I only found this out after the fact; and to make matters worse I have no idea exactly how my post was originally found because I had just gone through a web host change and had forgotten to set up analytics again. So to that poster, if you’re reading this - thanks\! :)
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2011/05/Visitors-Overview-Google-Analytics.png "Visitors Overview - Google Analytics")](http://oromis.davidwilemski.com/blog/191/webpastr-revisited/visitors-overview-google-analytics/)As you can see, it was by no means a huge amount of traffic (huge in the sense of reddit or Digg front page huge), but it was a ton for my tiny site that only gets maybe a handful of views a month. It was the first time I really thought about the fact that I can produce things that people want to know about though.
So why this post now? Months later, nobody cares about this right? Yeah, well pretty much. Something I planned on doing a while back was releasing the source to this so others could learn from it. I’ve progressed so far since then, but even though it’s really basic, it was valuable to me in understanding the structure of a App Engine/Python web app. So that’s what I’m finally doing today. WebPastr is now up [on github](http://github.com/davidwilemski/webpastr); Anyone can see just how simple this little idea of mine was to implement and hopefully it will help others in some small way - and who knows maybe I’ll actually end up doing more with it now that it’s public so that I can be less embarrassed about how simple it really is\!
Alex has made a demo video of Draw With Me to give a preview of where we are at with it. We’re in the process of tidying up some last minute bugs before we submit to the App Store’s approval process. Hopefully we can announce its availability soon\! For now, take a look:
Last week was pretty crazy. I’ve been working on an iPad app with [Alex Haefner](http://alexhaefner.com/) that we call Draw With Me and we spent most of last week preparing to unveil it to the world. It lets people draw together on an iPad simply and instantaneously. On Thursday, we finally got to unveil it to the community at U of M and some Apple employees at the first ever [iOS App Showcase](http://www.eecs.umich.edu/eecs/etc/events/showevent.cgi?2045).
I think it went really well. We got lots of people excited by the possibilities of what we’ve made and got some good feedback. We’re going to spend the next few days doing a last few bug fixes, move to a more stable server, and then submit it to the App Store. Very soon everyone will know the joy of drawing with others using Draw With Me\!
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/17062618690698475181264253926322577613035691_o-300x206.jpg "17062618690698475181264253926322577613035691o")](http://oromis.davidwilemski.com/blog/152/winter-update/17062618690698475181264253926322577613035691o/)So...
It’s been a while. Here’s a update: I’m taking 18 credits this term and involved in more projects and groups than ever before.
Last week we had a huge snow storm, a friend and I went out to take photograph the storm. It was a lot of fun.
Over the weekend I participated in a hackathon and helped build an iPad app. Everyone had really cool projects and I learned a lot. Hopefully I’ll get a video demo of the app soonish.
Lastly I joined a team that is going on a trip to the Bay area over spring break to pitch their idea to some VCs and people from companies like Google and Facebook. We also get a tour of their campuses and some other cool things. I’m really looking forward to it, and, it will be the first time I’ve done anything for spring break in probably forever.
Tonight, I went to the men’s Glee Club performance. They did an outstanding job and I don’t think that anyone was disappointed. I particularly enjoyed the Friars.
Afterward, I spent the evening working on my 280 project. My desk is now a disaster - that picture will come tomorrow though. I have completed a decent portion of it and finished the first season of Dexter. With any luck I’ll be close to done by Tuesday night so that when I get home on Wednesday I won’t have to think about it.
But, for now, sleep. Goodnight, World.
The magic bus kiosk has been having issues for a few days now...
My breakfast
Bryan won a pseudo snuggie at hacku this weekend
Alec being Alec
Mmm raspberry lemonade
Alec’s old phone
Boo calc homework
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101014221653-300x225.jpg "IMG20101014221653")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101014221653.jpg)
I finally got a new power supply for my broken desktop today, works great on the first try.
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101012231522-300x225.jpg "IMG20101012231522")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101012231522.jpg)
I received a Magic Mouse last month as a gift. So far I’m loving it, except it is not meant for gaming. I have a gaming mouse for that anyways.
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101012004821-300x225.jpg "IMG20101012004821")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101012004821.jpg)
The food that keeps tired college kids going during late night study sessions
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101010202200-300x225.jpg "IMG20101010202200")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101010202200.jpg)
We got some new aloe plants over the weekend, and yes those are CD spindle lids being used as pots.
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101010194025-300x225.jpg "IMG20101010194025")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101010194025.jpg)Ann Arbor at night
Update (5/26/2011): I’ve now [open sourced the code](http://github.com/davidwilemski/webpastr) and [wrote a little more about WebPastr](http://oromis.davidwilemski.com/blog/191/webpastr-revisited/)
[Earlier this week](http://twitter.com/#!/davidwilemski/status/26662857425) I though that it would be nice to post a text snippet to PasteBin via Alfred on my Mac. I looked into how a new pastebin post is posted and was dismayed by the fact that I couldn’t set it up just with Alfred’s wonderful custom search feature.
Last night I decided to build a quick [app engine](http://code.google.com/appengine/) application as an intermediate step between me and PasteBin.
I’ve done a little (unpublished) work with Python on app engine before so I quickly chose that as my language. I knew enough to get me going. I had to write a main page (this still must be done) and the script that would handle my requests and forward them on to PasteBin.
I am happy to say that in under an hour I got it working with a very rough set of features; Namely, posting to PasteBin. I have tested that this works if you setup a custom search inside of both Alfred and Google Chrome. I have posted some screenshots of the settings windows for each app here.
One of the things I keep hearing from both people who work at startups and even presenters from Google is to actually release something, to build and iterate. Â Even though I don’t even have a webpage designed for the service yet there are no other features yet, I am releasing this. This is not something I would have done before.
In order to create a new post you send a request to this URL: http://webpastr.appspot.com/post?code={yourtexthere}
\[caption id="attachment\_95" align="alignleft" width="300" caption="Alfred Preferences"\][![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/Alfred-Preferences-300x89.jpg "Alfred Preferences")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/Alfred-Preferences.jpg)\[/caption\]
\[caption id="attachment\_96" align="alignleft" width="300" caption="Chrome Preferences"\][![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/Google-Chrome-300x136.jpg "Google Chrome")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/Google-Chrome.jpg)\[/caption\]
What are my plans for the future of WebPastr?
First, I’d like to support other services. Definitely Gist when GitHub finally allows the API to post new Gists.
I’d also like to support PasteBin’s post expiration feature.
Lastly, I plan on designing a homepage for the app that clearly explains how to use it in various applications. I believe I could easily get Firefox working with this and maybe a Bash script as well.
Feel free to use this now, and be on the lookout for new features soon\!
I made it home this weekend and spent some time in downtown St. Clair. I took this picture just across from the St. Clair river and I really like the fall colors mixed with the deep blue of the river. [![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101009121147-300x224.jpg "IMG20101009121147")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101009121147.jpg)
I’ve always been a big fan of up close photography, I may try doing more soon.
This would be a great building to photograph with a good camera. I will have to remember that.
This is just a quick update. No photo yesterday - first exam of the term and my lack of sleep+frantic studying made me forget to take a picture. There is one today though.
CSE Building, UM North Campus
A very interesting pair of shoes to say the least.
Today I was messing around with various options on my camera. I’m pretty happy with this shot - It definitely encapsulates two things that are pretty important to me as a college student. I’ve been thinking about getting a DSLR and maybe I will - but probably not till about Christmas. I know, given some practice, I could make a much better shot with a better camera. For now I’m extremely happy with my phone’s 8MP camera (what I’ve been using for all of my shots thus far and into the foreseeable future)[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101003140240-300x225.jpg "IMG20101003140240")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/10/IMG20101003140240.jpg)
Today was yet another rainy day here and so I thought that this wouldn’t turn out so well. Surprisingly, the color details are pretty darn good - perhaps I should have stopped for a bit longer to improve the focus.
These gigantic leaves really caught my eye today, even while rushing through the rain. Also, the contrast between the giant green leaves and tiny colorful flowers is just great. I must go by here a good few times per week, so just think about what else I could be missing\!
EDIT: my phone didn’t publish this yesterday when I asked it to. Oops\! Sorry.
I’ve never seen anywhere with more board games that aren’t mainstream. It’s pretty awesome.
I believe it was called vault of midnight and is in downtown Ann Arbor if anyone is curious.
Today, I came up with a great idea for my personal website. I just had to draw it out though because otherwise I would never have stopped thinking about it until I built it. It got me thinking about just how thin I’ve been spreading myself. Between work and school and personal projects like mschedule and my Android development, I just don’t have time for all of it. Especially when I consider all of the things I’d like to do with other people as well.
My solution? After I catch up to my current backlog, I am going to reorganize my priorities. Whether this uses the GTD method or something else, I need to better use my time.
Now my photo of the day: my awful sketch of how I’d like my new site to look.
Today I went to a tech talk given by Google about Chrome as a Google product and a large scale open source project. It was really interesting to get some insight into how Google works and to hear about all the different things that go into Chrome.
Plus, I got this awesome Google t-shirt and a pretty cool bag. Free stuff is always nice.
It’s just another rainy day here in Ann Arbor. So that means we need to water the new landscaping.
Fall colors; coming to a tree near you\!
A very skillfully made paper airplane.
Oh the joys of north campus
Cat in the hat shoes are pretty cool
Tree face
Today, [Daring Fireball posted a link](http://daringfireball.net/linked/2010/09/22/fox-market)to Kevin Fox’s analysis of how to fix the android Marketplace.
I disagree with both of them. I don’t believe the launcher needs to be killed. Let’s face it, a phone is no longer just a phone. It does 80% of the tasks that I used to do on a computer.
While using my mac I do not have a desktop (analogous to the home screen) shortcut to every single application that is installed. I have a few shortcuts on my desktop and dock and then use a ’launcher’ to execute anything else that I might need.
Having every icon on my ’home screen’ makes the same amount of sense on my phone as it does on my computer.
Now that being said, I do like Fox’s analysis of what Apple does to get users to open the app store and Android handset designers could learn from that.
One of my favorite things about UM’s campus is how blended the campus is with surrounding Ann Arbor.
Now this doesn’t count as the first one, but it’s just so awesome I had to share it.
My friend Alec made this awesome kool aid cake for me and Matt on our birthday weekend. So. Tasty.
As I’ve had this blog setup for a few months now and have just a few posts I’m setting some new goals for me to stick with. I don’t wish to commit too much time here as I have lots of exciting other things that are grabbing my attention - these you will hear about soon enough. There are also lots of other things that need my attention which are not as exciting - such as keeping up with my classwork.
"So", you might ask, "what can I expect to see here?"
Starting tomorrow, I plan on taking one photo per day and posting it here. This photo will be taken with a camera and will not be a screenshot of any type. Furthermore, I will also be writing at least one post a week. It may be more but I am going to stick to at least the one. I am not committing to a day as of yet but if need be I will - and that would be announced here of course. These posts will be about the projects I am working on and any observations I make through out the days that I may want to record for the future. Lastly, as time permits, I will be working on a custom theme for this blog that will closely resemble that of DavidWilemski.com. I just have to populate that site with my own projects first.
Be on the look out for my photo of the day and first weekly post\! If you were feeling really generous you could even subscribe to my feed...
This is something that’s been on my mind for a while now.
Anyone that I know who has ever spent time creating a recipe in the kitchen or that has a treasured family recipe that has been passed down for generations is always extremely willing to pass it on so that others may enjoy the dish. It takes a certain amount of skill and talent to develop a good recipe as well as many hours spent perfecting it.
This is not to say that these chefs and bakers should never receive monetary reimbursement for their hard work, however those who choose to make a living through cooking are able to in other ways - such as building a restaurant around their own recipes or even selling a book with a large collection of their recipes. Just because they like to share a recipe for free here or there does not mean they do not value their work. Instead I believe the willingness to share causes others to look into the chef in question and support them more.
This is in direct conflict with what we see today in the digital media industry. Certainly the culture of sharing recipes has been around much longer, but today we see a push in the opposite direction. Artists are urged to not share their work to get their name well known but instead sign content deals with large corporations to market the work and make profit of their own.
There are artists and film directors out there that try to succeed through this method of sharing instead attempting to make a living off of live shows and movie viewings. I feel that this is frowned upon because of the industry we have built around the selling of content.
So, what do you think? Are cooks everywhere missing out? Should there be corporations built around marketing and selling recipes or should modern day artists rethink the accepted way to gain success?
*This article has been in response to articles like [one published on The Guardian’s website](http://www.guardian.co.uk/media/2010/mar/10/murdoch-illegal-dowloading-stealing-handbag) this past March. Even though I was directing this article towards artists, it could also apply to content producers who have this same business model.*
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/07/Screen-shot-2010-07-22-at-9.34.23-PM-250x300.png "UMich Dining Menu")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/07/Screen-shot-2010-07-22-at-9.34.23-PM.png)I’ve been interested in Android development for some time now. However, I never actually got around to setting up the development environment - until now. Before I give my impressions on the documentation that covers how to do all of this, I’d like to clarify that I’m not new to the development world at all - I’ve had a good few years experience - mostly with C++ and PHP and a few programming courses by now as well. I have not, however, had any experience with Java up until now. These are my immediate thoughts about developing on Android after spending some time (only a few hours so far, maybe 5 or 6) getting familiar with the SDK.
So far, I’ve been very pleased with how well the documentation is written. I followed the [setup guide at the Android Developers website](http://developer.android.com/sdk/installing.html). My first step was to download and install Eclipse for my platform. I’m running Mac OS, and that’s the platform that my Android development experience has been on thus far. After getting Eclipse set up I then proceded to follow the instructions to get the latest (2.2 as of this writing) Android SDK and setup the SDK tools.
I also took the time to setup a virtual Android device (AVD) and test it for a few minutes (not as fast as my [HTC Incredible](http://www.htc.com/us/products/droid-incredible-verizon#tech-specs)\!) Even though I did this here, the next guide walks you through the steps. I found them quite self explanatory and took the liberty to do it on my own.
Once finished setting up my development environment I dove right into the [Hello, Android](http://developer.android.com/guide/tutorials/hello-world.html) demo application. This tutorial shows how a very minimalistic is built and what components it needs. As with the previous tutorial, this one was written very well with a good mix of beginner tips and intermediate developer centric portions. There were few, if any, portions of the guide that left me lost or feeling unsure of what to do next. In addition, I felt comfortable exploring [SDK documentation](http://developer.android.com/reference/packages.html) immediately after completing the Hello, Android program and started messing with other Views and Actions available from the SDK.
Whenever I got stuck or wasn’t exactly sure of how to do something (keep in mind that at this point I’ve only been using the SDK and Java for an hour or two now) the excellent [Hello, Views](http://developer.android.com/resources/tutorials/views/index.html) tutorials helped set me on the right path.
After only reading those three tutorials and a few documentation pages I felt ready to begin work on my own application. What I have so far is an application that has two layers of tabbed menus with scrollable lists in each section. That’s all I can show you for now - more on that in a later post.
[![](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/07/Screen-shot-2010-07-23-at-1.52.58-AM-188x300.png "UMich Menu 2")](http://oromis.davidwilemski.com/blog/wp-content/uploads/2010/07/Screen-shot-2010-07-23-at-1.52.58-AM.png)Overall, I’m extremely impressed with how easy it is to develop Android applications so far. It really is quite easy to go from not knowing a thing about Android programming to being ready to build your own application all within a single afternoon or maybe a couple of nights. I’m super excited to continue working on my application and hope that this article helps to alleviate any fears that others may have about getting started with Android development.
I’m starting this blog to record my thoughts on various topics.
As I am a Computer Science student and have been doing lots of web and mobile development as of late, many of my posts will focus on my experiences with web development, Android development, and the occasional news item as I see fit. However, I am not claiming this is a tech or development blog and as such I may also write about other topics.