I spent some time this weekend noodling around with Github Copilot, using the copilot-cli tool.

A while back I had created a little Electron app that searched for and displayed information about Unicode characters. I didn’t really know what I was doing, either with Electron or Javascript, and couldn’t find much in the way of correct and up-to-date beginner-level documentation on things as simple as how to wire an application together - the doco was good enough for a "Hello World" but nothing beyond that. As an aside, I have the same gripe about lots of languages and frameworks - good luck figuring out how to drive things like make and the linker from any online C documentation. Anyway, I muddled through, and you can see what I did by hand here. It works, and the same code produces both an Electron app, and a stand-alone web page, and a CLI tool, but it’s vile, especially the build process.

The updates since then are here. Copilot wrote pretty much all of that with only a few very small manual tweaks. The initial prompt that produced most of the work was something like “Make this project more idiomatically Electron and Javascript-ish”. I worked with the evil hallucination engine for a few hours, including prompting it to add tests and a Github workflow for CI. I mostly gave it its head and let it do whatever it wanted although there were a few small things where I told it to do things slightly differently. Overall, I was very impressed with the result. With my lack of knowledge of modern HTML/CSS and Javascript there’s no way I could have done this myself.

I then thought “there’s so much Javascript/Electron stuff out there for it to have learned from, let’s try something different” so I took the next little tool improvement on my to-do list and asked it to write it for me. I’d previously had a shell alias alias highlight="ack --color --passthru" - you pipe stuff in, give it a perl regex, and it highlights anything matching. I wanted to be able to control the colours, by saying something like cal | highlight --red '25|26' --green '27|28'. So I prompted copilot with something along the lines of “write a shell script that will take an optional --color_name parameter followed by a perl regex, read input from STDIN, and spit it to STDOUT with any matches highlighted in that colour. It should allow any number of such matches”. It took a few attempts, mostly it iterating towards a solution while it tested and re-tested what it had done, and required very little in the way of corrective input from me. It also wrote the documentation when I told it to. I thought I’d try giving it something that is confusing as hell even for experienced shell scripters, and asked it to make the script able to spit out its own shell tab-completion function, but it managed that too. I can now type highlight --gr<tab> and it auto-completes green.

For another little task I gave it in perl, I told it to make sure that all the changes it made to an existing code base were 100% covered by tests, including branch and condition coverage. It went away and did that. In fact it also wrote some missing tests for stuff that I had previously written. And then it wrote a test that will fail if I ever make code changes in the future that aren’t 100% covered.

At various points it said things like “I need to consult the documentation” and (after asking permission) went off and read the docs for things like Devel::Cover, and the Electron docs. It even said “I can’t figure this out from the Electron docs” (oh good, it’s not just me that thought it wasn’t very good!) and went and read the Electron source code to figure out what’s what.

I was on a bit of a roll by now, so I asked it to fix a long-standing bug in one of my perl modules, which consistently failed its tests on OpenBSD, NetBSD and OmniOS, and intermittently failed on FreeBSD, while always passing on Linux and Mac. I’ve been banging my head on this for over a year, and no-one I’ve asked for help has been able to figure it out either. It looked at the code, and at the failure reports in the OpenBSD test workflow on Github, and told me what the problem was and how to fix it. I didn’t believe it, so I said “verify that, here’s the address of an OpenBSD machine you can ssh to”. So it did. I wasn’t happy with its suggested fix - it would have fixed the test, but subtly broken interactive use of the tool by buffering STDERR output until vomiting it all up at the end - so I explained that to it and it did a massive rewrite of my code. I was rather dubious but having tested it myself and very carefully reviewed what it did - yep, it got it right. The code it produced is a little hard to read, but that’s because it’s dealing with some complicated inter-process communication and control. It wrote tests without prompting. Perhaps my only criticism is that it didn’t comment the code.

This thing is FUCKING MAGIC. I will be using it a lot for creating new stuff, and for updating small code bases. I don’t think it stands a chance in hell of being particularly useful for the software archaeology part of maintaining a large legacy application - which is what most devs spend most of the time doing at work - but I am going to at least try using it to make small updates once I have figured out what/where they should be. The quality of the code it produces is fairly good. Many of us could of course do better, but what it produces is generally readable. Copilot does require a little hand-holding and its output needs a light editing pass before it’s worthy of being put up for code review by other humans, but no more than the code I write myself does.

I strongly recommend that you give it a go.