Nestoria Interview: Scott Manley, star gazer and DJ, with an interest in social search

A couple of weeks ago I spent a bit of time, outside of office hours of course, looking at lovely data visualisations.  I found a few that were quite relevant to what we do at Nestoria, but there was also one beautiful visualisation that was more relevant to all of mankind, but less relevant to Nestoria specifically.  It's a visualisation of asteroid discoveries made in the past 30 years, shown chronologically and adding to the spinning heavens. 

Here it is if my explanation isn't clear enough.

Of course after seeing this I wanted to ask Scott Manley, who's the man behind the visualisation, about the whys and hows of his project, so I took the liberty of expanded my remit to interview interesting and Nestoria relevant sorts to interviewing sorts that are interesting and relevant to us as earthlings as well.

Here are my Qs and his As:

I came across you via your beautiful data visualisation of asteroid discoveries over the past 30 years.  Can you tell me how you came to make it?  Both the reasons behind it's creation and how you did it in practical?

The video goes back to my student days in the late 1990's when I was fascinated by asteroids and trying to make a case that there was a potential impact threat which needed to be assessed. Telling people that there are thousands of asteroids regularly crossing the Earth's path is one thing, showing them a picture is much more effective, so I created a site http://szyzyg.arm.ac.uk/~spm/neo_map.html which would update with the locations of all known objects. Back then, we knew of less than 50,000 asteroids, but we were pretty sure that there are perhaps a billion or so substantial objects in the main belt.

Anyway, the site ran mostly automated, with the sysadmin at Armagh maintaining it after I moved to California. Last year I found someone posting a decade old video to youtube and decided that it was time to do better, so I dug up the code. There's actually 2 versions, the old one is written in TCL/TK and uses a canvas widget to build the image and then dumps it as a postscript object, this is nice because it provides high level drawing primitives and the resulting image scales nicely for printed publication, it did however take about half an hour to render an image.

The newer version (dating from 1998) is a lean bit of C code, it doesn't have any graphics library, it has a framebuffer in memory which I dump to disk and then shell out to ImageMagick to convert to a final image. This is much faster, fast enough that I could render hundreds of frames and make a very basic video, but it was a far more primitive system, the drawing essentially consists of setting the pixel at coordinates x-y to colour RGB. Over time I've had a few requests from people for various videos and had to add features to the rendering like lines, motion blur and 3d transforms, however it's always been a process of iteration rather than replacing the whole thing with a 3rd party rendering engine like Open GL, If I can make that leap then it's possible we'll see this kind of thing at speeds approaching realtime.

The actual asteroid data came from Ted Bowell's project at the Lowell Observatory, they compile a file with the most up to date orbital elements of every observed minor planet, this kind of vizualization would be a whole lot harder were it not for this data. One thing that's missing is the discovery date, and this information wasn't conveniently available in one place, instead I identified a couple of sources and wrote a perl script to crawl the sites in question and merge the data into the asteroid database.  For some objects with provisional designations there was no discovery date information readily available and so the script guesses the discovery dates based upon the temporary designation.

Now I had all the data, rendering the movie took a few hours and the first versions of the movie weren't quite right, so I'd tweak the settings and try again until I arrived at the final version which you see on youtube.

What did you personally learn from creating this?

First thing I learned is if you're uploading a video to youtube you probably want to make it private until you're sure it's ready. The first version uploaded had no audio, and ran at 60 fps, I primarily uploaded it to check what youtube would do to the video encoding. I knew that all those tiny dots moving in different directions represented something of a pathologically hard case for most encoding systems and I feared it would be unwatchable after youtube got its hands on it.

Well it was watchable enough that it's just passed 800,000 views, it got popular very quickly and there was no chance to replace it with a polished version. The best I could do was add some audio using youtube's audio swap, and I was really lucky to find 'Emergence' by Trifonic, they were local producers from San Francisco and when I put the sound next to the video it just seemed to go well together. Oddly enough there was a scientific explanation for why the music seemed to work well with the video.

Remember I had to scrape all those sites for discovery dates and make guesses for others? When I ran my first test renders I noticed that the discovery rate seemed very bursty rather than smooth, I hadn't expected this and at first I thought the discovery dates were subject to some sort of bias that was being introduced by my scraping or guessing. I tried fixing this in a number of different ways, but couldn't eliminate it.  It wasn't until after the video had been up a few days that a smart person pointed out that there was a pulsing in the discovery rate that was most likely due to the Lunar cycle. When there's a full moon the moonlight makes it harder to see faint objects, so the discovery rate rises and falls on a monthly basis. The video was running at 1 day per frame, 60 frames per second, which meant that the pulsing in the discovery rates was almost 120bpm (beats per minute), and sure enough the music I'd gravitated to was recorded at 60 bpm, with the piano motif driving along at 120bpm.

It was a rare moment where by astronomical experience aligned with my DJ experience.

Genius!  What a nice moment.
Once I'd seen your asteroid video I started poking around and came across the animations you created of what it might look like if an asteroid came close to us, and hit us.  Is this a sort of public service broadcast, or is animation the natural end result when you start visualising this sort of data?

Well, I'm a big proponent of educating the public about the potential dangers due to celestial hazards, it's more like the reverse was true, the images and videos came about as an educational tool to help people visualize just how busy the solar system actually is.

It's very easy to take simple physics and figure out the energy released in the impact of even a small asteroid is quite staggering, this is simple high school physics - a 2000meter object moving at 20km/sec contains the same energy as 800billion tons of TNT,. The consequences of an impact could end life as we know it on earth, as the dinosaurs discovered 65million years ago. However we are not the dinosaurs, we have awareness of science now, and we can recognize the danger and moreover we have some pretty good ideas on how to avert such an event should we know about it. There have been big impacts throughout Earth's history that have had terrible consequences for the inhabitants of the planet, we're the first species that has come along with the ability to do something about it. But before you go and design your asteroid diverting rocket you have to know where everything is, and truthfully, we'll probably find that there's nothing likely to be a threat in the next hundred years of so, until we look though, we won't know.

I noticed that you're also involved in web search.  Can you let us know what your interests and goals in search are and how you got into this field?

I work at Topsy.com, we're providing a realtime search engine which indexes data from the social web, because we're aware of the social networks we factor in the influence and expertise of the participants when ranking  content.
Personally I'm forever fascinated with the data that is being shared and how the sharing dynamic can inform us about the quality and relevance of web content that's being discussed.

Thanks very much to Scott for enlightening me - hopefully, Nestorialings, you'll feel the same way I do and are glad I veered slightly off topic.