Friday, October 29, 2010

The future's cool #5- SMT for everyone

Okay, hardcore nerdery here.  Sorry to the non-believers in the audience, but...

5.  SMT technology for hobbyists.

Some will argue the point, looking back with nostalgia at the days when every IC was available in through-hole and sockets abounded, but I for one welcome our minuscule highly integrated overlords.

Ten years ago, when I bought my first parts from Digi-key, it was inconceivable to me that SMT parts could be usable on an at-home basis.  Now, the proliferation of adapters (such as Capital Advanced Technologies' excellent "Surfboard" series), cheap hot-air and fine-point rework stations, excellent reflow tutorials, and extremely low-cost low quantity PCB fabrication services (like BatchPCB and the DorkbotPDX group buy), it's now completely and totally reasonable for a hobbyist to use SMT parts to realize extremely advanced designs in surprisingly small amounts of space.

All that's really needed is the right attitude.

Thursday, October 28, 2010

Two for one: the future is awesome 3 and 4

I missed yesterday (inauspicious, missing a goal two days in) but I'll make up by doing two today:

3.  CPU cycles are so cheap as to be practically free

While excessive wasting of CPU cycles still bugs me (do I REALLY need something checking to see if updates for Acrobat/Flash/iTunes are available 24/7?), I LOVE the fact that CPU cycles are so cheap that we can waste them on GOOD stuff- operating system eye candy, virtual machines that let me write extremely simple code to do REALLY complex stuff that is highly portable (*cough* PYTHON *coughcough*), and data compression that lets high bandwidth analog come through even fairly narrow (by modern standards) pipes with great quality.

4.  Text messaging

I LOVE text messaging.  Not the "omg lol i cant bleve u sed that" kind (as a grammar snob that stuff annoys me, although from a technical standpoint, it shouldn't), but the simple fact that it is (arguably) as information dense a means of communication as I can imagine.

The 160 character limit (artificial in this world of essentially unlimited bandwidth that's too cheap to meter) as well as the clunky UI (a limit of the small input device) means that we've adapted to pack a lot of information into a short message.  In my case, it tends to be "Pick up milk" or "Missed my bus"- messages that, transmitted by speech over the phone would involve lots of noise (chit-chat about how the day was, greetings, farewells, etc etc- stuff that can wait until I get home).  The ratio of truly important data to time required to read/write it is astronomical; I can read a text message in less time than it takes to dial the phone.

Tuesday, October 26, 2010

The future's pretty cool, day 2

2:  I <3 nearly-free data storage!

The first hard drive I remember owning was 40MB on the Mac LC my parents got when I was in 6th grade.

Four years later we upgraded to a Performa with a 500MB drive.

Two years after that, we hit 1.2GB.  Surely that will never fill up, I thought to myself.  Of course, then I got to college right in the midst of the MP3 boom...

Now I have two 250GB drives in my PC, and I can add another two terabytes (an unimaginable 50,000 times larger than my first drive) for less than $100.

Relative to that first drive, that's too cheap to even calculate.

Monday, October 25, 2010

The future's pretty cool, day 1

Public exercise:

I am going to attempt to provide an example of why living in the future is awesome every day for one year.  I'll never come up with 365 totally unique reasons, but maybe, JUST MAYBE, it'll help me remember that my life in the future is not as bad as I sometimes think it is and maybe I should be more grateful.

And no, this is not going to be a "neat stuff you probably already saw" series of posts- if I provide a link to an item, it's because I want to provide an example of what I'm talking about.

Day 1:  Cultural remixes

I really can't say I like R. Kelly, but I love this video, and I love living in an era where that level of remix is possible.

Tuesday, October 19, 2010

Parsing the Current, part 2

More data...and a pretty graph!

Okay, not pretty.

The three lines represent weekly tallies for total number of songs played, total unique artist names, and total unique tracks. Fun things this shows up:
1. Beginning in late 2007, the total number of songs per week trends up. I suspect that this represents a reduction in the amount of named programming in favor of either listing the tracks played during such programs or generally choosing to play songs instead of programs.
2. Dips in number of tracks at pledge drive time. Every six months or so (late May/early June and late October) there's a clear downward dip in the total number played, to the tune of about 15-20% or so. Hardly surprising, and they have to pay the bills. I still can't help but wondering if there's some kind of Laffer curve in pledge drives, where talking more decreases revenue because it alienates listeners, but talking less decreases revenue because you create less guilt.
3. The weekly diversity of tracks and artists clearly HAS been decreasing. There's a slow decline in diversity from early Q2 of 2006 on, with a sharp drop in track diversity in Q4 of 2007 and artist diversity in Q4 of 2008.
4. Really interesting are the three spikes in track diversity in Q2 '09, Q1 '10, and Q2 '10. After trying out a few theories about after-effects of the pledge drive, I remembered that they tend to do weekend countdown lists during Memorial Day weekends, which correspond to at least two of those blips. The third, in January 2010, is a mystery to me.

And now, some data: artist diversity across three semi-randomly* chosen weeks.
For the week beginning 23 Dec 2005 (2166 tracks):
1. Leela James (played 13 times, last on 2007-04-14 19:34:00)
2. Gang of Four (played 8 times, last on 2010-10-07 19:37:00)
3. Blackalicious (played 8 times, last on 2010-10-08 18:43:00)
4. Sun Kil Moon (played 7 times, last on 2010-10-12 13:40:00)
5. Matt Pond PA (played 7 times, last on 2010-10-09 04:34:00)
6. Teenage Fanclub (played 7 times, last on 2010-10-14 23:01:00)
7. David Bowie (played 7 times, last on 2010-10-14 22:46:00)
8. Diamond Nights (played 7 times, last on 2010-08-02 14:55:00)
9. Fiona Apple (played 7 times, last on 2010-10-13 19:11:00)
10. Spoon (played 7 times, last on 2010-10-14 17:12:00)

For the week beginning 6 Oct 2007 (2110 tracks):
1. Jose Gonzalez (played 14 times, last on 2010-09-26 22:52:00)
2. Spoon (played 13 times, last on 2010-10-14 17:12:00)
3. Band of Horses (played 13 times, last on 2010-10-14 07:19:00)
4. American Routes (played 12 times, last on 2007-11-11 08:32:00)
5. The Enemy (played 12 times, last on 2010-08-12 14:57:00)
6. Ryan Adams (played 12 times, last on 2010-10-08 18:00:00)
7. Steve Earle (played 12 times, last on 2010-10-13 15:55:00)
8. Radiohead (played 11 times, last on 2010-10-13 22:09:00)
9. Hot Hot Heat (played 11 times, last on 2010-10-11 14:22:00)
10. Sharon Jones and the Dap Kings (played 10 times, last on 2010-06-19 22:42:00)

Finally, for the week beginning 6 Oct 2010 (2286 total tracks) (get ready for a shocker!):
1. Cloud Cult (played 27 times)
2. Lissie (played 25 times)
3. LCD Soundsystem (played 23 times)
4. Robert Plant (played 23 times)
5. Of Montreal (played 22 times)
6. The Black Keys (played 19 times)
7. John Lennon (played 19 times)
8. Atmosphere (played 19 times)
9. Ray LaMontagne and The Pariah Dogs (played 19 times)
10. Mumford and Sons (played 19 times)

Holy cow! Back in 2005, the most popular artist for the week was Leela James, and she wasn't played quite two times a day. Fast forward to 2010, and you can guess that you'll hear FOUR Cloud Cult songs a day! Not only that, but some artists have gone "extinct"- Leela James went from being a top 10 to nothing in April of 2007. ("American Routes" is a radio program.)

Just as a measure, the 10 most popular tracks of the week in 2005 accounted for 78 0f 2166 tracks (3.6%), in 2007 it was 120 of 2110 (5.7%), and in 2010, 215 of 2286 (9.4%!). This meshes well with my observation that I hear the same songs over and over- I listen on the bus, mostly, for probably 2 hours a day, which I estimate to be about 20 songs a day or 100 a week. If 10% of those songs are coming from a list 10 songs long, I'm bound to hear the same songs several times over.

Again, I will reiterate that I really dig the Current. The DJs seem to REALLY love music, and their enthusiasm is infectious. I'm curious, however, as to how this Clear-Channel-esque playlist arose (anyone at MPR want to comment anonymously? :-)

I will also mention that at least one person I shared this data with responded along the lines of "I could have told you that" and indicated that he had dropped his membership because he no longer felt it was worth his support due to this drop in diversity (no, it's not me- I'm a sustainer and would be even if the Current goes away just for the news stations).

*I say "semi-randomly" because I know full well what a truly random choice would require; what I mean is that I chose them because the first one is at the beginning of my data set, the second is near the middle, and the third is near the end.

Monday, October 18, 2010

Data wrangling the Current

Arguably, one of the better things about Minnesota is Minnesota Public Radio. MPR is a giant on the national public radio scene (producing shows like "The Splendid Table", "A Prairie Home Companion", and "Marketplace", although "Marketplace" is recorded out in CA).

One of their interesting recent endeavors has been a pop music format station, "The Current". It's at 89.3 in the metro Twin Cities region, and it is (for better or worse)(and let's not argue the point) pretty much the only radio station I listen to.

I've noticed over the last couple of years (I've been listening since January of 2006; they came online in January of 2005) that the diversity of music seems to be declining somewhat. Not in the sense that they play more mainstream stuff, but in the sense that they are playing fewer songs more often.

So, I decided I'd do a little data wrangling. See, one of the outstanding features of the Current is their religious devotion to citing the songs they play. The artist and title are ALWAYS given on the air, AND they have a complete web accessible playlist of (pretty much) every song they've played since 5:00 AM, Dec 22, 2005. I say "pretty much" because they occasionally have a program or "guest DJ" which is listed on the playlist rather than the songs played during that period.

I wrote a Python script which fetched every day's playlist data since it was first available, does some formatting (stripping leading and trailing whitespace, replacing a few meta-characters with their intended, removing entries with empty song/artist entries), and concatenates it into one long file.

This post is all about the first pass I've made over the data- it's really quite basic stuff. I'll try and post some more interesting things later; many friends on Twitter seem to be interested in this little project and have suggested some excellent metrics to investigate. So, without further ado, here's what I can tell you so far:

Total number of unique artists: 13438
Total number of unique songs: 50369
Top 10 songs by number of plays:
1. [323] My Girls- Animal Collective (2009-01-10 22:12:00)
2. [317] Dominos- The Big Pink (2009-08-03 18:36:00)
3. [309] Two Weeks- Grizzly Bear (2009-05-09 16:13:00)
4. [307] Kids- MGMT (2008-04-01 21:05:00)
5. [304] 2080- Yeasayer (2008-01-08 03:50:00)
6. [299] French Navy- Camera Obscura (2009-04-11 16:36:00)
7. [299] Ambling Alp- Yeasayer (2009-11-06 19:26:00)
8. [298] Home- Edward Sharpe and the Magnetic Zeros (2009-09-21 20:07:00)
9. [298] Time To Pretend- MGMT (2007-12-04 11:07:00)
10. [296] How You Like Me Now?- The Heavy (2009-10-26 23:35:00)

Top 10 artists by number of plays:
1. Wilco (played 2413 times)
2. Spoon (played 2370 times)
3. The Hold Steady (played 2153 times)
4. Beck (played 2074 times)
5. Atmosphere (played 2006 times)
6. Radiohead (played 1943 times)
7. The Flaming Lips (played 1915 times)
8. Prince (played 1866 times)
9. Cloud Cult (played 1791 times)
10. Arcade Fire (played 1790 times)

Commentary- first, this data is PRETTY good but not VERY good. Ferinstance, the 22nd most played band is "R.E.M.". But, if their database entry for, say, "Stand", has the band as "REM" or "R. E. M." or any of a hundred other possible variations, my program is not (as yet) sophisticated enough to notice. So R.E.M. might well deserve to be higher if one of their songs is reasonably popular.

Second, I think it's interesting (and it meshes well with why I started this project) that all of the top 10 songs in terms of popularity were first played since the end of 2007 (the oldest in the top 10 has a "birthday" of 4 Dec, 2007). In fact, 22 of the top 25 were released since the beginning of 2009. I think I need a better way of characterizing this metric, though- perhaps by number of times played in the first month after the song premieres.

Third, and but perhaps most importantly, I don't want to sound like I'm down on these guys. I still love the station (for the music they play and for the commercials they DON'T play). I hope this doesn't tweak anyone's nose or get anyone mad at me- more than anything, I like parsing data to assuage my own curiosity, and this whole thing is also an excuse for me to play around with using Python to collect and parse a lot of data.

I'll add a few other items to this list over the coming days- half-life of particular songs, time-of-day correlation, playlist diversity for one day/week/month versus a given other day/week/month, and maybe a few more. If you have ideas, let me know!