Punch Through Design Blog

Founded in Minneapolis, based in San Francisco. We love the Internet of Things, Bluetooth Low Energy, and surfing.


Guerrilla Market Research - Scraping SparkFun


Once upon a time I had an idea for a business in the hobby electronics market.  Assuming the market was too small, I tried to do some market research.  Not finding any good existing research, I looked to SparkFun.  They are the undisputed champ of the hobby electronics market, so it makes sense to use them as a gauge.  They do release some business info (a lot more recently) but no specific numbers on individual products, inventory turnover, etc.

However, their product pages display live stock numbers… 


What’s that you ask?  What if we write a script that pulls these numbers every day, giving a day by day, product by product account of all inventory (and therefore sales)?  Well… now that is an idea.

So I wrote a script that reads sparkfun’s live inventory numbers, all of them, every night, and record them in daily spreadsheets.  Subtracting yesterdays inventory from today’s gives a rough idea of sales.  Once set up with a cronjob on my ubuntu server, it would run at 3am every day and I could forget about it for awhile.  So began my guerrilla style market research.

I’m writing this post after collecting over 9 months of data.  It would be more but sparkfun changed their website a bit, breaking my cheaply written scraping script.  I have since fixed it, for anyone interested.  Below I’ll link you to the data from March 13th, 2009 until December 5th, 2009.  The charts are based off data from July 2009.

The script isn’t perfect, if there are timeouts on a category, it is skipped.  So an entire category may be left out for a day.  However, the code is provided, fix it!  As lazy textbooks say: “it is left as an exercise for the reader”.  I’ve always wanted to say that.

What sells?

The simple stuff.  1k Resistors, LEDs, screws, jumper wires.  The first 9 days I recorded saw sparkfun going through 5,122 1k resistors, and 3,184 RGB LEDs. Also, as expected, Arduinos, which made up 14% of the revenue during the month I analyzed.  


Remember we don’t know SparkFuns profit margin, so these are pure revenue numbers.  This is important because certain products, like breakout boards, are high profit products, while others are resold products with a retailer-like margin.  In the time frame I analyzed, here are the top selling products:


Take this with a grain of salt.  It’s not exact.  There are times when the script failed- either because of a time out, or for a month when SparkFun changed their site and I failed to update the script.  I compared my monthly revenue recorded against the numbers SparkFun has made public, and I am short.  This means not everything is getting counted.  

However, if you have a product that you sell on Sparkfun, or just want to get an idea of what sells when, it’s worth looking at.  The code may or may not work out of the box, but I’ll leave that as a protection to SparkFun to let their servers breathe easier.  

Here is a link, to the code and my results for the above graphs.  The data is from July 2009.  You’ll have to pull prices from the masterList-prices file to generate revenue.   I have a little more data from earlier, here is a link to that.

For more info on the hobby electronics market, go here: http://www.sparkfun.com/news/358.  Sparkfun is over 10M in annual revenue, and supposedly there are 10 companies with over 1M in revenue.  Not bad!  It makes me really happy to see there are so many developing builders,makers, engineers, etc.