Fetching Android Market Stats with Selenium RC

(2010)

Finally.. I've got a reasonably decent way to pull Android Market stats. For some reason I keep coming back to this topic. This time, the way forward is to use Selenium RC, part of the Selenium browser testing suite.

My example will be in Python, but Selenium has bindings for several languages.

First of all, you gotta download Selenium RC from here: http://seleniumhq.org/download/

Then, extract it someplace you can remember. I've been putting things in ~/opt lately.

Okay, now create a new python script, comma ca:

import sys
sys.path.append('/the/path/to/selenium-python-client-driver-1.0.1')

from selenium import selenium

email = 'YOUR_GOOGLE_LOGIN'
passwd = 'YOUR_PASSWORD'

s = selenium("localhost", 4444, "*firefox", "http://market.android.com")
s.start()
s.open("/publish/Home")
s.type("Email", email)
s.type("Passwd", passwd)
s.click("signIn")
s.wait_for_page_to_load("30000")

n = int(s.get_xpath_count("//div[@class='listingRow']"))
for i in range(3,n):
try:
 title = s.get_text("xpath=(//div[@class='listingRow'])[%s]/div[1]/div[1]" % i)
 downloaded = s.get_text("xpath=(//div[@class='listingRow'])[%s]/div[2]/div[1]/span[1]" % i)
 installed = s.get_text("xpath=(//div[@class='listingRow'])[%s]/div[2]/div[2]/span[1]" % i)
 comments = s.get_text("xpath=(//div[@class='listingRow'])[%s]/table" % i)[1:-1]
 print title, downloaded, installed, comments
except:
 pass

This script is a bit of a trainwreck.. but it works and I don't feel like screwing with it..

To run the script, you need to start the Selenium RC server. Go to the place you downloaded it:

cd /path/to/selenium
java -jar selenium-server.jar

Then, you should be able to run this script from a terminal and it will start firefox, log you in to the Android Developer Console, wait a few seconds til the Ajax all loads, then use xpath to scrape each row of data from the table and print it to the terminal.

From there it should be pretty simple to export the results into a CSV file or make pretty charts or whatever it is you wanna do.

It does pop up a window on the screen, which is kinda annoying. Cooler to run firefox headless, maybe some other time..