ProRanks history video
-
- Series 72 Champion
- Posts: 333
- Joined: Fri Nov 07, 2008 9:42 pm
ProRanks history video
I made one of those moving bar charts showing ProRanks over time: https://www.youtube.com/watch?v=YfWpClM ... e=youtu.be
Only shows the top 20 as some weeks don't have a full top 25 because of deactivated accounts. Very low-effort recording technique as well.
Only shows the top 20 as some weeks don't have a full top 25 because of deactivated accounts. Very low-effort recording technique as well.
- Ronan M Higginson
- Enthusiast
- Posts: 292
- Joined: Fri Apr 19, 2019 5:11 pm
Re: ProRanks history video
Great vid!! Thanks very much
- Matt Morrison
- Post-apocalypse
- Posts: 7822
- Joined: Wed Oct 22, 2008 2:27 pm
- Location: London
- Contact:
Re: ProRanks history video
nice work Tom!
What was Ned up to with his 2348? Who is Jake Thornhill?
What was Ned up to with his 2348? Who is Jake Thornhill?
-
- Post-apocalypse
- Posts: 13318
- Joined: Mon Jan 21, 2008 10:37 pm
Re: ProRanks history video
Cool. Is this a "thing" then? I saw something like this relating to Formula One only a couple of days ago.
- Jon O'Neill
- Ginger Ninja
- Posts: 4551
- Joined: Tue Jan 22, 2008 12:45 am
- Location: London, UK
Re: ProRanks history video
Yeah it's a thing, and this is a great example of such a thing.
I remember Pendleton's rise to dominance. Amazing part of apterous history.
Nice.
I remember Pendleton's rise to dominance. Amazing part of apterous history.
Nice.
- Jon O'Neill
- Ginger Ninja
- Posts: 4551
- Joined: Tue Jan 22, 2008 12:45 am
- Location: London, UK
Re: ProRanks history video
It's so nice that I won't take it personally that I have been excluded! I hope it's because of my stupid punctuated name rather than a personal vendetta.
- Thomas Carey
- Kiloposter
- Posts: 1481
- Joined: Sun Jan 09, 2011 4:17 pm
- Location: North-West of Bradford
- Contact:
-
- Series 72 Champion
- Posts: 333
- Joined: Fri Nov 07, 2008 9:42 pm
Re: ProRanks history video
Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.Jon O'Neill wrote: ↑Tue Nov 19, 2019 12:05 am It's so nice that I won't take it personally that I have been excluded! I hope it's because of my stupid punctuated name rather than a personal vendetta.
- Ian Volante
- Postmaster General
- Posts: 3967
- Joined: Wed Sep 03, 2008 8:15 pm
- Location: Edinburgh
- Contact:
Re: ProRanks history video
Nice, but I must grumble about the excision of my week in 25th!
meles meles meles meles meles meles meles meles meles meles meles meles meles meles meles meles
-
- Devotee
- Posts: 539
- Joined: Thu May 14, 2009 8:39 am
- Location: Seated at a computer
Re: ProRanks history video
Would this non-alpha-character gremlin also affect Tom Chafer-Cook, who had a peak ProRank of 3rd?Thomas Cappleman wrote: ↑Tue Nov 19, 2019 10:08 am Oh whoops. Yes, that actually is the reason - my code gathering the data from each page expected names to contain letters and spaces only. Also means that any week that you should be in is wrong, as everyone else's rating will be shifted by one. Will upload a fixed version later.
Also, I have no time for Ian's gripe... some of us round here have a peak ProRank of 21st.
-
- Newbie
- Posts: 27
- Joined: Fri Jun 30, 2017 1:54 pm
Re: ProRanks history video
This is great Tom. I was considering trying to do something similar a few weeks back as part of my efforts to learn R, but couldn't raise the effort to scrape the data - what did you use for that piece?
-
- Series 72 Champion
- Posts: 333
- Joined: Fri Nov 07, 2008 9:42 pm
Re: ProRanks history video
My Python code's below. I used requests (a nice intuitive module) to just get each page in turn, then regexed the output to find usernames and rankings.Neil A Collins wrote: ↑Sun Dec 01, 2019 2:09 pm This is great Tom. I was considering trying to do something similar a few weeks back as part of my efforts to learn R, but couldn't raise the effort to scrape the data - what did you use for that piece?
I'm sure there'd be a better way to query the actual database, especially as my method can't get anything beyond the top 25, as it's all the same URL.
Code: Select all
import requests
import re
import xlsxwriter
from datetime import timedelta, date
# Fill in your details here to be posted to the login form.
payload = {
'inUserName': 'thomas@cappleman.co.uk',
'inUserPass': <password>
}
# Use 'with' to ensure the session context is closed after use.
with requests.Session() as s:
p = s.post('https://www.apterous.org/proranks.php?period=105392', data=payload)
# print the html returned or something more intelligent to see if it's a successful login page.
print p.text
user_dict = {}
# An authorised request.
# 104804
for period in range(104804, 105393):
r = s.get('https://www.apterous.org/proranks.php?period=%s' % period)
users = re.findall( r'href="viewuser\.php\?user=[0-9]*">([^<]*)</a><span class="rating">', r.text)
ratings = re.findall(r'"rating">(?:<br />Pro Rank: )?([0-9]*) <span', r.text)
proranks_date = re.findall(r'<h3>Published ([0-9a-zA-z ]*)</h3>', r.text)
for x in range(min(len(users), len(ratings))):
user = users[x]
if user not in user_dict:
user_dict[user] = {proranks_date[0]: ratings[x]}
else:
user_dict[user][proranks_date[0]] = ratings[x]
workbook = xlsxwriter.Workbook('proranks.xlsx')
worksheet = workbook.add_worksheet()
def daterange(start_date, end_date):
for n in range(int ((end_date - start_date).days) / 7):
yield start_date + timedelta(n) * 7
start_date = date(2008, 8, 11)
end_date = date(2019, 11, 25)
worksheet.write(0, 0, "Name")
column = 1
for single_date in daterange(start_date, end_date):
worksheet.write(0, column, single_date.strftime("%-d %B %Y"))
column += 1
row = 1
for user, user_info in user_dict.items():
worksheet.write(row, 0, user)
column = 1
for single_date in daterange(start_date, end_date):
formatted_date = single_date.strftime("%-d %B %Y")
if formatted_date in user_info:
worksheet.write(row, column, user_info[formatted_date])
column += 1
row += 1
workbook.close()