Side-game data project / FPL data collection

A Forum for all Fantasy PremierLeague.com (FPL) Links, Mini-Leagues and Side Games (i.e. inter-FISO competitions run by FISO users)
Post Reply
User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

I swear you could at least access the player history page, certainly not the squad for obvious reasons. Maybe I'm going senile, it is a possibility given the way work has gone this week :D

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Tacalabala wrote:
To: support@mail.fantasy.premierleague.com
Subject: Other player's history pages
Date: Wed, 20 Jul 2016 22:13:03 +0100

Good evening,

Firstly, may I take the opportunity to congratulate you on a visually stunning update of the game.

Please can you resolve a problem that I appear to be having with the new set-up? I have previously been able to access other player's history pages by clicking on squad/usernames on league pages, but this is not possible at this time. Is this only a temporary restriction which will be reinstated in the coming days?

Kind regards
Will let you know what the response is - they really have to bring this back in, crazy to stop this.
My hazy memory is the same as Stem, in that the links only appear for entrants, not pending entrants.

The thing I have been working through initially with MoSe has links to the pending entrants teams if it's helpful to you though.

User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

Mav3rick wrote:
Tacalabala wrote:
To: support@mail.fantasy.premierleague.com
Subject: Other player's history pages
Date: Wed, 20 Jul 2016 22:13:03 +0100

Good evening,

Firstly, may I take the opportunity to congratulate you on a visually stunning update of the game.

Please can you resolve a problem that I appear to be having with the new set-up? I have previously been able to access other player's history pages by clicking on squad/usernames on league pages, but this is not possible at this time. Is this only a temporary restriction which will be reinstated in the coming days?

Kind regards
Will let you know what the response is - they really have to bring this back in, crazy to stop this.
My hazy memory is the same as Stem, in that the links only appear for entrants, not pending entrants.

The thing I have been working through initially with MoSe has links to the pending entrants teams if it's helpful to you though.
Image

User avatar
SuperGrover
Grumpy Old Man
Posts: 1528
Joined: 12 Sep 2012, 15:38
Location: Chicago, USA

Re: Side-game data project / FPL data collection

Post by SuperGrover »

thesilkworm wrote:I'm just curious, what do the people who are contributing to this thread do for a living? Do you use these skills in your work? Obviously feel free to ignore this question :)
Used to be my job, now I tell people who do this for a living what to do :D

My skills are antiquated but still there to some extent.

User avatar
MoSe
Dumbledore
Posts: 9385
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

IIRC
1) you could click on Team names from a League Standings, which don't exist yet at this stage, leading to their team page for the current GW
2) and you could click on Manager names for the teams listed under Pending Entrants (to be added after the next points update) leading to their GW History

1) while I agree that it would be moot in the first case anyway, as it there'd be nothing to show yet before the 1st deadline,
2) I might instead like to check the Season History for players who'd gonna be playing in my League,
and that page IS definitely accessible even at this stage
2) it's indeed regrettable that we can't reach that info now, unless a team owner discloses his new ID, or we get the link to Mav3rick's goldmineImage
1) but it would be outright ABSURD if we couldn't check other teams from league standings once the season started

I can't indeed recall how it used to be tho before the 1st deadline, hopefully BOTH features will resume working as I recalled once the season properly starts
______

  • PS:
    the :goob: wasn't directed at SG, but it's a mock code conveying the trademark gooberman's "update your FPL button"
:mrgreen:

User avatar
MoSe
Dumbledore
Posts: 9385
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

BTW, if we get a League ID now, but we don't know the League owner FPL ID, we can't "hack" anymore the League CODE to join it....

which we know it was simply "OwnerID-LeagueID"
for instance for this season Fiso Forum League
viewtopic.php?f=82&t=121329
it's Spencer4 ID 95692 dash LeagueID 31821

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

MoSe wrote: 2) it's indeed regrettable that we can't reach that info now, unless a team owner discloses his new ID, or we get the link to Mav3rick's goldmine
Just to be clear, that works right now for any league id, not just the FISO example I gave.

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

I just want to say you guys are amazing and doing a great service! Especially Mav3rick! Sorry I can't contribute as I don't know these stuff (yet). Good luck for the season!


Sent from my iPhone using Tapatalk

ChrisA
Kevin and Perry
Posts: 12
Joined: 04 Jun 2015, 20:41

Re: Side-game data project / FPL data collection

Post by ChrisA »

Hey- does anyone have a link that can help one scrape the fixtures?

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

ChrisA wrote:Hey- does anyone have a link that can help one scrape the fixtures?
The fixture data is all contained here (online parsed version).

You'll need the lookup data here (online parsed version) (or a manual note of the team IDs) to work the data into a meaningful structure.

If you need an HTML scrape, I can't see a list of all fixtures, but you can step through all 38 GWs like so
https://fantasy.premierleague.com/a/fixtures/1
https://fantasy.premierleague.com/a/fixtures/2

ChrisA
Kevin and Perry
Posts: 12
Joined: 04 Jun 2015, 20:41

Re: Side-game data project / FPL data collection

Post by ChrisA »

The fixture data is all contained here (online parsed version).

That's exactly what I wanted- thank you very much! :)

User avatar
RomynPG
Dumbledore
Posts: 8070
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

Mav3rick wrote:
MoSe wrote: 2) it's indeed regrettable that we can't reach that info now, unless a team owner discloses his new ID, or we get the link to Mav3rick's goldmine
Just to be clear, that works right now for any league id, not just the FISO example I gave.
Awesome - that's made 5AS setup alot easier. (Captains not being able to get the squadIDs would have presented cat-herding possibilities)

Thanks dude :P

...not happy about my predicted score for GW1 though :? - https://ratemyteam.uk/Fiso/SideGameData/17443/1

User avatar
Surprised
FISO Jedi Knight
Posts: 26413
Joined: 13 Oct 2005, 18:32
Location: Home
FS Record: TFFOSM MotW in 2008 and MotM in 2003. 78th overall in TFFO for 2002/3 and 2003/4

Re: Side-game data project / FPL data collection

Post by Surprised »

Stemania wrote:Are you sure it isn't always like this before the first update since it isn't practical to see what team anyone has picked yet? The list of teams under each league aren't strictly speaking actually in he league yet, just to be added after the first update I think. :?
I think this is right. The leagues are not populated until the 1st update is completed

User avatar
RomynPG
Dumbledore
Posts: 8070
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

Surprised wrote:
Stemania wrote:Are you sure it isn't always like this before the first update since it isn't practical to see what team anyone has picked yet? The list of teams under each league aren't strictly speaking actually in he league yet, just to be added after the first update I think. :?
I think this is right. The leagues are not populated until the 1st update is completed
You used to be able to go to the gw history page for squads listed as pending-join in leagues.

User avatar
Surprised
FISO Jedi Knight
Posts: 26413
Joined: 13 Oct 2005, 18:32
Location: Home
FS Record: TFFOSM MotW in 2008 and MotM in 2003. 78th overall in TFFO for 2002/3 and 2003/4

Re: Side-game data project / FPL data collection

Post by Surprised »

RomynPG wrote:
Surprised wrote:
Stemania wrote:Are you sure it isn't always like this before the first update since it isn't practical to see what team anyone has picked yet? The list of teams under each league aren't strictly speaking actually in he league yet, just to be added after the first update I think. :?
I think this is right. The leagues are not populated until the 1st update is completed
You used to be able to go to the gw history page for squads listed as pending-join in leagues.
sorry yes...pending members you could see but if you tried to download the league details it wouldn't work

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

Mav3rick wrote:
thesilkworm wrote:I'm just curious, what do the people who are contributing to this thread do for a living? Do you use these skills in your work? Obviously feel free to ignore this question :)
Yep, it's my day job (which rather irritatingly eats into my FPL time).
MoSe wrote:webqueries from excel were a handy and readymade built-in tool to use for us non-coders
Ok MoSe, I had a couple of hours spare after lunch, so I put this together as a starting point, the URL contains the FPL League Id and the gameweek number, I get all the participants in the league, get their GW scores and return everything in a nice simple table:

Here's an example using the FISO league.

The gameweek data is randomised for now obviously, I'll need to return real data for the listed fields for my own site, but I could add other columns with relative ease I think. It's all static and simple HTML so Excel should be fine, if anybody wanted the actual JSON then I can expose that too. The data (which is all public info) is cached for 10 minutes but not persisted anywhere other than the FPL website.

Format of the URL is simply
https://ratemyteam.uk/Fiso/SideGameData ... /<GWNumber>

Is that useful to side game admins?
Would there be any chance to include hits if they aren't already included as the GW score isn't the H2H score when hits are taken? Hits and Net Score would be great additions.
I wouldn't mind the actual JSON used if possible?

Cheers and keep up the great work.

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.

User avatar
Stemania
FISO Jedi Knight
Posts: 20363
Joined: 27 Aug 2006, 11:54
Location: On the Iron Throne of xG, the seat of The Crown Prince of the Stat Perverts. Or if not, in the STC!
FS Record: Best: TFF 321st. FPL 129th. FFS Career HoF 2nd.

Re: Side-game data project / FPL data collection

Post by Stemania »

If FISO could do knighthoods...

Ok, you already have one, but a real one. :mrgreen:

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
Cheers, had a play around with the ones you posted earlier. A JSON for a league in a similar format to the RMT/leagueID/GW would be great for any league runners.
Just over 2.5 weeks to go :)

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

FPLCasual wrote:
Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
Cheers, had a play around with the ones you posted earlier. A JSON for a league in a similar format to the RMT/leagueID/GW would be great for any league runners.
Just over 2.5 weeks to go :)
Try doing a GET from the same URL format as above, but modify your client to send an "Accept" header with the value "application/json". That should trigger content negotiation and send you back a JSON result rather than HMTL.

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
Seems that your table got the table order correct but has the wrong scores.

User avatar
RomynPG
Dumbledore
Posts: 8070
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

Now that the first update has been done to the league pages I find my download code - which has been good (with tweaks) for 10 years now - no longer works :?

Not sure how best to sort it as there's no useful data in the raw html - it looks like it's all enhanced after init which doesn't work for Excel.

How are others getting on? ....

...going to have to look at Excel 2000's json capabilities :)

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

FPLCasual wrote:
Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
Seems that your table got the table order correct but has the wrong scores.
The scores are deliberately random rather than wrong. I've not updated the code to pull the real scores yet (the data was only available today so give me a chance :D ). I should be able to do it before the end of the weekend though as I've already been looking at the data available.

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

Mav3rick wrote:
FPLCasual wrote:
Mav3rick wrote:Once we get an actual set of FPL gameweek data I'll add those things. The score is inclusive of hits, so it would be easy to add a gross and net GW score with the hit amount shown.
Seems that your table got the table order correct but has the wrong scores.
The scores are deliberately random rather than wrong. I've not updated the code to pull the real scores yet (the data was only available today so give me a chance :D ). I should be able to do it before the end of the weekend though as I've already been looking at the data available.
Looks like you have already mastered it :)

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

RomynPG wrote:How are others getting on? ....
I've updated the earlier post with details of accessing the gameweek data.

The sidegame data link has been updated too, to include the GW scores. I haven't completely checked all the data (to ensure it's correct) so let me know if you spot anything that's wrong. Here's an example for one of the FISO 5AS leagues:

https://ratemyteam.uk/Fiso/SideGameData/462148/1

(As I said in an earlier post, if you can modify your client to send an "Accept" header with the value "application/json" then you can use the same url to return the table data in JSON rather than HTML).

User avatar
MoSe
Dumbledore
Posts: 9385
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

At first glance, it works perfectly
I assume Hits are TC from the GW history, of course,
then Score should be GP from history, that is the GW whole pts without deduction
While Points should be Score - Hits (GP-TC aka the Net Score)
I say should as of course on GW1 there are no TC to check which is which ;)
I also see that your GS or minutes or BP totals for a team, only counts the men on the pitch, not the benchers, which is correct
GS and GC in particular are used that way as tiebreakers fou H2H matches
the other summary data are indeed very interesting when comparing team totals in a league, although I don't know of any sidegame using them
_______

showing GP and TC separately was indeed a general requirement for Sidegames, as that's one of the usually customised options: I mean, several games use the GP without TC, several other sidegames use the net score, deducting TC
It was important to present both data, to allow any sidegame runner to use them in the appropriate way for their game format.
For instance, in Divisionals I follow the FPL classic scoring for Phases (i.e. Months)
TC get normally deducted, except on the 1st GW of a phase (GW4 scores won't be deducted from your September FPL score)

At this point, I figure it would be easy to add more fields in the table, from the gw history page

TM
(that's not the same as hits, as TC depends whether you had 1FT or 2FT)
FPL use TM as tiebreaker
Most sidegames don't
Mine does: I just reinstated TM as 1st tiebreaker, after it had been dropped for a few seasons.
If I can't access that data in the table too, I'd have to change the tiebreaker rules once again :mrgreen:

PB / £
I don't know of any sidegame using that data, but as they're available, someone might eventually create a new sidegame using them as a scoring/ranking/tiebreaking element...

User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

:| My web queries don't work!

I'm going to need help, otherwise I really don't know what I'm going to do. Spreadsheet and iqy files attached.
You do not have the required permissions to view the files attached to this post.

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

MoSe wrote: ... Lots of good work...
Thanks MoSe. I've added in the properties you suggested and also an indicator for any active chips. I've also made the table a lot wider, but beware that it really is only suitable for desktop browsers as you need quite a lot of horizontal space. I think that's fine though as I can't imagine many people try to run their side games on mobile devices!

User avatar
Mav3rick
FISO Knight
Posts: 18587
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Tacalabala wrote::| My web queries don't work!

I'm going to need help, otherwise I really don't know what I'm going to do. Spreadsheet and iqy files attached.
I've had a look at your spreadsheet, it looks like most of the data you need on the "Data-By-GW" tab would be available on the link above. So for example, the FISO 5AS Bad Boys league has things like the GW score, points benched, transfer hits, team value etc.

If you can link up your FISO manager name to the FPL Team Id, then you should be able to import all the GW score data you need? I'd think that one web query to pull in all the GW scores from the FISO league into your Excel workbook would be your best bet, and then you can work out any massaging of the data within Excel if you need to keep any existing sheet formats?

This is the FISO league (31821) (it can take 20-30 seconds to load as there are 1,000+ entrants to get the GW scores for). Can you update your web query to pull in the information from that table?

If there is data that you're missing, let me know what you need and I'll try to retrieve it from the FPL JSON and display it as an extra column in the table for you to pull down. SG offered earlier to help out creating Excel files from the data, I suspect that some sort of standard FISO format downloadable Excel file would be useful, but if I'm honest I don't want to take on maintenance of that on top of the raw data and JSON parsing. I don't mind collaborating though and I can extract and present the relevant FPL data, if someone else can mould it into a useful Excel workbook.

User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

Ignore Data by GW, its superfluous - GRID is the vital data. I'll attached last season's and then you'll see what I mean

View Latest: 1 Day View Your posts
Post Reply

Return to “FPL Links, Mini-Leagues & Side Games”