Side-game data project / FPL data collection

A Forum for all Fantasy PremierLeague.com (FPL) Links, Mini-Leagues and Side Games (i.e. inter-FISO competitions run by FISO users)
Post Reply
User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

RomynPG wrote:.
Tacalabala wrote:Romyn, could you include the 'missing' data, in that what is present on the history page should be the baseline?
At the moment your sheet does the ones in bold:

GW points - GW points on bench - GW rank - GW transfers - GW cost - Overall points - Overall ranking - Value - Change from previous week
If "GW Cost" is tranfer penalties then that's the same as GW transfers (/4 ... WC excluded).
NO it's not the same

User avatar
RomynPG
Dumbledore
Posts: 8066
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

MoSe wrote:
RomynPG wrote:.
Tacalabala wrote:Romyn, could you include the 'missing' data, in that what is present on the history page should be the baseline?
At the moment your sheet does the ones in bold:

GW points - GW points on bench - GW rank - GW transfers - GW cost - Overall points - Overall ranking - Value - Change from previous week
If "GW Cost" is tranfer penalties then that's the same as GW transfers (/4 ... WC excluded).
NO it's not the same
What is it then?

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

MoSe wrote:
RomynPG wrote:.
Tacalabala wrote:Romyn, could you include the 'missing' data, in that what is present on the history page should be the baseline?
At the moment your sheet does the ones in bold:

GW points - GW points on bench - GW rank - GW transfers - GW cost - Overall points - Overall ranking - Value - Change from previous week
If "GW Cost" is tranfer penalties then that's the same as GW transfers (/4 ... WC excluded).
This theory only works if there was no such thing as a free transfer.

GW1 = 0
GW2 = 0 transfers/WC = 0, 1 transfer = 0, 2 transfers = (2-1FT)*4 = 4 etc
GW3 = 0 transfers/WC = 0, 1 transfer = 0, 2 transfers = (2-2FT) or (2-1FT due to WC in GW2)*4

User avatar
RomynPG
Dumbledore
Posts: 8066
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

FPLCasual wrote:
MoSe wrote:
RomynPG wrote:.
Tacalabala wrote:Romyn, could you include the 'missing' data, in that what is present on the history page should be the baseline?
At the moment your sheet does the ones in bold:

GW points - GW points on bench - GW rank - GW transfers - GW cost - Overall points - Overall ranking - Value - Change from previous week
If "GW Cost" is tranfer penalties then that's the same as GW transfers (/4 ... WC excluded).
This theory only works if there was no such thing as a free transfer.

GW1 = 0
GW2 = 0 transfers/WC = 0, 1 transfer = 0, 2 transfers = (2-1FT)*4 = 4 etc
GW3 = 0 transfers/WC = 0, 1 transfer = 0, 2 transfers = (2-2FT) or (2-1FT due to WC in GW2)*4
ah I get ya - thx - that was my fault for being confusing...

I should have said is GW Cost the same as transfer penatlies - and by transfer penalties I mean the points that may have been spend over-an-above any free transfers or WC.

I calc transfer penalties by using overall points and gw score compared to the previous weeks overall points. I don't know the actual transfer numbers - just whether it cost points or not so I can get Points Scored and Points nett of transfers

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Tacalabala wrote:Thank you :) thanks to you, FISO Cup is now up to date, so pat yourself on the back
I've added in the "zero rows" for future GWs as requested.

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

There was something weird in the FPL GW data (which had the old points value). I've changed the code to look in a different place to get the points tally.[/quote]

Is GWS correct or an error? Currently it total points instead of GW - hits.

https://ratemyteam.uk/Fiso/SideGameData/17443/2

User avatar
Moist von Lipwig
FISO Knight
Posts: 18228
Joined: 08 Jun 2011, 16:08
Location: The Eyrie
FS Record: FPL Spring 16 Winner 2010-11. Murfs F1 Predictions 2012 Winner. Pick Quick 2012-13 Winner. SP4s Predictions League A & Champions League

Re: Side-game data project / FPL data collection

Post by Moist von Lipwig »

Right, I definitely need help now. Does anyone know how to just automatically get the league table and results from this sample page? in a form that will go into excel.

https://fantasy.premierleague.com/a/lea ... 287451/h2h


Nothing I've tried to date has worked, however I'm relatively simple in this area. If it can't be done then I think I'd have to learn how to use the parsed data collection (well done on that by the way) and essentially replicate the entire set of H2H leagues fixtures myself and table it from that.

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

FPLCasual wrote:There was something weird in the FPL GW data (which had the old points value). I've changed the code to look in a different place to get the points tally.
Is GWS correct or an error? Currently it total points instead of GW - hits.

https://ratemyteam.uk/Fiso/SideGameData/17443/2[/quote]

No it wasn't correct, as you say the UI was rendering total points rather than total gameweek points. I've corrected the error. I have noticed that the connection(s) to the FPL site seem to fail on occasion, so if you get an error accessing the url the first time, then do a refresh and hopefully it will load at which point it gets cached. The error is that one of the connections to the FPL site fails and since I connect once per league and then once per league entrant, the failure has multiple chances of happening.

I wonder if they have some sort of throttling or something else that is causing it, I might put a bit of investigation into it over the weekend.

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Moist von Lipwig wrote:Right, I definitely need help now. Does anyone know how to just automatically get the league table and results from this sample page? in a form that will go into excel.

https://fantasy.premierleague.com/a/lea ... 287451/h2h


Nothing I've tried to date has worked, however I'm relatively simple in this area. If it can't be done then I think I'd have to learn how to use the parsed data collection (well done on that by the way) and essentially replicate the entire set of H2H leagues fixtures myself and table it from that.

What data do you need from it? Just a full copy of the data in a page that you can reference in Excel?

User avatar
Moist von Lipwig
FISO Knight
Posts: 18228
Joined: 08 Jun 2011, 16:08
Location: The Eyrie
FS Record: FPL Spring 16 Winner 2010-11. Murfs F1 Predictions 2012 Winner. Pick Quick 2012-13 Winner. SP4s Predictions League A & Champions League

Re: Side-game data project / FPL data collection

Post by Moist von Lipwig »

Mav3rick wrote:
Moist von Lipwig wrote:Right, I definitely need help now. Does anyone know how to just automatically get the league table and results from this sample page? in a form that will go into excel.

https://fantasy.premierleague.com/a/lea ... 287451/h2h


Nothing I've tried to date has worked, however I'm relatively simple in this area. If it can't be done then I think I'd have to learn how to use the parsed data collection (well done on that by the way) and essentially replicate the entire set of H2H leagues fixtures myself and table it from that.

What data do you need from it? Just a full copy of the data in a page that you can reference in Excel?

pretty much, yes

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

Mav3rick wrote:
FPLCasual wrote:
Mav3rick wrote:There was something weird in the FPL GW data (which had the old points value). I've changed the code to look in a different place to get the points tally.
Is GWS correct or an error? Currently it total points instead of GW - hits.
https://ratemyteam.uk/Fiso/SideGameData/17443/2
No it wasn't correct, as you say the UI was rendering total points rather than total gameweek points. I've corrected the error.
Well...
Net GW Points could be easily calculated as (P-H)
Total Overall Points can't be obtained from a single GW page

even if it wasn't the original intention, I realised that having the OP too in a GW summary could be useful
H2H leagues use OP in their ranking as tiebreaker for teams with the same H2H points
yes, OR can be used to the same effect, although the figure is bigger and less immediate to read

I know we're never satisfied, but at this point, could it be possible to have both? :mrgreen:
something like
P T H Net Tot
___

BTW, i realised your original Data Aggregator
https://ratemyteam.uk/Fiso/SideGameData/17443/2
doesn't work for H2H leagues
https://ratemyteam.uk/Fiso/SideGameData/287451/2
which could've been useful to MvL

neither does the League version GRID
https://ratemyteam.uk/Fiso/GRID/287451
I thought that just needed to get the IDs list from a League, and then would've worked from there in the same way...
...are H2H IDs stored differently than in Classic ones?
____

AH, but I see in the meanwhile you're already directly addressing MvL predicament :mrgreen:

BTW, automatically not,
but the H2H tables can be manually copy-pasted into excel
although the Team & Manager names on two disticnt rows are a little nuisance wfich forces you to a bit more toil
FISO H2H Champ gw2 into excel.gif
You do not have the required permissions to view the files attached to this post.

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

MoSe.... Yeah the H2H league data is different and comes from a different URL unfortunately.

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Moist von Lipwig wrote:
Mav3rick wrote:
Moist von Lipwig wrote:Right, I definitely need help now. Does anyone know how to just automatically get the league table and results from this sample page? in a form that will go into excel.

https://fantasy.premierleague.com/a/lea ... 287451/h2h


Nothing I've tried to date has worked, however I'm relatively simple in this area. If it can't be done then I think I'd have to learn how to use the parsed data collection (well done on that by the way) and essentially replicate the entire set of H2H leagues fixtures myself and table it from that.

What data do you need from it? Just a full copy of the data in a page that you can reference in Excel?

pretty much, yes
Any good:

https://ratemyteam.uk/Fiso/H2HData/287451

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

Mav3rick wrote:
FPLCasual wrote:There was something weird in the FPL GW data (which had the old points value). I've changed the code to look in a different place to get the points tally.
Is GWS correct or an error? Currently it total points instead of GW - hits.

https://ratemyteam.uk/Fiso/SideGameData/17443/2
No it wasn't correct, as you say the UI was rendering total points rather than total gameweek points. I've corrected the error. I have noticed that the connection(s) to the FPL site seem to fail on occasion, so if you get an error accessing the url the first time, then do a refresh and hopefully it will load at which point it gets cached. The error is that one of the connections to the FPL site fails and since I connect once per league and then once per league entrant, the failure has multiple chances of happening.

I wonder if they have some sort of throttling or something else that is causing it, I might put a bit of investigation into it over the weekend.[/quote]

Cheers for that. I just copy/paste into Excel then do Total - Hit in another column.
Is there any way to sort by Net GW score instead of Gross score? Would save the copy and paste but not a big issue.

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

No, I won't be adding sorting. It's just a maintenance thing that I don't want to deal with really.

User avatar
FPLCasual
Kevin and Perry
Posts: 19
Joined: 21 Feb 2016, 01:37

Re: Side-game data project / FPL data collection

Post by FPLCasual »

Mav3rick wrote:No, I won't be adding sorting. It's just a maintenance thing that I don't want to deal with really.
That's cool. Just a matter of copy/paste into Excel.
Cheers for all the hard work. Makes running big leagues much easier.

User avatar
Moist von Lipwig
FISO Knight
Posts: 18228
Joined: 08 Jun 2011, 16:08
Location: The Eyrie
FS Record: FPL Spring 16 Winner 2010-11. Murfs F1 Predictions 2012 Winner. Pick Quick 2012-13 Winner. SP4s Predictions League A & Champions League

Re: Side-game data project / FPL data collection

Post by Moist von Lipwig »

Mav3rick wrote:
Moist von Lipwig wrote:
Mav3rick wrote:
Moist von Lipwig wrote:Right, I definitely need help now. Does anyone know how to just automatically get the league table and results from this sample page? in a form that will go into excel.

https://fantasy.premierleague.com/a/lea ... 287451/h2h


Nothing I've tried to date has worked, however I'm relatively simple in this area. If it can't be done then I think I'd have to learn how to use the parsed data collection (well done on that by the way) and essentially replicate the entire set of H2H leagues fixtures myself and table it from that.

What data do you need from it? Just a full copy of the data in a page that you can reference in Excel?

pretty much, yes
Any good:

https://ratemyteam.uk/Fiso/H2HData/287451

Yes, that's ideal - thankyou, much appreciated.

The fixtures I can build up myself in advance using the rest of your data should I have the time or I'll just have to copy paste them as MoSe (thanks for looking at it also btw) suggests. Which is where I'd got to myself but was trying to avoid having to do that for everything. I suppose I could copy it in and sort out the name row/columns and record it as a macro and save a bit of time that way.

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

Moist von Lipwig wrote:....was trying to avoid having to do that for everything. I suppose I could copy it in and sort out the name row/columns and record it as a macro and save a bit of time that way.
I did the example for a single GW, but if you go to the "Matches" tab
https://fantasy.premierleague.com/a/lea ... h2h?page=1
you can copy-paste the whole season fixtures at once

OK you'll have to browse 8 pages to get to GW38
https://fantasy.premierleague.com/a/lea ... h2h?page=8
but once you copied the whole fixtures, copying the managers on a new column aside the teams (just copy the whole column 1 row up) then selecting the empty cells (with F5 Special) and removing their lines, can be done once for the whole season

Of course that's to be repeated for 6 Leagues :mrgreen:
it depends on your macro-threshold ;)
(how many steps you're willing to endure performing manually, times the required repetitions, before you prefer to automate it)
mine is rather high :oops:

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

posting it here also to share the info with Taca and other sidegame admins who might eventually stumble in this

todayI kept getting Error (" X There was an error!") when using
https://ratemyteam.uk/Fiso/GRID
to get the Divisionals data

at first I though it might be something related to the number of IDs inserted (we're now at 101, I had used it without problems after last GW with ~70)
but after some trials I found out there's only one ID causing this error, even when inserted alone
147250
that's chris2k_11 team, which history page opens otherwise normally
he's in 5AS Kanglas but RPG s using his custom VBA data fetch
otoh he was definitely in the 70-ish list last GW too, when the GRID loaded normally

removing that one now, the other 100 IDs got the grid loaded in less than half a minute

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Thanks for investigating it so thoroughly MoSe... Issue should be fixed now, it was to do with Janmaat, a new player in the league, causing an issue with player caches.

User avatar
Mystery
FISO Knight
Posts: 12474
Joined: 13 Oct 2005, 18:33
Location: Just about here
FS Record: 116th in TFF 05/06
Contact:

Re: Side-game data project / FPL data collection

Post by Mystery »

Do any of the boffins in here know the address for the JSON that drives the weekly points page - e.g. https://fantasy.premierleague.com/a/team/399668/event/3

The data in there is a prerequisite for me even considering running the Decathlon again.


EDIT: found it from earlier in this thread. Ta

User avatar
eagle224
Grumpy Old Man
Posts: 2069
Joined: 10 Nov 2005, 22:14
Location: England
FS Record: Average

Re: Side-game data project / FPL data collection

Post by eagle224 »

This has been such a helpful thread for getting my python scripts back up and running this season.

Here are a couple more JSON data URL's that folks may find useful:

All player, Club and Lookup data: https://fantasy.premierleague.com/drf/bootstrap-static
Next Gameweek Information: https://fantasy.premierleague.com/drf/bootstrap-dynamic
GameWeek Stats: https://fantasy.premierleague.com/drf/events/

Fixture History which includes players info for YC,RC,Pen,PenSave etc for finished gameweeks: https://fantasy.premierleague.com/drf/fixtures/
Fixtures Info for a gameweek: https://fantasy.premierleague.com/drf/f ... ?event={GW}

Not too sure what this one is for but seems to be a summarised list of fixtures, regarless of the number used at the end of the URL
https://fantasy.premierleague.com/drf/fixtures/1


All Player Stats to Date: https://fantasy.premierleague.com/drf/elements
Individual Player Hitory: https://fantasy.premierleague.com/drf/e ... /{PlayerID}

User avatar
eagle224
Grumpy Old Man
Posts: 2069
Joined: 10 Nov 2005, 22:14
Location: England
FS Record: Average

Re: Side-game data project / FPL data collection

Post by eagle224 »

Just found this information on GitHub - maybe useful for anyone running leagues via coding

User avatar
baganboy
Comfortably Dumb(ledore)
Posts: 5698
Joined: 05 Aug 2008, 06:59
FS Record: Double-Nelson

Re: Side-game data project / FPL data collection

Post by baganboy »

Read through this link, and got a little sentimental. You guys, I hope you know how amazing you are. Really.

I wish I could help, but 'I cannot code for toffees' is applicable for me too. It would not be for long though, My boy will be six months old in a few days, and by the time he grows up enough to pick up a computer, I promise to learn a programming language well enough to teach him.

User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

Mav, I don't think I ever properly thank you for setting up GRID, honestly that has been a lifesaver this season! Have we been able to sort out the H2Hs at all, or is it down to copy-paste? There is every chance that I may run the Spring League off Excel and not FPL next season, it's just less of a ball ache each week!

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

Hi guys, would anyone be able to use the JSON data to get the FPL points scored by each player each week? If it is possible, would it be very difficult to do that? I am looking for data in the excel spreadsheet format of first row being the headers: eg Gameweek, Name, Position, Appearances, Goal Scored, Assist, Yellow card, Mins played, etc. Then the rows below it would be the values for each gameweek. I am still unable to learn and make sense of using language like python to parse the JSON data out from the FPL data links mentioned in this thread.

User avatar
RomynPG
Dumbledore
Posts: 8066
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

My download code has stopped working again :(

I believe it's to do with the https encryption and my system is so old (XP & Excel2000) that it can't deal with what's coming now. (I'm aware various ciphers are being withdrawn over time).

Long story short though ... fixing this is not something that I can do overnight and as there's only a couple of GWs left would it be possible for someone with the wherewithall can give me a csv for the 5AS?

My needs are simple (for the squads in 5AS) ...squadID, GW36 points scored, overall points ... from that I can do the rest for H2H & Champs.

Hoping someone can help me avoid manual labour :)

Thanks
Romyn

edit: I can provide a list of leagueIDs/squadIDs

User avatar
Tort
Grumpy Old Giant
Posts: 1794
Joined: 22 May 2006, 22:17
Contact:

Re: Side-game data project / FPL data collection

Post by Tort »

can't help sorry :)

thesilkworm
Grumpy Old Man
Posts: 1554
Joined: 01 Oct 2009, 17:59

Re: Side-game data project / FPL data collection

Post by thesilkworm »

RomynPG wrote:My download code has stopped working again :(

I believe it's to do with the https encryption and my system is so old (XP & Excel2000) that it can't deal with what's coming now. (I'm aware various ciphers are being withdrawn over time).

Long story short though ... fixing this is not something that I can do overnight and as there's only a couple of GWs left would it be possible for someone with the wherewithall can give me a csv for the 5AS?

My needs are simple (for the squads in 5AS) ...squadID, GW36 points scored, overall points ... from that I can do the rest for H2H & Champs.

Hoping someone can help me avoid manual labour :)

Thanks
Romyn

edit: I can provide a list of leagueIDs/squadIDs
Hi Romyn - the attached should do it. Let me know if you need anything else. Just started dabbling in this stuff and it's a good learning exercise for me (especially to confirm the Aristocrats as champions :D).
You do not have the required permissions to view the files attached to this post.

User avatar
RomynPG
Dumbledore
Posts: 8066
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

thesilkworm wrote:Hi Romyn - the attached should do it. Let me know if you need anything else. Just started dabbling in this stuff and it's a good learning exercise for me (especially to confirm the Aristocrats as champions :D).
thanks thesilkworm - just what I needed 8-)

(... I hope you don't mind if I request the same again once GW37 is done)

View Latest: 1 Day View Your posts
Post Reply

Return to “FPL Links, Mini-Leagues & Side Games”