Side-game data project / FPL data collection

A Forum for all Fantasy PremierLeague.com (FPL) Links, Mini-Leagues and Side Games (i.e. inter-FISO competitions run by FISO users)
Post Reply
User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

RomynPG wrote:
thesilkworm wrote:
RomynPG wrote:My download code has stopped working again :(

I believe it's to do with the https encryption and my system is so old (XP & Excel2000) that it can't deal with what's coming now. (I'm aware various ciphers are being withdrawn over time).

Long story short though ... fixing this is not something that I can do overnight and as there's only a couple of GWs left would it be possible for someone with the wherewithall can give me a csv for the 5AS?

My needs are simple (for the squads in 5AS) ...squadID, GW36 points scored, overall points ... from that I can do the rest for H2H & Champs.

Hoping someone can help me avoid manual labour :)

Thanks
Romyn

edit: I can provide a list of leagueIDs/squadIDs


Hi Romyn - the attached should do it. Let me know if you need anything else. Just started dabbling in this stuff and it's a good learning exercise for me (especially to confirm the Aristocrats as champions :D).

thanks thesilkworm - just what I needed 8-)

(... I hope you don't mind if I request the same again once GW37 is done)

excellent :D (as a 5AS player! :mrgreen: )

btw, I think the GRID tool Mav had provided for Taca's and mine sidegames, would have helped in this case as well
I had anyway all teams ID in my "who plays what" topic and online file
would just require a bit data shuffling and referencing, but no coding

of course I can't compete with ease-of-use of ad-hoc written code,
but you can hold me for sure as backup, just in case

thesilkworm
Grumpy Old Man
Posts: 1554
Joined: 01 Oct 2009, 17:59

Re: Side-game data project / FPL data collection

Post by thesilkworm »

Happy to do again next week. PM or quote-reply to this at end of gameweek if I forget :)

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

MoSe wrote:btw, I think the GRID tool Mav had provided for Taca's and mine sidegames, would have helped in this case as well
...
you can hold me for sure as backup, just in case
here, took the 5AS template you had shared earlier in this topic, trying to keep your formatting and architecture.
I only expunged all teams not in 5AS, for compactness
FISO 5AS 2016-17.zip
it took me, say, 20 minutes work
it actually passed more than that in "clock time" but I only dedicated to it my "hourglass time" :wink: , i.e. the pauses between one work task and the other :mrgreen: , so I figure 20' or at most 30'
I copied there verbatim the "How to Update" instructions I had prepared for young Fiso member samyadav to help him update his Fiso Huddle H2H league which started in January, so sorry for the maybe patronising tone but I wanted clarity for an unexperienced sidegame manager ;) which is of course not your case

only data which might not be completely exact is the League rank for past GWs, as I know you don't completely follow FPL Leagues ranking, and I didn't go back now to review eventual ties according to your 5AS rules there

it contains current GW37 provisional points as of yesterday matches, but I left the formulas in for the GW in progress, so all that's needed is to run Mav's GRID tool again at its end, and update the GRID in the file, for all the scores to update autmatically

of course I leave it to silkworm to provide the same data if that's easier, the above is just a backup ...and an "exercise" ;)
and mind,
haven't done this to be meddling or prove anything,
but shared it in this topic just to show how Mav's grid tool, developed following Taca's requirements, it's actually a rather versatile and effective tool, allowing to tackle the management needs of several and diverse sidegames! :D
...hoping it won't have to get rehauled anew next season at FPL's whim :o
You do not have the required permissions to view the files attached to this post.

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

FISO 5AS 2016-17 GW37.zip
UPDATED with GW37 final scores
(fyi, it took less than 5 minutes, 2 thereof for Mav's tool to load the data)
You do not have the required permissions to view the files attached to this post.

thesilkworm
Grumpy Old Man
Posts: 1554
Joined: 01 Oct 2009, 17:59

Re: Side-game data project / FPL data collection

Post by thesilkworm »

Thanks MoSe - saved me having to do an FPL related task on a morning that I'm trying to forget about FPL after choosing Costa over Kane :)

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

thesilkworm wrote:Thanks MoSe - saved me having to do an FPL related task on a morning that I'm trying to forget about FPL after choosing Costa over Kane :)
I provided that as backup, and as excercise to demonstrate what Mav's tools can allow admins do
in your place I'd check with RPG whether he's actually happy with my format, or if he preferred yours :)

PS: I chose both... :?

User avatar
RomynPG
Dumbledore
Posts: 8066
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

MoSe wrote:... I'd check with RPG whether he's actually happy with my format, or if he preferred yours :)
Computer says no

(XP Excel2000 can't cope with the format - even with an xlsx converter plugin)

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

I mostly kept your format, must be some minor and irrlelvant quirk, blocking it

I tired saving it now as .xls myself (97-2003), but I'm not that confident....
FISO 5AS 2016-17 GW37 (2).zip
You do not have the required permissions to view the files attached to this post.

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

https://docs.google.com/spreadsheets/d/ ... p=drivesdk
I saved in google spreadsheets format from mobile, but not sure it's in my Google Drive...

Try opening that and download a copy :?

User avatar
RomynPG
Dumbledore
Posts: 8066
Joined: 13 Oct 2005, 18:32
FS Record: FPL 134 overall 06/07 ... never close since

Re: Side-game data project / FPL data collection

Post by RomynPG »

MoSe wrote:I mostly kept your format, must be some minor and irrlelvant quirk, blocking it

I tired saving it now as .xls myself (97-2003), but I'm not that confident....
FISO 5AS 2016-17 GW37 (2).zip
Cheers MoSe - got what I needed from this one.

Thanks again

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

Hi, I would like to ask a question. If I use Mav GRID tool during the middle of the season, would the first column shows all the 38GWs with blanks in those GWs yet to play? Thanks in advance.

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

kidzio wrote:Hi, I would like to ask a question. If I use Mav GRID tool during the middle of the season, would the first column shows all the 38GWs with blanks in those GWs yet to play? Thanks in advance.
1st. we have to hope that FPL doesn't change its formats once again for next season, and that Mav's tools go on working without needing a major rehaul ;)

2nd. yes, it's as you say, or rather, not blanks but actual 0s
it was an explicit requirement by Taca
viewtopic.php?p=2930141#p2930141
the "GRID" was how Taca had organised the data in his spreadsheet in past seasons, when he retrieved data on his own using excel web queries (.iqy) from teams History page
for ease of use and consistency of reference, he organised the whole table with 38 rows per stat (and a column for each team) at the start of the season
and being able to copy the whole output and paste it without needing to bother how many gw/rows have been played, makes things easier
Tacalabala wrote:... Please can you add the zeros for the weeks with no data?
It sounds mad, but that way I can literally copy and paste into a sheet and my GRID could then feed off the raw numbers. Once up and running, I could do the whole thing in a matter of a minute, which is perfect for me. The fact that is can do this during the GW is brilliant, Saturday night updates are back on!

I'm blown away, I can't thank you enough to be honest :)
I had things organised differently, but had no problems to adapt to it,
in some cases it made thigs actually easier, and it's rather flexible (I used it for H2H leagues too)
Doesn't matter if half of the retrieved data I never use ;)

there's only one glitch:
  • most GWs, randomly, one or a few teams return 0s for a GW
    say, you extract a grid with 100 IDs as input, in GW22, and you find out that, say, team #21 GW3 and team #67 GW8 have 0s in all the corresponding stats
    you retry and it's the same
    you maybe retry in the evening, and it's OK again
    you retry next GW, and those ID/GW are correctly retrieved, but maybe another one gets 0s...
    I assume it's FPL fault, maybe random transitory time-outs on the data fetch... :?:
    so I took the habit of not pasting directly on the stored GRID data, but on a new temporary sheet, check for 0s, and eventually erase the 0 cells and then paste on the "official" GRID using "skip empty cells", so that previous collected data in those ID/GWs doesn't get overwritten with 0s

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

Thank you so much MoSe for the detailed explanation. I noticed the glitches too as sometimes those chrome's apps are not working well too. We will know more when next season starts.

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Anyone who had been using my website tools may have noted issues with the SSL certificates over the last few weeks. I've finally had time to fix the issues so everything should be back to normal now.

I've also added a new URL, which aggregates FPL league data along with FPL Statistico Information for each team:

Team Performance Data (FPL Statistico) (TeamPerformanceData/<LeagueId>/<PageNumber>)

This is in addition to the existing URLs that aggregate weekly GW scores to assist Side Game Administrators.

FISO Sidegame Data (SideGameData/<LeagueId>/<GameweekNumber>)
FISO H2H Data (H2HData/<LeagueId>)
FISO Sidegame GRID (GRID/<LeagueId>)

After having the code running for most of last season, I'm open to either consolidating those URLs into one or putting up a brand new definitive format that fits everyone's purpose. The GRID is maybe the most useful tool from discussions I've had with people here, but it is slow due to all the live grabbing of data that it does (and having no storage means it takes longer and longer as the GWs go on). Now, I'm happy for google spreadsheets or similar to be the online storage mechanism, perhaps a side game admin just extracts the data week-by-week from GRID, or perhaps I extend the tool to provide storage (and hence speed) but at the cost of becoming a single point of failure personally.

It's likely that I'll have to do something when FPL returns for the new season as there will probably be some change or other to the FPL data that causes the current URLs to break (indeed GRID currently seems to be broken, but I can't see much point fixing it until the new season starts). So, if we want to fix some single useful and universal data extraction tool, after having a season of working with the FPL data under our belts, I think now would be a perfect time to discuss it.

User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

Mav3rick wrote:Anyone who had been using my website tools may have noted issues with the SSL certificates over the last few weeks. I've finally had time to fix the issues so everything should be back to normal now.

I've also added a new URL, which aggregates FPL league data along with FPL Statistico Information for each team:

Team Performance Data (FPL Statistico) (TeamPerformanceData/<LeagueId>/<PageNumber>)

This is in addition to the existing URLs that aggregate weekly GW scores to assist Side Game Administrators.

FISO Sidegame Data (SideGameData/<LeagueId>/<GameweekNumber>)
FISO H2H Data (H2HData/<LeagueId>)
FISO Sidegame GRID (GRID/<LeagueId>)

After having the code running for most of last season, I'm open to either consolidating those URLs into one or putting up a brand new definitive format that fits everyone's purpose. The GRID is maybe the most useful tool from discussions I've had with people here, but it is slow due to all the live grabbing of data that it does (and having no storage means it takes longer and longer as the GWs go on). Now, I'm happy for google spreadsheets or similar to be the online storage mechanism, perhaps a side game admin just extracts the data week-by-week from GRID, or perhaps I extend the tool to provide storage (and hence speed) but at the cost of becoming a single point of failure personally.

It's likely that I'll have to do something when FPL returns for the new season as there will probably be some change or other to the FPL data that causes the current URLs to break (indeed GRID currently seems to be broken, but I can't see much point fixing it until the new season starts). So, if we want to fix some single useful and universal data extraction tool, after having a season of working with the FPL data under our belts, I think now would be a perfect time to discuss it.
I've been using this - https://ratemyteam.uk/Fiso/GRID - and I have to say that it's done the job for me with season, so thank you :)

In terms of next season, Google Sheets would be my preference as I know it's reliable and easy to use. I'm not too concerned with speed, I just need to know that it's going to be there and work.

GRID seems to work for me this morning, don't know if you've fixed for the hell of it though :mrgreen:

This may be a bit cheeky, but if you have time after sorting out GRID, do you think there is something you could do in terms of Chips and Season history as well? Just an idea mind.

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Chips and season history? In what way?

User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

This available on the Gameweek History page, so I imagine it can be worked with in a similar way?

For history, I'd imagine something like ID-Manager-Year1 Ranking-Year1 Points-Year2 Ranking-Year2 Points etc etc, and for chips something similar Date-Name-GW Active

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

So added in as extra sections in the GRID you mean? I'm sure I could do something with that, I'll have a look at doing so the next time I open up the source code.

User avatar
Tacalabala
FISO Knight
Posts: 19010
Joined: 07 Sep 2008, 01:03

Re: Side-game data project / FPL data collection

Post by Tacalabala »

Yes like a GRID extra if you like

Cheers Mav, you're a genius!

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

Firstly, I want to thank Mav for what you have done! They are very useful.

If it's a single useful and universal data extraction tool, then I think the GRID is the best, most universal and can cater to different league formats. I personally would be using the data from the GRID and copying them into google sheet and using functions to present the data in the format and order that I require for my leagues. I haven't used the GRID for the 2016/17 season but I have just completed a google sheet that uses the data from the GRID to generate the results that I need for my new league next season.

I also agree that adding the chips and season history would be great as these are also information that I am looking to use for my league next season.

I used the sidegame data page for a different purpose this season. I copied and pasted the league standings so as to generate a history of their league positions.

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

Hi Mav, do you think the GRID will continue to work with the new FPL? Are there any differences/problems in the data collection? Hope it works :)


Sent from my iPhone using Tapatalk

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

I've not had a chance to look yet, but I doubt the underlying data has changed much given the site itself is broadly the same.

There might be some changes to the format of data, I would think if you tried running the other tools on my site against a new mini league, that will give us an indication.

A full test of grid can't be done until the season is underway, but I'll make sure it works still.

User avatar
Mystery
FISO Knight
Posts: 12474
Joined: 13 Oct 2005, 18:33
Location: Just about here
FS Record: 116th in TFF 05/06
Contact:

Re: Side-game data project / FPL data collection

Post by Mystery »

The elements player list json still seems the same so signs are good

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

Great to know that the json appears to be the same. I have checked the sidegame data tool and it is showing up except for the manager name and region which I think is because it is still pre-season now.

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

Mav3rick wrote:I've not had a chance to look yet, but I doubt the underlying data has changed much given the site itself is broadly the same.

There might be some changes to the format of data, I would think if you tried running the other tools on my site against a new mini league, that will give us an indication.

A full test of grid can't be done until the season is underway, but I'll make sure it works still.
I keep tho getting Error with https://ratemyteam.uk/Fiso/SideGameData/3688/
this is a Renewed tiny Mini-League with just Joccki and RNZ https://fantasy.premierleague.com/a/lea ... 88/classic
and the 3 of us have already entered a new team which is listed in the league as pending entrant (although RNZ hasn't updated his button yet)
the same on other Leagues (you can see a few FISO ones in my button)

IIRC the league tool did work last season even before 1st deadline, showing random data
or did you change it to require actual data once the season started?
Could it be again a Certificate problem as it happened last season towards the end?

thanks!

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

I'll have a look later, most likely it's a data format change, or something to do with the gameweek dates on my side (I've not imported the new ones yet which is quite possibly the problem).

User avatar
MoSe
Dumbledore
Posts: 9384
Joined: 10 Sep 2014, 12:25
Location: next door S.Siro stadium
FS Record: FISODAS CUP Winner Season 25
FISO H2H Winner: 15/16 Div2 - 16/17 Div1
FISO Mirror: 16/17 PL Winner

Re: Side-game data project / FPL data collection

Post by MoSe »

Mav3rick wrote:I'll have a look later, most likely it's a data format change, or something to do with the gameweek dates on my side (I've not imported the new ones yet which is quite possibly the problem).
D'oh!

I saved the link in favs without the GW # and forgot I had to add it! :oops: :lol:

https://ratemyteam.uk/Fiso/SideGameData/3688/1 works as a charm :D
same with FISO MIRROR PREMIER LEAGUE
https://ratemyteam.uk/Fiso/SideGameData/32386/1
which might be useful to andybarrel to fetch his sidegame teams ID for those who were auto-renewed in the league but haven't posted yet

sorry for my thickness, glad to see it works :D

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

:lol:

Thanks for checking it though, you've saved me a job which means I can watch game of thrones in peace tonight :D :D

User avatar
kidzio
Wideboy
Posts: 51
Joined: 06 Sep 2015, 14:42

Re: Side-game data project / FPL data collection

Post by kidzio »

You all missed my last post on the 13th that I have checked that the SideGameData is working except that the managers' names are not showing up (which sld be available after gw1 I guess) :)

And I think the upgraded GRID (with chips played etc) can definitely wait till after watching GoT ep1 :)


Sent from my iPhone using Tapatalk

User avatar
Mav3rick
FISO Knight
Posts: 18574
Joined: 20 Jul 2009, 20:35
FS Record: FPL: 1082, 1201, 1800, 10203

The stats are dark and full of errors.

Re: Side-game data project / FPL data collection

Post by Mav3rick »

Sorry yeah I did miss your post, thank you as well kidzio, winter is definitely coming then :)

View Latest: 1 Day View Your posts
Post Reply

Return to “FPL Links, Mini-Leagues & Side Games”