Page 1 of 10

Side-game data project / FPL data collection

Posted: 18 Jul 2016, 10:54
by Mav3rick
Tacalabala (in the 'When does the game start' thread) wrote:This has got me worried, as if they have massively changed the way the data is accessed/displayed, it will make sidegames near impossible to run. Hopefully I'm just worrying for nothing!
Me too, although it depends which data you're looking at. As far as I can tell, it's still ISM running FPL, so that suggests to me that the JSON data would still be used (although obviously the form might change), especially so if it's being re-designed for better mobile capabilities.

It's going to wreck my RMT site if the JSON for an individual FPL team can't be accessed anymore, and if the individual player JSON files are lost then I'd expect many auto imports will fail, possibly affecting the price change sites too.

Re: When does the new FPL season start?

Posted: 18 Jul 2016, 17:45
by MoSe
Mav3rick wrote:
Tacalabala wrote:This has got me worried, as if they have massively changed the way the data is accessed/displayed, it will make sidegames near impossible to run. Hopefully I'm just worrying for nothing!
Me too, although it depends which data you're looking at. As far as I can tell, it's still ISM running FPL, so that suggests to me that the JSON data would still be used (although obviously the form might change), especially so if it's being re-designed for better mobile capabilities.

It's going to wreck my RMT site if the JSON for an individual FPL team can't be accessed anymore, and if the individual player JSON files are lost then I'd expect many auto imports will fail, possibly affecting the price change sites too.
I'll just need the Gameweek History table to be referrable as a table and imported wholly as a web query into excel
Can't do JS and didn't need it, the very basic level of xls import was enough
Failing that, I'd have to relinquish the Divisionals management, or hire some dev to provide me with the weekly score, hits and transfers data for each team involved (129 last season)

Re: When does the new FPL season start?

Posted: 18 Jul 2016, 17:49
by Tacalabala
Mav3rick wrote:
Tacalabala wrote:This has got me worried, as if they have massively changed the way the data is accessed/displayed, it will make sidegames near impossible to run. Hopefully I'm just worrying for nothing!
Me too, although it depends which data you're looking at. As far as I can tell, it's still ISM running FPL, so that suggests to me that the JSON data would still be used (although obviously the form might change), especially so if it's being re-designed for better mobile capabilities.

It's going to wreck my RMT site if the JSON for an individual FPL team can't be accessed anymore, and if the individual player JSON files are lost then I'd expect many auto imports will fail, possibly affecting the price change sites too.
Yeah, it's the way that the Euro one was designed, deliberately it seems to stop data capture. We'll see, for all we know they might make it easier!

Re: When does the new FPL season start?

Posted: 18 Jul 2016, 18:18
by RomynPG
As I've probably been data collecting in various forms from FPL for 10-ish years now I probably use the most primitive method - I parse the html in VBA :o

I used to do lots - players, fixtures, squads etc - enough to fill in some holes when D+ took up the Crack-the-Code stuff a while ago and support a whole load of games and nerdy things. Now I only do varoius league pages for the 5AS.

Regardless of whether anything changes this year maybe it's time to pool resources and put something together that collects all the data we all use and make it available via an API of some sort.

Making that side of things easier may encourage more side-games - I know I'd like to bring back a game that needs full squad info that I don't collect any more.

Re: When does the new FPL season start?

Posted: 18 Jul 2016, 19:11
by Tacalabala
RomynPG wrote:As I've probably been data collecting in various forms from FPL for 10-ish years now I probably use the most primitive method - I parse the html in VBA :o

I used to do lots - players, fixtures, squads etc - enough to fill in some holes when D+ took up the Crack-the-Code stuff a while ago and support a whole load of games and nerdy things. Now I only do varoius league pages for the 5AS.

Regardless of whether anything changes this year maybe it's time to pool resources and put something together that collects all the data we all use and make it available via an API of some sort.

Making that side of things easier may encourage more side-games - I know I'd like to bring back a game that needs full squad info that I don't collect any more.
Agreed, though I'll hold my hands up and admit I can't code for toffees haha

Re: When does the new FPL season start?

Posted: 18 Jul 2016, 19:21
by Mystery
Happy to help if I can although pretty much all the work I do is in Google Sheets.

Re: When does the new FPL season start?

Posted: 18 Jul 2016, 19:27
by Mav3rick
RomynPG wrote: Regardless of whether anything changes this year maybe it's time to pool resources and put something together that collects all the data we all use and make it available via an API of some sort.

Making that side of things easier may encourage more side-games - I know I'd like to bring back a game that needs full squad info that I don't collect any more.
If it's something people want then I'm happy to help. Based on last year's structures, my RMT site was pulling back (for any given FPL team id and GW number) things like total score, total goals scored, total yellow cards, total red cards, penalties missed, total BPs etc. If that still broadly works post-redesign, then I'd be happy to make it available to everyone via a simple API hosted on my web server.

Assuming that the additional data requirements aren't too numerous, I could add in extra bits of data as needed by us here if we can agree what we need.

Probably needs a new thread for further discussion (mods - can we get a thread split please?), and we may need to wait until GW1 is underway before we can be sure that all the data needed is available, but I'll do what I can if there's popular demand.

Re: When does the new FPL season start?

Posted: 18 Jul 2016, 19:34
by RomynPG
Mav3rick wrote:
RomynPG wrote: Regardless of whether anything changes this year maybe it's time to pool resources and put something together that collects all the data we all use and make it available via an API of some sort.

Making that side of things easier may encourage more side-games - I know I'd like to bring back a game that needs full squad info that I don't collect any more.
If it's something people want then I'm happy to help. Based on last year's structures, my RMT site was pulling back (for any given FPL team id and GW number) things like total score, total goals scored, total yellow cards, total red cards, penalties missed, total BPs etc. If that still broadly works post-redesign, then I'd be happy to make it available to everyone via a simple API hosted on my web server.

Assuming that the additional data requirements aren't too numerous, I could add in extra bits of data as needed by us here if we can agree what we need.

Probably needs a new thread for further discussion (mods - can we get a thread split please?), and we may need to wait until GW1 is underway before we can be sure that all the data needed is available, but I'll do what I can if there's popular demand.
Sounds like there's some interest - nice one.

Good plan for a thread split - maybe call it "Sidegame data project"? ... leave it here for a bit - to hopefully be seen by general data collectors - and then move to sidegames later?... or whatever.

I have some thoughts I'll put together.

Re: Side-game data project / FPL data collection

Posted: 18 Jul 2016, 20:17
by Mav3rick
I'll quickly describe what I already have (well, had last season):

Gameweek Details for a given team
Input: FPL Team ID, Gameweek Number
Output: A gameweek summary for the starting XI (or XV in case of BB) that contains the total GW score and the total number of GoalsScored, GoalsConceded, MinutesPlayed, Saves, YellowCards, Cleansheets, Assists, BonusPoints, OwnGoals, PenaltiesSaved, PenaltiesMissed, RedCards for all the starting players, including auto-subs. I don't store any data, it's just parsing data on the FPL site and returning it.

I'll need to try to get that working again for my own site once the new FPL site launches, if I manage to do that then I could easily expose the data via a WebApi. Any interested clients would need to be able to parse some web friendly content types, such as JSON or XML so it wouldn't be suitable for everyone.

I think sidegames care most about GW data and events, in many cases comparing the events of one team to events of another. Unfortunately, GW data is the least programatically accessible data on the old FPL site at least. I don't know how much a more dedicated API would help side game maintainers if they run their games through Excel currently using table imports (I really don't know much at all about Excels abilities here). I was able to auto run automated (simple) sidegames on my site via the single method above, but that obviously involved other code auto-processing the game weeks as they passed:

Twenty20
Heptathlon
World Cup

Re: Side-game data project / FPL data collection

Posted: 18 Jul 2016, 21:42
by Tacalabala
FISO DATA 2015-16.zip
Ideally, we'd have something like this online for all registered FISOers. Obviously this is all dependent on .iqy queries.

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 13:29
by MoSe
I'm positive now that I can NOT manage my sidegames on my own anymore
(unless I individually check and import each team scores :o )

the problem lies in the .iqy queries and in the browser version on the PC I'm using

as far as I can tell, you can't pick the browser used by the .iqy, excel it's an MS Office program, and it uses MS IExplorer,
couldn't find any way to customise it to Chrome, regardless I made Chrome the predefined browser on my PC

Bigger problem, I only have the work PC, don't have one at home.
And on the work PC I have to do with the company certified SW
And we're stuck to IE 8, can't upgrade it (it's indeed to circumvent that that I installed Chrome on my own)

Don't know wheter updating to the latest IE version would solve it,
but IE8 it's not able to correctly display the green banner and its buttons, so I can't navigate to the team page and the GW history page
it lets me login, but then even pasting the /a/team/my or the /a/event/xxx/history link into the url line, it always redirects me to the /a/#home page
(after an avalanche of pulselive and ims script errors)

That's in IE alone, leaving alone setting up a web query in excel, where the available MS browser version problems of course propagate.
Mind, I have no problem with explorer from the Android, but there in the android excel I don't have iqy to manage data imports...
___________

As I said, I only need a very limited data set from each player to run the FISO Divisionals (GP, TC, and TM)
Tacalabala's sheet would be overkill ....if his IE version allows him to keep it updated I could piggyback it :oops:
of course it would be ..."humiliating" ;) to depend on someone else's work to run and manage a sidegame

an auto-updating website as Mav3rick proposed would be optimal as it would only require the initial setup, and no maintenance,
but it would of course be useful only if it can present data from ALL the teams in a sidegame (on in FISO) in one view,
otherwise it would be the same as checking them manually and individually on the FPL site

another option, if Mav3 can't present a data table for a given League, or List of teams,
it could also work as he suggested as a data-fetch page for a single team, as long as the "Input: FPL Team ID, Gameweek Number" can be embedded in the URL, and not having to insert it manually in a field in the page
that way I'd set up and customise the individual web queries to query Mav's site for each single player, as I did so far with FPL

it's either that, or forsaking running the sidegame (hoping someone else takes on), or checking data manually every GW for each team (also error-prone)...
...or buying a PC with Win10 & IE11 (provided someon confirms FPL correctyl runs there) for the sole purpose of runinng an FPL sidegame :o :lol:

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 13:58
by MoSe
btw, i've checked Mav's RMt site, excellent work, how could've I missed it in two seasons here?

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 14:19
by MoSe
MoSe wrote:Bigger problem, I only have the work PC, don't have one at home.
And on the work PC I have to do with the company certified SW
And we're stuck to IE 8, can't upgrade it (it's indeed to circumvent that that I installed Chrome on my own)
btw, dropped a hint to my boss, and he said IE11 has been tested on some pilot users
it worked fine, except for some company applications (don't know whether the ones out unit is using)
I doubt it could get "certified" before september, nothing happens in august in italy :(

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 15:01
by Aldershot Rejects
As a non-IT person, thus thread is complete gobbledygook, but it just makes me more thankful for the work you guys do on sidegames .

Thank you and good luck in whatever you decide :)

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 15:06
by jpk
just my 2pence worth .... rather than use webqueries from excel - have you tried VBA and XMLHTTP e.g.

Code: Select all

Dim xmlhttp As Object
Dim response As String
Set xmlhttp = CreateObject("Microsoft.XMLHTTP")
xmlhttp.Open "GET", "https://www.someadrress.com/" + Trim(Str(some variable)), False
xmlhttp.setRequestHeader "Connection", "close"
xmlhttp.send
If xmlhttp.readyState = 4 Then
       response = xmlhttp.responseText
End If 

then all the html is in the response variable and you can parse that in code.

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 16:20
by MoSe
IF ONLY had I ever learned VBA.....

RomynPG does the VBA html parsing, but as Taca had put it "I can't code for toffees haha" :oops:
  • last time I coded something it was in C++ over than 20 years ago
    I know VBA shouldn't be rocket science, but I lost too many neurons in the last 2 decades ;)
webqueries from excel were a handy and readymade built-in tool to use for us non-coders

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 18:46
by Mav3rick
MoSe wrote:webqueries from excel were a handy and readymade built-in tool to use for us non-coders
In general the site redesign has worked out better than expected for me as they have simplified all the data access that I have so far been able to see (an actual gameweek score page will be the big test, but I can't see that until GW1 is in progress).

Everything so far looks to be available in a nice JSON format:

Team History:
https://fantasy.premierleague.com/drf/e ... d>/history

League data:
https://fantasy.premierleague.com/drf/l ... &ls-page=1

(example)

All player, club and lookup data:
https://fantasy.premierleague.com/drf/bootstrap-static

Current Squad Selection (when logged in only):
https://fantasy.premierleague.com/drf/transfers

Fixtures:
The fixture data is all contained here (online parsed version). You'll need to cross reference the team ids with the lookup data (which has the teams in it).

Gameweek Data:
The gameweek player data can be found at
https://fantasy.premierleague.com/drf/e ... mber>/live
(this is a set of data for all players in FPL)

Then for an individual FPL team, you can get their gameweek selection from
https://fantasy.premierleague.com/drf/e ... weekNumber>
(and cross check the player ids with the live player data if you want to find out individual gameweek data, such as goals scored).

Even the login process is a (tiny) bit simpler.

The way things are working is a bit like a single page app, where the HTML loads and then triggers a request to get the JSON from the URLs above, and the page is then populated by javascript. I don't know how your web queries work in terms of the rendering engine, but if the javascript isn't executed then the Excel "load from this url" feature might stop working, as at least some of the data is being loaded dynamically and isn't rendered in the HTML.

Again, we'll need to wait until GW1 is underway before we can see the full league data structures, but if your Excel can parse the data returned from the "League data" url above then you should be laughing, especially as it should change less than the HTML in future seasons.

Re: Side-game data project / FPL data collection

Posted: 19 Jul 2016, 19:02
by Mav3rick
MoSe wrote: ... it would of course be useful only if it can present data from ALL the teams in a sidegame (on in FISO) in one view, otherwise it would be the same as checking them manually and individually on the FPL site
I can't say for definite until we see the GW1 in progress pages (and associated data) but I think it will be very possible for me to take an FPL league id and GW number, and then to do all the parsing and cross checking to provide the GW event data I mentioned above for all participants in a league.

That's assuming everything on the GW page in the new FPL site works like the others, but I think you'd be able to request a URL that includes the LeagueId and GW number and get back a list of JSON or XML data.

Input: https://somesite/<FplLeagueId>/<GameweekNumber>
Output: A list of FPL team ids (and team/manager name) with GoalsScored, GoalsConceded, MinutesPlayed, Saves, YellowCards, Cleansheets, Assists, BonusPoints, OwnGoals, PenaltiesSaved, PenaltiesMissed, RedCards for all the starting players, including auto-subs for each of those team ids.

You'll be able to get something similar with maybe less finesse from the FPL json links above.

If you need CSVs or HTML or similar then there's probably some sort of tool/content type provider we can use to do a conversion but I'd struggle to replicate (and test/maintain) something as complex as the multi-sheet excel list currently in use.

Re: When does the new FPL season start?

Posted: 20 Jul 2016, 03:49
by SuperGrover
Mav3rick wrote:
RomynPG wrote: Regardless of whether anything changes this year maybe it's time to pool resources and put something together that collects all the data we all use and make it available via an API of some sort.

Making that side of things easier may encourage more side-games - I know I'd like to bring back a game that needs full squad info that I don't collect any more.
If it's something people want then I'm happy to help. Based on last year's structures, my RMT site was pulling back (for any given FPL team id and GW number) things like total score, total goals scored, total yellow cards, total red cards, penalties missed, total BPs etc. If that still broadly works post-redesign, then I'd be happy to make it available to everyone via a simple API hosted on my web server.

Assuming that the additional data requirements aren't too numerous, I could add in extra bits of data as needed by us here if we can agree what we need.
I have something similar written in Java (last language I used as a developer, so easier for me than learning PHP or something more appropriate). I could aslo add additional fields if needed or at least help bounce ideas around for a design of any solution that does so.

I am a bit late to the thread though; do we know for certain if the JSON feed has been eliminated?

EDIT: I see the player JSON data is still readily available at https://fantasy.premierleague.com/drf/elements/. Looks to me like the format has changed but is now simpler (used to have a map of value in English to a code, which then was later utilized as the key in the key/value pair). I don't see any issue making this work again.

EDIT X2: Missed previous post by Mav3rick which explains all the available data in the new site. Looks great to me.

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 04:16
by SuperGrover
Mav3rick wrote:If you need CSVs or HTML or similar then there's probably some sort of tool/content type provider we can use to do a conversion but I'd struggle to replicate (and test/maintain) something as complex as the multi-sheet excel list currently in use.
If Excel is absolutely required I can assist (although probably not for a couple months due to a new home purchase). I already spit out multi-tabbed, well formatted Excels using the Apache POI - Java API for Microsoft Documents (https://poi.apache.org/index.html). I am able to produce multi-formats (numbers rounded to decimals, bolding, italics, ec.), can adjust cell formatting (horizontal/vertical alignment, cell joins/splits), and even column/row groupings. I should be able to produce the aforementioned spreadsheet from an input file (JSON preferably) without too much issue (although not sure how well the solution would scale to excessively large Excel documents).

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 09:18
by Mav3rick
Sounds like we have options then. I missed that "elements" file SG, didn't see it being used by the site, but it looks like another useful data source.

They've really made is easier for us!

I like the idea of your excel outputs, we definitely have the ability here to present the data now, we just need to see if the side game runners can use a single format, and if their existing import routines are still working.

At the very least, I feel confident we can help MoSE.

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 09:38
by DrBunker
Mav3rick wrote:They've really made is easier for us!
The idea that they'd deliberately make it harder for game developers always seemed confusing to me. I'm glad to hear it sounds like our sidegames shouldn't be affected negatively.

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 11:08
by thesilkworm
I'm just curious, what do the people who are contributing to this thread do for a living? Do you use these skills in your work? Obviously feel free to ignore this question :)

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 15:32
by Mav3rick
thesilkworm wrote:I'm just curious, what do the people who are contributing to this thread do for a living? Do you use these skills in your work? Obviously feel free to ignore this question :)
Yep, it's my day job (which rather irritatingly eats into my FPL time).
MoSe wrote:webqueries from excel were a handy and readymade built-in tool to use for us non-coders
Ok MoSe, I had a couple of hours spare after lunch, so I put this together as a starting point, the URL contains the FPL League Id and the gameweek number, I get all the participants in the league, get their GW scores and return everything in a nice simple table:

Here's an example using the FISO league.

The gameweek data is randomised for now obviously, I'll need to return real data for the listed fields for my own site, but I could add other columns with relative ease I think. It's all static and simple HTML so Excel should be fine, if anybody wanted the actual JSON then I can expose that too. The data (which is all public info) is cached for 10 minutes but not persisted anywhere other than the FPL website.

Format of the URL is simply
https://ratemyteam.uk/Fiso/SideGameData ... /<GWNumber>

Is that useful to side game admins?

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 16:50
by MoSe
Mav3rick wrote:Ok MoSe, I had a couple of hours spare after lunch, so I put this together as a starting point,
....
It's all static and simple HTML so Excel should be fine,
I'm moved that you took the effort! :D
I found out that excel can import from the web in HTML or XML, but not from the /drf csv text for instance

In can thus confirm now that I could easily import the whole Pending Entrants table values neatly arranged in their rows and columns
I was even puzzled at first that I imported different values from the page I had displayed in my browser, then I realised that the query imported the latest randomised values!

now I'm beginning to realise tho the actual and handier requirements specific to the Divisionals sidegame
for instance, could I find all the needed data in a single table for all the sidegame teams, I could even just Copy it from the web page and Paste it into Excel, forsaking the need to set up and albeit simple query to be updated :mrgreen:

or, as modifying the url in a webquery amounts as redefining it,
in order to link it instead to a variable value I'd just insert in a cell, I'd have to make it parametric,
but as I got it, it would require the gw in the url to be in the format ?GW=1 instead of /1
https://ratemyteam.uk/Fiso/SideGameData/31821?GW=1 a feature I'm not familiar with (yet ;) )

As the reason for all this is basically my being stuck to an obsolete IE version, quite a personal limitation,
I'd be happy to present you via PM my actual requirements to properly run my sidegame, should you allow me to continue abusing of your patience :oops:
maybe the next week?

Of course the project and endeavour to collect and provide a comprehensive and more accessible data source and collection for ANY sidegame runner, goes beyond my single personal case and predicament
Maybe Taca and RomynPG could better guide and discuss such wider project

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 17:09
by RomynPG
That's really useful Mav - thx.

What's especially useful is the fact that all the data is so easily available in JSON.

Not sure about login etc - do you need to be logged in to FPL to access from a external server for example?

My main aim, and something that's been building for a few years now, is that I want to move everything I do wrt sidegames onto the web - in particular I want to do that with the 5AS H2H. At present, and for historical reasons, my sidegames are run on an 11 year old PC running XP and Office 2000 - grabbing raw html and parsing it and storing it in a main lookup table. On top of this fairly generic import are all the extra worksheets that, after initial season config, allow the H2H to be run with a couple of clicks a week. It was done this way so Krispy could bring his awesome game to life by allowing the admin of what is an extremely complex game relatively straightforward.

That method is well past it's sell-by date and I want to get in a position where I can pass-on the H2H admin when I finally get bored of it. I also want to bring back another game that needs a level of data I don't now collect - full player stats - and I'm not redoing it Access again.

I won't be able to get this in place for this season - at least as far as the H2H goes - so I'm looking at getting something together as the season progresses. My first thoughts are to collect and store whatever data is available - fixtures/results, player stats (some frequency), squad stats and anything else that could feasibly be used for a sidegame. (

(It's also more practical that way, permanent storage, when you want to add additional data to fixtures or players to use. Example - adding referee to fixtures and birthday to players/refs allowed for Pointless Sweepstake - The Birthday Problem - which was fun :D )

I'd also like to move my fixture/results grid online - allowing me to add user defined team rankings etc.

I currently work with LAMP and JSON/AJAX stuff so am comfortable with what I'd need to do there. The data already in JSON makes the whole thing pretty easy - I hope.

... and if I was doing it for myself ...then why not just open that data up a bit - in the same manner as your examples - for output via JSON or just plain csv (as alot of data would be tabular).

That's how I got here - looking forward to where I'm headed.

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 17:25
by DrBunker
RomynPG wrote:I want to get in a position where I can pass-on the H2H admin when I finally get bored of it.
Let's hope this project works then as there aren't many scarier sentences on FISO than that!

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 22:08
by Tacalabala
I can see what has changed - you can't click on squad names to access player history, at least not at this stage anyway. This is a massive pain in the arse for me as now I'm essentially relying on people to provide me with the URLS themselves.

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 22:14
by Tacalabala
To: support@mail.fantasy.premierleague.com
Subject: Other player's history pages
Date: Wed, 20 Jul 2016 22:13:03 +0100

Good evening,

Firstly, may I take the opportunity to congratulate you on a visually stunning update of the game.

Please can you resolve a problem that I appear to be having with the new set-up? I have previously been able to access other player's history pages by clicking on squad/usernames on league pages, but this is not possible at this time. Is this only a temporary restriction which will be reinstated in the coming days?

Kind regards
Will let you know what the response is - they really have to bring this back in, crazy to stop this.

Re: Side-game data project / FPL data collection

Posted: 20 Jul 2016, 22:22
by Stemania
Are you sure it isn't always like this before the first update since it isn't practical to see what team anyone has picked yet? The list of teams under each league aren't strictly speaking actually in he league yet, just to be added after the first update I think. :?