Side-game data project / FPL data collection

[RomynPG]

Bear with me if you can while I sort it out, although if you side game runners have any alternate ways of scraping the data and/or have no further need for my site, now would be a good time to let me know

I do still use it for 5AS but don't fork out for me if you aren't using the data yourself.

I've been wanting to move 5AS off Excel( ) and set it up online for years so maybe now is the time if you're on a LAMP setup and want someone to help share the cost. I have a few other ideas I've been wanting to explore so pm me if you get time and are interested in chat.

[Mav3rick]

Cheers RomynPG, unfortunately all my code is .NET though!

I've been looking at keenly priced hosting options, the best of them seems to be a basic package with winity for about $9 a month (or $7 per month if I pay for a year up front). That would give me a UK based virtual server again (that I could do other things with too), so is probably the front runner for now.

I'll see what other replies we get here though before I sign up and move things over.

[bspittles]

I use tmdhosting.com for my (low traffic) website, costs $3 per month. I know you can scale it up if you need something above the basic package.

[Oxford NZ]

Currently using a script to generate the grid written by Talkie Toaster for the divisional championships and the FISO cup. Still using your site as a back up as the grids are close to being the same. Many thanks Mav and TT

[andybarrell]

I use it for my Fiso Mirror but don't pay out on my behalf as this is possibly the last season i will run this sidegame

I am not savvy enough to get the data any other way , except clicking on every members team history and adding the data manually.....i don't have the time or inclination to do that though

[Brightwater]

My 2 cents worth, if you are looking for a web host 1&1 / ionos.co.uk have hosted my villa website trouble free and reliable for 12 years. Negotiable pricing, mine was increased last year, knocked down to £3 per month including VAT.

[Smurphy Paw]

I use it for my Fiso Mirror but don't pay out on my behalf as this is possibly the last season i will run this sidegame

Possibly the wrong thread for discussion but I’d be very sorry if that is the case. It’s a great game

[Mav3rick]

Thanks for the hosting ideas, I was using a £2 a month host this last year - they just didn't like the CPU usage I can get it hosted for a reasonable cost to the end of the season at least though, the plan is to get the site back up over the weekend.

Currently using a script to generate the grid written by Talkie Toaster for the divisional championships and the FISO cup. Still using your site as a back up as the grids are close to being the same. Many thanks Mav and TT

Is this script something that is or can be widely used by side game runners? I don't want to leave people without a solution as the side games are a vital part of FISO.

[Mav3rick]

The site is back up and running now, eg.

https://www.emerald-solutions.co.uk/fiso

I've put enough in the meter to see the season through, and perhaps we can have a discussion in the close season about what's needed going forward.

[Oxford NZ]

Many tanks Mav TT's script will work for others but I believe we will be kicked out from using the cors development site due to the usage.
Unfortunately I have no disposable income to fund side games as I am getting close to retirement and have spent my savings due to a recent illness, covid lock down and a down turn in the economy.

[Smurphy Paw]

Sponsor a month? There are enough of us on here who love the side games you guys run. I suspect it’d have to be voluntary - I’ll not name names but a couple of years ago had an interesting exchange with an overseas poster about the relative cost of accessing the likes of FFScout depending where you are in the world. And a fair proportion can probably relate to points you make ONZ. The sums you’re talking about are modest enough for the crowd to cover though, I would have thought?

[Mav3rick]

Thanks SP, although given that we're now back on a proper virtual server (its always useful having a web server hanging around to chuck stuff on or experiment with), I'm ok with the relatively small increase in hosting costs.

I suppose what I'd like to explore more at the close season is if there are any other ways people have found or settled into to run their side games that make the site redundant now. It's coming up on five years running now , and is low but not zero maintenance for me - especially so if FPL towers make any significant API changes. Perhaps we can do something with TT's scripts hosted on a cloud platform (if they are partly running via Heroku this might be an option).

[Smurphy Paw]

No worries
Good to hear from you, hope you’re well

[Talkie Toaster]

The site is back up and running now, eg.

https://www.emerald-solutions.co.uk/fiso

I've put enough in the meter to see the season through, and perhaps we can have a discussion in the close season about what's needed going forward.

Many thanks Mav. It's much appreciated.

[Oxford NZ]

S17 Divisional champs registration

Just dropping this here for safe keeping ( succession planning in case I get hit by a bus/digger/ covid delete as applicable).
Divisional championship registration sheet updated with qualifying points and players sorted into the divisions they qualify for if / when they sign up.
Format is the same as MoSe used to run but Team of the Season points were changed by me last season to reward those coming up through the divisions.

I trust the grid will work in some form or another in the coming season and many thanks to Talkie Toaster and Mav3rick for the tools to do the job

[Mav3rick]

The billing cycle for my site is up on the 19th June, and as mentioned I'm interested in having a bit of a discussion about what we want or need to do with it going forward.

If it's just a case of keeping the lights on, then I am ok with doing this. The hosting (with Winity.io) costs about $25 per quarter, which is a cost I can bear so financially there is no particular issue in keeping the site up.

What I am concerned about though is if and when FPL decide to change their API, as I have less time these days to spend on FPL and if there was a need for some substantial rework in the code, then that's where I'd start to be worried about being able to keep things running.

So my feeling is that if I just purely keep things running, that when FPL do change their API it will be a surprise, and a bit of an emergency, needing a fix in probably a time frame of a month or so. If I'm not able to assist, then side games may all of a sudden fail. I don't really want the sole responsibility of that so what I'm looking to do is establish a bit of redundancy.

I don't mind handing the code to FISO to re-brand and run as a FISO site, or I could give copies of the code to anyone who wants to volunteer to run something similar to my site going forward. It's .NET framework code, an MVC application specifically, so can run on any windows hosting provider, but as evidenced in this thread, some of the cheaper shared hosting sites might not like hosting it due to the network and CPU use when someone is requesting the grid data for a big league (it's all done on the fly so is quite HTTP request intensive).

I'm also happy to strip out the fundamental parts that make up the FISO grid, and publish that to github, or if TT's approach needs a server component I could potentially work that into the site and publish that to github instead of or as well as the grid. That would at least give others a chance to use (and host) the components as they see fit.

So, what do people prefer?

[Oxford NZ]

Thanks Mav.
I use the TT script and your grid tool to generate the grid for the divisionals and the FISO cup. I have tried to understand how it all works so I could change or re write at if required but If the host boots us out then I am lost. I have run the game manually for a few weeks when we were booted out before and your grid tool was not working, but that took up a huge amount of time I do not have.
I will have to leave it up to TT and the other script / code writers to make comment and come up with a cunning plan.
Thanks again for all the work over the years and the offer of your site and tools. The side games are unique to FISO and should be saved if possible.

[RomynPG]

I did some work towards the end of the season on finally moving the H2H off Excel and on to the web. Got a ittle bit more to do on displaying the data and some final automation but I'm effectivity grabbing everything, fixtures, players etc, selected league level data and also detailed data for individual teams - and processing and caching it all. I tested it with the H2H leagues and the Rounded Keeper and hope to run the H2H with it next season, with LiveScore , and get some fun stats for the pub league. I ran this season H2H through and got the same results which is a good sign.

I should(?) be able to provide datasets if needed. My setup is LAMP with memcached.

[Mav3rick]

That's great new RomynPG, having more than one source is really the key to ensuring we don't lose the sidegames at any point.

I just checked the original post in this thread and it's been 5 years now since FPL made their significant change. I don't really see any particular reason for the underlying API to change much, but they might move it to a new technology one year. The more people we have with the ability to recover the GRID when that happens the better!

ONZ - I think with TT's script he has issues mainly on actually accessing the APIs. Hopefully he can drop into the conversation later and explain the errors he's getting. I know authentication has been an issue for him, and he's been having issues with CORS which suggests his scripts might be pure JS running in a browser. I suspect either myself or Romyn or both of us could provide a reverse proxy for him to use that just connects the FPL site for him (we're all accessing the same underlying APIs afterall), or we could find a browser plugin to help him through the CORS issues if that's the only real blocker.

That would be three solutions then, which would provide the redundancy we need.

[Smurphy Paw]

I tested it with the H2H leagues and the Rounded Keeper and hope to run the H2H with it next season, with LiveScore , and get some fun stats for the pub league. I ran this season H2H through and got the same results which is a good sign.

Just imagine, someone competent on the case!
Some of the lack of update last year was real life related. But there’s no way I could do ^^ this.

Meanwhile, LiveScore for H2H That’d be amazing news

Thanks again each of you involved in this thread, this is the behind the scenes work that really does keep FISO special

[andybarrell]

all the above is well over my pay grade

all i require to run my side games is a weekly score from all the entrants less transfer hits....and maybe world rank but that is not a major factor

[andybarrell]

I have noticed on the eve of the new season that this is not working....is it a case of wait til tomorrow or is it broken?

[Oxford NZ]

I have noticed on the eve of the new season that this is not working....is it a case of wait til tomorrow or is it broken?

Only just seen this, sorry. Email sent with an alternative sheet that can be linked to your Mirror game.
Linking the data should help to save you an rsi from data entry

edit to correct auto spell and a glass of wine that swamped my keyboard
edit 2 same as above

[kidzio]

Hi Oxford, can I get it too? Thank you so much.


Sent from my iPhone using Tapatalk

[Oxford NZ]

Hi Oxford, can I get it too? Thank you so much.


Sent from my iPhone using Tapatalk

Email sent
Why not join a few FISO side games like the divisional championships as you play FPL?

[kidzio]

Thanks man! Shall take a look at both the FISO side games and the link tonight


Sent from my iPhone using Tapatalk

[wonkypenguin]

That's great new RomynPG, having more than one source is really the key to ensuring we don't lose the sidegames at any point.

I just checked the original post in this thread and it's been 5 years now since FPL made their significant change. I don't really see any particular reason for the underlying API to change much, but they might move it to a new technology one year. The more people we have with the ability to recover the GRID when that happens the better!

ONZ - I think with TT's script he has issues mainly on actually accessing the APIs. Hopefully he can drop into the conversation later and explain the errors he's getting. I know authentication has been an issue for him, and he's been having issues with CORS which suggests his scripts might be pure JS running in a browser. I suspect either myself or Romyn or both of us could provide a reverse proxy for him to use that just connects the FPL site for him (we're all accessing the same underlying APIs afterall), or we could find a browser plugin to help him through the CORS issues if that's the only real blocker.

That would be three solutions then, which would provide the redundancy we need.

So I got some stupid idea in my head to write some FPL data extraction tools as part of learning some new (to me) languages/tech, and I came across this page (no idea how I've missed it before!). Was gonna have a peruse at what people have done, but I see a lot of the older links seem to be out of action nowadays... is anyone in here still actively maintaining any source in Python/Ruby/anything other interesting languages?

TIA!

[Mav3rick]

I think there was/is an API access library on GitHub, you can obviously just hit the end points yourself with an http client, but I assume it takes care of some of the data transformation and authentication issues for you.

My stuff is all .NET though.

[eagle224]

So I got some stupid idea in my head to write some FPL data extraction tools as part of learning some new (to me) languages/tech, and I came across this page (no idea how I've missed it before!). Was gonna have a peruse at what people have done, but I see a lot of the older links seem to be out of action nowadays... is anyone in here still actively maintaining any source in Python/Ruby/anything other interesting languages?

TIA!

I maintain a basic Python script that collects information for an FPL league using the FPL API. It creates a gameweek report for each member in the league (first page only, though can be adapted). I could PM you a copy to play with. I'm not a pro Python programmer so it is probably not pythonic but worked perfectly well for the past few years for my purposes as the admin of a league.

Example of my manager summary report from GW38 last year:

Code: Select all

Manager Summary
  Entry ID: 841311
   Manager: [name removed]
      Team: Just an illusion
      Rank: 1
    Points: 2369

Gameweek Stats
     Total Points: 76
        Transfers: 2
    Transfer Cost: 4
     Bench Points: 18
  Auto-sub Points: 0
    Gameweek Rank: 527289
      Active Chip: -
          Captain: Son
     Vice Captain: Vardy

Gameweek Transfers
  In: Vardy                 Out: Richarlison           Nett: 10
  In: Toney                 Out: Watkins               Nett: -4

Current Stats
  Pts GS GA CS OG PS PM YC RC BN CP VP
   76  6  2  3  0  0  0  2  0  7 12 10

I also have an end-of-season script that summarises all the members performances, again using the API

Code: Select all

    
    Player: [name removed]
 Team Name: Just an illusion
      Rank: 1
    Points: 2369
GW Started: 1

High Score: 102 in game week 37
 Low Score:  20 in game week  2

Transfer Stats:
  Total:  55, Hits:  19, Cost:  88

Captain Stats:
  Pts: 643 from  37 active, High:  57 in game week 37

Vice-Captain Stats:
  Pts:   4 from   1 active, High:   4 in game week 25

Auto-sub Stats:
  Pts:  94 from  28 active, High:  11 in game week 10

Bonus Point Stats:
  Pts: 297, Captain Extra:  46, Bench:  25

Starting Selection Stats:
  YC:  47, RC:   1, GS: 103, GA:  85, PM:   1, CS: 134

Bench Selection Stats:
  YC:   9, RC:   0, GS:   9, GA:   2, PM:   0, CS:  11

By Position:   GKP   DEF   MID   FWD
       Goals     0    10    64    29
     Assists     0    18    46    21
Clean Sheets    10    60    64    29
Bonus Points    16    73   131    77
Total Points   148   664  1142   503

Above average gameweek scores: 30

[eagle224]

The API has been updated to include the new player profile statistics that have been mentioned on the The Rounded Keeper thread. These can be found under the player elements (sample attached).

NewStats.png