Server Changes / Notes for Add-on Creators

Ah yes I understood all that perfectly

Is there any plan to release a public API? Fetching JSON will be lighter and easier to manipulate than loading full HTML page.

A high amount of our traffic comes from various user add-ons and scripts

Which could be greatly reduced by providing a proper API for uncritical public data (like for the bundled-games-list, giving an equivalent no-cv list (which is still completely missing), user-/group giveaway-listings and so on)

ISP's with long delay's should take you as an example, and renew their services....

http://www.speedtest.net/result/7847888851

having these kinds of internet speeds, is sooooo annoying when the rest of the world has a hard time catching on...

Thanks for keeping us informed about infrastructure changes, it's not a usual thing to do and it's greatly appreciated.

SGTools already uses gzipped data for SG parsing and frankly there is no reason for other scripts to not support it, since it's trivial and heavily reduces the amount of bandwidth used. Forcing only gzipped responses is perfect in my opinion.

To reduce even more bandwidth, and it as already been suggested several times though those last years and a couple of times in this thread already, an API with public information in json would be perfect.
Scripts would need to do far less petitions, they response would be smaller and you could fine control the amount of petitions that each script or site is doing.

A high amount of our traffic comes from various user add-ons and scripts, and reasonably priced bandwidth is not a strong point with Amazon.

We've been asking for API access for years now, no script would need to put pressure on the site and force SG server to render it if it had better way to access the required data. If your motivation is truly cutting the traffic that comes as a result of userscripts, please consider providing an API way to access SG data - this will not just cut the bandwidth being wasted as a result of site render, it'll also put less pressure on SG servers.

It doesn't have to be anything fancy or huge, you can start small with what generates the most traffic, which probably is a giveaway itself. For instance, here is the data that I'm making use of in ArchiBoT that could be available as /Api/Giveaway/ABCDE instead:

bool Bundled // is the game bundled? it'd probably make more sense to use something like RealCV percentage that could be set at 100, 15 and 0 respectively for unbundled, bundled and free
bool CanJoin // Is the account eligible for joining this giveaway? If API assumes no account-based access, this can be removed
DateTime CreatedAt
ulong CreatorID
DateTime EndsAt
HashSet<ulong> Entries
HashSet<SteamGroup> Groups // If SteamgiftsType includes groups, just ID of the group as seen on Steam would be enough for me
byte Points
byte RequiredLevel
EStatus Status // e.g. Begins, Running, Ended
SteamGame SteamGame // Some basic data about game like its ID (appID/subID) and type (app, sub, custom for gift cards etc)
ESteamgiftsType SteamgiftsType // Public, invite-only, group, wishlist?
Dictionary<ulong, bool> Winners // Map that maps steamIDs of the winners to a bool "received" value

Stuff that can't be easily requested (e.g. due to expensive SQL calls) could be further separated from general giveaway info, for example /Api/Giveaway/ABCDE/Winners and /Api/Giveaway/ABCDE/Entries. There is a lot of possibilities how you want to design your APIs in the most efficient and natural way in regards to your database, and regardless what you stick with, the improvements will be huge.

If you have concerns in terms of who should be permitted to access the API and what info it should expose, I don't see anything against linking API to the user's account, similar to how Steam is doing that. Afterwards you can easily impose any limits you want, for example inability to request giveaway data from creators that have blacklisted you (or you blacklisted them).

In any case thanks for keeping us informed. I doubt enforcing gzip will help much because all modern browsers and tools should already prefer to ask for gzipped content by definition, while everything that doesn't most likely do not support it to begin with, so you'll most likely cut old browsers, tools and scripts and do not affect majority of users and usages, with a very small fraction being in-between of those two where forcing gzip will actually make web browser ask for it while it didn't do that before.

Thanks for the info on the updates, and for seeking improvements in performance and usability! Most sysadmins would have just informed of server maintenance, but knowing the SG community indeed has a good component of technically-oriented users, it shows care for all the crowds.

Meanwhile, a GIF to cover all bases ^^

View attached image.

'll be testing some different servers for our database with AMD processors

EPYC news :)

wow. didn't know we were on Amazon.

<3

grazie, cg

Could someday throw some light over how SteamGifts and SteamTrades run? How many servers, balancers, services...some diagrams porn and stuff like that :D

Happy Holidays!

If most of traffic comes from addon users, why not release some API to reduce server load?

Attention to NodeJS devs, just add gzip: true to request options.

I would also love to see an api.

I'm glad I saw this thread just before my script failed. Otherwise I would have had no idea why this suddenly failed and was returning gibberish. I could see it was a binary blob that that was it.

Note Autoit users.

This is one library of many. I just happened to pick this one as it does not use an external dll.
Written by "Ward".

you will need to include his file.

#include <ZLIB.au3>

and the lines in your code similar to

$web = InetRead($url, 1)
$web = BinaryToString($web)

will become

$web = InetRead($url, 1) 
If StringInStr($url, 'steamgifts') Then $web = _ZLIB_GZUncompress($web)
$web = BinaryToString($web)

adding the one line in between. I'm sure there is a better test of if gzip is used but I'm not seeing it at first glance. I'd rather be more universal but this is the only site I've encountered that uses gzip. Perhaps If I looked at headers? I've never had a need but I know they are returning more data that I'm not currently looking at.

The reason for the if statement is because this is a web page monitor that looks at more then just steamgifts pages. I have several threads I want to know if the original post is modified. They get checked once a day and I get emailed if a change is detected.

Did not see this post in time, so had to learn it the hard way :(
Anyhow, the fix was indeed super easy - so no harm, no foul.

I also agree with what others wrote - API would make my life easier (in the long run).

Thank you for the great work
but steamtrades is down ( 504 Gateway Time-out )

Good work!

Thanks for the updates, cg.

As a lot of people already mentioned, that high amount of traffic would become very small with a proper API. Endpoints to get information about giveaways and users should already reduce major traffic, at least from ESGST.

One of the most heavy features from ESGST in terms of traffic is the feature that checks if a user has whitelisted / blacklisted you (because people just want to know), so if you provided that information natively, that would already reduce a lot.

So...a server is some kind of waitress for computers?

┬──┬◡ﾉ(° Θ°ﾉ)　Thank You！　Maintenance！

nice

A quick update on the original post since all of the maintenance has been finished for a couple of days.

1. The servers all moved to the different availability zone. Moving the databases resulted in some minor downtime of about 30 minutes as the data volumes were snapshotted and recreated.

2. As expected, most people were moved to the new load balancers almost instantly, and the previous ones received some minor traffic which faded out over 24 hours. They were then shutdown.

3. The new instance types are working well. There are AMD processors on the SteamGifts database server, and all of the other servers were moved to newer instances and better optimized.

4. The sites are now only serving gzipped data. This is a screenshot of data transfer from our load balancers before and after the changes, so we're seeing about 50% savings.

Overall, everything went smoothly and the servers are paid for another year. The performance is the same as before, we still have room to grow, and our annual bill will be a couple of thousand less than before.

if all goes well, you won't notice a thing

has anyone noticed anything?

:D & bump