HTTP Shopping API Documentation

[Stretchious]

Does that work when there is no beacon name or guild alignment, or a mixture of both?

[Mayumichi]

I haven’t actually tested it with guild alignment, but theoretically it should. Just found a bug in it by skimming through it though, could be more. Fixed it ofc.

edit: It should respect the string length, so if it’s 0, the resulting beacon/guild name is just an empty string.

[Dhusk]

How are you instantiating the ArrayBuffer from the binary data? DataView complains that it requires an ArrayBuffer (which obviously it does) but ArrayBuffer requires a set length and the length of the response will vary and I don’t see any constructor that allows setting an ArrayBuffer based on binary.

Are you perhaps instantiating an ArrayBuffer with a length that is well beyond what is needed and then looping through and putting the api response in byte by byte?

[Mayumichi]

I added an edit with a snipped how I do it in node.js, but it’s a bit special since it’s node.js so if you’re using something else it’ll probably be different.

It’s a few posts above, here HTTP Shopping API Documentation

[Dhusk]

I’m building a new website for Portal Seekers and as part of it I wanted to have a little area to browse the shopping API. It wouldn’t be anywhere as detailed as what you’ve already got for quite a while as I’m doing this part time in my free time, which I don’t have much of at present

https://joshdboyle.github.io/ps/

[Mayumichi]

Looks very professional! The more tools the merrier imo

[Dhusk]

For anyone wanting to do this in vanilla js, here’s how I accomplished (with a huge assist from Mayumichi):

function makeApiCall(url) {
    let oReq = new XMLHttpRequest();

    oReq.open("GET", url, true);
    oReq.responseType = "arraybuffer";

    oReq.onload = function (oEvent) {
      let arrayBuffer = oReq.response;
      if (arrayBuffer) {
        let byteArray = new Uint8Array(arrayBuffer);
        orders = parseShoppingApiResponse(byteArray);
      }
    };

    oReq.send(null);
  }

The variable “orders” is declared at a higher scope and thus accessible in the method.

[Dhusk]

Is there a list of item IDs out there somewhere somebody could share? Thanks

[Mayumichi]

Luca gave us a list

HTTP Shopping API Documentation

this is the list of all items that should be possible to ever have in a shopping plinth, noting it also contains the “cheat” items normally impossible to get (you may want to just ignore those…), it is 1112 items, not 1114 (I took out a couple items that are technically in-game, but are just dev experiments and not actually attainable). — edit: looks like some of these are not quite correct, eg ive listed the LED_1,2,3 etc blocks which you cant get in your inventory, so ignore those too, same…

[james]

This is an example script for processing the binary responses in Python:

https://gist.github.com/jamesaustin/2be04dc868cca9d5b5fea1f6e9dfc67c

I used it to dump and validate the output from the servers during development.

We don’t plan to offer a JSON response whilst the live game servers are returning the data - as we don’t want to risk stalling the game simulation. If we want JSON then we’ll likely switch to a different architecture where the data is collated via another service. But the disadvantage of this would be that the Live servers and local Creative servers would then deliver different responses.

[Dhusk]

Makes sense. Thanks, @james. Testing is down atm as well. Testing update inc???

[Stretchious]

Would it be possible to clarify a few things on the Shopping API?

1. Regarding request limits - Is it 1 request per second, per API Key or is that per server (regardless of where the request comes from)?
   1b. If per server, will there be anything in place to distribute requests fairly?

2. With cached responses on live, will there be a header in the response to show the timestamp of the cache, so that we know how fresh the data is?

[lucadeltodecso]

its 1 request per second’ish (strictly, 1 second since the server last responded to a shopping request, rather than since last received a request), and per server, not per API key. This is about preventing the shopping http api from being able to throttle the normal in-game requests as they share the same “resource pool”.

there is nothing to try and distribute across requesters… I would hope there are not going to be so many competing users in the first place, but if it became a problem could do something about it then.

[Nightstar]

@lucadeltodecso

I would hope there are not going to be so many competing users in the first place

The desire to discourage use shows in the design.

I gather from some side conversations a little information about how the shop scanner is implemented into the server and can really only put it into context one way.

However this API is only minimally useful and it’s set up in such a way that it’s very restrictive and difficult to use, and can’t actually be relied upon to provide consistent, much less “real time” data.

So you have a situation where not many people are going to be interested in working with it, and there won’t be really room for interactive applications or any sort of integration other than a minimal data logger that’s always running behind.

@lucadeltodecso

if it became a problem could do something about it then.

If more than one person attempts to integrate with this, it’s going to be a problem.

[lucadeltodecso]

the CDN cache layer should smooth that over… the limit is in the server, not the cdn so you can only hit the limit if requesting uncached data; so there would be some small amount of fighting between multiple clients over which bit of uncached data is generated next, but once hit itd be cached for 30mins and next hits would just hit the cdn and get the cached data without hitting the rate limiting.

[lucadeltodecso]

and the cdn does return an “age” header which is the number of seconds since the data was cached in the cdn.

[Stretchious]

I think that’s going to be quite difficult when any new app comes along and wants to initially populate it’s own database with all the most up-to-date prices.

This news kinda blows my php implementation out of the water too. I’m going to have to move the requests to the front end now and delay showing data to a user.

[lucadeltodecso]

(assuming there is an existing app already polling all this like crazy) then the new app comes along and almost all of its requests just hit the cdn and get the last cached data.

[DreamEvil]

Just trying to test my understanding of the rate limiting and the cache.

• Each individual request get cached
  – I do a request for stands selling item x, gets cached for 30 mins
  – I do a request for baskets requesting x, gets its own 30 minute cache timer
• If its in cache, you will always be served the cache data
• If it’s not in cache and server is not on cooldown, you will get live data which is then cached
• If it’s not in cache and server is on cooldown, what is the error response?
  – Response code?
  – Error message?
• Do too many requests while server is on cooldown result in some kind of ban?

Did I get any of the above wrong?

[Stretchious]

Ah ok, not as much an issue as I assumed then.