per planet, there is no cross-planet interactions
Okay, thatās a heck of a lot less concerning! 
So if one was to keep a full list of all 1114 items across 50 planets, it would indeed only take ~40 minutes to get everything.
I can definitely live with that.
Zero latency / 50 threads?
Should be better than days, thoughā¦
I mean yeah, itās not a perfect solution and thereās room for improvement but itās a significant leap forward. At the moment, if I want to find the best price on an item I have to manually run to all 50 planets (thank heavens for TNT!) and do a scan where usually 60+% donāt even have any at all of the item I want.
Now, I can trust that the data I am getting is anywhere between 0-60 minutes old so I would be more willing to use it.
itās ok if only one do the request, when many request will happen from different interfaces/users there will be higher time than 40 min delay for everything.
Sure its better than nothing i guess.
I think the 1 sec delay globally is pretty limiting.
Is this limted to only shopstand and request basket ?
Can it be used for storage block aswell ?
I couldnāt agree more. Iām sure itāll evolve into something even more useful though.
You are right though, with other people also requesting and you have no idea what they requested or when it will absolutely take longer than 40 minutes. Depending, of course, on the number of users trying to hit that planetās API at the same time
So, whatās the procedure for acquiring/requesting an API key? @lucadeltodecso @james
Should we be PMing someone specific?
Or will a topic be created?
1 Like
This is a message to the Devs (hope they are watching)ā¦
Iām sure you guys have thought of this, and maybe iām totally wrong with what iām about to say. .but, wouldnāt it have been easier for both you guys and for members that are trying to help code to this to have a standard JSON packet as the return response and to have the API with a āget allā command?
50 worlds
1100 total items (roughly)
There is my thought/logic behind itā¦
-
doing each query one by one, per item, per server etc ā¦ means for 1 item, it would have to send 100 fetch commands to pull from all servers. if you have 20 users that do that, searching for all the hammer types (for example) ā¦thatās 6 hammers. in total thats 100x20x6 = 12,000 queries to hit the server for only 6 hammers, and only 20 players.
-
maybe itās just me, but #1 above doesnāt seem very scalable, if you are planing on supporting multiple searches, multiple APIs, multiple users, etc. I realize that right now the servers have at most between 5-25 users, but if you are thinking of scaling up to support a larger customer base, I would think you would want something with much less queries.
My Thoughts / Original Plans
So the following was my plan to try and help out the community ā¦but itās on hold till we see how this API progressing, in hopes that it changes some.
I was thinking of making a google app to compliment this, to allow users to search via there phone and/or PCs. It would use BigQuery as the DataWarehouse and DataStudio as the visualization. Was planing on having the system query all 50 servers with a 5mins delay between queries so that the game servers werenāt taxed. The data collected would store into the DataWarehouse ā¦ and allow for tracking of market trending, searching per planet, per town, per location (lat/long distance), etc. All of this would query not the game serversā¦ but the DataWarhouse that would hold all the data.
To make the above possible, would need the API to return all Buy/Sell orders in 1 packet. Or at least, all buys in 1 ā¦ and all sells in 1. for example, āGetAllDataā from server to datawarehouse ā¦ that parses through the data and populates into the DataWarehouse. This would mean max of 2 queries per 5mins to get the item details. I would think this data response would be in JSON, or at least the byte data would be in JSON ā¦
I realize on a coding level, if you allready have the functions setup to return a per item, based on ingame code that is used for the current searching method ā¦ the easiest method is to open that ingame code to the API which is what it looks like you guys did, however if you want for the community to pretty much create your shopping center for you and to expand on it a bit ā¦ to make more people want to play your game so you can make more moneyā¦ I feel the method really needs to be adjustedā¦
Also, with the API, Iām so surprised that we havenāt gotten some method of search/filtering based on item stats. for example, if i do a search for a hammer ā¦ I have to travel around to 20 different shops to find one that is what I am looking forā¦ or the current API and ingame searching works OK for resources, etc that donāt have added statsā¦ but it doesnāt really work well for items that have bonus stats from forging, etc. Some people would argue the fact that adding the ability to search for that. .would remove the exploration and shopping around in the gameā¦ however, I argue that because the API and shopping/searching in the game ā¦any amount of this will limit exploration to a degree. Adding some specific filters would limit that, it will mean players can progress more and then explore at there lesure ā¦ as it is, I donāt explore when looking for an itemā¦ I mark it on the map, find the closest portal, and run like crazy to it, without stopping. then rinse and repeat.
I feel like this searching for custom items brings me back to the old āSWGā days (for anyone that remembers those) where you could search for customer items, filter by stats, etc. it didnāt ruin the economy, it actually stabilized it because sellers were able to compare there items to others similar and sell like that. Right now, we have sellers that think ā¦ āohh, someone else is selling this weapon for 5000, well Iām going to do the sameā ā¦ not realizing that itās a forged item vs a non-forged. it makes for new players to get frustrated as well. my wife and 3 kids can contest to that.
Anyways, thatās my feedback, rant, or whatever you want to call me. take it or leave it, but i wanted to put my thoughts and 2cents out there. I like the game, the direction it has gone, and the fact of the API, I just feel there needs to be tweaks, if they want it to invite new player base, etc.
2 Likes
Iād love to have this, would make the forge pricing in line with supply/demand, now itās all over the place.
1 Like
-
We proposed exposing the shopping API via a key as a prototype as it was relatively simply once we had the system. Players repeatedly requested access to the shopping data and were snooping it which is even less efficient.
-
A āgetAllā JSON API is also not scalable as the universe grows the shopping offers will also grow. āgetAllā would already be a huge response if it returned all offers. Ideally we would expose a āstreamingā API that allows players to track all changes to the markets. This would allow you to track all offers as theyāre made available and removed.
-
The current API will be shielded by a cache so that requests donāt hit the game servers.
-
There is no need to constantly request all items, as some items are never bought and sold. The API should be used in relation to the item market liquidity. Likewise many worlds have very few shops. So any calculation that shows 50 worlds x 2 (buy + sell) x 1000 items isnāt being efficient at maintaining active market status.
-
An advantage of using the game API is that itās portable between Testing, Live and Creative universes. So any tools created can also run in the future on your own universe.
-
All items stats (forged abilities) and colours are supported by the server API. But theyāre currently not accessible via the client as we donāt currently have a mechanism to define the more complex search in the client GUI.
2 Likes
Searchable exo colors!!!
hey James,
thanks for the response. I get why you guys did what you did, on a program level, as I work with very large companies to setup/create multiple API and web services integrations, so I get it. I guess I was thinking more of a higher level of what āIā was trying to do, vs what āyouā were trying to do, lol ā¦ sounds selfish, but hey, being honest 
ā¦ also, I was thinking about web services that charge for there traffic, ā¦ for example, X amt per 1000 POST or GET calls after the first 100,000 ā¦ thatās standard web services costs for most I have worked with for large companies, so thatās the approach I was attempting, but with the āpossibleā quantity this method could turn into, the cost for said servers/ hosting services would be extreme. Not looking to take a huge cost on of my own pocket right now for this which is why I was trying to do it a little different, but fully understand your logic and the reasons, makes since.
I see that some other people are in the process of working on there own APIs and requesting keys, so Iām taking a back seat for now since my method wouldnāt exactly be cost effective as a web service. If i could request that we can get two abilities added into the API and maybe even the knowledge section ā¦
- the ability to search / filter by block color
- the ability to filter by weapon/item stats ā¦ or even forge status (example :: filter all hammers that have X dmg or higherā¦ OR filter all hammers that have lvl 10 Damage boon and lvl 4 adjacent boon) ā¦ after all, you guys have it listed in the weapons, would think it would be relatively easy to add it as a return value for the API ā¦ now adding it to the filter ability into the game where there are no filters? I realize, thatās step askā¦ but canāt blame me for trying
2 Likes
Iād like to argue that worst case scenarios are a valid performance metric, but I hope in practise the 30min cache will be the limiting factor and most items have low enough velocity to be shuffled down in priority.
This is the most interesting part to me, does this mean itād be relatively easy to expose the āmetadataā via the API? How would shop stands with varying forge boons be aggregated? I think right now theyāre all in the same entry. Theyāre obviously more expensive than non-forged stuff so theyāre often trimmed from the result list on busy planets.
You can argue it - but what youāll be doing is making sure everything is updated every 30 minutes - whereas if you prioritise popular items at the expense of less popular (or absent) items then youāll could have everything updated every 10 minutes (say).
If you query for all items on all worlds then a good 75% of them will rarely give any new data back or will come back empty.
Try something like:
- Query everything - to generate a base line.
- Start querying stuff again - if an item hasnāt changed then increase the delay until itās queried again. If it has changed then decrease the delay until itās queried again.
Weād need to decide if itās fair to have data available outside of the game versus in it.
6 Likes
As the back end system has never really been explained, Iād really like to know if the following is correct:
The way the system is implemented, if you were to enable these features (requesting by an itemās forge traits or color) in the API, any list returned would be a subset of the cheapest 20 stands/baskets and not the full subset of the planetās available items.
Iām not sure if thatās confusing so, for instance, if someone requested the list of hammers forged with 3x3 and speed, they wold not necessarily get all of the sellers of this item on the planet, but only those items that meet the criteria and also are among the list of the cheapest 20 vendors for diamond hammers already being returned.
Even more critically, those that want to search for an item like a rock in an uncommon exo color. As these items are being excluded (on some planets) from the list already due to price, the API would return a result saying that nobody is selling the item if the lowest price on it is 8c but the scanner is currently only returning shop stands with prices below 5c.
Is that correct?
Isnāt the lowest update cycle 30 minutes atm because of the cache? In any case I meant that I like to base my estimates on the worst case scenario. If I said items are going to be updated every 10 minutes and then the worst case scenario does come true, Iād have to eat my words. Right now Iām doing something similar to what youāre suggesting, I calculate a volume delta which determines how often an item should be updated. But as I have no idea what the activity looks like in reality, especially in the future, itās hard to estimate what the update cycle will be.
1 Like
Iām wondering if this would be more beneficial to have on your end, so any apps/keys accessing the service will all just hit that server and nothing else. Then there is no need for external apps to handle potential collisions when more than one is requesting new data from the same server at the same time.
Youāll also likely get faster round-trip responses as well if theyāre on the same server centre.
2 Likes
The game server API was designed to give instant, real time responses to the live data on request. So change any plinth -> query the data -> and itāll be instantly correct. Which was our usage pattern with players making individual requests via the Knowledge.
If weād designed the system for players to mass export all data weād have implemented something different. Likely a daily(?) snapshot + stream of deltaās API.
2 Likes
Iāve been racking my brain with this thinking whether it would be better than what I currently have.
The idea is that I should increase/decrease the delay between the queries to item X by Y amount, depending on if there was market movement?
But wouldnāt this cause the items with an exact 50%/50% active/inactive scan cycle hover at 0, those with less activity would eventually reach +infinity delay, those with more activity would be at -infinity.
Or am I missing something obvious 