Boundlexx API: An API to get all things related to Boundless

Boundlexx is an API that provides all kinds of information related to Boundless. I am a backend software engineer professionally and my goal with the project is to do what I do best to provide all of the information the community needs to do what they do best and build the awesome tools they do.

If there is data missing from the API you need or the data that is available does not quite meet your requirements, let’s talk so I can try to meet your needs! If it is in the wrong format, it is too hard to get everything you need quickly, or you just need help figuring something out. Just DM me here on the forums or on Discord (Angellus Mortis#1769).

Special thanks to @Mayumichi, @Simoyd, and @Gorillastomp for helping me figure out all of the related data bits to get this all up and running. Also, thanks to @DreamEvil for setting up the Discord Webhook notifications on the official TNT Discord server and ironing out the kinks in it as well.

URL for Testing Universe: https://testing.boundlexx.app/
URL for Live Universe: https://api.boundlexx.app/

Repo for Issue Tracking:

Since I have not been able to open source the Boundlexx API yet, please use the repo for Boundlexx UI for creating or tracking the progress of an feature/issue.

Current features

Overview:

To get started using the API, checkout the API docs and the individually endpoints for the browsable API. All of the possible searches, filters and related URLs are directly in the API for easy discovery.

  • Rich API docs and API schema generation (OpenAPI 3 schema + ReDoc API Renderer): https://api.boundlexx.app/api/v1/.
  • Full browsable API for all endpoints. Just go to an endpoint and check everything out.
  • Full Localization support for in game objects (localization data pulled directly from the game, so it is the same localization as the game)
  • A color API: https://api.boundlexx.app/api/v1/colors/?format=api
    • Color API has a list of all colors and their color ID, including their real RGB HEX values
    • Get all known for a given color
  • An item API: https://api.boundlexx.app/api/v1/items/?format=api
    • Includes game ID, string ID
    • Resource counts for the item on known worlds (if the item is a “resource”)
    • Get all known colors for a given block/item
    • Get current buy/sell prices of items
    • Get mint value of items
  • A world API: https://api.boundlexx.app/api/v1/worlds/?format=api
    • List of all of the known (worlds the user that is doing the polling has access to + active)
    • All of the data for the world from the game API including name, region, tier, size, atmosphere and water colors
    • A preview image of the world and forum post URL for the world if it is available.
    • Full support for Sovereign/Creative worlds from the 245/246 updates
    • World Poll data including counts of resources, leader board, total prestige, plots, beacons, etc. for each world
    • Get all known block colors for a given world
    • Get the distance (in Blinksecs) between any two given worlds as well as the warp cost and details for making a portal
    • Ability for players (mostly designed for Exoworld explorers like @Gorillastomp) to be able to upload color data
    • Get current buy/sell prices of items on given world
  • APIs for Skills, Skill Groups (so you can construct a full skill tree) and Emojis
  • “Full text fuzzy search” - I am using the Postgres trigram and btree_gin so for just about all of the endpoints, you can just do ?search=query and get results related to what you are likely looking for.
  • Stats for timeseries data (more below)
  • A notification system that allows users to subscribe to specific events (more below)

Timeseries/Statistics Data:

Everyone love stats! Well, I have a boatload of them for you. Unfortunately I am starting see holes in the auto-generated API docs, so at the moment the API docs do not cover everything you can do with them. So here is a in quick crash course.

For the timeseries data I have, there are three endpoints to retrieve the data related to the data easily:

  • /latest/ - easy shortcut to get the latest result in the timeseries if you do not care about any of the historical data.
  • / - a listing of all of the items in the timeseries, paged to 50 results at a time.
  • /stats/ - a list of varies stats data for the timeseries

For / and /stats/ you can provide filters to specify the window you want to pull results for. The format for both of these are YYYY-MM-DD or YYYY-MM-DD HH:MM. These are based on your local timezone and will automatically be converted properly on the backend so no worries there! The actual filters are

  • time_after - will only give results after the given time. For example, time_after=2020-08-02 will only give results after August 8th, 2020.
  • time_before - will only give results before the given time. For example, time_before=2020-08-05 04:00 will only give results before 4 AM, August 5th, 2020

time_after and time_before can be combined to select a very specific datetime range you want timeseries data on.

/stats/ also has another specials filter you can provide to split all of the results into “buckets” and give stats on each bucket. The format for the filter is a “human readable time interval”. So bucket=4 hours will take your results and split them into even buckets of 4 hours each and then run all of the stats on each bucket and give you the results for each one.

If there are any additional stats you would like added to /stats/, let me know! As a note for implementation, I am using a Timeseries PostgreSQL database for all of the timeseries data, so ideally there should be a PostgreSQL function to calculate the value of whatever stat you want.

Current Timeseries Endpoints:

Discord Notifications

Polling a REST API for new data can be expensive. As a way to get around that, I have added the start of a robust notification system. It is just a start, so right now I only support posting data for new Exoworlds to Discord Webhooks. In the future, I plan to support more notification types and I also plan to allow users to manage their own notification subscriptions. For now, if you would like to get this Webhook set up for a Discord server you manage, just DM me to get it set up. If there are any custom event types/notifications methods you would like support for, let’s start a conversation and work something out!

Discord Webhook - New Exoworlds

There is a lot of data for Exoworlds, so messages are not always in a single message since Discord has a hard limit on message length. Pings will how ever only happen for a single group of messages though.

Data from Exoworlds comes from two sources. So it is not always possible to post all of the data in a single ping. If the color and extra data has not be received yet, a single ping with a group of messages will be posted ASAP with the core Exoworld data and the other message with color/extra data will be posted later. If the other color/extra data exists when the core Exoworld data is received, it will only do a single ping with all of the data.

The “bot” user for the Webhook is controlled by the owner of the Webhook. I do not add a name or image to the user. That is up to you when you create the Webhook

Initial Exoworld found message (but without color/extra data available yet):

Follow up message when color and extra data is available:

What I will need from you:

Uploading Color Data:
Anyone can upload “planet json” files if they want. For security purposes, I do have the endpoint protected by an Auth Token (the Internet is a dangerous place), so just DM me to snag yourself one.

Ideally I would love to be fully automated, but right now you have to retrieve the planet JSON file yourself. I cannot help you get the JSON file, but check out @Simoyd’s Price Harvester.

Once you get your auth token and your Planet JSON file, you can us this Python script to upload it:

https://gist.github.com/AngellusMortis/b9cd435f2148afad0e29df7a173dd346

Dashboards

I have a Metabase instance set up and running on top of the data. If there are any dashboard/graphs you would love to see put together for Boundless data, just shoot me a DM and I can get something set up for you.

Current Dashboards:


Planned Features
  • Shop prices for items for live Universe! This is already implemented for Testing Universe, I just need to get an API key going for the Live Universe.
  • More stats views for shop pirices!
  • Improved API docs. The auto-generated docs are starting to show some holes with the more complicated endpoints (like the timeseries related ones). I need to go in and clean them up and make the docs more accurate for how the endpoints work.
  • A Discord bot that can query the data and also trigger notifications (see next feature)
  • A notification system built on top of the Discord bot
    • The notification system will be able allow users to configure what types of events that want to be notified for (or server owners to configure what types of events channels can be notified for).
    • Types of notifications would include (just some initial ideas):
      • a discovery of a new world (Sovereign, Creative, Exo, or even a new perm world)
      • a Shop Stand/Request Basket for a specific price range/item/count

Open Source and Collaboration

I want this tool to be useful to everyone in the community that wants to use it, so please give feedback. I want to collaborate with you all as much as possible to make it as great as possible. I would love to make the project open source so you all can collaborate for real, but right now that is not possible. I am currently using private game APIs to gather World data so I do not want to open source the code until those APIs become public or I get approval from @james, et. al. for me to make the project it public. Until the project is open sourced, please post feedback, bug reports, feature requests here in the topic.

If you want to discuss anything more in depth, feel free to DM me here or on Discord (Angellus Mortis#1769).


Supporting Boundlexx

I have had a few people say “if you need anything in game, let me know” or “if there is anything I can do to help out, let me know”. While I am not asking anything from anyone for Boundlexx some people really enjoy showing their appreciation. I will not be accepting any real world money donations, but if you really so inclined to donate, you are more than welcome to donate resources (I always love me some nice mining AoE hammers!) or coin in-game. DM me on here or Discord (Angellus Mortis#1769) to meet up with me in-game.

If you would like to remain anonymous, that is totally cool as well. For those people, I have setup a shop stand at my shop at DK Mall (location: X1). Feel free to “buy” my horribly overpriced rock.


Screenshots

All screenshots from the Testing App except for any Shop Price related ones. Those are from my local Creative universe.

API Docs






Actual API







Admin Site











Admin Site (Shop API from local Creative)



33 Likes

Architecture Details
(for the nerds):

The full dev stack is set up using Docker + Docker Compose. So I can quickly make changes, create new environments and deploy changes. There is also Cloudflare sitting in front of the Testing and Live Universe apps to protect against attacks and help cache the API endpoints. Endpoints are cached for up to 30 minutes (browsers will cache them for 30 minutes, server side cache will automatically expire when changes are made to any of the data).

The induvial Docker Compose services are as follows:

  • django: the main Python/Django application server
  • postgres: a Postgres/TimescaleDB server
  • redis: a redis instance for Caching/KV store/Celery Tasks Backend
  • celery / celerybeat: a Celery instance for async task running
  • flower: A Celery Flower instance to help track Celery Tasks

Since this is a brand new project, everything is using the latest and greatest versions of everything (Python 3.8, Django 3.0, DRF 3.11, etc.). I was also planning on adding the Discord bot as another stand alone Docker Compose service as well.

I try to automate as much of the data collection as possible (right now 100% of it is). I do that with the following:

  • BoundlessClient - a nice class designed to interface with the Boundless game APIs. All requests to rate limited to only a single request per second to prevent the app from overloading the servers.
  • import_items - a Django management command to parse Boundless game files and pull in data for Colors/Items (and metals, but those are not really used for anything fun).
  • try_update_prices - a Celery task to iterate over all Items and Worlds and pull new shop data for them. The task itself runs every 5 minutes
    • It does not pull all Item/World combinations every run. Each Item/World combination is given a “rank” for both buying and selling.
    • The rank ranges from 1 to 30. With rank 1 being update every 10 minutes and rank 30 being update every 6 hours.
    • If no changes for the Item/World combination, the rank goes down. If there are changes, it goes up.
    • Since this runs very often (every 5 minutes) and can take a very long time to run (up to ~45 minutes), only a single instance of this task can every run.
  • discover_all_worlds / search_exo_worlds - Finds all possible worlds in the given game universe.
  • poll_worlds - Runs a poll on all “active” worlds. If world is no longer active (no longer in the Game API), it is deactivated so it is no longer included. Runs once a hour.
  • ingest_exo_world_data - A task to scan the Exoworld category and parse out block colors for the worlds. Runs once a week.
  • ingest_world_data - Pulls all World data from @Gorillastomp’s awesome Google Sheet of world data. Runs every 10 minutes.
  • calculate_distances - Determines the missing distance calculations for worlds and calculates them (using the Game API). Automatically runs when any new worlds are discovered.
  • An API endpoint that allows you to post a “planet json” file (see Price Harvester) to for ingesting of World Color data as well. The API endpoint is protected by an auth token. If you want to ability to post to it for either the Testing Universe or the Live Universe, DM me and I can get you set up.

I also have a nice fancy Boundless Account authentication backend for the Admin Site, but right now it is really only for my own use since I cannot possibly expect anyone to give me their Boundless credentials to login if they cannot see the source code and verify it is “safe” (Note for Boundless devs: maybe give us an OAuth 2/Open ID Connect API?)

8 Likes
  • Added live Universe instance link (https://api.boundlexx.app/)
  • Added API endpoints for block color data, including endpoint to post a “planet json” file to
  • Added task to parse Exoworld forums for block color data
  • Adds RGB HEX values to the Color API
  • Adds Cloudflare caching layer (for Testing and Live)
3 Likes
  • Added information about the Discord Exoworld notification system.
1 Like

I am a programmer too and would be interested in getting involved to help advance this further as I have a desktop and mobile app planned for boundless players.

I could also assist in the bandwidth issues of the rest api as i have several unmetered fully dedicated servers that i would be willing to allow the use of to leverage to reduce REST request expense. As well as allow for further advancement of the api.

I have 15 years programming experience and would be really stoked to get involved!

1 Like

I appreciate the help for the offer for hosting, but bandwidth or server space are not any issue for me. I have a nasty server habit myself and have like 4 hardcore servers (like 100+GB of RAM and as much storage space as I need to throw into it) laying around. I also have a symmetric full gigabit connection as well.

I basically cannot get “better” hosting without going to AWS/Azure/GCP.

1 Like

how hard would it be to make a mod that lets you input a bit off tekst next to the alt names in the alt list so i can write what they do there lol
something like
moebius - hunter
moebs - crafter
mackab - portals/plants
i will be forever gratefull :smiley:

@Angellus thx for this epic work love it
you da best much apreciated

Good lord man where do you live, I need that!

1 Like

For real thats insane Ive got fiber but dang.

I would have to look but I dont think that is modable as those areas seem to be inaccessable via code. However ill take a look and confirm.

1 Like

This API can help me to find to find item filtered by color?

https://api.boundlexx.app/api/v1/colors/1/blocks/?format=api where 1 is the color ID of the color you are looking for. The full list of colors and their color IDs are at https://api.boundlexx.app/api/v1/colors/?format=api.

Or if you want to look at all of the color for a given item (like Gleam): https://api.boundlexx.app/api/v1/items/9555/colors/?format=api where 9555 is the ID of the item. The full list of items and their IDs are at https://api.boundlexx.app/api/v1/items/?format=api.

Just know the “block colors” API list all possible worlds. Even the previous Exoworlds that use to have them. If the entry for the block color has not URL for the World, that means it is an expired Exoworld that is no longer available.

1 Like

And if i want to locate a store that sells, as example, deep blu gleams?

I do not yet support the Shop API. I still need to get an API key for the Shop API from the devs. Also, the Shop API does not provide color data. Meaning it can tell you where shops sell gleam, but not which ones sell Deep Blue Gleam.

That being said, it looks like Deep Blue Gleam has not even existed on a world (https://api.boundlexx.wl.mort.is/api/v1/colors/235/blocks/?format=api) meaning it is only available from Gleambow. So I recommend checking out row X at DK Mall or Elvira’s Gleambow Emporium as those are the two best places I can think of. Personally, I would recommend picking a different color since Gleambow stuff is a expensive to buy.

3 Likes

Thanks for the shout out. A little more gleam yet to stock. Will be out later today along with glass and none sprayable objects.

Another good gleambow gleam store is the metal store in the back of gyosha mall near the middle hub.

Right next to the Gyosha mall guild HQ tower.

@DarkSkies

Your shop is now empty of deep blu gleam :slight_smile: let me know if u acquire more
For the other one, i visited lot of shops in the gyosha mall but no luck

1 Like

I will check whats left and let you know. Also about do for a gleambow hunt.

1 Like
  • Adds full text/fuzzy search and full feature filters to all of the endpoints. Thanks to @DreamEvil for using the API for the TNT bot and helping figure out the new features.
2 Likes

The live resource counts, if they are what I think they are, seem pretty useful for miners/gatherers.

It is currently straightforward to combine multiple polls to create time series of resource counts so people can see when a planet is relatively “depleted” versus “refreshed.” For planet ID = 5, you can get a list of polls with https://boundlexx.app/api/v1/worlds/5/polls/, which gives you a bunch of resource URLs to get the counts.

Maybe it would be better if the API itself has time series (or does it already?) so a website can get the relevant data for a world-resource pair in 1 query rather than N = number of polls.

If I understand correctly, right now:

  • one type of request will give you counts for a vector of resources for a single world at a single poll (like the exoworld posts)
  • another type of request gives counts for a vector of worlds for a single resource and a single poll (which world is best for a resource?),
  • but it could also be useful/fun to have counts for a vector of polls/times for a single resource and a single world (time series).