For those looking for historical EDDN data, I have built a cache system and a new protocol to deal with the problem of sending commodities one at a time. This system is based on protocol buffer, which in testing reduced the bandwidth taken by the messages from about 240 bytes zlib compressed JSON to around 95 bytes uncompressed protocol buffers. In addition, this system supports batching. When you need data, you call the endpoint with the last Unix timestamp of data you receive and it will send you everything since then. The code for the server is located at https://github.com/huadianz/elite-eddn-history and includes the protocol buffer schemas you need to receive the data. In addition, it also contains an example program that shows you how to unpack the protocol buffer binary stream.
The current server I am running is located at http://elite.servers.henryzh.com/api/v0/update/
The API will send you up to the last 1000 missed updates with Unix timestamps in MS >= the timestamp in the system. If you need more you will need to query the server again.
As this project is open source I welcome any suggestions or if you have any code you want to commit, that would be great. As for already planned features:
- Update the network using batched commodity updates
- Current status server (query the status of a station in its most recent state)
- Timeseries server with influxdb
- Using commodity IDs instead of names, if we get those from Frontier at any point