Flox Data Export

Daniel Sperl on February 10, 2020

As I mentioned in the previous blog post, our no-fuzz backend is sadly reaching the end of its lifetime. Flox will shut down on August 1st, 2020.

Some of you are using Flox purely for its simple analytics and logging capabilities. If that’s the case, I just want to assure you that your games won’t run into any problems, even if you leave them untouched. Naturally, you won’t get any analytics any more, but the games themselves will just continue to run without any errors. The AS3 library will attempt to reach the Flox server, which will fail. Then it will store a tiny compressed file with the logs on the user’s phone, and that’s it.

With that out of the way, I want to reach out to those of you who make use of the Entity system and need to migrate their data to a different backend. I promised you an easy way of exporting the data from Flox – and here it is: an update to the good, old Flox Ruby Gem.

Some of might already have used that gem in the past; it’s a small Ruby library (plus command-line tool) that helps with certain Flox tasks. In the latest version, it can act as a very handy exporter for all your player and entity data.

Preparations

Adding an index

First of all, there’s a small caveat: all the entities you want to export need to have an index on the property updatedAt. I had to make this a requirement in order to make sure no entities are missed, even if you run the exporter multiple times.

E.g. you could start the exporter today to get all current entities safely exported. When you run it again in, say, two weeks, only changed or new entities will be fetched. The same mechanism also kicks in if an export was aborted for some reason (deliberately or unwittingly): just start it again, and it will continue where it left of.

So the first thing you should do is log into the Flox backend and navigate to the “Indices” section of your game. Click on “Add Index”, select the desired entity type and enter “updatedAt” as Property 1, while leaving Property 2 empty.

This will immediately trigger indexing of this entity table. Depending on the number of entities, this will take a while – although there’s unfortunately no way to tell when it’s finished. For really big datasets, I recommend you wait a day before moving on, just to be on the safe side.

Adding a Hero

In Flox, super-users are called “Heroes”. They are basically just the same as regular “Players” with a predefined “Key”-login. Different to regular players, though, heroes don’t care about access rights. A hero doesn’t ask for permission, he simply does what’s necessary, right?

That data export is the job of a hero, so create one in the “Heroes” section of the web interface and note the auto-generated “key”.

(BTW, did you know Chuck Norris once broke the law? It still isn’t fixed.)

Installing the Flox Gem

The command-line tool that handles the actual export is part of the “Flox” Ruby Gem. (Gems are like NPM packages, just for Ruby.)

To use it, you need to have Ruby installed on your computer. You’ll easily find tutorials online showing how to install it, but here’s a quick recap:

  • On Windows, the easiest route is using RubyInstaller.

    • When setup is complete, start the “Command Prompt with Ruby” and navigate to the directory in which you want to store the exported data (cd my\export\dir).
    • Install the Flox gem via gem install flox

  • On macOS, Ruby will already be available (alternatively, install it via Homebrew).

    • Open the Terminal and navigate to the directory in which you want to store the exported data (cd my/export/dir).
    • Install the Flox gem via gem install flox

That’s it with the preparations!

Running the exporter

Now that everything is prepared, the export is really straight-forward. Just call the export_entities sub-command and pass in the respective game ID, keys and entity type, like this:

flox export_entities --game_id="id" --game_key="key" --hero_key="key" --type="Player"

Hit enter and watch as the script creates JSON files with all player data in a sub-folder of the current directory, starting with the oldest ones.

Exporting entities of type 'Player'. Beware, this will take a while!
  -> Precondition: the entity MUST have an index on the property 'updatedAt'!
  -> If you stop and restart, the exporter will continue where it left off.
  -> Running the exporter again will download only the changed / new entities.
Progress: .............................................................
Export completed!

From there, you can pick them up and process them for import into whatever new backend you want to migrate to. As mentioned above, if the process finished and you want to get only the changed or new entities, just run the same command again, from the same directory.

That’s it!

Again, I’m fully aware that having to do all this in the first place is a real pain. But I hope that this exporter at least simplifies the migration process for all of you.

Let me know if you’ve got any questions or if you run into problems during the export. Cheers!