Hmm... an interesting question!
The code registers reactions by going through your ability set, so the correct answer should be "in whatever order they were in, from 1 to 8, the last time reactions were updated." HOWEVER! Each reaction triggers a "use ability" result... which are processed in "first in, last out" order, so the actual correct answer is "in whatever order they were in, from 8 to 1, the last time reactions are updated."
Reactions are updated fairly frequently, so this will usually be correct. However, note that re-ordering your ability hotkey buttons does not trigger such an update, so doing so may not have an immediate effect on the order they are applied in.
That said, reactions are updated fairly often, so the de-sync from moving abilities around will not last long... and I will probably force a re-check after swapping abilities just so that this little wrinkle vanishes.