This forum uses cookies
This forum makes use of cookies to store your login information if you are registered, and your last visit if you are not. Cookies are small text documents stored on your computer; the cookies set by this forum can only be used on this website and pose no security risk. Cookies on this forum also track the specific topics you have read and when you last read them. Please confirm whether you accept or reject these cookies being set.

A cookie will be stored in your browser regardless of choice to prevent you being asked this question again. You will be able to change your cookie settings at any time using the link in the footer.

Thread Rating:
  • 0 Vote(s) - 0 Average
  • 1
  • 2
  • 3
  • 4
  • 5
War Game Downtime and Recovery (Rollback)
#1
Hello everyone,

EDIT 11:20:
- The server has stabilized as planned.
- All missing payments have been processed.
- We have added 20 premium to all accounts that were logged in in the last 4 days.
- We will be closely monitoring the situation.
- We understand that some players may be emotional about the situation, but this does not warrant insulting behavior. Please stay constructive.

EDIT 03:00: 

- War Game is back online!
- Missing payments will be processed during today, approximately 12:00 CEST.
- There will still be some performance issues until the system settles back to normal operation.
- We are sorry for the inconvenience.

More Information:

In short, the reason for this was a large scale database corruption, which required us to reinstall a backup, dating back around 08:30 CEST. (A couple of hours).

A long description of what happened (for technical people) will be added below.

In layman terms, what happened is this: The War Game Database containing all information of the game started acting up over the weekend, but was not in a bad state. Early this morning we started with some optimizations to restore the database to full performance. While this went well over the course of the day, we didn't know of an underlying problem that went unnoticed. This problem caused a breakdown of the game, forcing us to recover. This recovery failed and we needed to roll back to the last un-corrupted backup, and then re- apply the changes we made to fix the issue we fixed over the course of the day.

We are sorry for any inconvenience this may have caused and will try our best to build additional protection against such cases.

Missing payments will be fixed over the course of Tuesday. If you have any other issue, please contact support@evilgrog.com.

Your Evil Grog Team

A more technical explanation:

The War Game database is secured via replication into "replication logs". These logs, together with periodical "full backups" allow a so- called "point in time" recovery. We make such a full backup every 4 hours, and then store the replication logs along with it. Theoretically, this allows us to apply a backup from any 4 hour checkpoint, and then re- apply the replication log forward until any point we desire. In the past this has worked flawlessly on a couple of occasions, usually with a "rollback" of less than a second.

In this special case, these replication logs and the last full backups have become slowly corrupted to a point that we were able to restore the database, but the information within in was logically not sound anymore. This was caused because we store a lot of information in our database, multiple tens of millions of fights are stored in this database, and a couple of processes that usually run far apart from each other started interacting in unexpected ways.

The initial cause for the corruption was a problem called a "race condition". Two processes on the server, that usually run well apart from another, suddenly competed for file resources on the system drive, and one process practically "stole" the resources of the other while he was trying to process it, causing the problem.

Once the problem had occurred, the system War Game is running on began behaving strangely, but not critically and went unnoticed until the database finally crashed later in the day because the corruption hit a vital spot in the database. This is where we started to recover the server, and you may have notices small downtimes during this period. We tried to fix the issue until about 23:00 CEST, when we made the call to stop and recover from a backup and re- apply the changes we had made during the day.

This backup restore is currently running, and once we have made final checks we will reactivate the server.

We decided to roll back to a full backup to keep the risk of inconsistencies to a minimum, as we're unsure if the replication logs had been affected between the full backups we made and identified as corrupt.

We believe that this was the best call to make, opting for a faster restore and getting the game back up vs. a potential risk of even longer downtimes and inconsistencies that may affect some players.

We also hope that this post serves as a transparent explanation of what has happened and why it happened.

Your Evil Grog Team
Chief Evilness Officer and Founder of Evil Grog Games
Reply
#2
grazie per le spiegazioni nel frattempo le classifiche di gioco di guerra sono impazzite. grazie all'incidente i giocatori hanno trasferito miliardi da un accountagliato. l'agricoltura rimane impunita. il primo in cima è già 800 miliardi. non richiesta ai rapporti. non intraprendere alcuna azione. stai perdendo utenti. aiuti come che cercano trucchi. devi bandire questi giocatori il prima possibile.
Reply
#3
The original post has been updated with news.
Chief Evilness Officer and Founder of Evil Grog Games
Reply


Forum Jump:


Users browsing this thread: 2 Guest(s)