On March 21, 2022 10:59:12 AM UTC time, our V2 testnet experienced a service blackout regarding its FOX/OWL pool. After investigation, we discovered that the suspension of service occurred due to an overload issue in the microservice on Qilin’s subgraph. We’re disclosing the details of this incident and our fixes in this release.
PnL
Each position’s PnL is critical information for traders to determine their actions as well as for the protocol to determine liquidatable positions in the pool. On Qilin, the PnL of each position is updated based on market price movements and is handled by the graph node in The Graph. At the time of the overload incident, there are 14,474 positions in the smart contract pool (0xa094c5f24991991b4edcfe4d389e3348fe135e86). This led to an overloaded graph node that caused the backend suspension.
Subgraph
Currently, the graph node would store updated data from positions which takes up storage space. When the data reaches a certain size, the subgraph would break down and not be accessible. In its current iteration, we have found that the subgraph architecture is unsuitable for live updating data states.
Some fixes
We will be continuously updating the subgraph and updating the live states in an external microservice or frontend as a long-term fix.
The second round of testnet has been deployed such as testnet testing would continue and be unaffected.
Follow us on the channels below to stay connected:
Twitter: https://twitter.com/QilinProtocol
Telegram: https://t.me/QilinOfficial
Medium: https://medium.com/qilinprotocol
