A few months ago, members of the LotusFlare Server Engineering Team reflected on the benefits of using OpenResty and Lua. In this tech blog, we meet up with the Team again to talk about the latest Lua library for Apache Cassandra called “Lucas”, which is a component toolset that was built by the LotusFlare team.
LotusFlare is one of the few companies that uses the Lua programming language and, above all, uses it to build its own libraries to improve functionality over existing ones. In this tech blog, the LotusFlare Engineering Team describes these challenges and how this new library solves the problems.
The Challenge
Like many other engineering teams, the LotusFlare team had relied on thibaultcha/lua-cassandra for all of its OpenResty-based services. This library had served LotusFlare’s work for customers well for the past couple of years but, as we put it through its paces over time, we found it to be lacking in a number of areas. In the further text, we will highlight these areas and describe their pain points.
- Topology Updates – The main challenge was that the existing driver was not able to respond dynamically to topology changes in the Cassandra cluster. This turned out to be a big headache for operations and infrastructure teams because it meant whenever we needed to perform database maintenance, it had to be scheduled during off-peak times to minimize the impact of service disruptions. Our team attempted to fix this by modifying the thibaultcha/lua-cassandra library to accept manual topology refresh commands but this was fragile and cumbersome to manage across hundreds of application instances. This challenge defined our goal for the team always to have a robust, fault-tolerant system that could handle routine maintenance tasks without downtime.
- Monitoring – Another area of difficulty was troubleshooting database issues within Lua services. The team had previously addressed this using logging and metrics tools before and after database queries but this still left us guessing much of the time. When the team decided to build its own version, we had in mind that Lucas (the new library) would expose internal logs through a Lua callback and give us access to a wider range and more accurate array of metrics.
- Security – As part of regular security hardening initiatives, the team reviewed service-to-service communications and determined that enforcing authentication and encryption between service and database was a high priority. The old driver did not support encryption, which was some of the main requirements LotusFlare customers had. The team struggled to bring this feature to life using the old library so it was directly enabled by creating a new driver.
- Maintenance – Current drivers are not maintained to the standard that LotusFlare requires. The team was aware that the bulk of development was done more than 5 years ago and missed other capabilities in addition to the ones mentioned above. The team had concerns about the long-term viability of the old driver, especially since Cassandra itself is an actively maintained project with new versions continuing to come.
Solution: Creating Lucas
Given all of the challenges, the team decided to build a LotusFlare-developed driver which we dubbed “Lucas”. Lucas wraps the DataStax C/C++ driver using the Lua 5.1 C SDK. The reason we took this direction is that DataStax C/C++ is one of the officially supported drivers and will receive new development alongside Cassandra. By doing so, the team can focus on providing a clean API for Lua rather than dealing with the complex internal Cassandra protocols.
Also, the library is compatible with Lua 5.1, LuaJIT and OpenResty. When using the library in an OpenResty-backed web service, the session state is maintained within the worker process. Each worker process has its own session. The session is thread-safe and can handle multiple requests from different Lua contexts.
We’re in the early stages of using the new driver but we feel it was a good decision to develop our own based on what we saw as challenges for the driver that was currently available. LotusFlare Engineering plans to continue working on and bring up “Lucas” into production in future projects.
For reference, please visit: https://github.com/lotusflare/lucas
