Scientific Financial Systems values the power of Snowflake for the scalability, performance, and burstability it offers to its customers. Snowpark Container Service (SPCS) Snowflake now offers more value. The appeal of SPCS lies not only in its promise of lower latency and improved security, but also in its capacity for easier scalability and streamlined deployment. SnowSQL as Infrastructure as Code (IaC)The team at SFS is excited to share that we: Migrating Quotient to the Snowpark Container Services Model. This migration provides us with a tremendous opportunity to leverage advanced database interactions and strengthen our partnership with Snowflake.
Understanding Old Architecture
Quotient previously operated as a containerized application. Docker Compose. This setting allowed us to bundle application components. (Nginx, Redis, Postgres, and Django) Isolated containers make deployment and management more consistent across environments. To support this architecture, we have developed the following product families: Python script It provided several important features such as deployment automation, update automation, CI/CD operations, Django management automation, data environment population, and other features.
This architecture provided a solid foundation and was much better than what we started with, but it also presented several challenges. The complexity of managing numerous scripts and ensuring compatibility between internal and client environments could be cumbersome. Furthermore, as our application and user base grew, we faced limitations in security, scalability, and orchestration capabilities that Docker Compose alone could not address.
We also wanted to enhance our security assurance for our customers. By migrating to SPCS, we were able to leverage advanced security features such as granular access control, strong encryption mechanisms, and secure data sharing capabilities. These features ensure that our customers’ financial data is protected, compliant with industry standards, and mitigates the risk of breaches and unauthorized access. This change not only strengthened our security posture, but also increased our customers’ confidence in Quotient’s ability to protect their sensitive information.
Finally, piecing together all these scripts with an installer that could support a variety of environments and configurations proved to be a significant time and resource burden. This experience highlighted the need for a more robust solution, which led us to explore Snowpark Container Services as a potential upgrade to our infrastructure.
See further, faster with Snowpark Container Services
The migration to SPCS was a multi-step process. The initial phase involved migrating the existing supporting infrastructure, including secrets management, image storage, and logic to populate the Quotient tables in Snowflake. This phase was relatively straightforward, thanks to Snowflake’s intuitive SnowSQL commands. By modularizing each phase into its own SQL file and folder, we greatly improved the readability, documentation, and understandability of our infrastructure code. For our custom scripts that populate tables in Snowflake that require various Quotients, we wrote some basic SQL to upload the necessary CSV files to an encrypted phase. The SQL code then ingested this data into tables based on the CSV header and column data types.
The next step involved writing the infrastructure code to deploy the containers, a complex task that required standardizing a lot of parts. Django and Nginx codeFor example, we appreciated everything about ourselves. remain We used endpoints and Django views to return standardized response codes and objects, which allowed our app to comply with Snowflake’s proxy service that sits between our public endpoints and our containers. We also modified our Snowflake connector code to leverage the OAuth flow, which provides faster and more secure connectivity to Snowflake data.
We also revamped our CI/CD pipeline. Previously, we used GitHub Actions code that SSHed into our servers and ran various scripts. We replaced this with GitHub Actions code that works with SnowSQL to update and run tests in containers. This transition not only streamlined our deployment process, but also improved the efficiency and reliability of our CI/CD operations.
The biggest benefit of migrating to SPCS is that you can decouple your Docker Compose application into multiple SPCS services. This granularity allows you to scale different parts of your application independently, optimizing resource allocation and reducing operational costs. For example, you can spin down a compute service while you run your frontend and accept requests. When a request comes in, you can spin up the expensive compute service again. This dynamic scaling capability allows you to manage your resources more efficiently, significantly reducing costs while maintaining performance.
Migrating to Snowpark Container Services transformed our infrastructure, delivering improved performance, scalability, and security. The migration process involved many complex steps, but the benefits gained have been worth the investment. From modularizing our supporting infrastructure to standardizing our codebase and implementing a robust CI/CD pipeline, the move to SPCS has positioned Scientific Financial Systems for future growth and success. We are now offering a private preview of this combined offering by integrating the Quotient application on Snowflake Native App and Snowpark Container Services. Stay tuned for an announcement when we publish Quotient on the Snowflake Marketplace!