As a small team of data engineers and data scientists, we often work on projects that involve designing and implementing data processing solutions for various customers. Recently, we had an interesting challenge: one of our customers had developed an optimisation algorithm that they wanted to run on a container-based platform in the cloud. The algorithm was designed to solve a complex optimisation problem that required some specific libraries installed in a Linux environment. The customer wanted us to design and implement an architecture that could run a data processing job on a daily basis, using their optimisation algorithm as the core component. The job would take about 10 to 20 minutes to complete, and the customer expected to run about five jobs per day. It all had to be extremely cost-effective and secure, meaning that only authorized users or systems could create job runs and view the results. Once rolled out, we would have to maintain the architecture, so monitoring and observability were also important aspects for us.
Staff Service
https://www.friend007.com/read-blog/224036 Get world-class Staff Service services from eHR! By hiring us, you can keep your operational costs lower. Contact us today! For more information,…