The main objective of this project is to provide a data engineering solution to generate valuable insights in terms of real-time user specific statistics about a typical industrial software product which will aid in betterment of product and also improve the user experience. Project Quantify focuses on building a robust data pipeline which can generate descriptive analytics by aggregating service logs, engine logs, api responses, network events, frontend and system level events etc., which are usually found in any software system. Although this project is confined to demonstrating a system working on top of a music streaming application such as Spotify or Youtube Music, the main goal is to provide a generic solution which can cover areas other than music. With this scope this project integrates a lot of tools constructing a Data Lake with the help of Spark and then outputs few value adding dashboard screens using Data Warehouse for a business specific end user who aspires for deeper insights about their product.
- Link to public snapshot of grafana dashboard https://rajaravindra919.grafana.net/dashboard/snapshot/wGqVPaDV6u1BkEJFOWKljXMrIp3OzKNH?orgId=0&refresh=30s
If you already have a Google Cloud account and a working terraform setup, you can skip the pre-requisite steps.