Technical Stack

Core Backend

• Flask: Flask
• Structured Data: PostgreSQL

Connectors && Data Processing

Connectors can be written in any language. However, in this project is:

• Python 3.9.0 or upper

Frontend

• Node.js 16 or upper
• JavaScript
• Web Framework/Library: React

Additional Tools

• DataOps: Dbt
• CI/CD: GitHub Actions
• Containerization: Docker and Docker Compose
• Linter (Frontend): ESLint
• Formatter (Frontend): Prettier
• Formatter (Backend): Spotless
• Spell Checker: VSCode Extension

Encourage for learning, Data-Stack

1. Storage

• File object storage MinIO
• Non-Structured Data: MongoDB

2. Processing

• Streaming and Batching: Flink
• Big Data Processing: Spark or Arrow

3. Warehousing

• Databricks

4. Programming

• Programming OOP & Functions Scala
• Backend & Infra Go
• Fundamentals Data Engineering and Software Development

5. Protocol

• Sync Rest API
• ASync Message Queue

6. Semantic

• Modeling Power BI or SuperSet

7. Cloud Providers Any of cloud providers AWS , Azure, GCP

8. Infrastructure

• Infrastructure as Code Terraform
• Monitoring Grafana
• Logging Prometheus or ELK Stack

Supporting Development Tools

• VSCode VSCode
• Dbeaver Dbeaver
• Postman Postman
• MongoDBCompass MongDb
• CLI

FAQ

Why are most REST API connectors written in Python?

Most contributors felt comfortable writing in Python and most of libraries are written in Python.

What is using dbt for DataOps tools?

Dbt stands for data build tool and uses for modeling and manipulating data as well as checking data quality is the most benefit of dbt. dbt has supporting Enterprise Cloud version, this is simplified version of dbt cloud using Flask to wrap up dbt-core.