📊 computer_sales_data_engineering_project

🚀 Project Overview

This project demonstrates the design and implementation of an end-to-end data engineering pipeline for a computer sales business. The goal is to transform raw transactional data into high-quality, analytics-ready datasets that support data-driven decision-making.

🧠 Business Problem The business lacked:

• Centralized sales reporting

• Visibility into product profitability

• Insights into customer purchasing behavior

• Performance tracking for sales employees

This project solves these challenges by building a scalable and automated data platform.

Repository structure

DATA ARCHITECTURE

DATA SECURITY

🔐 Data Security & Governance

To ensure secure and reliable data handling:

🔒 Access Control

• Role-Based Access Control (RBAC):

  • Data Engineer → Full access
  • Data Analyst → Read-only
  • Business manager → Restricted views

DATASET

SCRIPTS

VERSION CONTROL

🔁 Version Control

Version control was implemented using Git:

Key Practices:

• Feature-based branching
• Meaningful commit messages
• Code version tracking
• Collaboration-ready structure

README.md

``

Components:

• Data Source: Raw PC sales dataset
• **Ingestion: SQL scripts
• Storage: Data lake (raw, processed, curated layers)
• Transformation: SQL + Python
• Warehouse: Structured star schema

---

🔄 Data Engineering Workflow

1. Data Ingestion

• Loaded raw CSV data using Python
• Stored data in the raw layer of the data lake

---

2. Data Storage (Medallion Architecture)

• Raw Layer: Original dataset**
• Processed Layer: Cleaned and standardized data
• Curated Layer: Business-ready datasets

---

3. Data Transformation

• Handled missing values and inconsistencies
• Standardized formats (dates, pricing, categories)

4. Data Modelling (Star Schema ⭐)

Fact Table:

Dimension Tables:

🧪 Data Quality Checks

Implemented validation checks:

• Null value detection
• Duplicate removal
• Invalid pricing checks

Ensures high data reliability and trust.

---

⚙️ Pipeline Automation

• Automated workflows using orchestration (Airflow-ready design)
• Scheduled data refreshes
• Reduced manual intervention

---

☁️ Scalability & Future Improvements

• Cloud migration (AWS / Azure)
• Real-time streaming pipelines
• Integration with APIs
• Advanced analytics (ML models)

---

📈 Business Impact

This solution enables:

• Better pricing strategies
• Improved sales performance tracking
• Data-driven decision making
• Increased operational efficiency

---

🛠️ Tech Stack

• Python
• SQL
• Ssms
• Git & GitHub
• (Optional: Airflow, Snowflake, AWS/Azure)

---

👨‍💻 Author

Andile Dube