# Go Data Warehouse Framework - SKILL.md

> **Engineering Guide - Go Microservices + PostgreSQL**
> Version: 1.0
> Stack: Go 1.21+, PostgreSQL, Kafka, Redis, Prometheus, ELK
> Last Updated: 2026-07-24

---

## 1. Purpose

Dokumen ini merupakan panduan resmi implementasi **Go Data Warehouse Framework** untuk standardisasi development di 30-person engineering team.

Tujuan:
- Standardisasi architecture untuk semua data warehouse services
- Consistency di seluruh codebase
- Production-ready patterns dan best practices
- Reference untuk AI Coding Assistant
- Maintain performance dan reliability

---

## 2. Stack Overview

```
Data Sources
    ↓
Services (Go + Gin)
    ├── Ingestion Workers (batch via Kafka)
    ├── Query API
    └── Stream Processors
    ↓
PostgreSQL (Data Warehouse)
    ↓
Redis (Query Cache)
    ↓
Prometheus + Grafana (Metrics)
ELK Stack (Logs)
Jaeger (Tracing - optional)
    ↓
Clients
```

---

## 3. Business Capabilities

Data Warehouse Platform provides:

1. **Data Ingestion** - Accept data from multiple sources (batch, streaming)
2. **Data Storage** - PostgreSQL as primary warehouse
3. **Data Query** - API layer untuk access warehouse
4. **Data Quality** - Validation, deduplication, audit trail
5. **Observability** - Metrics, logs, traces

---

## 4. Folder Structure

### Project Level
```
go-dw-platform/
├── framework/                      # Shared library (Go module)
│   ├── go.mod
│   ├── ingestion/
│   │   ├── batcher.go
│   │   ├── deduplicator.go
│   │   └── retry.go
│   ├── query/
│   │   ├── cache.go
│   │   └── analyzer.go
│   ├── db/
│   │   ├── pool.go
│   │   ├── batch.go
│   │   └── tx.go
│   ├── metrics/
│   │   └── prometheus.go
│   ├── logger/
│   │   └── structured.go
│   ├── middleware/
│   │   ├── auth.go
│   │   ├── tracing.go
│   │   └── logging.go
│   └── config/
│       └── loader.go
│
├── services/
│   ├── ingestion-{domain}/        # Per-domain ingestion service
│   │   ├── main.go
│   │   ├── handler/
│   │   ├── service/
│   │   ├── repository/
│   │   ├── transformer/
│   │   ├── domain/
│   │   ├── entity/
│   │   ├── dto/
│   │   ├── validator/
│   │   ├── migrations/
│   │   ├── tests/
│   │   ├── go.mod
│   │   ├── config.go
│   │   └── README.md
│   │
│   ├── query-api/                 # Unified query API
│   │   ├── (same structure)
│   │
│   └── template-service/          # Boilerplate untuk new service
│
├── deployments/
│   ├── docker/
│   │   ├── Dockerfile.base
│   │   └── Dockerfile.service
│   ├── k8s/                       # (if upgrade to K8s later)
│   └── docker-compose.yml
│
├── docs/
│   ├── ARCHITECTURE.md
│   ├── DATABASE.md
│   ├── PERFORMANCE.md
│   ├── API.md
│   └── DEPLOYMENT.md
│
├── scripts/
│   ├── new-service.sh
│   ├── migrate.sh
│   └── benchmark.sh
│
├── go.work                        # Go workspace
├── Makefile
└── README.md
```

### Service Level (Ingestion Example)
```
services/ingestion-orders/
├── go.mod
├── main.go                        # Entry point
├── config.go                      # Configuration
│
├── handler/                       # HTTP + Kafka handlers
│   ├── ingest.go                 # POST /ingest endpoint
│   └── health.go                 # Health check
│
├── service/                       # Business logic
│   └── orders.go                 # Order ingestion logic
│
├── repository/                    # Database access
│   └── orders.go                 # CRUD + batch operations
│
├── transformer/                   # Data mapping
│   └── orders.go                 # Raw → Domain → Entity
│
├── domain/                        # Business objects
│   └── order.go                  # Order domain model
│
├── entity/                        # Database entities
│   └── order.go                  # DB table mapping
│
├── dto/                           # Request/Response
│   ├── request.go
│   └── response.go
│
├── validator/                     # Input validation
│   └── orders.go
│
├── migrations/                    # Database schema
│   ├── 001_create_orders.up.sql
│   └── 001_create_orders.down.sql
│
├── tests/                         # Unit & integration tests
│   ├── service_test.go
│   ├── repository_test.go
│   ├── handler_test.go
│   └── fixtures.go
│
└── README.md
```

---

## 5. Architecture Layers

```
HTTP/Kafka Request
      ↓
  ┌─────────────────┐
  │   HANDLER       │ ← Receive request, auth, validation
  └─────────────────┘
      ↓
  ┌─────────────────┐
  │  TRANSFORMER    │ ← Map data, convert types
  └─────────────────┘
      ↓
  ┌─────────────────┐
  │   SERVICE       │ ← Business logic, batching, dedup
  └─────────────────┘
      ↓
  ┌─────────────────┐
  │  REPOSITORY     │ ← Database operations
  └─────────────────┘
      ↓
   PostgreSQL
```

All requests MUST follow this flow. **No shortcuts allowed.**

---

## 6. Layer Responsibilities

### 6.1 Handler (HTTP/Kafka)

**Responsibilities:**
- Receive request (HTTP POST atau Kafka message)
- Authentication & Authorization
- Request validation (format, required fields)
- Call Service layer
- Return HTTP Response atau handle errors
- Metrics: request count, latency

**DO ✅**
```go
- Bind request to DTO
- Validate request format
- Extract auth token
- Call service.Process()
- Return standardized response
- Log request_id + action
```

**DON'T ❌**
```go
- Business logic
- SQL queries
- Database transactions
- Data transformation logic
- Direct repository access
```

**Example:**
```go
func (h *Handler) IngestOrders(c *gin.Context) {
    ctx := c.Request.Context()
    requestID := c.GetString("request_id")
    
    var req dto.IngestRequest
    if err := c.ShouldBindJSON(&req); err != nil {
        h.logger.Error("validation failed", 
            "request_id", requestID,
            "error", err)
        c.JSON(400, errorResponse(err))
        return
    }
    
    result, err := h.service.Process(ctx, &req)
    if err != nil {
        c.JSON(500, errorResponse(err))
        return
    }
    
    c.JSON(200, successResponse(result))
}
```

### 6.2 Transformer

**Responsibilities:**
- Map data between layers
- Type conversion
- Normalization (trim, lowercase, etc)
- Timestamp handling

**Transformations:**
```
Request DTO → Domain Model
Domain Model → Entity (for DB)
Entity → Response DTO
```

**DO ✅**
```go
- Map Request → Domain
- Convert types
- Format dates/timestamps
- Validate data format
- Handle null values
```

**DON'T ❌**
```go
- Business rules/logic
- Database access
- HTTP operations
```

**Example:**
```go
func (t *Transformer) RequestToDomain(req *dto.IngestRequest) (*domain.Order, error) {
    order := &domain.Order{
        OrderID:    req.OrderID,
        CustomerID: req.CustomerID,
        Amount:     req.Amount,
        CreatedAt:  time.Now(),
    }
    
    // Validation
    if err := order.Validate(); err != nil {
        return nil, fmt.Errorf("transform: %w", err)
    }
    
    return order, nil
}

func (t *Transformer) DomainToEntity(domain *domain.Order) *entity.Order {
    return &entity.Order{
        OrderID:    domain.OrderID,
        CustomerID: domain.CustomerID,
        Amount:     domain.Amount,
        CreatedAt:  domain.CreatedAt,
    }
}
```

### 6.3 Service (Business Logic)

**Responsibilities:**
- All business rules
- Batching strategy
- Deduplication logic
- Transaction control
- Retry logic
- Metrics calculation

**DO ✅**
```go
- Implement business rules
- Control transactions
- Call repository methods
- Validate business constraints
- Handle retries
- Log business events
```

**DON'T ❌**
```go
- JSON parsing (use Handler)
- HTTP operations
- SQL queries (use Repository)
- Data mapping (use Transformer)
```

**Example:**
```go
func (s *Service) Process(ctx context.Context, req *dto.IngestRequest) (*dto.ProcessResult, error) {
    // Transform
    domain, err := s.transformer.RequestToDomain(req)
    if err != nil {
        return nil, fmt.Errorf("process: %w", err)
    }
    
    // Business logic: deduplication
    exists, err := s.repository.ExistsByKey(ctx, domain.OrderID)
    if err != nil {
        return nil, fmt.Errorf("check duplicate: %w", err)
    }
    if exists {
        return nil, ErrDuplicate
    }
    
    // Accumulate to batch
    s.batcher.Add(domain)
    
    // Auto-flush on size or timeout
    if s.batcher.IsFull() {
        return s.FlushBatch(ctx)
    }
    
    return &dto.ProcessResult{Status: "queued"}, nil
}

func (s *Service) FlushBatch(ctx context.Context) (*dto.ProcessResult, error) {
    batch := s.batcher.Get()
    
    // Transform to entities
    entities := make([]*entity.Order, len(batch))
    for i, d := range batch {
        entities[i] = s.transformer.DomainToEntity(d)
    }
    
    // Insert with retry
    rows, err := s.retrier.Do(ctx, func() (int64, error) {
        return s.repository.InsertBatch(ctx, entities)
    })
    if err != nil {
        return nil, fmt.Errorf("flush batch: %w", err)
    }
    
    // Metrics
    s.metrics.RecordBatchInsert(len(batch), time.Since(start))
    
    s.batcher.Clear()
    return &dto.ProcessResult{
        Status:        "success",
        RowsInserted:  rows,
    }, nil
}
```

### 6.4 Repository (Data Access)

**Responsibilities:**
- CRUD operations
- Query execution
- Batch operations
- Connection management
- Transaction handling (initiated by Service)

**DO ✅**
```go
- CRUD operations
- Execute queries
- Manage batch inserts
- Use prepared statements
- Log slow queries
- Handle connection errors
```

**DON'T ❌**
```go
- Business logic
- Validation (beyond schema)
- Response mapping
- HTTP operations
```

**Example:**
```go
func (r *Repository) InsertBatch(ctx context.Context, entities []*entity.Order) (int64, error) {
    const batchSize = 5000
    
    tx, err := r.pool.Begin(ctx)
    if err != nil {
        return 0, fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback(ctx)
    
    stmt := `INSERT INTO orders (order_id, customer_id, amount, created_at)
             VALUES ($1, $2, $3, $4)`
    
    batch := &pgx.Batch{}
    for _, e := range entities {
        batch.Queue(stmt, e.OrderID, e.CustomerID, e.Amount, e.CreatedAt)
    }
    
    results := tx.SendBatch(ctx, batch)
    defer results.Close()
    
    var rowsInserted int64
    for i := 0; i < len(entities); i++ {
        tag, err := results.Exec()
        if err != nil {
            return 0, fmt.Errorf("exec batch: %w", err)
        }
        rowsInserted += tag.RowsAffected()
    }
    
    if err := tx.Commit(ctx); err != nil {
        return 0, fmt.Errorf("commit: %w", err)
    }
    
    return rowsInserted, nil
}

func (r *Repository) FindByFilters(ctx context.Context, filters *repository.Filter) ([]*entity.Order, error) {
    ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()
    
    query := `SELECT order_id, customer_id, amount, created_at 
              FROM orders 
              WHERE created_at >= $1 AND created_at <= $2
              LIMIT $3`
    
    rows, err := r.pool.Query(ctx, query, filters.StartDate, filters.EndDate, filters.Limit)
    if err != nil {
        return nil, fmt.Errorf("query: %w", err)
    }
    defer rows.Close()
    
    var orders []*entity.Order
    for rows.Next() {
        var o entity.Order
        if err := rows.Scan(&o.OrderID, &o.CustomerID, &o.Amount, &o.CreatedAt); err != nil {
            return nil, fmt.Errorf("scan: %w", err)
        }
        orders = append(orders, &o)
    }
    
    return orders, rows.Err()
}
```

---

## 7. Dependency Rules

**Allowed:**
```
Handler
  ↓
Transformer
  ↓
Service
  ↓
Repository
  ↓
PostgreSQL
```

**FORBIDDEN (will be caught in code review):**
- Handler → Repository (bypass Service)
- Repository → Service (circular)
- Any layer → Handler (upward dependency)
- Handler → Database (direct SQL)

---

## 8. Data Warehouse Specific Rules

### 8.1 Batching

```go
// Service layer manages batching
batcher := ingestion.NewBatcher(config.BatchSize)

for _, record := range records {
    batcher.Add(record)
    
    if batcher.IsFull() || batcher.IsExpired() {
        err := s.flush(ctx)
    }
}
```

**Configuration (per service in config.go):**
```go
type BatchConfig struct {
    Size        int           // 5000 rows
    TimeoutSec  int           // 30 seconds
    MaxRetries  int           // 3
    RetryDelay  time.Duration // exponential backoff
}
```

### 8.2 Deduplication

**Natural Key:**
```go
// In domain/order.go
func (o *Order) NaturalKey() string {
    return fmt.Sprintf("%s_%s_%d", o.OrderID, o.CustomerID, o.CreatedAt.Unix())
}
```

**In Service:**
```go
exists, err := s.repository.ExistsByKey(ctx, order.NaturalKey())
if exists {
    // Skip atau return error based on business rule
}
```

### 8.3 Kafka Integration

**Handler untuk Kafka:**
```go
type KafkaHandler struct {
    service Service
    logger  Logger
}

func (h *KafkaHandler) Handle(ctx context.Context, msg *kafka.Message) error {
    var dto dto.IngestRequest
    if err := json.Unmarshal(msg.Value, &dto); err != nil {
        h.logger.Error("unmarshal failed", "error", err)
        return err // will be retried
    }
    
    _, err := h.service.Process(ctx, &dto)
    return err
}
```

**Configuration (in config.go):**
```go
type KafkaConfig struct {
    Brokers         []string
    Topic           string
    ConsumerGroup   string
    MaxConcurrency  int
}
```

### 8.4 Redis Caching (Query API)

**In Service:**
```go
func (s *QueryService) Execute(ctx context.Context, q *repository.Query) ([]Result, error) {
    // Check cache
    cacheKey := q.CacheKey()
    if cached, err := s.cache.Get(ctx, cacheKey); err == nil {
        return cached, nil
    }
    
    // Query database
    results, err := s.repository.Query(ctx, q)
    if err != nil {
        return nil, err
    }
    
    // Store in cache (TTL: 5-60 minutes)
    s.cache.Set(ctx, cacheKey, results, 5*time.Minute)
    
    return results, nil
}
```

**Cache Strategy:**
- Hot queries: 60 minutes
- Moderate queries: 15 minutes
- Cold queries: 5 minutes
- Invalidate on data update

---

## 9. Naming Conventions

### Package
```go
github.com/yourorg/go-dw-platform/framework
github.com/yourorg/go-dw-platform/services/ingestion-orders
```

### Files
- Lowercase, underscore separated
- `orders.go`, `orders_test.go`, `orders_integration_test.go`

### Types
```go
type OrderService struct{}        // PascalCase
type IngestRequest struct{}
type ErrOrderNotFound struct{}
```

### Functions/Methods
```go
func (s *Service) Process() error       // PascalCase (exported)
func (s *Service) process() error       // camelCase (private)
func (s *Service) FindByID() error
func (s *Service) CreateOrder() error
```

### Constants
```go
const (
    DefaultBatchSize = 5000
    MaxRetries       = 3
)
```

### Errors
```go
var (
    ErrOrderNotFound     = errors.New("order not found")
    ErrDuplicateOrder    = errors.New("duplicate order")
    ErrInvalidBatchSize  = errors.New("invalid batch size")
)
```

**Error wrapping (always):**
```go
if err != nil {
    return fmt.Errorf("process order: %w", err)
}
```

---

## 10. Error Handling Standard

**Pattern:**
```go
// Always wrap with context
if err != nil {
    return fmt.Errorf("insert batch: %w", err)
}

// Custom errors for business logic
if !order.IsValid() {
    return ErrInvalidOrder
}

// Never panic in production code
// Only panic if application cannot continue
panic("database pool initialization failed")
```

**Error Response Format:**
```json
{
    "success": false,
    "message": "Failed to process order",
    "error_code": "DUPLICATE_ORDER",
    "request_id": "req-12345"
}
```

---

## 11. Logging Standard

**Structured Logging (JSON):**
```go
logger.Info("order processed",
    "request_id", "req-12345",
    "user_id", "user-456",
    "order_id", "ord-789",
    "duration_ms", 125,
    "batch_size", 100,
)
```

**Minimal Fields (required):**
- `request_id` - Trace requests
- `module` - Service name
- `action` - Operation (insert, query, etc)
- `duration_ms` - Timing
- `error` - Error message (if applicable)

**Log Levels:**
- `ERROR` - Errors that need attention
- `WARN` - Warnings (retries, slow queries)
- `INFO` - Business events (batch inserted, dedup)
- `DEBUG` - Detailed debugging (not in production)

**NEVER log:**
- Passwords, tokens, API keys
- PII (passwords, emails in logs)
- Full database records (only IDs)
- Sensitive configuration

```go
// ❌ Wrong
logger.Info("user login", "password", "secret123")

// ✅ Correct
logger.Info("user login", "user_id", "user-123")
```

---

## 12. Validation Standard

**Layer:**
- **Handler** - Format validation (required fields, JSON format)
- **Transformer** - Type validation (dates, numbers)
- **Service** - Business validation (business rules)

**Example:**
```go
// Handler - format
if req.OrderID == "" {
    return ErrMissingOrderID
}

// Transformer - type
orderID, err := strconv.ParseInt(req.OrderID, 10, 64)
if err != nil {
    return fmt.Errorf("invalid order ID: %w", err)
}

// Service - business rule
if !s.IsValidCustomer(ctx, order.CustomerID) {
    return ErrInvalidCustomer
}
```

---

## 13. Transaction Standard

**Rule:** Transactions ONLY in Service layer.

**Repository accepts context, Service creates tx:**
```go
// ❌ Wrong - Repository creates tx
func (r *Repository) Process(ctx context.Context) error {
    tx, _ := r.pool.Begin(ctx)
    // ...
}

// ✅ Correct - Service creates tx
func (s *Service) Process(ctx context.Context) error {
    tx, err := s.pool.Begin(ctx)
    if err != nil {
        return err
    }
    defer tx.Rollback(ctx)
    
    // Call repository with tx context
    err = s.repository.Insert(ctx, entity)
    if err != nil {
        return err
    }
    
    return tx.Commit(ctx)
}
```

---

## 14. Database Rules

**Connection Pooling (in framework/db/pool.go):**
```go
type Config struct {
    MaxConns        int32         // 25 (tune per load)
    MinConns        int32         // 5
    MaxConnLifetime time.Duration // 15 minutes
    MaxConnIdleTime time.Duration // 5 minutes
}
```

**All SQL in Repository:**
```go
// ❌ Wrong - SQL di Handler
func (h *Handler) GetOrder(c *gin.Context) {
    rows, _ := db.Query("SELECT * FROM orders")
}

// ✅ Correct - SQL di Repository
func (r *Repository) FindByID(ctx context.Context, id string) (*Order, error) {
    row := r.pool.QueryRow(ctx, "SELECT * FROM orders WHERE id = $1", id)
    // ...
}
```

**Prepared Statements (always for batch):**
```go
stmt := `INSERT INTO orders (id, customer_id, amount) VALUES ($1, $2, $3)`
batch := &pgx.Batch{}
for _, order := range orders {
    batch.Queue(stmt, order.ID, order.CustomerID, order.Amount)
}
```

**Query Timeout:**
```go
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()

rows, err := r.pool.Query(ctx, query, args...)
```

---

## 15. API Response Format

**Success:**
```json
{
    "success": true,
    "message": "Data retrieved successfully",
    "data": {
        "orders": [...],
        "total": 100,
        "page": 1
    },
    "request_id": "req-12345"
}
```

**Error:**
```json
{
    "success": false,
    "message": "Validation failed",
    "error_code": "VALIDATION_ERROR",
    "details": {
        "field": "order_id",
        "error": "required"
    },
    "request_id": "req-12345"
}
```

**Pagination:**
```json
{
    "data": [...],
    "pagination": {
        "page": 1,
        "limit": 10,
        "total": 150,
        "pages": 15
    }
}
```

---

## 16. Security Rules

**MANDATORY:**

1. **Authentication** - JWT verification
```go
func AuthMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        token := c.GetHeader("Authorization")
        if token == "" {
            c.JSON(401, errorResponse("missing token"))
            c.Abort()
            return
        }
        // Verify token
    }
}
```

2. **Authorization** - Role-based access
```go
func RoleMiddleware(required []string) gin.HandlerFunc {
    return func(c *gin.Context) {
        role := c.GetString("user_role")
        // Check if role in required
    }
}
```

3. **Input Validation** - Prevent injection
```go
// Always use parameterized queries
stmt := "SELECT * FROM users WHERE id = $1"  // ✅
stmt := "SELECT * FROM users WHERE id = " + id  // ❌
```

4. **Secret Management** - Use Vault
```go
type Config struct {
    DBPassword string `vault:"db.password"` // Loaded from Vault
}
```

5. **Audit Trail** - Log sensitive operations
```go
logger.Info("data accessed",
    "user_id", userID,
    "action", "export",
    "records", 1000,
    "timestamp", time.Now(),
)
```

**NEVER:**
- Hardcode credentials
- Log passwords/tokens
- Store plaintext secrets
- Skip validation

---

## 17. Performance Rules

### Pagination (always)
```go
const DefaultLimit = 100
const MaxLimit = 10000

func (h *Handler) List(c *gin.Context) {
    limit := c.DefaultQuery("limit", "100")
    page := c.DefaultQuery("page", "1")
    
    // Validate
    if limit > MaxLimit {
        limit = MaxLimit
    }
}
```

### Context & Timeout
```go
// Always use context with timeout
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

rows, err := r.pool.Query(ctx, query)
```

### Batch Processing
```go
// For large datasets, use batching
const BatchSize = 5000

for i := 0; i < len(records); i += BatchSize {
    batch := records[i:min(i+BatchSize, len(records))]
    r.InsertBatch(ctx, batch)
}
```

### Query Optimization
```go
// ✅ Good - With index
SELECT * FROM orders WHERE customer_id = $1 AND created_at > $2 LIMIT 1000

// ❌ Bad - Full table scan
SELECT * FROM orders WHERE amount * quantity > 1000

// Index needed on (customer_id, created_at)
```

### Avoid N+1
```go
// ❌ Wrong
for _, order := range orders {
    customer := r.GetCustomer(order.CustomerID)  // N queries
}

// ✅ Correct
customerIDs := extract(orders, "CustomerID")
customers := r.GetCustomersBatch(customerIDs)   // 1 query
```

### Connection Pooling
```go
// Pool configured in framework
pool, _ := pgxpool.New(ctx, dsn)
defer pool.Close()

// Connection automatically managed
row := pool.QueryRow(ctx, query)
```

---

## 18. Testing Standard

**Minimum Coverage: 80%**

**Layers to Test:**
- ✅ Service (business logic)
- ✅ Repository (database access)
- ✅ Transformer (data mapping)
- ⚠️ Handler (integration test, if needed)

**Testing Framework: testify**

### Service Test
```go
func TestOrderService_Process(t *testing.T) {
    // Arrange
    mockRepo := &MockRepository{}
    service := NewService(mockRepo)
    
    req := &dto.IngestRequest{OrderID: "ord-123"}
    
    // Act
    result, err := service.Process(context.Background(), req)
    
    // Assert
    assert.NoError(t, err)
    assert.Equal(t, "success", result.Status)
    assert.Equal(t, 1, mockRepo.InsertCallCount)
}
```

### Repository Test
```go
func TestOrderRepository_InsertBatch(t *testing.T) {
    db := setupTestDB(t)
    defer db.Close()
    
    repo := NewRepository(db)
    
    entities := []*entity.Order{
        {OrderID: "1", CustomerID: "cust-1", Amount: 100},
        {OrderID: "2", CustomerID: "cust-2", Amount: 200},
    }
    
    rows, err := repo.InsertBatch(context.Background(), entities)
    
    assert.NoError(t, err)
    assert.Equal(t, int64(2), rows)
}
```

### Integration Test
```go
func TestOrderHandler_Ingest(t *testing.T) {
    router := setupTestRouter()
    db := setupTestDB(t)
    
    payload := `{"order_id": "123", "customer_id": "cust-1", "amount": 100}`
    
    req, _ := http.NewRequest("POST", "/api/ingest", strings.NewReader(payload))
    req.Header.Set("Content-Type", "application/json")
    
    resp := httptest.NewRecorder()
    router.ServeHTTP(resp, req)
    
    assert.Equal(t, 200, resp.Code)
}
```

**Test File Naming:**
```
orders.go          → orders_test.go
orders_handler.go  → orders_handler_test.go
```

**Run Tests:**
```bash
go test ./... -cover              # All tests
go test ./service -cover          # Specific package
go test -run TestOrderService -v  # Specific test
```

---

## 19. Observability

### 19.1 Metrics (Prometheus)

**Framework provides:**
```go
metrics.RecordHTTPRequest(method, endpoint, statusCode, duration)
metrics.RecordDatabaseQuery(operation, duration)
metrics.RecordBatchInsert(rowCount, duration)
metrics.RecordCacheHit(key, duration)
```

**In your code:**
```go
start := time.Now()
rows, err := r.InsertBatch(ctx, entities)
metrics.RecordBatchInsert(len(entities), time.Since(start))
```

**Prometheus Queries:**
```
rate(http_request_duration_seconds[5m])
histogram_quantile(0.95, rate(db_query_duration_seconds[5m]))
```

### 19.2 Logging (ELK)

**Framework provides structured logger:**
```go
logger.Info("batch inserted",
    "module", "ingestion-orders",
    "batch_size", 5000,
    "duration_ms", 250,
)
```

**Kibana can parse JSON:**
```json
{
  "timestamp": "2026-07-24T10:30:45Z",
  "level": "INFO",
  "message": "batch inserted",
  "module": "ingestion-orders",
  "batch_size": 5000,
  "duration_ms": 250,
  "request_id": "req-12345"
}
```

### 19.3 Tracing (Optional - Jaeger)

**Add trace context:**
```go
ctx, span := tracer.Start(ctx, "ProcessOrder")
defer span.End()

span.SetAttributes(
    attribute.String("order_id", order.ID),
    attribute.Int64("customer_id", order.CustomerID),
)
```

---

## 20. CI/CD & Deployment

### GitHub Actions
```yaml
on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v4
        with:
          go-version: '1.21'
      - run: go test ./... -cover
      - run: golangci-lint run
```

### Docker Build
```dockerfile
FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o service .

FROM alpine:latest
COPY --from=builder /app/service .
ENTRYPOINT ["./service"]
```

### Deployment Checklist
- [ ] All tests pass
- [ ] Code review approved
- [ ] Linting passes
- [ ] No vulnerabilities (gosec)
- [ ] Database migrations ready
- [ ] Configuration set
- [ ] Monitoring configured
- [ ] Rollback plan defined

---

## 21. Common Patterns

### Pattern: Graceful Shutdown
```go
func main() {
    server := gin.Default()
    
    go func() {
        sigChan := make(chan os.Signal, 1)
        signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
        <-sigChan
        
        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()
        
        server.Shutdown(ctx)
    }()
    
    server.Run(":8080")
}
```

### Pattern: Health Check
```go
func (h *Handler) Health(c *gin.Context) {
    health := map[string]interface{}{
        "status": "healthy",
        "database": h.db.Ping(c.Request.Context()),
        "cache": h.cache.Ping(c.Request.Context()),
    }
    c.JSON(200, health)
}
```

### Pattern: Middleware Chain
```go
router := gin.New()
router.Use(middleware.Logger())
router.Use(middleware.Recovery())
router.Use(middleware.Auth())
router.Use(middleware.Tracing())

router.POST("/api/ingest", handler.Ingest)
```

---

## 22. Code Review Checklist

Before submitting PR:

- [ ] Follows layer architecture (Handler → Transformer → Service → Repository)
- [ ] No business logic in Handler
- [ ] Mapping in Transformer
- [ ] Business logic in Service
- [ ] SQL only in Repository
- [ ] All tests pass (≥80% coverage)
- [ ] Logging complete (request_id, action, duration)
- [ ] Error handling correct (wrapped)
- [ ] Context used in all I/O
- [ ] No hardcoded values
- [ ] Passes linting (`golangci-lint run`)
- [ ] No SQL injection vulnerabilities
- [ ] No secrets in code
- [ ] Performance acceptable (EXPLAIN ANALYZE for queries)
- [ ] Metrics added
- [ ] README updated (if new feature)

---

## 23. Definition of Done

Service is production-ready when:

- ✅ All unit tests pass (≥80% coverage)
- ✅ Code review approved
- ✅ Linting passes (golangci-lint)
- ✅ No security vulnerabilities (gosec)
- ✅ Database migrations tested
- ✅ Configuration documented (.env.example)
- ✅ API documented (OpenAPI/Swagger)
- ✅ Logging configured (ELK integration)
- ✅ Metrics configured (Prometheus)
- ✅ Performance benchmarked
- ✅ Deployment tested (Docker build succeeds)
- ✅ Rollback plan documented
- ✅ README complete

---

## 24. Project Structure Script

**New Service Generation:**
```bash
./scripts/new-service.sh ingestion-payments

# Generated:
# services/ingestion-payments/
# ├── go.mod
# ├── main.go
# ├── config.go
# ├── handler/
# ├── service/
# ├── repository/
# ├── transformer/
# ├── domain/
# ├── entity/
# ├── dto/
# ├── migrations/
# ├── tests/
# └── README.md
```

---

## 25. References

- **Go Best Practices**: https://golang.org/doc/effective_go
- **PostgreSQL**: https://www.postgresql.org/docs/
- **pgx Documentation**: https://github.com/jackc/pgx
- **Gin Web Framework**: https://gin-gonic.com/
- **Prometheus**: https://prometheus.io/docs/
- **Clean Architecture**: Robert C. Martin
- **Domain-Driven Design**: Eric Evans
- **Microservices Patterns**: Chris Richardson

---

## 26. Change History

| Version | Date       | Changes |
|---------|------------|---------|
| 1.0     | 2026-07-24 | Initial Version - Go DW Framework |

---

**Last Updated:** 2026-07-24
**Owner:** Engineering Team
**Next Review:** 2026-10-24