pranajaya161 d2d8add667 first commit

2026-08-07 12:21:42 +07:00

30 KiB

Raw Blame History

Go Data Warehouse Framework - SKILL.md

Engineering Guide - Go Microservices + PostgreSQL Version: 1.0 Stack: Go 1.21+, PostgreSQL, Kafka, Redis, Prometheus, ELK Last Updated: 2026-07-24

1. Purpose

Dokumen ini merupakan panduan resmi implementasi Go Data Warehouse Framework untuk standardisasi development di 30-person engineering team.

Tujuan:

Standardisasi architecture untuk semua data warehouse services
Consistency di seluruh codebase
Production-ready patterns dan best practices
Reference untuk AI Coding Assistant
Maintain performance dan reliability

2. Stack Overview

Data Sources
    ↓
Services (Go + Gin)
    ├── Ingestion Workers (batch via Kafka)
    ├── Query API
    └── Stream Processors
    ↓
PostgreSQL (Data Warehouse)
    ↓
Redis (Query Cache)
    ↓
Prometheus + Grafana (Metrics)
ELK Stack (Logs)
Jaeger (Tracing - optional)
    ↓
Clients

3. Business Capabilities

Data Warehouse Platform provides:

Data Ingestion - Accept data from multiple sources (batch, streaming)
Data Storage - PostgreSQL as primary warehouse
Data Query - API layer untuk access warehouse
Data Quality - Validation, deduplication, audit trail
Observability - Metrics, logs, traces

4. Folder Structure

Project Level

go-dw-platform/
├── framework/                      # Shared library (Go module)
│   ├── go.mod
│   ├── ingestion/
│   │   ├── batcher.go
│   │   ├── deduplicator.go
│   │   └── retry.go
│   ├── query/
│   │   ├── cache.go
│   │   └── analyzer.go
│   ├── db/
│   │   ├── pool.go
│   │   ├── batch.go
│   │   └── tx.go
│   ├── metrics/
│   │   └── prometheus.go
│   ├── logger/
│   │   └── structured.go
│   ├── middleware/
│   │   ├── auth.go
│   │   ├── tracing.go
│   │   └── logging.go
│   └── config/
│       └── loader.go
│
├── services/
│   ├── ingestion-{domain}/        # Per-domain ingestion service
│   │   ├── main.go
│   │   ├── handler/
│   │   ├── service/
│   │   ├── repository/
│   │   ├── transformer/
│   │   ├── domain/
│   │   ├── entity/
│   │   ├── dto/
│   │   ├── validator/
│   │   ├── migrations/
│   │   ├── tests/
│   │   ├── go.mod
│   │   ├── config.go
│   │   └── README.md
│   │
│   ├── query-api/                 # Unified query API
│   │   ├── (same structure)
│   │
│   └── template-service/          # Boilerplate untuk new service
│
├── deployments/
│   ├── docker/
│   │   ├── Dockerfile.base
│   │   └── Dockerfile.service
│   ├── k8s/                       # (if upgrade to K8s later)
│   └── docker-compose.yml
│
├── docs/
│   ├── ARCHITECTURE.md
│   ├── DATABASE.md
│   ├── PERFORMANCE.md
│   ├── API.md
│   └── DEPLOYMENT.md
│
├── scripts/
│   ├── new-service.sh
│   ├── migrate.sh
│   └── benchmark.sh
│
├── go.work                        # Go workspace
├── Makefile
└── README.md

Service Level (Ingestion Example)

services/ingestion-orders/
├── go.mod
├── main.go                        # Entry point
├── config.go                      # Configuration
│
├── handler/                       # HTTP + Kafka handlers
│   ├── ingest.go                 # POST /ingest endpoint
│   └── health.go                 # Health check
│
├── service/                       # Business logic
│   └── orders.go                 # Order ingestion logic
│
├── repository/                    # Database access
│   └── orders.go                 # CRUD + batch operations
│
├── transformer/                   # Data mapping
│   └── orders.go                 # Raw → Domain → Entity
│
├── domain/                        # Business objects
│   └── order.go                  # Order domain model
│
├── entity/                        # Database entities
│   └── order.go                  # DB table mapping
│
├── dto/                           # Request/Response
│   ├── request.go
│   └── response.go
│
├── validator/                     # Input validation
│   └── orders.go
│
├── migrations/                    # Database schema
│   ├── 001_create_orders.up.sql
│   └── 001_create_orders.down.sql
│
├── tests/                         # Unit & integration tests
│   ├── service_test.go
│   ├── repository_test.go
│   ├── handler_test.go
│   └── fixtures.go
│
└── README.md

5. Architecture Layers

HTTP/Kafka Request
      ↓
  ┌─────────────────┐
  │   HANDLER       │ ← Receive request, auth, validation
  └─────────────────┘
      ↓
  ┌─────────────────┐
  │  TRANSFORMER    │ ← Map data, convert types
  └─────────────────┘
      ↓
  ┌─────────────────┐
  │   SERVICE       │ ← Business logic, batching, dedup
  └─────────────────┘
      ↓
  ┌─────────────────┐
  │  REPOSITORY     │ ← Database operations
  └─────────────────┘
      ↓
   PostgreSQL

All requests MUST follow this flow. No shortcuts allowed.

6. Layer Responsibilities

6.1 Handler (HTTP/Kafka)

Responsibilities:

Receive request (HTTP POST atau Kafka message)
Authentication & Authorization
Request validation (format, required fields)
Call Service layer
Return HTTP Response atau handle errors
Metrics: request count, latency

DO ✅

- Bind request to DTO
- Validate request format
- Extract auth token
- Call service.Process()
- Return standardized response
- Log request_id + action

DON'T ❌

- Business logic
- SQL queries
- Database transactions
- Data transformation logic
- Direct repository access

Example:

func (h *Handler) IngestOrders(c *gin.Context) {
    ctx := c.Request.Context()
    requestID := c.GetString("request_id")
    
    var req dto.IngestRequest
    if err := c.ShouldBindJSON(&req); err != nil {
        h.logger.Error("validation failed", 
            "request_id", requestID,
            "error", err)
        c.JSON(400, errorResponse(err))
        return
    }
    
    result, err := h.service.Process(ctx, &req)
    if err != nil {
        c.JSON(500, errorResponse(err))
        return
    }
    
    c.JSON(200, successResponse(result))
}

6.2 Transformer

Responsibilities:

Map data between layers
Type conversion
Normalization (trim, lowercase, etc)
Timestamp handling

Transformations:

Request DTO → Domain Model
Domain Model → Entity (for DB)
Entity → Response DTO

DO ✅

- Map Request → Domain
- Convert types
- Format dates/timestamps
- Validate data format
- Handle null values

DON'T ❌

- Business rules/logic
- Database access
- HTTP operations

Example:

func (t *Transformer) RequestToDomain(req *dto.IngestRequest) (*domain.Order, error) {
    order := &domain.Order{
        OrderID:    req.OrderID,
        CustomerID: req.CustomerID,
        Amount:     req.Amount,
        CreatedAt:  time.Now(),
    }
    
    // Validation
    if err := order.Validate(); err != nil {
        return nil, fmt.Errorf("transform: %w", err)
    }
    
    return order, nil
}

func (t *Transformer) DomainToEntity(domain *domain.Order) *entity.Order {
    return &entity.Order{
        OrderID:    domain.OrderID,
        CustomerID: domain.CustomerID,
        Amount:     domain.Amount,
        CreatedAt:  domain.CreatedAt,
    }
}

6.3 Service (Business Logic)

Responsibilities:

All business rules
Batching strategy
Deduplication logic
Transaction control
Retry logic
Metrics calculation

DO ✅

- Implement business rules
- Control transactions
- Call repository methods
- Validate business constraints
- Handle retries
- Log business events

DON'T ❌

- JSON parsing (use Handler)
- HTTP operations
- SQL queries (use Repository)
- Data mapping (use Transformer)

Example:

func (s *Service) Process(ctx context.Context, req *dto.IngestRequest) (*dto.ProcessResult, error) {
    // Transform
    domain, err := s.transformer.RequestToDomain(req)
    if err != nil {
        return nil, fmt.Errorf("process: %w", err)
    }
    
    // Business logic: deduplication
    exists, err := s.repository.ExistsByKey(ctx, domain.OrderID)
    if err != nil {
        return nil, fmt.Errorf("check duplicate: %w", err)
    }
    if exists {
        return nil, ErrDuplicate
    }
    
    // Accumulate to batch
    s.batcher.Add(domain)
    
    // Auto-flush on size or timeout
    if s.batcher.IsFull() {
        return s.FlushBatch(ctx)
    }
    
    return &dto.ProcessResult{Status: "queued"}, nil
}

func (s *Service) FlushBatch(ctx context.Context) (*dto.ProcessResult, error) {
    batch := s.batcher.Get()
    
    // Transform to entities
    entities := make([]*entity.Order, len(batch))
    for i, d := range batch {
        entities[i] = s.transformer.DomainToEntity(d)
    }
    
    // Insert with retry
    rows, err := s.retrier.Do(ctx, func() (int64, error) {
        return s.repository.InsertBatch(ctx, entities)
    })
    if err != nil {
        return nil, fmt.Errorf("flush batch: %w", err)
    }
    
    // Metrics
    s.metrics.RecordBatchInsert(len(batch), time.Since(start))
    
    s.batcher.Clear()
    return &dto.ProcessResult{
        Status:        "success",
        RowsInserted:  rows,
    }, nil
}

6.4 Repository (Data Access)

Responsibilities:

CRUD operations
Query execution
Batch operations
Connection management
Transaction handling (initiated by Service)

DO ✅

- CRUD operations
- Execute queries
- Manage batch inserts
- Use prepared statements
- Log slow queries
- Handle connection errors

DON'T ❌

- Business logic
- Validation (beyond schema)
- Response mapping
- HTTP operations

Example:

func (r *Repository) InsertBatch(ctx context.Context, entities []*entity.Order) (int64, error) {
    const batchSize = 5000
    
    tx, err := r.pool.Begin(ctx)
    if err != nil {
        return 0, fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback(ctx)
    
    stmt := `INSERT INTO orders (order_id, customer_id, amount, created_at)
             VALUES ($1, $2, $3, $4)`
    
    batch := &pgx.Batch{}
    for _, e := range entities {
        batch.Queue(stmt, e.OrderID, e.CustomerID, e.Amount, e.CreatedAt)
    }
    
    results := tx.SendBatch(ctx, batch)
    defer results.Close()
    
    var rowsInserted int64
    for i := 0; i < len(entities); i++ {
        tag, err := results.Exec()
        if err != nil {
            return 0, fmt.Errorf("exec batch: %w", err)
        }
        rowsInserted += tag.RowsAffected()
    }
    
    if err := tx.Commit(ctx); err != nil {
        return 0, fmt.Errorf("commit: %w", err)
    }
    
    return rowsInserted, nil
}

func (r *Repository) FindByFilters(ctx context.Context, filters *repository.Filter) ([]*entity.Order, error) {
    ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
    defer cancel()
    
    query := `SELECT order_id, customer_id, amount, created_at 
              FROM orders 
              WHERE created_at >= $1 AND created_at <= $2
              LIMIT $3`
    
    rows, err := r.pool.Query(ctx, query, filters.StartDate, filters.EndDate, filters.Limit)
    if err != nil {
        return nil, fmt.Errorf("query: %w", err)
    }
    defer rows.Close()
    
    var orders []*entity.Order
    for rows.Next() {
        var o entity.Order
        if err := rows.Scan(&o.OrderID, &o.CustomerID, &o.Amount, &o.CreatedAt); err != nil {
            return nil, fmt.Errorf("scan: %w", err)
        }
        orders = append(orders, &o)
    }
    
    return orders, rows.Err()
}

7. Dependency Rules

Allowed:

Handler
  ↓
Transformer
  ↓
Service
  ↓
Repository
  ↓
PostgreSQL

FORBIDDEN (will be caught in code review):

Handler → Repository (bypass Service)
Repository → Service (circular)
Any layer → Handler (upward dependency)
Handler → Database (direct SQL)

8. Data Warehouse Specific Rules

8.1 Batching

// Service layer manages batching
batcher := ingestion.NewBatcher(config.BatchSize)

for _, record := range records {
    batcher.Add(record)
    
    if batcher.IsFull() || batcher.IsExpired() {
        err := s.flush(ctx)
    }
}

Configuration (per service in config.go):

type BatchConfig struct {
    Size        int           // 5000 rows
    TimeoutSec  int           // 30 seconds
    MaxRetries  int           // 3
    RetryDelay  time.Duration // exponential backoff
}

8.2 Deduplication

Natural Key:

// In domain/order.go
func (o *Order) NaturalKey() string {
    return fmt.Sprintf("%s_%s_%d", o.OrderID, o.CustomerID, o.CreatedAt.Unix())
}

In Service:

exists, err := s.repository.ExistsByKey(ctx, order.NaturalKey())
if exists {
    // Skip atau return error based on business rule
}

8.3 Kafka Integration

Handler untuk Kafka:

type KafkaHandler struct {
    service Service
    logger  Logger
}

func (h *KafkaHandler) Handle(ctx context.Context, msg *kafka.Message) error {
    var dto dto.IngestRequest
    if err := json.Unmarshal(msg.Value, &dto); err != nil {
        h.logger.Error("unmarshal failed", "error", err)
        return err // will be retried
    }
    
    _, err := h.service.Process(ctx, &dto)
    return err
}

Configuration (in config.go):

type KafkaConfig struct {
    Brokers         []string
    Topic           string
    ConsumerGroup   string
    MaxConcurrency  int
}

8.4 Redis Caching (Query API)

In Service:

func (s *QueryService) Execute(ctx context.Context, q *repository.Query) ([]Result, error) {
    // Check cache
    cacheKey := q.CacheKey()
    if cached, err := s.cache.Get(ctx, cacheKey); err == nil {
        return cached, nil
    }
    
    // Query database
    results, err := s.repository.Query(ctx, q)
    if err != nil {
        return nil, err
    }
    
    // Store in cache (TTL: 5-60 minutes)
    s.cache.Set(ctx, cacheKey, results, 5*time.Minute)
    
    return results, nil
}

Cache Strategy:

Hot queries: 60 minutes
Moderate queries: 15 minutes
Cold queries: 5 minutes
Invalidate on data update

9. Naming Conventions

Package

github.com/yourorg/go-dw-platform/framework
github.com/yourorg/go-dw-platform/services/ingestion-orders

Files

Lowercase, underscore separated
orders.go, orders_test.go, orders_integration_test.go

Types

type OrderService struct{}        // PascalCase
type IngestRequest struct{}
type ErrOrderNotFound struct{}

Functions/Methods

func (s *Service) Process() error       // PascalCase (exported)
func (s *Service) process() error       // camelCase (private)
func (s *Service) FindByID() error
func (s *Service) CreateOrder() error

Constants

const (
    DefaultBatchSize = 5000
    MaxRetries       = 3
)

Errors

var (
    ErrOrderNotFound     = errors.New("order not found")
    ErrDuplicateOrder    = errors.New("duplicate order")
    ErrInvalidBatchSize  = errors.New("invalid batch size")
)

Error wrapping (always):

if err != nil {
    return fmt.Errorf("process order: %w", err)
}

10. Error Handling Standard

Pattern:

// Always wrap with context
if err != nil {
    return fmt.Errorf("insert batch: %w", err)
}

// Custom errors for business logic
if !order.IsValid() {
    return ErrInvalidOrder
}

// Never panic in production code
// Only panic if application cannot continue
panic("database pool initialization failed")

Error Response Format:

{
    "success": false,
    "message": "Failed to process order",
    "error_code": "DUPLICATE_ORDER",
    "request_id": "req-12345"
}

11. Logging Standard

Structured Logging (JSON):

logger.Info("order processed",
    "request_id", "req-12345",
    "user_id", "user-456",
    "order_id", "ord-789",
    "duration_ms", 125,
    "batch_size", 100,
)

Minimal Fields (required):

request_id - Trace requests
module - Service name
action - Operation (insert, query, etc)
duration_ms - Timing
error - Error message (if applicable)

Log Levels:

ERROR - Errors that need attention
WARN - Warnings (retries, slow queries)
INFO - Business events (batch inserted, dedup)
DEBUG - Detailed debugging (not in production)

NEVER log:

Passwords, tokens, API keys
PII (passwords, emails in logs)
Full database records (only IDs)
Sensitive configuration

// ❌ Wrong
logger.Info("user login", "password", "secret123")

// ✅ Correct
logger.Info("user login", "user_id", "user-123")

12. Validation Standard

Layer:

Handler - Format validation (required fields, JSON format)
Transformer - Type validation (dates, numbers)
Service - Business validation (business rules)

Example:

// Handler - format
if req.OrderID == "" {
    return ErrMissingOrderID
}

// Transformer - type
orderID, err := strconv.ParseInt(req.OrderID, 10, 64)
if err != nil {
    return fmt.Errorf("invalid order ID: %w", err)
}

// Service - business rule
if !s.IsValidCustomer(ctx, order.CustomerID) {
    return ErrInvalidCustomer
}

13. Transaction Standard

Rule: Transactions ONLY in Service layer.

Repository accepts context, Service creates tx:

// ❌ Wrong - Repository creates tx
func (r *Repository) Process(ctx context.Context) error {
    tx, _ := r.pool.Begin(ctx)
    // ...
}

// ✅ Correct - Service creates tx
func (s *Service) Process(ctx context.Context) error {
    tx, err := s.pool.Begin(ctx)
    if err != nil {
        return err
    }
    defer tx.Rollback(ctx)
    
    // Call repository with tx context
    err = s.repository.Insert(ctx, entity)
    if err != nil {
        return err
    }
    
    return tx.Commit(ctx)
}

14. Database Rules

Connection Pooling (in framework/db/pool.go):

type Config struct {
    MaxConns        int32         // 25 (tune per load)
    MinConns        int32         // 5
    MaxConnLifetime time.Duration // 15 minutes
    MaxConnIdleTime time.Duration // 5 minutes
}

All SQL in Repository:

// ❌ Wrong - SQL di Handler
func (h *Handler) GetOrder(c *gin.Context) {
    rows, _ := db.Query("SELECT * FROM orders")
}

// ✅ Correct - SQL di Repository
func (r *Repository) FindByID(ctx context.Context, id string) (*Order, error) {
    row := r.pool.QueryRow(ctx, "SELECT * FROM orders WHERE id = $1", id)
    // ...
}

Prepared Statements (always for batch):

stmt := `INSERT INTO orders (id, customer_id, amount) VALUES ($1, $2, $3)`
batch := &pgx.Batch{}
for _, order := range orders {
    batch.Queue(stmt, order.ID, order.CustomerID, order.Amount)
}

Query Timeout:

ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()

rows, err := r.pool.Query(ctx, query, args...)

15. API Response Format

Success:

{
    "success": true,
    "message": "Data retrieved successfully",
    "data": {
        "orders": [...],
        "total": 100,
        "page": 1
    },
    "request_id": "req-12345"
}

Error:

{
    "success": false,
    "message": "Validation failed",
    "error_code": "VALIDATION_ERROR",
    "details": {
        "field": "order_id",
        "error": "required"
    },
    "request_id": "req-12345"
}

Pagination:

{
    "data": [...],
    "pagination": {
        "page": 1,
        "limit": 10,
        "total": 150,
        "pages": 15
    }
}

16. Security Rules

MANDATORY:

Authentication - JWT verification

func AuthMiddleware() gin.HandlerFunc {
    return func(c *gin.Context) {
        token := c.GetHeader("Authorization")
        if token == "" {
            c.JSON(401, errorResponse("missing token"))
            c.Abort()
            return
        }
        // Verify token
    }
}

Authorization - Role-based access

func RoleMiddleware(required []string) gin.HandlerFunc {
    return func(c *gin.Context) {
        role := c.GetString("user_role")
        // Check if role in required
    }
}

Input Validation - Prevent injection

// Always use parameterized queries
stmt := "SELECT * FROM users WHERE id = $1"  // ✅
stmt := "SELECT * FROM users WHERE id = " + id  // ❌

Secret Management - Use Vault

type Config struct {
    DBPassword string `vault:"db.password"` // Loaded from Vault
}

Audit Trail - Log sensitive operations

logger.Info("data accessed",
    "user_id", userID,
    "action", "export",
    "records", 1000,
    "timestamp", time.Now(),
)

NEVER:

Hardcode credentials
Log passwords/tokens
Store plaintext secrets
Skip validation

17. Performance Rules

Pagination (always)

const DefaultLimit = 100
const MaxLimit = 10000

func (h *Handler) List(c *gin.Context) {
    limit := c.DefaultQuery("limit", "100")
    page := c.DefaultQuery("page", "1")
    
    // Validate
    if limit > MaxLimit {
        limit = MaxLimit
    }
}

Context & Timeout

// Always use context with timeout
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()

rows, err := r.pool.Query(ctx, query)

Batch Processing

// For large datasets, use batching
const BatchSize = 5000

for i := 0; i < len(records); i += BatchSize {
    batch := records[i:min(i+BatchSize, len(records))]
    r.InsertBatch(ctx, batch)
}

Query Optimization

// ✅ Good - With index
SELECT * FROM orders WHERE customer_id = $1 AND created_at > $2 LIMIT 1000

// ❌ Bad - Full table scan
SELECT * FROM orders WHERE amount * quantity > 1000

// Index needed on (customer_id, created_at)

Avoid N+1

// ❌ Wrong
for _, order := range orders {
    customer := r.GetCustomer(order.CustomerID)  // N queries
}

// ✅ Correct
customerIDs := extract(orders, "CustomerID")
customers := r.GetCustomersBatch(customerIDs)   // 1 query

Connection Pooling

// Pool configured in framework
pool, _ := pgxpool.New(ctx, dsn)
defer pool.Close()

// Connection automatically managed
row := pool.QueryRow(ctx, query)

18. Testing Standard

Minimum Coverage: 80%

Layers to Test:

✅ Service (business logic)
✅ Repository (database access)
✅ Transformer (data mapping)
⚠️ Handler (integration test, if needed)

Testing Framework: testify

Service Test

func TestOrderService_Process(t *testing.T) {
    // Arrange
    mockRepo := &MockRepository{}
    service := NewService(mockRepo)
    
    req := &dto.IngestRequest{OrderID: "ord-123"}
    
    // Act
    result, err := service.Process(context.Background(), req)
    
    // Assert
    assert.NoError(t, err)
    assert.Equal(t, "success", result.Status)
    assert.Equal(t, 1, mockRepo.InsertCallCount)
}

Repository Test

func TestOrderRepository_InsertBatch(t *testing.T) {
    db := setupTestDB(t)
    defer db.Close()
    
    repo := NewRepository(db)
    
    entities := []*entity.Order{
        {OrderID: "1", CustomerID: "cust-1", Amount: 100},
        {OrderID: "2", CustomerID: "cust-2", Amount: 200},
    }
    
    rows, err := repo.InsertBatch(context.Background(), entities)
    
    assert.NoError(t, err)
    assert.Equal(t, int64(2), rows)
}

Integration Test

func TestOrderHandler_Ingest(t *testing.T) {
    router := setupTestRouter()
    db := setupTestDB(t)
    
    payload := `{"order_id": "123", "customer_id": "cust-1", "amount": 100}`
    
    req, _ := http.NewRequest("POST", "/api/ingest", strings.NewReader(payload))
    req.Header.Set("Content-Type", "application/json")
    
    resp := httptest.NewRecorder()
    router.ServeHTTP(resp, req)
    
    assert.Equal(t, 200, resp.Code)
}

Test File Naming:

orders.go          → orders_test.go
orders_handler.go  → orders_handler_test.go

Run Tests:

go test ./... -cover              # All tests
go test ./service -cover          # Specific package
go test -run TestOrderService -v  # Specific test

19. Observability

19.1 Metrics (Prometheus)

Framework provides:

metrics.RecordHTTPRequest(method, endpoint, statusCode, duration)
metrics.RecordDatabaseQuery(operation, duration)
metrics.RecordBatchInsert(rowCount, duration)
metrics.RecordCacheHit(key, duration)

In your code:

start := time.Now()
rows, err := r.InsertBatch(ctx, entities)
metrics.RecordBatchInsert(len(entities), time.Since(start))

Prometheus Queries:

rate(http_request_duration_seconds[5m])
histogram_quantile(0.95, rate(db_query_duration_seconds[5m]))

19.2 Logging (ELK)

Framework provides structured logger:

logger.Info("batch inserted",
    "module", "ingestion-orders",
    "batch_size", 5000,
    "duration_ms", 250,
)

Kibana can parse JSON:

{
  "timestamp": "2026-07-24T10:30:45Z",
  "level": "INFO",
  "message": "batch inserted",
  "module": "ingestion-orders",
  "batch_size": 5000,
  "duration_ms": 250,
  "request_id": "req-12345"
}

19.3 Tracing (Optional - Jaeger)

Add trace context:

ctx, span := tracer.Start(ctx, "ProcessOrder")
defer span.End()

span.SetAttributes(
    attribute.String("order_id", order.ID),
    attribute.Int64("customer_id", order.CustomerID),
)

20. CI/CD & Deployment

GitHub Actions

on: [push, pull_request]

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: actions/setup-go@v4
        with:
          go-version: '1.21'
      - run: go test ./... -cover
      - run: golangci-lint run

Docker Build

FROM golang:1.21-alpine AS builder
WORKDIR /app
COPY . .
RUN go build -o service .

FROM alpine:latest
COPY --from=builder /app/service .
ENTRYPOINT ["./service"]

Deployment Checklist

All tests pass
Code review approved
Linting passes
No vulnerabilities (gosec)
Database migrations ready
Configuration set
Monitoring configured
Rollback plan defined

21. Common Patterns

Pattern: Graceful Shutdown

func main() {
    server := gin.Default()
    
    go func() {
        sigChan := make(chan os.Signal, 1)
        signal.Notify(sigChan, syscall.SIGINT, syscall.SIGTERM)
        <-sigChan
        
        ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
        defer cancel()
        
        server.Shutdown(ctx)
    }()
    
    server.Run(":8080")
}

Pattern: Health Check

func (h *Handler) Health(c *gin.Context) {
    health := map[string]interface{}{
        "status": "healthy",
        "database": h.db.Ping(c.Request.Context()),
        "cache": h.cache.Ping(c.Request.Context()),
    }
    c.JSON(200, health)
}

Pattern: Middleware Chain

router := gin.New()
router.Use(middleware.Logger())
router.Use(middleware.Recovery())
router.Use(middleware.Auth())
router.Use(middleware.Tracing())

router.POST("/api/ingest", handler.Ingest)

22. Code Review Checklist

Before submitting PR:

Follows layer architecture (Handler → Transformer → Service → Repository)
No business logic in Handler
Mapping in Transformer
Business logic in Service
SQL only in Repository
All tests pass (≥80% coverage)
Logging complete (request_id, action, duration)
Error handling correct (wrapped)
Context used in all I/O
No hardcoded values
Passes linting (golangci-lint run)
No SQL injection vulnerabilities
No secrets in code
Performance acceptable (EXPLAIN ANALYZE for queries)
Metrics added
README updated (if new feature)

23. Definition of Done

Service is production-ready when:

✅ All unit tests pass (≥80% coverage)
✅ Code review approved
✅ Linting passes (golangci-lint)
✅ No security vulnerabilities (gosec)
✅ Database migrations tested
✅ Configuration documented (.env.example)
✅ API documented (OpenAPI/Swagger)
✅ Logging configured (ELK integration)
✅ Metrics configured (Prometheus)
✅ Performance benchmarked
✅ Deployment tested (Docker build succeeds)
✅ Rollback plan documented
✅ README complete

24. Project Structure Script

New Service Generation:

./scripts/new-service.sh ingestion-payments

# Generated:
# services/ingestion-payments/
# ├── go.mod
# ├── main.go
# ├── config.go
# ├── handler/
# ├── service/
# ├── repository/
# ├── transformer/
# ├── domain/
# ├── entity/
# ├── dto/
# ├── migrations/
# ├── tests/
# └── README.md

25. References

Go Best Practices: https://golang.org/doc/effective_go
PostgreSQL: https://www.postgresql.org/docs/
pgx Documentation: https://github.com/jackc/pgx
Gin Web Framework: https://gin-gonic.com/
Prometheus: https://prometheus.io/docs/
Clean Architecture: Robert C. Martin
Domain-Driven Design: Eric Evans
Microservices Patterns: Chris Richardson

26. Change History

Version	Date	Changes
1.0	2026-07-24	Initial Version - Go DW Framework

Last Updated: 2026-07-24 Owner: Engineering Team Next Review: 2026-10-24

30 KiB Raw Blame History

Go Data Warehouse Framework - SKILL.md

1. Purpose

2. Stack Overview

3. Business Capabilities

4. Folder Structure

Project Level

Service Level (Ingestion Example)

5. Architecture Layers

6. Layer Responsibilities

6.1 Handler (HTTP/Kafka)

6.2 Transformer

6.3 Service (Business Logic)

6.4 Repository (Data Access)

7. Dependency Rules

8. Data Warehouse Specific Rules

8.1 Batching

8.2 Deduplication

8.3 Kafka Integration

8.4 Redis Caching (Query API)

9. Naming Conventions

Package

Files

Types

Functions/Methods

Constants

Errors

10. Error Handling Standard

11. Logging Standard

12. Validation Standard

13. Transaction Standard

14. Database Rules

15. API Response Format

16. Security Rules

17. Performance Rules

Pagination (always)

Context & Timeout

Batch Processing

Query Optimization

Avoid N+1

Connection Pooling

18. Testing Standard

Service Test

Repository Test

Integration Test

19. Observability

19.1 Metrics (Prometheus)

19.2 Logging (ELK)

19.3 Tracing (Optional - Jaeger)

20. CI/CD & Deployment

GitHub Actions

Docker Build

Deployment Checklist

21. Common Patterns

Pattern: Graceful Shutdown

Pattern: Health Check

Pattern: Middleware Chain

22. Code Review Checklist

23. Definition of Done

24. Project Structure Script

25. References

26. Change History

30 KiB

Raw Blame History