mhc-algorithm

1# mHC: Manifold-Constrained Hyper-Connections
2 
3## Overview
4 
5mHC (Manifold-Constrained Hyper-Connections) stabilizes deep network training by constraining residual mixing matrices to be **doubly stochastic**. It provides:
6 
7- **Stable Training**: Lower gradient norm variance via doubly stochastic constraints
8- **Multiple Streams**: Hyper-Connections with learnable mixing across residual streams
9- **Sinkhorn Projection**: Log-space Sinkhorn-Knopp algorithm for doubly stochastic projection
10- **GPT Integration**: Pattern for wrapping attention and MLP layers
11 
12Two components:
13- **HyperConnections Module**: Core PyTorch module with H_res, H_pre, H_post matrices
14- **Sinkhorn-Knopp**: Log-space projection to doubly stochastic manifold
15 
16## Quick Reference
17 
18| Topic | Reference |
19|-------|-----------|
20| Core Concepts & Math | [Core Concepts](references/core-concepts.md) |
21| Sinkhorn Algorithm | [Sinkhorn-Knopp](references/sinkhorn-knopp.md) |
22| HyperConnections Module | [Module Implementation](references/module-implementation.md) |
23| GPT Integration | [GPT Integration](references/gpt-integration.md) |
24| Common Pitfalls | [Pitfalls](references/pitfalls.md) |
25 
26## Installation
27 
28```python
29# Required packages
30pip install torch einops numpy
31```
32 
33## Minimal Example
34 
35```python
36import torch
37import torch.nn as nn
38from einops import rearrange, einsum
39 
40def sinkhorn_knopp(logits, num_iters=20, tau=0.05):
41    log_alpha = logits / tau
42    for _ in range(num_iters):
43…