liger() — pytorch Function Reference

Architecture documentation for the liger() function in kernels.py from the pytorch codebase.

Function python DynamoBenchmarks Configuration calls 1

Entity Profile

DynamoBenchmarks→ Configuration→ liger() — pytorch Function Reference

Dependency Diagram

graph TD
  ac72338f_8a94_8d37_87cb_d74c1927bca1["liger()"]
  94e8675f_d7e9_407b_4e04_6f5632247962["compute_mean_rstd()"]
  ac72338f_8a94_8d37_87cb_d74c1927bca1 -->|calls| 94e8675f_d7e9_407b_4e04_6f5632247962
  style ac72338f_8a94_8d37_87cb_d74c1927bca1 fill:#6366f1,stroke:#818cf8,color:#fff

Relationship Graph

Source Code

benchmarks/dynamo/genai_layers/kernels.py lines 708–725

    def liger(self, args, kwargs) -> Any:
        """
        Call layer_norm_backward directly rather than calling
        liger_kernel.transformers.layer_norm.LigerLayerNorm and
        torch.autograd.grad.

        The latter fashion saves mean/rstd in x.dtype which can fail
        accuracy test. We call layer_norm_backward with fp32 mean and
        rstd.
        """
        from liger_kernel.ops.layer_norm import layer_norm_backward

        x, w, dy = args
        eps = 1e-6
        mean, rstd = self.compute_mean_rstd(x, eps)
        M, N = x.shape

        return lambda: layer_norm_backward(dy, x, w, None, mean, rstd)[0:2]