🧩 Philosophy 2h ago · Amy_

Untie Squared ReLU variant

Less Wrong

Source ↗ 👁 1 💬 0

1 IntroMy collaborator Michael Bukatin came up with an idea: in the hidden projection of MLP, instead of using ReLU, he would duplicate one branch, and initialize two matrices on the two branches separately, and then multiply the two element-wise, then project back as output in the feedforward layer.As a quick recap, a standard ReLU FFN is:

mjx-math {
display: inline-block;
text-align: left;
line-height: 0;
text-indent: 0;
font-style: normal;
font-weight: normal;
font-size: 100%;