Blogs

This week I read papers that blows my mind. I write this blog to share my understanding about Lottery Ticket Hypothesis. As always, suggestions and corrections are always welcome 😃 Drawing Winning Tickets Model pruning is a commonly used practice to reduce the number of parameters in a model while still maintaining performance. The question is, if a smaller model with fewer parameters can achieve good evaluation results, why do we need to train the entire model?...

Last Sunday, I was waiting for my order at the food court with my family. That second I decided to skim the FNet paper to kill time. In this article, I will explain everything I know about FNet: Mixing Tokens with Fourier Transforms. Please correct me if you find something incorrect 😀 Introduction In the FNet paper, the author uses the fourier transform as a sublayer that replaces the multi head attention sublayer in transformer....

I always forget things I’ve learned; It doesn’t seem right. I wrote this blog mainly to refresh my own knowledge about transformer. I hope someone who reads this blog will find it useful. If something isn’t correct, please let me know 😀 Architecture The transformer architecture from the famous “Attention Is All You Need” paper looks like this: Basically, it has two main parts: the encoder block (left) and the decoder block (right)....

If you familiar with linear regression, we need to define the slope and intercept to make the regression. What if we want to fit our linear regression model to numerical data? How do we get the best combination of slope and intercept with the least errors among the data? One of the answers is through gradient descent. Gradient descent may be one of the most important concepts that machine learning practitioners or enthusiasts must understand....

Introduction In neural networks, we use gradients to examine and tweak our weight so it can converge and hopefully have a good general performance. Neural networks were invented around 1944 by Warren McCullough and Walter Pitts. But why is it just trending recently? In the early 2000s, deep neural networks were considered as abandoned technology because they tend to have unstable gradients. This problem is known as vanishing gradient or the opposite, exploding gradient problem....