Global Effects in Netflix Prize

忙活了几天，终于把那个global effects给搞定了，不容易啊，嘿嘿

Today, I have implement alogrithm of global effects in Netflix. My method is similar to Koren's method. However, I think, his paper is not very clearly. I have tested many methods and at last get same result as him. Therefore, I will give detail description of my algorithms.

Related Papers:
The BigChaos Solution to the Netflix Prize 2008
Scalable Collaborative Filtering with Jointly Derived Neighborhood Interpolation Weights

0. global average

b0(u,i) = global_mean

1. movie effect

x(u,i) = average(i) = sum_u {r(u,i) - b0(u,i)} / n(i)

we calculate average rating of movies. In order to avoid, we also shrink average(i) by:

average(i) = average(i) * n(i) / (n(i) + alpha)

In my implement, we choose alpha = 22 and get RMSE = 1.0527

b1(u,i) = b0(u,i) + p(u) x(u,i)

Here, p(u) is estimate by Koren's method.

p(u) = sum_i (b1(u,i) - b0(u,i)) * x(u,i) / sum_i x(u,i) * x(u,i)

and shrink by:

p(u) = p(u) * n(u) / (n(u) + alpha)
2. user effect

x(u,i) = average(u) = sum_u {r(u,i) - b1(u,i)} / n(u)

we also shrink x(u,i), choose alpha = 7.5 and get RMSE = 0.9840

Then, we get predictor b2:
b2(u,i) = b1(u,i) + p(u) x(u,i)

3. user X time(user)

Let t(u) be the first time when user u give ratings. t(u,i) is the time when u assign rating to i. Then, the effect x(u,i) is:

x(u,i) = sqrt(t(u,i) - t(u)) ; t(u,i) - t(u) > beta
x(u,i) = 0 ; otherwise

In our algorithm, we choose beta = 18 and shrink parameter alpha = 160. We get RMSE = 0.9801.

Then, we get predictor b3:
b3(u,i) = b2(u,i) + p(u) x(u,i)