Research is Different

This is how lab seems sometimes:

Person1: "To make your system go faster, you should use method Z, I believe person 2 has written an implementation of it"

Person2: "I am working on an implementation of X, but it isn't finished yet. Person3 found a bug in method Y that I need to use, so you should ask them"

Person3: "Yeah, method Y uses method Z, but method Z gives different answers depending on if you use CPU or GPU, and neither is correct. I don't actually know anything about how method Z works"

Person1: "Well, you could use Method X instead, person4 worked on it"

Person4: "Yeah... it doesn't work yet"

Person1: "Maybe Method W?"

Me: "It's written in a completely different language... :(((("

Life is fun though. I am enjoying the detective aspect of searching through code and  documentation filled with "TODO"s and "Writeme"s. At school, we use established programming languages for which we can easily find complete documention online and answers to questions on Stack Overflow. At my lab, part of the lab’s contribution to science is building a system to easily set up Machine Learning experiments. This system is always being added on to, and each lab member has their own private version of the system, so nobody actually knows how everything works. If someone writes a useful add-on, they try to get it merged into the official version. However, because we are all human and communication is hard,  multiple people end up trying to write similar things independently, and a number of features don’t get finished or don’t work. Life on the edge, man.

Research is hard

Before I start on actual content: I changed my blog template! I had had the original one since approximately January 12, 2008. That's 6 years ago. This one is kind of boring, but the gray background won't mess up pictures too much.

But anyways, research is hard. I'm trying to making a neural net that will read in a bunch of series of words (called ngrams) and try to use the middle word to predict the words around it. You might say, "that doesn't sound very useful, why would you want to predict words around a word?" and I totally agree; it isn't very useful. What is useful, is that to predict the surrounding words, the system first projects the middle word into a vector in a continuous space (at first to a random spot). Then, when it predicts wrong, it adjusts the vector a little bit in the direction that would have predicted the right word. After many iterations, these vectors end up containing information about the word that is useful to the system when predicting nearby words. Since similar words tend to be surrounded by the same words, similar words get clumped together. Once we have these projections, we can do cool stuff like translate phrases from English to French semantically rather than just using the most common words.

So I write the code for this neural node, run it on a small piece of data, and it seems to learn (the number it gets wrong while training goes down). Cool beans. I get the actual data and holy moley, there are 350,000,000 samples inside. My code can run about ~225 samples per second, so that would take about 3 weeks... Right now I am trying to run 10,000,000 samples, and it is not going so well. Not only is it taking longer than the predicted 12 hours, but after going down at first, the cost is now wavering back and forth very far from 0. This probably means my learning rate is too high, but the cost does seem to be decreasing slowly if I look at it from afar. 

To make us all feel better, here are some pictures from my weekend wandering:

A bakery with no sugar and white flour. It has integral bread, but that just sounds derivative.

Don't cook, just eat!

This picture isn't funny, but that bread looks delicious.

I too drink my mushrooms as tea.

I'm in Canada!

After many months of stressing about whether I was going to be allowed to work in Canada, I successfully got my work permit and started my internship at the beginning of this week. Yay! I'm doing computer science research in Montreal, which is pretty darn awesome. My coworkers are a fun bunch: to celebrate someone's Doctorate, one student got him a machete, and we hiked up the mountain to play real life fruit ninja! Turns out coconuts are really hard to slice, but melons work surprisingly well.

Montreal is an interesting city. In the US, I generally know what language to use when talking to people: if they are my parent's French friends, I use French, otherwise I use English. In Montreal, most people speak both, but would be more comfortable using one language over the other. It's a confusing guessing game.

Nonetheless, I have definitely been enjoying life here. I love walking around my neighborhood and checking out the little grocery stores and cafes. I seem to be living at the intersection of a Greek neighborhood and a Hasidic neighborhood, so there are a lot of Greek restaurants and kosher grocery stores. Even the generic grocery store has interesting greek cheeses that I had never tried before (including Mizithra, which my Greek friend keeps talking about). This morning I also saw freshly grilled Souvlaki being sold outside one of the Greek restaurants! I'll have to try that next week (or perhaps tomorrow if it is still there).

I've been lazy about bringing my camera places, but I promise I'll be better next week. In the meantime, here are some things I've found with my phone camera:

There is a fruit store near my lab that has some nice looking and rather inexpensive fruit. It's "La Sucrerie" and it sells fruit, pastries, and maple syrup. It also does not seem to exist on the internet (...?) Unfortunately, it is really far from my apartment, so I won't be going shopping there very often.

There is a used bookstore about 5m from my apartment building! It does not seem to have much Sci-Fi/Fantasy, but as I only brought 2 books with me, I'll probably end up getting some books from there :)

As a final treat, here is a video of the post-doc I'm working with slicing through a cantaloupe :)