Thursday, March 29, 2012

Some updates...

So I wasn't super productive in the last couple of days, but I did manage to watch some lectures and do a little work.

I did a bit of re-thinking of my schedule. During some of my GRE study where I encountered some combinatorics and statistics, I decided that perhaps taking some classes that addressed this would be a good use of time, so I entered a bit earlier than I thought into the Discrete Math course from ArsDigita that I was planning to take. How much I treat these as equal to my other classes remains to be seen, but at the very least it looks like a fascinating course, with a wide range of subjects covered.

I also briefly thought that perhaps I wasn't taking the right class to satisfy my Data Structures requirement, and I considered switching from the Stanford CS106B course I'm in now to the Berkeley 61B course, which has the benefit of being specifically titled "Data Structures." But after looking at the syllabus and seeing a fair amount of overlap with 106B, I decided to stick with that one. Besides the fact that I'm already a few lectures into 106B, it has a few benefits as well. For one, it seems to be a slightly more complete course, with problem sets, libraries, handouts etc all readily available. For another, it's good to learn C++, even if it's somewhat of a PITA compared to Python (that feeling has diminished slightly since I've gotten going, but it still is more cumbersome). The 61B course teaches Java, which no doubt is worth knowing as well, but while I can be quite confident that it's worthwhile for someone who only knows Python to also learn C++, I'm not so sure the same thing holds with Java. I don't really know that much about Java, but from what I do know it seems to be quite similar to python, both being high-level object-oriented programming languages. On the other hand, there are some courses I plan to take which involve Java so I'm sure I'll pick it up sooner or later. I hear it's syntactically similar to C++ anyway.

I watched the fourth lecture of 106B and did another problem (problem 2) of the first set. Here's the code for it:
106B Problem set 1, problem 2: http://pastebin.com/nwpTu6yW

Like I mentioned I've begun to be a bit more comfortable with C++. I still forget to initialize variables and put in the semicolons at the end of the lines, but all in all it really isn't shaping up to be that different than python deeper than the surface. There are a few exceptions - for example, python has really handy data types like lists, tuples, dictionaries etc which are very simple to understand and manipulate, and are also incredibly powerful. C++'s data types seem to be a bit more restricted, or perhaps just a bit more obcure (third option: I just don't understand them well enough yet). The new data type at work in this code is a struct, which is somewhat like a very anemic class object that only contains variables and not functions.

I watched lecture 3 and partway through lecture 4 of the Linear Algebra class. I'm working my way through the first problem set and about 2/3 finished. I'm finding it difficult to get an intuitive grasp on how matrices work. The lecturer obviously has no trouble looking at two fairly complicated matrices and seeing immediately how multiplying one into the other will result. I'm not at that level as of yet. But I've been able to figure out some basic concepts of my own which are helping me understand it.

Rules for LR = F, where L, R and F are two n X n matrices:

  1. The kth row of F depends on all of R but only the kth row of L.
  2. Similarly, the ith column of F depends on all of L but only the ith column of R .
  3. Think of L as a bunch of n-vectors laid out horizontally. Think of R as a bunch of n-vectors laid out vertically. Matrix multiplication is a bunch of dot products between the vectors in L and those in R. 
  4. In order for a dot product to happen, two vectors must have the same number of elements (namely n), which is why the horizontal length of L must match the vertical length of R. The other dimensions are independent.
It's still a work in progress, but this kind of thinking helps me a bit.

In the Algorithms class I watched the explanation of the Master Method, which is interesting. I have to admit algorithms have yet to really spark my enthusiasm, but apparently they're the bee's knees for a lot of computer scientists, so we'll wait and see. 

I haven't visited udacity in a few days, so I'll probably go today. At some point when I've cleaned up the code, I hope to upload the web crawler code I've written. I'm mostly using the same concepts as taught in the class, but I'm using object-oriented programming, with objects like web pages, keyword dictionaries, etc. Fun stuff.

Random musings, unexplored ideas:
I'm curious if there's a fast algorithm for finding prime numbers.
In a physics tangent, it occurs to me that the reason objects under a damping or friction force stop, instead of slow down asymptotically like models often show, must have to do with some sort of energy well - if the kinetic or other relevant energy decreases below the threshold of a given energy well, the object won't proceed. Consider a ball rolling over a rough surface. Initially its speed will decrease smoothly, but at some point it won't have the kinetic energy to pass over some pit or hill on the surface it's traversing, and its speed will decrease from some non-zero amount directly to zero. How non-smooth the end of this curve is will depend not only on initial speed, but also on the relationship between the geometries of the surface and the ball.

Tuesday, March 27, 2012

My lesson plan

I shared what I have done up until now; at this point maybe it would be nice to share my tentative plan of what I'm going to study over the next year.

With the rise of free online lectures provided by the likes of KhanAcademy, TheNewBoston and of course a variety of universities like MIT, Stanford, Berkeley and IIT Kharagpur, there really is an embarrassment of riches to be found for a self-learner, and if anything a problem is that it can be simply too much to choose from. Right now I find myself on vacation from work for a few weeks, and I still don't feel like there's nearly enough time in the day to cover all of the courses I'm taking now, let alone all that I want to take. But as is often the case, the first step to success is a plan; even if you don't know the best way, simply having a plan in mind is said to be helpful. So here we go.

Since I'm looking to enter graduate school, it would seem natural to want to cover the content of an undergraduate computer science education. Unfortunately, there is a wide degree of variability in such majors, both between schools and within the same school, based on a student's goals. Listing every course taken by computer science undergrads, let alone grad students, would be futile. But at the absolute basic level, they seem to share some fundamentals:

Computer Science topics:

Introduction to Computer Science
Data Structures and Algorithms
Computer Organization
Systems Programming

Mathematical topics:
Discrete Math
Linear Algebra
Probability and Statistics


From what I've seen looking at various department websites, these seem to make a fairly solid base. I start from here and then add on what I find particularly useful/relevant to me. The field I'm most interested in is artificial intelligence, computer creativity and computer vision. Basically I want to design the brains of a creative robot. So, things that are most relevant to me are things like artificial intelligence, machine learning, differential equations, digital signal processing, natural language processing, and the like. There are also a few I've starred (*) to indicate things that perhaps might not be essential, but that I'm just interested in.

So let's expand our list. In addition to what we had before, let's add some. Note that with some of these, the distinction between math and CS is a bit blurry.

Computer Science topics:

Introduction to Computer Science
Data Structures and Algorithms
Computer Organization
Systems Programming

Artificial intelligence
Machine learning
Natural Language Processing

Mathematics:

Discrete Math
Linear Algebra
Probability and Statistics

Differential Equations
Digital Signal Processing

OK, we now have quite a lot on our plate. But this still isn't everything that I want to cover - I'm also interested in program design, smartphone and web application development, computer music and algorithmic composition, and others, so we'll just have to find a way to accommodate as many of those things as possible. So, let's turn our course outline into a plan of online courses to take. We'll get to ordering and scheduling later. For now I just want to know that all of this stuff can be covered with existing online lectures. Here we go:


Computer Science topics:

Introduction to Computer Science
I've already taken two classes in this vein, the 6.00 course from OCW and the CS101 course from Udacity. With such a broad topic, practically anything that adds fundamental programming knowledge helps, so after this point I won't really worry too much about satisfying this requirement specifically.
Data Structures and Algorithms
I'm taking a really good-seeming Design and Analysis of Algorithms class from the Stanford Coursera website. It doesn't seem to cover data structures, but the Stanford Programming Abstractions class I'm taking, I believe, does address that. If not, then we might have to find something else to cover DS, but for now we'll consider this base being covered.
Computer Organization
I admit, I'm not to excited about this class. The inner workings of computers don't interest me nearly as much as what those workings can accomplish. But just for completeness, let's find a course on this. The Berkeley 61C course - Machine Structures would seem to do the trick.
Systems Programming
Here we turn again to Berkeley, as they seem to have a nice one there called Operating Systems and Systems Programming. Once again I'm not to psyched about this class, but if I want to develop for an OS, it would seem to be necessary.

Artificial intelligence
The free AI Course offered by Stanford looks nice, although I'm not sure I'll actually be able to access it. I suppose we'll cross that bridge when we come to it. Alternatively, I can take the robotic car class from Udacity.
Machine learning
Stanford's SEE site has a Machine Learning class that looks great!
Natural Language Processing
Once again, it's Stanford to the rescue with their Natural Language Processing course. As I'm looking at the course list on the SEE website, I notice that it might be best to take the natural progression of three courses beginning with Introduction to Robotics. Once again, we'll wait to cross that bridge.

Mathematics:

Discrete Math
The Berkeley one of this seems nice, and broad-ranging.
Linear Algebra
The one I'm taking right now, 18.06 of OCW, fits, but there's a further one I'd like to take if I can, which is introduction to linear dynamical systems at Stanford.
Probability and Statistics
For now, I'm looking at the Probability and Random Processes lecture series from NPTEL. But I might want to switch to another course that has accompanying homework assignments. We'll see.

Differential Equations
The 18.03 course from MIT seems ideal.
Digital Signal Processing
MIT to the rescue again, with their eponymous course.


OK, there we go! Let's see how it shakes out.

First assignment in C++

I feel like my real first assignment in the Stanford 106B course was just getting a working compiler set up. There were library issues... it was a process. It really makes one value Python. Oh, I know C++ has its advantages over Python, but on the other hand, for the kind of programming, I doubt many if any of those advantages will be relevant. But whatever.

My first assignment completed, I move on to the first "true" assignment, which was the first one of problem set 1. This asked me to write a function that would remove select bits from a string. Here's what I came up with:

http://pastebin.com/K8c6qLb7

I initially wrote it with a cin command instead of fgets. This was OK but it crashed when I input a string with whitespace in it. I asked the good folks at my hangout openstudy.com about it, and they told me to use gets, but then another said that gets should be avoided in favor of fgets. So, that's what I used. My choice of 200 characters is quite arbitrary; I feel like it's long enough to incorporate most sentences so that's what I did. Anyway, it works. Here's an example output:

http://pastebin.com/aC2WXMFa

Monday, March 26, 2012

The beginning: better late than never

Introduction:

This is a blog for me to track my progress as I try to become a computer scientist beginning at age 26. I don't know how extensive this blog will be, but I hope that I can keep it up long enough to be meaningful to me and to whomever might read it. If things turn out ideally, perhaps reading this can be a source of pride for me and inspiration for others. If not, at best it will be interesting and at worst, embarrassing.

I might as well start with an introduction. This will be long and will probably not be worth reading unless you really want to get to know me, or my story. But I suppose it will help understand where I'm coming from. The "short version" is that at age 26, I suddenly realized I wanted to be a computer scientist. The long version is below.

Background: 1985-2007

I come from an accomplished family. My father was a MIT PhD and a professor at the Booth school of business at the University of Chicago (he passed away in 1995, when I was 10). His father is a Harvard PhD and an emeritus professor. My mother was college-educated, and is currently in graduate school at age 53. My sister is in medical school. My brother is soon to enter a biology PhD program.

As for me, well on the surface I don't seem so unaccomplished. Like my siblings, After graduating from a top private high school, I attended the University of Chicago, receiving a degree in Physics with General Honors after four years. But while I was there, I didn't really apply myself, content to work in an easy and (relatively) lucrative summer job instead of finding research positions; content to sit in a major that I knew I could do, even though it didn't excite me.

Fast forward. In the five years since I graduated, my sister has piled weighty accomplishments like post-bac degrees and Fulbright scholarships onto her resume and entered medical school (a top 10 one, and she's on a full scholarship), and my brother has been accepted into a biology PhD program, with full stipend of course. But I've been floating about like a boat in an eddy.

After Graduation: 2007-2009

Having no great love for physics, doubting my own abilities thereof (despite decent grades), and no research experience to boot, I didn't think to seek a position in a physics lab, or even at some company that could use a physics major. Instead, after graduation I somehow found myself a biology research position. Coming into the job with no experience, I managed to convince myself that I would summon both the motivation and the passion to learn what I needed to know, and succeed in a difficult research field. As it turns out, I was optimistic.

I had neither great aptitude nor great love for immunology. I learned enough to do my job (more or less), but I didn't find that spark that made me stay up late reading papers and textbooks, or plan new experiments while daydreaming at lunch. When you're in research, drive is as important as ability and knowledge - more so even - and I had little ability, less knowledge and even less drive.

After initially being hired with the possibility suggested that I might enter a PhD program after two years, when the time had elapsed, it was mutually understood that I wouldn't be working there any more. Seeing the end approaching, I dithered a bit thinking about what job I should do next, and then hit upon teaching abroad. I had studied Japanese in college, and I was always seeing those internet ads calling people to teach English. Why not? I wanted to be a teacher, I wanted to become fluent in a foreign language, I was interested in Japanese, and I needed a job. So after spending perhaps all of 15 minutes googling "teach English in Japan" and the like, I had found a company and applied for a job. A month later, I had been interviewed and accepted. Then I stopped worrying; I had a job, and that was all that mattered. Future? What future? It was all so far away anyway.

Doldrums in Japan: 2009 - 1/2012

Thus the Japan chapter of my life began, a chapter I'm still in now. I've been here for a bit over two and a half years now, teaching English. The salary is nothing amazing but it's enough, and the workload is very light - too light. In fact, you often feel completely worthless, as your job consists largely of sitting in the teacher's office pretending to be working (the job is technically an "assistant teacher" job - you're not actually responsible for a class' education). Perhaps it was this useless feeling that woke me up, but I'm more inclined to believe it was just a motivating factor. See, I had been in this job for a bit under two and a half years - until about January of this year - before everything changed. Although I had kind of a joke job, I was content with that, complacent. I had everything I needed for the time being, and I always boasted of being a non-planner; whatever happens, happens.

So what happened then? Well, to begin with, I don't have many friends here. I'm somewhat of a quiet, reserved guy, and on top of that I'm in a foreign country. So my number of close friends is quite small. Really, it's one - Noboru, a Japanese guy who's a co-guitarist in a band I'm in. Noboru and I get along famously, practically buddy-comedy levels. We hang out all the time, and there were many times when he was the only thing keeping me sane in Japan. But in January of this year, Noboru suddenly sent me a text message: "I've decided to quit my job next month." A further text: "I'm going to move to Australia and learn English." I was stunned. He had always complained about his job, a high-level position at a small instrument maker, but had never really indicated he even wanted to quit. He had been there for almost 10 years, had achieved a good deal of seniority, and made a good amount of money. Why did he want to quit? He told me he felt like he wasn't accomplishing anything, that he needed to do something that was meaningful. He said he always wanted to learn English, and at close to 30, his window of opportunity was closing. Furthermore he surprised me by telling me that he envied me; he envied that I was living abroad and learning a foreign language, that I was chasing my dream.

Except, I wasn't chasing my dream. What was my dream? I didn't have one. I just existed. I went to work, got my paycheck, went home. I never thought about the future because it was too unknown, and not the least bit exciting. In all of the emotions I felt in that time, like surprise and worry and loneliness, the biggest of them was jealousy. My best friend was going to be leaving, and even though he was older than me, though he was leaving behind a lot of money and security for an uncertain future, I felt jealous. But now I was woken up. The only thing missing was me knowing what my dream was.

Discovering What I Want To Do: 2/2012

Luckily, I didn't have to look too far. I've long been a listener of science podcasts, and on one of them, I heard someone discussing some of the cutting edge technologies that suggest what is around the corner in our scientific advancement as a species. Things like nanotechnology and genetics were interesting, but one stuck out especially: artificial intelligence. Specifically, it was the discussion of a robot built by Hod Lipson at Cornell, a robot that taught itself to walk, and could deduce equations from looking at data. It was fascinating. Hearing about it, I found myself thinking about all of the questions that I had, and lamenting that I didn't have the knowledge to better understand the research and the kind of things I was interested in. And maybe it was all timing, but that's when I realized I wanted to be a computer scientist. With my friend leaving, my life was at a crossroads. Knowing that I would soon be alone, simple routine and complacency were not an option. I had to think about what I was going to do, and this was it. I wanted to study AI.

But then the $64,000 question: now what? I need to learn. I decided that I wanted to get a PhD, but what school will accept me? I have a math background in the form of calculus from my physics education, but besides general savvy and a Scheme course in high school, I had nothing of computer science education. As a high schooler, I had flirted with the notion of learning programming, but being bored with learning to write simple functions when what I wanted to do was make video games, I never went far. Now, I was 10 years behind the curve. I contacted various professors looking for research positions, but as one might expect, my anemic resume didn't impress many of them. So, I figured I would have to go back to school. I briefly looked into various computer science master's degree programs, but my realization had come just a hair too late for the application periods of most schools. It would seem that I have at least a year to wait.

Self-Education begins: 2/2012 - Present

But at the same time, that's an opportunity. I have a brain, the internet, and a lot of free time. I have a decent job and low expenses. I can study and I can save. So now, for the first time in years, I have a plan. Study. Save money. Apply to CS master's programs late this year. Enter school in fall of 2013. Do research while getting my master's. Apply to PhD programs probably late 2014. Get my PhD 4-5 years later. Naturally, the further out I project, the foggier things get. But I don't to see the future; I just have to focus on doing the right thing now.

So, that's the story. Although I was long-winded, I left a lot of things out. What's pertinent now is what I'm doing to educate myself, and hopefully that's what the blog will focus on. I'm incredibly fortunate to be living in a time period where education, especially computer science education, is so widely available at a high level for free on the Internet. I can only hope and expect this will continue to expand as time goes on.


_________________________________________________________


Progress Overview:

So let's enumerate what I'm taking.

My first class was the Introduction to Computer Science course (6.00) on MIT's OpenCourseWare site: http://ocw.mit.edu/courses/electrical-engineering-and-computer-science/6-00-introduction-to-computer-science-and-programming-fall-2008/. I began this class on February 3rd, 2012. The course is in python, which is great because it's not only a super easy language but also widely used. The course taught or touched on a lot of fundamentals, like big-O notation, recursion and dynamic programming (still a nemesis for me), object-oriented programming etc. Doing the stuff in this course made me feel confident that at the very least I could do this. I've even tried some projects of my own; python is so easy to use. I've completed up through the 10th problem set, but at this point I somewhat feel that I've accomplished most of what I need to from this class, so I've since branched out. I might return to the class at some point to finish the remaining two problem sets, but who knows.

Now, I'm taking four classes:

  1. Stanford's Programming Abstractions (CS106B): http://see.stanford.edu/see/courseInfo.aspx?coll=11f4f422-5670-4b4c-889c-008262e09e4e
  2. Stanford's Coursera site's Design and Analysis of Algorithms 1: https://www.coursera.org/algo/lecture/index
  3. MIT's Linear Algebra (18.06): http://ocw.mit.edu/courses/mathematics/18-06-linear-algebra-spring-2010/
  4. Udacity's Building a Search Engine (CS101): http://www.udacity.com/overview/Course/cs101/CourseRev/feb2012


From here on I'll refer to them by shorter names. Here's the progress I've made in each:

106B: Three lectures, completed first problem of first problem set.
Algorithms: Watched three sections, completed first problem set.
Linear Algebra: three lectures
101: Watched about 4.5 units, various homework problems.

Comments on the courses thus far:

  1. I don't really know what to expect from 106B. I decided to take because it seemed like a logical "second programming class", and because it teaches a bit of C++, which seems like a useful language to know. Beyond getting used to the different syntax and other differences from Python, getting a C++ compiler working and with all of the libraries and whatnot was a big pain. I seem to have reached a stable state in that regard, but I continue to be wary. I can tell why Randall Munroe wrote a comic praising the simplicity of Python - it really is quite amazing, at least compared to my limited experience with C++. As for the class itself, like I said I don't really know what to expect. I hope they extensively cover data structures because I know that's a fundamental area for CS students to cover.
  2. The algorithms class seems interesting. The professor, Tim Roughgarden, has this somewhat CS-cool-badass attitude to him. His teaching style is clear but he speaks very fast and I often find myself having to rewind to gain a full understanding of what he's saying. Some of this has to do with my choice to take copious notes during all of these lectures (except for the udacity ones). It might seem silly to take notes on a pre-recorded class, but it's something that I've done since college and I find that if I just watch and don't write anything down, I often don't retain as much information. It's nice that videos lend themselves well to this. As for the course content, I've been able to understand everything so far and I got everything correct on the first problem set, but I wonder if things are about to get a lot more complicated. Maybe it's just the attitude of the professor that contributes to this. Not in a bad-way, but I feel like sooner or later he's going to cover some really complicated stuff and just breeze through it.
  3. I had settled on taking the MIT linear algebra course and bought the textbook - rather expensive, although average for a textbook. Then I found a class on Stanford's SEE site (same as the programming abstractions class) that was a more applied version, so I though to take that. But after a few lectures of that one, I determined that it was a bit over my head, so I switched back to this one. Glad that I'm not wasting the money I spent on the textbook. There are lots of math classes I plan to take - the three key ones being linear algebra, probability and statistics and differential equations. After I cover those bases, or at the very least after I finish this linear algebra class, hopefully I can return to that Stanford class and understand it all. The MIT professor Gilbert Strang is a good teacher, with some endearing mannerisms (for example whenever he's finished some thought or section, he says "OK" with a degree of finality as if to say "well you guys should understand that, so let's move on"). It's still somewhat difficult for me to conceptualize matrices - how they act on each other, what they mean - but with time and effort I expect it will get better.
  4. The 101 class hasn't introduced much new knowledge, but it is a lot of fun, and it's nice to have immediate practical use for what you're doing, instead of the more pedagogical feel of most of the other classes. And I'm really excited about some of the other classes coming up; I've enrolled in just about all of them so come April 16th my class load will increase a lot (but in a good way).