Ex Machina

When you are training a reinforcement learning system, there are two broad approaches: value based and policy based. For value based, you try to get the system to understand the value of each action in each state so that it has a running estimate of "how well it's doing". For policy based, you simply ask the machine at each point in time how it could do better, but don't expect it to be able to evaluate how well it's doing in any absolute sense.

For the past few days, I've been debating which of these two methods I should adopt at present. I've been feeling a bit lost from a value perspective, I don't really know how well I'm doing in any absolute sense. However, I can see before me lots of directions for improvement. It's tempting therefore to take a policy based approach: "I don't know where I am, but I do know how to do a bit better". (Parenthetically, let's expand on the specific strategies we each use for this type of thinking in our new blog.)

Despite this, perhaps it's worthwhile to try from time to time to take a values based approach and take a bigger picture evaluation of where I am in my life. The question becomes, then, both in machine learning and in real life, what baseline should I use? Should I compare to a random person my age? A random person in the world now? A random person of roughly the same demographics? It's funny that, for machines, the baseline doesn't matter in evaluating your progress (adding a constant value to all states doesn't really change anything), but it does affect how you explore your next actions and how much risk you take on.

Ultimately, I don't know which of these two paths can bring me to enlightenment, or whether enlightenment should be seen as a terminal goal to achieve rather than a path among the set of paths to converge towards. For me, I think it starts with realizing that I have enough in this moment (choosing a humble baseline), every moment, until that is no longer true.

What about you? Is there a specific step you think you could take towards more enlightenment?

Comments

Post a Comment

Popular posts from this blog

Pilgrim

 Your post is so rich - I'm not sure where to start. Perhaps first to say that yes, let us discuss offline a quotidian address 2.0. :) Your distinction between true progress and experienced progress is an interesting one, and one that rings true to me. If we are progressing, but lack the contextual supports to make us feel we are, is our solution in part a change in mindset? A different way of seeing our lives and world, a new way of counting those markers? I have been reading Joseph Campbell's The Hero with a Thousand Faces, which I find esoteric in the literal sense: I can't read it intensively, rather I scatter my attention across the page and see what leaps out at me. I suspect I could read it again and notice totally different things. It is a mix of philosophy, religion, Freudian psychoanalysis, and storytelling. Still, it has led me to reflect on enlightenment - what is it? It seems to me a common thread across many conceptions is that the enlightened realize that mos...
Read more

Cultivation

 I have no definition of enlightenment. As I gestured at in my last post, I think it has something to do with realizing what matters and what doesn't matter; and that a vast amount on which we spend time in our lives doesn't matter. Realizing it not just in the sense of being able to say it, but to truly feel it. Your point about a deeper connection to the world seems important too: that is something that does matter, I think.  On the other hand, the link to physical exercise troubles me. It's certainly possible that's enlightenment, but if it is simply exhaustion making you care less about things, then to my mind it is not. Enlightenment, I think, isn't just about not caring about what doesn't matter (which, temporarily, could result from exhaustion) but also caring deeply about what does. Let me try another angle. I tend to believe that competition is fundamentally important to self-improvement. When done well, it can push us to be better, to improve, whether ...
Read more

Rara Avis

How can we know if we're ready for black swans, and change course if we're not? Great question. I like to reason about collective human systems (societies, organizations, etc) using the same set of tools I would for studying a machine learning systems. Partly, this is because when you've got a nice new hammer, everything takes on a distinctively nailish aspect. However, it also seems that despite humans being agents exercising individual autonomy, collections of humans are awfully good at climbing uphill relative to metrics you can get them to agree to care about. We are all living longer, children are dying less frequently, and we are on average more wealthy. Also like a machine learning model, we often improve these metrics to the detriment of things outside of our collectively agreed upon reward function (externalities) and we don't distribute the average across all people, but that is a discussion for another time. So, I guess my first answer to your question ab...
Read more