Jekyll2018-05-18T09:44:53+00:00http://herbert.id/BlogMy personal homepageHerbert Ilhan TanujayaThe Array of Options/Optionals2017-12-03T00:00:00+00:002017-12-03T00:00:00+00:00http://herbert.id/2017/12/the-array-of-options-optionals<p>This is a pattern that I saw again and again that I decided to make a quick post
about it.</p>
<h2 id="background">Background</h2>
<p>Consider this problem:</p>
<blockquote>
<p>Given a String, split it on spaces and parse each part into an integer. If a
part is not an integer, omit it from the result.</p>
</blockquote>
<p>For example, <code class="highlighter-rouge">'1 2 abc -3'</code> becomes <code class="highlighter-rouge">[1, 2, -3]</code>.</p>
<h2 id="initial-solution">Initial Solution</h2>
<p>Here’s a quick solution in Java:</p>
<div class="language-java highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">public</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="nf">parseIntegerArray</span><span class="o">(</span><span class="n">String</span> <span class="n">input</span><span class="o">)</span> <span class="o">{</span>
<span class="n">ArrayList</span><span class="o"><</span><span class="n">Integer</span><span class="o">></span> <span class="n">result</span> <span class="o">=</span> <span class="k">new</span> <span class="n">ArrayList</span><span class="o"><</span><span class="n">Integer</span><span class="o">>();</span>
<span class="k">for</span> <span class="o">(</span><span class="n">String</span> <span class="n">i</span> <span class="o">:</span> <span class="n">input</span><span class="o">.</span><span class="na">split</span><span class="o">(</span><span class="s">" "</span><span class="o">))</span> <span class="o">{</span>
<span class="k">try</span> <span class="o">{</span>
<span class="n">result</span><span class="o">.</span><span class="na">add</span><span class="o">(</span><span class="n">Integer</span><span class="o">.</span><span class="na">parseInt</span><span class="o">(</span><span class="n">i</span><span class="o">));</span>
<span class="o">}</span> <span class="k">catch</span> <span class="o">(</span><span class="n">NumberFormatException</span> <span class="n">e</span><span class="o">)</span> <span class="o">{</span>
<span class="c1">// Parse fail, do nothing</span>
<span class="o">}</span>
<span class="o">}</span>
<span class="k">return</span> <span class="n">result</span><span class="o">;</span>
<span class="o">}</span>
</code></pre></div></div>
<p>However, since we are working with data, usually there is a quick way to do it
in functional programming. Indeed, assuming that all parts are integers, we can
do it in one line in Ruby:</p>
<div class="language-ruby highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">def</span> <span class="nf">parse_to_integer_array</span><span class="p">(</span><span class="n">input</span><span class="p">)</span>
<span class="n">input</span><span class="p">.</span><span class="nf">split</span><span class="p">(</span><span class="s1">' '</span><span class="p">).</span><span class="nf">map</span><span class="p">(</span><span class="o">&</span><span class="ss">:to_i</span><span class="p">)</span>
<span class="k">end</span>
</code></pre></div></div>
<p>(By the way, <code class="highlighter-rouge">map(&:to_i)</code> is the same as <code class="highlighter-rouge">map { |s| s.to_i }</code>, and <code class="highlighter-rouge">to_i</code>
basically means “to integer”.)</p>
<p>Now what if there are non-integer parts? The <code class="highlighter-rouge">#to_i</code> method will return 0 here
(which sucks).</p>
<p>In more modern languages, there are constructs that can still do this in one
line. Before that, let’s talk about something else…</p>
<h2 id="optionals">Optionals</h2>
<p>If you have worked with Java, you must have known about the annoying
<code class="highlighter-rouge">NullPointerException</code>, which basically what happens when you try to call
methods/attributes on a <code class="highlighter-rouge">null</code> value.</p>
<p>To solve this, modern languages have a type called the Option type (sometimes,
the Optional type). Basically, it is either a None (representing a null value),
or a Some (representing the value).</p>
<p>The integer parsing above is an example. We would like to have <code class="highlighter-rouge">parse("3")</code> to
return <code class="highlighter-rouge">Some(3)</code> and <code class="highlighter-rouge">parse("-12")</code> to return <code class="highlighter-rouge">Some(-12)</code>, for example. Giving
invalid input, for example <code class="highlighter-rouge">parse("abc")</code>, will return <code class="highlighter-rouge">None</code> instead.</p>
<p>You can read more about Option types at
<a href="https://en.wikipedia.org/wiki/Option_type">this Wikipedia link here</a>.</p>
<p>Sample implementations in languages include:</p>
<ul>
<li>Swift: <a href="https://developer.apple.com/documentation/swift/optional">the Optional type</a></li>
<li>Scala: <a href="https://www.scala-lang.org/api/current/scala/Option.html">the Option type</a></li>
<li>Rust: <a href="https://doc.rust-lang.org/std/option/index.html">the Option type</a></li>
</ul>
<h2 id="going-back-to-the-problem">Going back to the problem</h2>
<p>Now, if we do something like our <code class="highlighter-rouge">map</code> solution, we will get an array of
Options. For example, in Swift, we can do something like:</p>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">parseToIntegerArray</span><span class="p">(</span><span class="nv">input</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="p">[</span><span class="kt">Int</span><span class="p">]</span> <span class="p">{</span>
<span class="n">input</span><span class="o">.</span><span class="nf">split</span><span class="p">(</span><span class="nv">separator</span><span class="p">:</span> <span class="s">" "</span><span class="p">)</span><span class="o">.</span><span class="n">map</span> <span class="p">{</span> <span class="kt">Int</span><span class="p">(</span><span class="nv">$0</span><span class="p">)</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>and running <code class="highlighter-rouge">parseToIntegerArray("2 a 3")</code> will return <code class="highlighter-rouge">[2, nil, 3]</code>. Now, how
do we remove the <code class="highlighter-rouge">nil</code> values?</p>
<p>One way to do it is by using <code class="highlighter-rouge">array.filter { $0 != nil }</code>. However, there is a
neater solution here…</p>
<h2 id="the-flat-map-function">The “flat map” function</h2>
<p>In most libraries that support some notion of “higher order functions”, one can
usually find methods like <code class="highlighter-rouge">map</code>, <code class="highlighter-rouge">reduce</code> (or <code class="highlighter-rouge">fold</code>, <code class="highlighter-rouge">inject</code>, etc), <code class="highlighter-rouge">filter</code>,
<code class="highlighter-rouge">flatten</code>, <code class="highlighter-rouge">for_each</code>, and so on. Another common method is <code class="highlighter-rouge">flat_map</code>, which
flattens the array after mapping them.</p>
<p>For example, let’s say we have an array of people: <code class="highlighter-rouge">["John Doe", "Luke Walt"]</code>.
Suppose we have a function which returns the children of a person. Calling the
function on this array will produce something like, for example,
<code class="highlighter-rouge">[["Jane Doe", "Jake Doe"], ["Nancy Walt", "Troy Walt", "Alice Walt"]]</code>.
However, if suppose we want to get an array of strings, we can use <code class="highlighter-rouge">flat_map</code>
instead of <code class="highlighter-rouge">map</code>, we can get
<code class="highlighter-rouge">["Jane Doe", "Jake Doe", "Nancy Walt", "Troy Walt", "Alice Walt"]</code> instead.</p>
<p>I personally have known this method for quite some time. What I didn’t know,
though, is that we can treat an Option type as an array: either consisting of
one element (for <code class="highlighter-rouge">Some</code>), or zero elements (for <code class="highlighter-rouge">None</code>)!</p>
<p>Most languages also implement <code class="highlighter-rouge">flat_map</code> to expand these Option types. Going
back to the string parsing example, we have these working solutions:</p>
<h3 id="scala-solution">Scala Solution</h3>
<div class="language-scala highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">import</span> <span class="nn">scala.util.Try</span>
<span class="k">def</span> <span class="n">parse</span><span class="o">(</span><span class="n">input</span><span class="k">:</span> <span class="kt">String</span><span class="o">)</span> <span class="k">=</span> <span class="n">input</span><span class="o">.</span><span class="n">split</span><span class="o">(</span><span class="s">" "</span><span class="o">).</span><span class="n">flatMap</span><span class="o">(</span><span class="n">s</span> <span class="k">=></span> <span class="nc">Try</span><span class="o">(</span><span class="n">s</span><span class="o">.</span><span class="n">toInt</span><span class="o">).</span><span class="n">toOption</span><span class="o">)</span>
</code></pre></div></div>
<p>The code might look quite complicated, but <code class="highlighter-rouge">Try(s.toInt).toOption</code> basically
just converts all exceptions to a <code class="highlighter-rouge">None</code>.</p>
<h3 id="rust-solution">Rust Solution</h3>
<div class="language-rust highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">fn</span> <span class="nf">parse</span><span class="p">(</span><span class="n">input</span><span class="p">:</span> <span class="o">&</span><span class="nb">str</span><span class="p">)</span> <span class="k">-></span> <span class="nb">Vec</span><span class="o"><</span><span class="nb">i32</span><span class="o">></span> <span class="p">{</span>
<span class="n">input</span><span class="nf">.split</span><span class="p">(</span><span class="s">" "</span><span class="p">)</span><span class="nf">.filter_map</span><span class="p">(|</span><span class="n">s</span><span class="p">|</span> <span class="n">s</span><span class="nf">.parse</span><span class="p">()</span><span class="nf">.ok</span><span class="p">())</span>
<span class="p">}</span>
</code></pre></div></div>
<p>Notice that in Rust, we use the special function <code class="highlighter-rouge">filter_map</code> instead. (Took me
two hours to figure this out ugh)</p>
<h3 id="swift-solution">Swift Solution</h3>
<div class="language-swift highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="kd">func</span> <span class="nf">parse</span><span class="p">(</span><span class="nv">input</span><span class="p">:</span> <span class="kt">String</span><span class="p">)</span> <span class="o">-></span> <span class="p">[</span><span class="kt">Int</span><span class="p">]</span> <span class="p">{</span>
<span class="k">return</span> <span class="n">input</span><span class="o">.</span><span class="nf">split</span><span class="p">(</span><span class="nv">separator</span><span class="p">:</span> <span class="s">" "</span><span class="p">)</span><span class="o">.</span><span class="n">compactMap</span> <span class="p">{</span> <span class="kt">Int</span><span class="p">(</span><span class="nv">$0</span><span class="p">)</span> <span class="p">}</span>
<span class="p">}</span>
</code></pre></div></div>
<p>UPDATE: in Swift 4.1 the method <code class="highlighter-rouge">compactMap</code> is used to differentiate it with
<code class="highlighter-rouge">flatMap</code>, which is now exclusively used as a map operation composed with a
flatten operation. This is similar with Rust above where <code class="highlighter-rouge">filter_map</code> v.s.
<code class="highlighter-rouge">flat_map</code> is used.</p>
<p>Note that we are using the <code class="highlighter-rouge">Int</code> constructor here, which actually returns <code class="highlighter-rouge">Int?</code>
(the option type for integers). In Swift, this is called “failable constructors”</p>
<ul>
<li>constructors that can fail.</li>
</ul>
<h2 id="afterthoughts">Afterthoughts</h2>
<p>I first saw this problem when I was doing my internship in Twitter. Basically,
I have a pipeline of data that I would like to process. However, some of these
data are anomalies which need to be discarded. Using Scala, all I need to do is
to use the <code class="highlighter-rouge">flatMap</code> function.</p>
<p>I saw this again when I was learning Rust on
<a href="https://adventofcode.com/2017/day/2">Advent of Code 2017 Day 2</a>. Even though
the sample data given consists of all numbers, I was wondering if I can remove
all bad input from the data. Which brings us back to our problem here!</p>
<p>To sum it up: if you need to process data that might have bad data you wish to
discard, check out the <code class="highlighter-rouge">flatMap</code> function from your programming language’s array
library. (If you can’t find them, try to read through all methods, just like
what I did in Rust!)</p>Herbert Ilhan TanujayaThis is a pattern that I saw again and again that I decided to make a quick post about it.Pivot Tables in SQL - Converting Rows to Columns2017-11-12T00:00:00+00:002017-11-12T00:00:00+00:00http://herbert.id/2017/11/pivot-tables-in-sql<p>A few weeks ago I gave a talk at NUS Hackers’s Friday Hacks on some advanced
tips in using SQL. (It was my first technical talk, btw!) Here, I will elaborate
upon one of those tips in more detail.</p>
<h2 id="background">Background</h2>
<p>I have been working on a website that hosts math olympiad contests monthly.
(Shameless plug - source code at https://github.com/donjar/kontes-terbuka,
website at https://ktom.tomi.or.id. Even though it is in Bahasa Indonesia,
Google Translate gives a good enough translation.) In each contest, there are
several structured questions that a contestant can solve. The schema is
something like this:</p>
<p>Submissions:</p>
<ul>
<li>id: primary key</li>
<li>user_id: foreign key to users</li>
<li>problem_id: foreign key to problems</li>
<li>score: integer</li>
</ul>
<p>(There is some sample SQL data that can be found
<a href="https://github.com/donjar/sql-wizardry">here</a>. The file <code class="highlighter-rouge">dump.sql</code> is of your
interest; check out the <code class="highlighter-rouge">contest_scores</code> table. The file
<code class="highlighter-rouge">contest_pivot_table.sql</code> is the solution to the problem below. This repo was
originally made for my SQL Wizardry presentation.)</p>
<h2 id="problem">Problem</h2>
<p>Let’s say the current data looks like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> id | user_id | problem_id | score
----+---------+------------+-------
1 | 1 | 1 | 3
2 | 1 | 2 | 8
3 | 1 | 3 | 1
4 | 2 | 1 | 7
5 | 2 | 2 | 3
6 | 2 | 3 | 3
7 | 3 | 1 | 4
8 | 3 | 2 | 2
9 | 3 | 3 | 3
10 | 4 | 1 | 7
11 | 4 | 2 | 5
12 | 5 | 1 | 4
13 | 5 | 3 | 7
14 | 6 | 2 | 3
15 | 6 | 3 | 4
16 | 7 | 1 | 4
17 | 7 | 2 | 4
18 | 7 | 3 | 9
19 | 8 | 1 | 7
20 | 8 | 2 | 2
21 | 8 | 3 | 4
22 | 9 | 1 | 8
23 | 9 | 2 | 0
24 | 9 | 3 | 4
25 | 10 | 1 | 3
26 | 10 | 2 | 3
27 | 10 | 3 | 6
</code></pre></div></div>
<p>Obviously data like this is not suitable to be shown to the end user. What we
want is something like this:</p>
<div class="highlighter-rouge"><div class="highlight"><pre class="highlight"><code> user_id | problem_1 | problem_2 | problem_3
---------+-----------+-----------+-----------
1 | 3 | 8 | 1
2 | 7 | 3 | 3
3 | 4 | 2 | 3
4 | 7 | 5 |
5 | 4 | | 7
6 | | 3 | 4
7 | 4 | 4 | 9
8 | 7 | 2 | 4
9 | 8 | 0 | 4
10 | 3 | 3 | 6
</code></pre></div></div>
<p>Notice that this involves converting rows of the table into columns: in this
case, the <code class="highlighter-rouge">problem_id</code> needs to be “split” into <code class="highlighter-rouge">problem_1</code>, <code class="highlighter-rouge">problem_2</code>, and
<code class="highlighter-rouge">problem_3</code>.</p>
<p>You can’t do this with a normal SQL SELECT statement! Normally, when you select
columns, you can only select based on operations on the existing columns. There
are no operations that allow you to convert rows into columns in this way.</p>
<p>You can work around this with table joins though, for example:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">WITH</span> <span class="p">(</span>
<span class="k">SELECT</span>
<span class="n">user_id</span><span class="p">,</span>
<span class="n">problem_1</span> <span class="k">AS</span> <span class="n">score</span>
<span class="k">FROM</span> <span class="n">submissions</span>
<span class="k">WHERE</span>
<span class="n">problem_id</span> <span class="o">=</span> <span class="mi">1</span>
<span class="p">)</span> <span class="k">AS</span> <span class="n">problem_1_table</span><span class="p">,</span> <span class="p">(</span>
<span class="k">SELECT</span>
<span class="n">user_id</span><span class="p">,</span>
<span class="n">problem_2</span> <span class="k">AS</span> <span class="n">score</span>
<span class="k">FROM</span> <span class="n">submissions</span>
<span class="k">WHERE</span>
<span class="n">problem_id</span> <span class="o">=</span> <span class="mi">2</span>
<span class="p">)</span> <span class="k">AS</span> <span class="n">problem_2_table</span><span class="p">,</span> <span class="p">(</span>
<span class="k">SELECT</span>
<span class="n">user_id</span><span class="p">,</span>
<span class="n">problem_3</span> <span class="k">AS</span> <span class="n">score</span>
<span class="k">FROM</span> <span class="n">submissions</span>
<span class="k">WHERE</span>
<span class="n">problem_id</span> <span class="o">=</span> <span class="mi">3</span>
<span class="p">)</span> <span class="k">AS</span> <span class="n">problem_3_table</span>
<span class="k">SELECT</span>
<span class="n">problem_1_table</span><span class="p">.</span><span class="n">user_id</span><span class="p">,</span>
<span class="n">problem_1</span><span class="p">,</span>
<span class="n">problem_2</span><span class="p">,</span>
<span class="n">problem_3</span>
<span class="k">FROM</span> <span class="n">problem_1_table</span>
<span class="k">FULL</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="n">problem_2_table</span>
<span class="k">ON</span> <span class="n">problem_1_table</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">problem_2_table</span><span class="p">.</span><span class="n">user_id</span>
<span class="k">FULL</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="n">problem_3_table</span>
<span class="k">ON</span> <span class="n">problem_1_table</span><span class="p">.</span><span class="n">user_id</span> <span class="o">=</span> <span class="n">problem_3_table</span><span class="p">.</span><span class="n">user_id</span>
</code></pre></div></div>
<p>Basically, <code class="highlighter-rouge">problem_1_table</code> is a table that contains the (<code class="highlighter-rouge">user_id</code>, <code class="highlighter-rouge">score</code>)
pair for submissions with <code class="highlighter-rouge">problem_id = 1</code>, and so on. Afterwards, we join with
them on the <code class="highlighter-rouge">user_id</code> to produce the table we want.</p>
<p>In fact, this was our original approach to the problem! The website was built
with Ruby on Rails, and we used a loop in Ruby to loop through all problems and
generate the corresponding SQL query. I would see SQL queries like this often:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">user_contests</span><span class="p">.</span><span class="o">*</span><span class="p">,</span> <span class="n">marks</span><span class="p">.</span><span class="n">short_mark</span><span class="p">,</span> <span class="n">marks</span><span class="p">.</span><span class="n">long_mark</span><span class="p">,</span> <span class="n">marks</span><span class="p">.</span><span class="n">total_mark</span><span class="p">,</span>
<span class="k">case</span> <span class="k">when</span> <span class="n">marks</span><span class="p">.</span><span class="n">total_mark</span> <span class="o">>=</span> <span class="n">gold_cutoff</span> <span class="k">then</span> <span class="s1">'Emas'</span> <span class="k">when</span> <span class="n">marks</span><span class="p">.</span><span class="n">total_mark</span> <span class="o">>=</span>
<span class="n">silver_cutoff</span> <span class="k">then</span> <span class="s1">'Perak'</span> <span class="k">when</span> <span class="n">marks</span><span class="p">.</span><span class="n">total_mark</span> <span class="o">>=</span> <span class="n">bronze_cutoff</span> <span class="k">then</span>
<span class="s1">'Perunggu'</span> <span class="k">else</span> <span class="s1">''</span> <span class="k">end</span> <span class="k">as</span> <span class="n">award</span><span class="p">,</span> <span class="nv">"long_problem_marks_269"</span><span class="p">.</span><span class="nv">"problem_no_269"</span><span class="p">,</span>
<span class="nv">"long_problem_marks_271"</span><span class="p">.</span><span class="nv">"problem_no_271"</span><span class="p">,</span>
<span class="nv">"long_problem_marks_270"</span><span class="p">.</span><span class="nv">"problem_no_270"</span><span class="p">,</span>
<span class="nv">"long_problem_marks_268"</span><span class="p">.</span><span class="nv">"problem_no_268"</span><span class="p">,</span> <span class="n">RANK</span><span class="p">()</span> <span class="n">OVER</span><span class="p">(</span><span class="k">ORDER</span> <span class="k">BY</span>
<span class="n">marks</span><span class="p">.</span><span class="n">total_mark</span> <span class="k">DESC</span><span class="p">)</span> <span class="k">AS</span> <span class="n">rank</span> <span class="k">FROM</span> <span class="nv">"user_contests"</span> <span class="k">INNER</span> <span class="k">JOIN</span> <span class="nv">"contests"</span> <span class="k">ON</span>
<span class="nv">"contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"contest_id"</span> <span class="k">INNER</span> <span class="k">JOIN</span> <span class="nv">"users"</span> <span class="k">ON</span>
<span class="nv">"users"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"user_id"</span> <span class="k">INNER</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">SELECT</span> <span class="n">user_contests</span><span class="p">.</span><span class="o">*</span><span class="p">,</span>
<span class="n">short_marks</span><span class="p">.</span><span class="n">short_mark</span><span class="p">,</span> <span class="n">long_marks</span><span class="p">.</span><span class="n">long_mark</span><span class="p">,</span> <span class="p">(</span><span class="n">short_marks</span><span class="p">.</span><span class="n">short_mark</span> <span class="o">+</span>
<span class="n">long_marks</span><span class="p">.</span><span class="n">long_mark</span><span class="p">)</span> <span class="k">as</span> <span class="n">total_mark</span> <span class="k">FROM</span> <span class="nv">"user_contests"</span> <span class="k">INNER</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">SELECT</span>
<span class="n">user_contests</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id</span><span class="p">,</span> <span class="k">sum</span><span class="p">(</span><span class="k">case</span> <span class="k">when</span> <span class="n">short_submissions</span><span class="p">.</span><span class="n">answer</span> <span class="o">=</span>
<span class="n">short_problems</span><span class="p">.</span><span class="n">answer</span> <span class="k">then</span> <span class="mi">1</span> <span class="k">else</span> <span class="mi">0</span> <span class="k">end</span><span class="p">)</span> <span class="k">as</span> <span class="n">short_mark</span> <span class="k">FROM</span> <span class="nv">"user_contests"</span>
<span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"short_submissions"</span> <span class="k">ON</span> <span class="nv">"short_submissions"</span><span class="p">.</span><span class="nv">"user_contest_id"</span> <span class="o">=</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"short_submissions"</span>
<span class="nv">"short_submissions_user_contests_join"</span> <span class="k">ON</span>
<span class="nv">"short_submissions_user_contests_join"</span><span class="p">.</span><span class="nv">"user_contest_id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span>
<span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"short_problems"</span> <span class="k">ON</span> <span class="nv">"short_problems"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span>
<span class="nv">"short_submissions_user_contests_join"</span><span class="p">.</span><span class="nv">"short_problem_id"</span> <span class="k">WHERE</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"contest_id"</span> <span class="o">=</span> <span class="mi">29</span> <span class="k">AND</span> <span class="p">(</span><span class="nv">"short_submissions"</span><span class="p">.</span><span class="nv">"short_problem_id"</span> <span class="o">=</span>
<span class="nv">"short_problems"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">OR</span> <span class="p">(</span><span class="nv">"short_submissions"</span><span class="p">.</span><span class="nv">"short_problem_id"</span> <span class="k">IS</span> <span class="k">NULL</span> <span class="k">AND</span>
<span class="nv">"short_problems"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">IS</span> <span class="k">NULL</span><span class="p">))</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span><span class="p">)</span> <span class="n">short_marks</span> <span class="k">ON</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"short_marks"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">INNER</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">SELECT</span> <span class="n">user_contests</span><span class="p">.</span><span class="n">id</span>
<span class="k">as</span> <span class="n">id</span><span class="p">,</span> <span class="k">sum</span><span class="p">(</span><span class="n">coalesce</span><span class="p">(</span><span class="n">long_submissions</span><span class="p">.</span><span class="n">score</span><span class="p">,</span> <span class="mi">0</span><span class="p">))</span> <span class="k">as</span> <span class="n">long_mark</span> <span class="k">FROM</span>
<span class="nv">"user_contests"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"long_submissions"</span> <span class="k">ON</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"user_contest_id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">WHERE</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"contest_id"</span> <span class="o">=</span> <span class="mi">29</span> <span class="k">GROUP</span> <span class="k">BY</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span><span class="p">)</span> <span class="n">long_marks</span> <span class="k">ON</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"long_marks"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">WHERE</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"contest_id"</span> <span class="o">=</span>
<span class="mi">29</span><span class="p">)</span> <span class="n">marks</span> <span class="k">ON</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"marks"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">SELECT</span>
<span class="n">user_contests</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id</span><span class="p">,</span> <span class="n">long_submissions</span><span class="p">.</span><span class="n">score</span> <span class="k">as</span> <span class="n">problem_no_269</span> <span class="k">FROM</span>
<span class="nv">"user_contests"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"long_submissions"</span> <span class="k">ON</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"user_contest_id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">WHERE</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"long_problem_id"</span> <span class="o">=</span> <span class="mi">269</span><span class="p">)</span> <span class="n">long_problem_marks_269</span> <span class="k">ON</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"long_problem_marks_269"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">SELECT</span>
<span class="n">user_contests</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id</span><span class="p">,</span> <span class="n">long_submissions</span><span class="p">.</span><span class="n">score</span> <span class="k">as</span> <span class="n">problem_no_271</span> <span class="k">FROM</span>
<span class="nv">"user_contests"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"long_submissions"</span> <span class="k">ON</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"user_contest_id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">WHERE</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"long_problem_id"</span> <span class="o">=</span> <span class="mi">271</span><span class="p">)</span> <span class="n">long_problem_marks_271</span> <span class="k">ON</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"long_problem_marks_271"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">SELECT</span>
<span class="n">user_contests</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id</span><span class="p">,</span> <span class="n">long_submissions</span><span class="p">.</span><span class="n">score</span> <span class="k">as</span> <span class="n">problem_no_270</span> <span class="k">FROM</span>
<span class="nv">"user_contests"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"long_submissions"</span> <span class="k">ON</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"user_contest_id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">WHERE</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"long_problem_id"</span> <span class="o">=</span> <span class="mi">270</span><span class="p">)</span> <span class="n">long_problem_marks_270</span> <span class="k">ON</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"long_problem_marks_270"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="p">(</span><span class="k">SELECT</span>
<span class="n">user_contests</span><span class="p">.</span><span class="n">id</span> <span class="k">as</span> <span class="n">id</span><span class="p">,</span> <span class="n">long_submissions</span><span class="p">.</span><span class="n">score</span> <span class="k">as</span> <span class="n">problem_no_268</span> <span class="k">FROM</span>
<span class="nv">"user_contests"</span> <span class="k">LEFT</span> <span class="k">OUTER</span> <span class="k">JOIN</span> <span class="nv">"long_submissions"</span> <span class="k">ON</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"user_contest_id"</span> <span class="o">=</span> <span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">WHERE</span>
<span class="nv">"long_submissions"</span><span class="p">.</span><span class="nv">"long_problem_id"</span> <span class="o">=</span> <span class="mi">268</span><span class="p">)</span> <span class="n">long_problem_marks_268</span> <span class="k">ON</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"id"</span> <span class="o">=</span> <span class="nv">"long_problem_marks_268"</span><span class="p">.</span><span class="nv">"id"</span> <span class="k">WHERE</span>
<span class="nv">"user_contests"</span><span class="p">.</span><span class="nv">"contest_id"</span> <span class="o">=</span> <span class="mi">29</span> <span class="k">AND</span> <span class="nv">"users"</span><span class="p">.</span><span class="nv">"province_id"</span> <span class="o">=</span> <span class="mi">35</span> <span class="k">ORDER</span> <span class="k">BY</span>
<span class="nv">"marks"</span><span class="p">.</span><span class="nv">"total_mark"</span> <span class="k">DESC</span>
</code></pre></div></div>
<p>It wasn’t very pretty Ruby and SQL code, but hey, it works. But of course, this
does not sit really well with me - using Ruby to generate SQL and using that
generated code to query data seems weird, isn’t it?</p>
<p>Of course we can just select all data in SQL and do the processing in Ruby on
Rails. However, as the saying goes - do the data processing in the database
level, where it is suitable for the job right?</p>
<p>Enter pivot tables.</p>
<h2 id="pivot-tables">Pivot Tables</h2>
<p>You might have heard about pivot tables from Microsoft Excel. To quote
Wikipedia:</p>
<blockquote>
<p>A pivot table is a table that summarizes data in another table, and is made
by applying an operation such as sorting, averaging, or summing to data in
the first table, typically including grouping of the data.</p>
</blockquote>
<p>The “grouping of the data” is of interest here.</p>
<h3 id="caveat">Caveat</h3>
<p>Pivot tables have different implementations across different databases. I am
only going to discuss how to do it in PostgreSQL, as that is the database I am
using in my application. You should be able to find implementations for other
databases by searching for something like “pivot tables MySQL”.</p>
<h3 id="crosstab">Crosstab</h3>
<p>In PostgreSQL, the relevant function is called “crosstab”. It is available as
an extension, and hence, you should install it first by running <code class="highlighter-rouge">CREATE
EXTENSION tablefunc;</code> as a superuser.</p>
<p>The relevant documentation can be found
<a href="https://www.postgresql.org/docs/10/static/tablefunc.html">here</a>. One thing you
might notice, though, is that there are actually four different crosstab
functions!</p>
<p>The functions are actually generating pivot tables, only with different
abstraction levels. I found that the <code class="highlighter-rouge">crosstab(text source_sql,
text category_sql)</code> function produces the result I needed. To quote the
documentation:</p>
<blockquote>
<p><code class="highlighter-rouge">source_sql</code> is a SQL statement that produces the source set of data. This
statement must return one <code class="highlighter-rouge">row_name</code> column, one category column, and one
value column. It may also have one or more “extra” columns. The <code class="highlighter-rouge">row_name</code>
column must be first. The category and value columns must be the last two
columns, in that order. Any columns between <code class="highlighter-rouge">row_name</code> and category are
treated as “extra”. The “extra” columns are expected to be the same for all
rows with the same <code class="highlighter-rouge">row_name</code> value.</p>
</blockquote>
<p>So basically, <code class="highlighter-rouge">source_sql</code> is in the form:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="k">key</span><span class="p">(</span><span class="n">s</span><span class="p">),</span> <span class="n">categories</span><span class="p">,</span> <span class="k">values</span> <span class="k">FROM</span> <span class="p">...</span>
</code></pre></div></div>
<p>and the <code class="highlighter-rouge">category_sql</code> matches the values that we want in the category to
separate.</p>
<p>An example will make these concepts clearer. Going to the example we discussed
previously, we notice that the key is the <code class="highlighter-rouge">user_id</code> - basically, this is the
column we want as our first column. The category is <code class="highlighter-rouge">problem_id</code>, since
this is what we want the next columns of the resulting pivot table to be. And
finally, the values to be filled in inside the resulting table would be <code class="highlighter-rouge">score</code>.
It is normal to use an <code class="highlighter-rouge">ORDER BY key</code> as well here, so that the resulting pivot
table is not jumbled. Combining all of them, we have this <code class="highlighter-rouge">source_sql</code> query:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="n">user_id</span><span class="p">,</span> <span class="n">problem_id</span><span class="p">,</span> <span class="n">score</span> <span class="k">FROM</span> <span class="n">submissions</span> <span class="k">ORDER</span> <span class="k">BY</span> <span class="n">user_id</span>
</code></pre></div></div>
<p>For the category, we know that the problem IDs we want are 1, 2, and 3. Hence,
the corresponding <code class="highlighter-rouge">category_sql</code> is:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="mi">1</span><span class="p">,</span> <span class="mi">2</span><span class="p">,</span> <span class="mi">3</span>
</code></pre></div></div>
<p>(in reality we would want something like <code class="highlighter-rouge">SELECT id FROM problems ORDER BY id</code>).</p>
<p>And at the end, we would need to specify the columns to be generated as well,
with this format:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nv">"column1"</span> <span class="n">category1</span><span class="p">,</span> <span class="nv">"column2"</span> <span class="n">category2</span><span class="p">,</span> <span class="p">...)</span>
</code></pre></div></div>
<p>In this case, it would be:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="p">(</span><span class="nv">"user_id"</span> <span class="n">int</span><span class="p">,</span> <span class="nv">"problem_1"</span> <span class="n">int</span><span class="p">,</span> <span class="nv">"problem_2"</span> <span class="n">int</span><span class="p">,</span> <span class="nv">"problem_3"</span> <span class="n">int</span><span class="p">)</span>
</code></pre></div></div>
<p>Hence, the resulting SQL query is:</p>
<div class="language-sql highlighter-rouge"><div class="highlight"><pre class="highlight"><code><span class="k">SELECT</span> <span class="o">*</span> <span class="k">FROM</span> <span class="n">crosstab</span><span class="p">(</span>
<span class="s1">'SELECT user_id, problem_id, score FROM submissions ORDER BY user_id'</span><span class="p">,</span>
<span class="s1">'SELECT 1, 2, 3'</span>
<span class="p">)</span> <span class="k">AS</span> <span class="p">(</span>
<span class="nv">"user_id"</span> <span class="n">int</span><span class="p">,</span>
<span class="nv">"problem_1"</span> <span class="n">int</span><span class="p">,</span>
<span class="nv">"problem_2"</span> <span class="n">int</span><span class="p">,</span>
<span class="nv">"problem_3"</span> <span class="n">int</span>
<span class="p">)</span>
</code></pre></div></div>
<p>This will return just what we wanted!</p>
<h3 id="problems">Problems</h3>
<p>While this query is “cleaner” than the SQL joins “workaround” I wrote
previously, there are still some problems with this solution. Firstly, there
is a need to specify the columns in the resulting pivot table. While this can
also be generated with the underlying framework (with Ruby etc.), we still need
to query two times: once to get the column definitions (for example, <code class="highlighter-rouge">SELECT
id FROM problems</code>), and once to do the actual crosstab query. If I remember
correctly, this problem is not specific to PostgreSQL; other databases also have
this problem. This solution will not solve the problem of using the application
to craft SQL queries.</p>
<p>The other problem which might not be obvious is about ORMs - it is highly
unlikely that ORMs have support for pivot tables. This may cause problems with
using ORM tools - for example, while ActiveRecord (the ORM in Ruby on Rails)
has a <code class="highlighter-rouge">find_by_sql</code> method, chaining it with another ActiveRecord method can
cause bugs. In my case, <code class="highlighter-rouge">Submissions.find_by_sql(crosstab_query).count</code> does
not work (as I assume Rails doesn’t know where to put the <code class="highlighter-rouge">COUNT</code> query).</p>
<p>With two queries running, performance might also be a problem. From my
observations, the SQL joins query takes around 70 ms, while these two queries
take around 50 ms in total; however, this does not account for network overheads
etc. A more systematic benchmarking is needed to confirm if this solution is
indeed faster than the joining solution.</p>
<h2 id="conclusion">Conclusion</h2>
<p>If you need to somehow convert rows into columns, and vice versa, to do
grouping/aggregation, pivot tables might be for you. While several problems
exist, I still believe that pivot table is the right tool for the job and the
resulting solution is “cleaner”.</p>Herbert Ilhan TanujayaA few weeks ago I gave a talk at NUS Hackers’s Friday Hacks on some advanced tips in using SQL. (It was my first technical talk, btw!) Here, I will elaborate upon one of those tips in more detail.Least Squares Policy Iteration pt. 1: Markov Decision Processes2017-04-24T00:00:00+00:002017-04-24T00:00:00+00:00http://herbert.id/2017/04/least-squares-policy-iteration-part-1-markov-decision-processes<p>In NUS CS3243 (Introduction to Artificial Intelligence) we had a project, which is to build an agent that plays Tetris. It was an interesting project, even though I sort of freerided (sorry my teammates haha).</p>
<p>I was tasked to look at this interesting algorithm called LSPI (Least Squares Policy Iteration) though. Took me some time to grasp it, and I finally understood it. Sadly, when I implemented it, the policy resulted clears exactly 0 lines. :/ Perhaps, the features were bad, or the samples were bad, or I just misunderstood the algorithm. My seniors told me that they can clear like millions of lines with LSPI…</p>
<p>Anyways, all the reason for me to recap about it, isn’t it? Maybe even juniors can refer to this in future iterations of the project! (It might be changed though, since I heard my batch is the last time Prof. Bryan’s teaching CS3243)</p>
<p>I forsee that this can be long, hence I decided to break it into a few parts. This part talks about the underlying model, which is Markov Decision Process (MDP). Probably in the next part I will talk about approximation algorithms and LSPI itself, and the final part I will try to re-implement it with a simpler game.</p>
<h2 id="basic-idea">Basic Idea</h2>
<p>The idea of a MDP is simple. We have a set of states <script type="math/tex">S</script>, as well as a set of actions <script type="math/tex">A</script>. An action brings you from a state to some other states with a given probablity. Doing an action can also give you a reward.</p>
<p><img src="http://herbert.id/images/blog/2017-04-24-markov-decision-process.png" alt="Example of a Markov Decision Process" /></p>
<p>Shown above is an example of a Markov Decision Process. Here, the states are indicated by <script type="math/tex">S1</script> until <script type="math/tex">S4</script>, and the actions are <script type="math/tex">a1</script>, <script type="math/tex">a2</script>, and <script type="math/tex">a3</script>. Listed in the brackets are the probability that doing an action will bring you to that state. For example, while action <script type="math/tex">a1</script> will always bring you from <script type="math/tex">S1</script> to <script type="math/tex">S2</script>, action <script type="math/tex">a2</script> has 0.9 probability of bringing you back to <script type="math/tex">S1</script>, and 0.1 probability of bringing you to <script type="math/tex">S4</script>. As you can see, an action can bring you back to the same state.</p>
<p>Also listed after the letter R is the reward. For example, the reward of doing action <script type="math/tex">a1</script> that brings you from <script type="math/tex">S2</script> to <script type="math/tex">S3</script> is 100, while the reward of doing action <script type="math/tex">a3</script> that brings you from <script type="math/tex">S4</script> to <script type="math/tex">S4</script> is 1.</p>
<p>So, what can be modeled by this, you may ask? Many things! Consider, for example, Tetris; here, the states are the position of the board as well as the next piece, and the action is the act of placing a piece into the board. The rewards would be the number of points you get. Most likely, this would be the number of rows cleared; in certain games, however, you might get rewarded for things like back-to-back Tetris or T-spins. In this case, the probability of any action will always be 1.</p>
<p>We can also look at <a href="https://en.wikipedia.org/wiki/Yahtzee">Yahtzee</a>, for example. Here, the states are everything that describes the current position. This includes points filled in the scoresheet, current dice position, turn number, and so on. The actions would be the dice that you choose to roll, or filling in a scoresheet. The reward would be the number of points you get on filling in a score, and the probablity varies according to the action taken. If you’re rolling all 5 dice, for example, there are <script type="math/tex">\binom{10}{5} = 252</script> states that you can reach!</p>
<p>What is the main objective of this MDP, anyways? The idea is to find a policy <script type="math/tex">\pi</script> that specifies what action should one take from a given state, so that we can maximize the total number of rewards over time. On the MDP above, for example, a policy would be to do action <script type="math/tex">a1</script> from <script type="math/tex">S1</script>, action <script type="math/tex">a1</script> from <script type="math/tex">S2</script>, action <script type="math/tex">a3</script> from <script type="math/tex">S3</script>, and action <script type="math/tex">a3</script> from <script type="math/tex">S4</script>.</p>
<h2 id="formal-definition">Formal Definition</h2>
<p>Before we proceed, it’s a good idea to formalize everything to remove any doubts.</p>
<p>A MDP consists of:</p>
<ul>
<li><script type="math/tex">S</script>, a set of states.</li>
<li><script type="math/tex">A</script>, a set of actions.</li>
<li><script type="math/tex">P: S \times A \times S \to [0, 1]</script>, the probability function. This function takes in the starting state <script type="math/tex">s</script>, the action <script type="math/tex">a</script> and the resulting state <script type="math/tex">s'</script>, and returns the probability that taking action <script type="math/tex">a</script> from state <script type="math/tex">s</script> brings you to <script type="math/tex">s'</script>. In the diagram above, <script type="math/tex">P(S1, a1, S2) = 1</script> while <script type="math/tex">P(S1, a3, S3) = 0</script>.</li>
<li><script type="math/tex">R: S \times A \times S \to \mathbb{R}</script>, the reward function. In the diagram above, <script type="math/tex">R(S2, a1, S3) = 100</script>, while <script type="math/tex">R(S1, a1, S2) = 0</script>.</li>
<li><script type="math/tex">\gamma \in [0, 1]</script>, the discount factor, which is described below.</li>
</ul>
<p>The objective is to find a function <script type="math/tex">\pi</script> that maps from <script type="math/tex">S</script> to <script type="math/tex">A</script>, so that the value</p>
<script type="math/tex; mode=display">\sum_{t = 0}^\infty \gamma^t R(s_t, \pi(s_t), s_{t + 1})</script>
<p>is maximized. Here <script type="math/tex">t</script> denotes the “time” and <script type="math/tex">s_0, s_1, s_2, \dotsc</script> are states the agent is in at time <script type="math/tex">0, 1, 2, \dotsc</script> respectively.</p>
<p>We can see that <script type="math/tex">\gamma</script> is a value that diminishes the importance of future rewards. This <script type="math/tex">\gamma</script> is typically close to 1, and is only there so that the infinite sum can converge. Indeed, if <script type="math/tex">S</script> and <script type="math/tex">A</script> are both finite, <script type="math/tex">R</script> has a upper bound and the sum will definitely converge if <script type="math/tex">% <![CDATA[
\gamma < 1 %]]></script>. There should be nothing wrong with setting <script type="math/tex">\gamma = 1</script> though!</p>
<h2 id="finding-the-policy">Finding the Policy</h2>
<p>So how do we actually find this optimal policy <script type="math/tex">\pi</script>? A simple idea might be to just take the action that maximizes the direct reward. For example, in the sample above, we can just take <script type="math/tex">a1</script> from <script type="math/tex">S2</script> since we can cash 100 instantly.</p>
<p>However, this might not be optimal! We can see that the optimal way is to actually take <script type="math/tex">a2</script> from <script type="math/tex">S2</script>, then take <script type="math/tex">a2</script> again from <script type="math/tex">S1</script>. This is so that we can eventually reach <script type="math/tex">S4</script>, where we can do <script type="math/tex">a3</script> an infinite number of times. (Of course, this assumes that <script type="math/tex">\gamma</script> is sufficiently big that it will be able to reach a total reward greater than 100 after a sufficient amount of time.)</p>
<p>Hence, we should define a value function <script type="math/tex">V</script> that determines how rewarding a state is, according to a given policy. Before that, it is nice to define a helper function. Let’s define an evaluation of how wise an action is, as follows: the wiseness of taking action <script type="math/tex">a</script> that moves the agent from state <script type="math/tex">s</script> to state <script type="math/tex">s'</script> is</p>
<script type="math/tex; mode=display">W(s, a, s') = P(s, a, s')\left[ R(s, a, s') + \gamma V(s') \right].</script>
<p>This wiseness is something I made up by the way :). This should make sense, though; if we want to consider how wise an action is, we should see the probability, the reward of taking the action, as well as how valuable the next state is.</p>
<p>Notice that, previously, <script type="math/tex">V</script> is still not properly defined yet. We are ready to do so now. Here, <script type="math/tex">V</script> maps from <script type="math/tex">\Pi \times S</script> to <script type="math/tex">\mathbb{R}</script>, where <script type="math/tex">\Pi</script> is the set of all policies possible, such that</p>
<script type="math/tex; mode=display">V(\sigma, s) = \sum_{s' \in S} W(s, \sigma(s), s').</script>
<p>Simple, right? We just sum up all the “wiseness” of taking action <script type="math/tex">\sigma(s)</script> from <script type="math/tex">s</script> to everywhere. Also, if the context of a policy is clear, we can drop the <script type="math/tex">\sigma</script> (the policy).</p>
<p>Now, to get the best policy, we just need to pick the action such that this sum of wiseness is maximized:</p>
<script type="math/tex; mode=display">\pi(s) = \max_{a \in A} \left\{ \sum_{s' \in S} W(s, a, s') \right\}.</script>
<p>This might seem weird; we get the values from the policy, yet we get the policy from the values. It seems that there is a chicken-and-egg problem here!</p>
<p>To solve this, the idea is to start from a random policy <script type="math/tex">\pi_0</script>. Then, we use both the value equation and the policy equation above to improve the policy again and again, until it converges. It should converge; after all, there are only finitely many policies available, assuming finitely many states and actions.</p>
<p>Several variants of the algorithms exist. Notable algorithms include the “value iteration” and the “policy iteration”.</p>
<h3 id="value-iteration">Value Iteration</h3>
<p>In value iteration, we don’t care about <script type="math/tex">\pi</script> at all; we just aim at maximizing <script type="math/tex">V</script>. In other words, we merge the definition of <script type="math/tex">\pi</script> into <script type="math/tex">V</script>, so that we get:</p>
<script type="math/tex; mode=display">V_{i + 1}(s) = \max_{a \in A} \left\{ \sum_{s' \in S} W_i(s, a, s') \right\}.</script>
<p>Here <script type="math/tex">i</script> is the iteration number, and we define <script type="math/tex">W_i(s, a, s')</script> to be <script type="math/tex">P(s, a, s') \left[ R(s, a, s') + \gamma V_i(s') \right]</script>. So, we start at <script type="math/tex">i = 0</script>, getting <script type="math/tex">V_0</script> as a value function from a random policy. Then, we iterate through the function above, stopping when we reach an <script type="math/tex">i</script> such that <script type="math/tex">V_{i + 1}(s) = V_i(s)</script>.</p>
<p>If you don’t want to merge the functions, this algorithm can also be viewed as running the policy and the value equations alternatingly.</p>
<h3 id="policy-iteration">Policy Iteration</h3>
<p>In policy iteration, we get a random policy at the start, then we calculate the <em>exact</em> value of all states by repeatedly getting the value from the value equation until the value of each state converges. Then, from the values, we get a new policy from the policy equation, then we get the exact value again. We do this until the policy converges.</p>
<p>Basically:</p>
<ul>
<li><code class="highlighter-rouge">old_policy</code> is initialized to a random policy, while <code class="highlighter-rouge">new_policy</code> is null</li>
<li>While <code class="highlighter-rouge">old_policy != new_policy</code>:
<ul>
<li><code class="highlighter-rouge">old_policy = new_policy</code> if not null</li>
<li>Get exact value of all states from the value equation until it converegs</li>
<li>Get <code class="highlighter-rouge">new_policy</code> from the policy equation and the values from the previous step</li>
</ul>
</li>
</ul>
<p>Notice that if we have a policy in hand, in the wiseness equation <script type="math/tex">W(s, a, s') = P(s, a, s') \left[ R(s, a, s') + \gamma V(s') \right]</script>, <script type="math/tex">P(s, a, s')</script>, <script type="math/tex">R(s, a, s')</script>, and <script type="math/tex">\gamma</script> are actually already known! This means that we can just solve for the values by solving the system of <script type="math/tex">n</script> <em>linear</em> equations, where <script type="math/tex">n</script> is the number of states. This equation is</p>
<script type="math/tex; mode=display">V(s) = \sum_{s' \in S} W(s, \pi(s), s')</script>
<p>with <script type="math/tex">V(s)</script> as the unknowns for all <script type="math/tex">s \in S</script>. This can be done with, for example, matrix inversion and multiplication. With Gauss-Jordan elimination this is <script type="math/tex">O(n^3)</script>; with Strassen algorithm this is around <script type="math/tex">O(n^{2.8})</script>.</p>
<p>Now, while these algorithms will definitely return the best policy that optimizes the MDP, we can see here the problem - these algorithms are <em>slow</em>. Imagine Tetris with 10 columns and 20 rows, for example; the number of possible states would be in an order of <script type="math/tex">2^{10 \times 20} \approx 10^{60}</script>. Even <script type="math/tex">O(n)</script> is slow!</p>
<p>Enter LSPI, to be described in the next post (hopefully).</p>Herbert Ilhan TanujayaIn NUS CS3243 (Introduction to Artificial Intelligence) we had a project, which is to build an agent that plays Tetris. It was an interesting project, even though I sort of freerided (sorry my teammates haha).Afterthoughts: CS2020 Teaching Assistant2017-04-11T00:00:00+00:002017-04-11T00:00:00+00:00http://herbert.id/2017/04/afterthoughts-cs2020-teaching-assistant<p>This semester (AY16/17 sem 2, Spring 2017) I decided to take up the job of teaching a bunch of students in NUS CS2020: Data Structures and Algorithms Accelerated.</p>
<p>How I decided to take up this job was quite funny, actually. Last year, after I failed to <em>come</em> to an exam, I was very, very sad. Then at that time Prof Ben is also not happy with me not taking CS2103, telling me that he might reject me from CS3217 due to that. I was like “no man, I cannot let my life become this messy” and I decided to email Prof Seth for CS2020 TA positions.</p>
<p>In the end, I took up the job of CS2020 TA, while getting accepted to and taking CS3217 at the same time. I don’t regret my decisions (even though I can die already from this workload haha).</p>
<p><img src="http://herbert.id/images/blog/2017-04-11-cs2020-discussion-group.jpg" alt="Me and my students" /></p>
<p><em>Me and my students in CS2020. There’s also the classic tactic of bribing your students pizza to get good feedback ;)</em></p>
<p>I also compiled 30 questions for the final discussion session, which can be viewed <a href="http://herbert.id/others/blog/2017-04-11-funquiz.pdf">here</a>.</p>
<p>It was quite a sad moment, actually. This module is offered for the last time in NUS, since CS1020, CS2010, and this is being replaced by CS2030 and CS2040. The professors for this module is Prof. Seth Gilbert and Prof. Alan Cheng, both very enthusiastic and charismatic professors. In the lectures you could feel the energy flowing out from them to the students, and they never seem to be tired at all!</p>
<p>I would say that everyone not taught by them is missing something in NUS School of Computing.</p>
<p><img src="http://herbert.id/images/blog/2017-04-11-cs2020-all-students.jpg" alt="The whole class" /></p>
<p><em>The whole class of CS2020 AY16/17</em></p>
<p>Oh, we also bought them these ultra cute Pusheen toys!!</p>
<p><img src="http://herbert.id/images/blog/2017-04-11-pusheen.jpg" alt="Pusheen toy" /></p>
<p><em>Prof. Seth seems to like it really much. :)</em></p>
<p>Tutoring students was also a very enjoyable moment. Sharing knowledge is fun. Provoking them with hard problems are also fun. I feel that CS2020 students are enthusiastic students, because they actually choose the hardcore mode (you need at least A- for the basic programming module, CS1010 or its equivalent).</p>
<p>I don’t know what other modules to tutor. Other students seem to be less enthusiastic with the other modules…</p>
<p>Oh well, good times come to an end. I wish all the students good luck for the CS2020 (very) final exam.</p>Herbert Ilhan TanujayaThis semester (AY16/17 sem 2, Spring 2017) I decided to take up the job of teaching a bunch of students in NUS CS2020: Data Structures and Algorithms Accelerated.An Interesting Problem from Indonesia’s National Math Olympiad2017-04-04T00:00:00+00:002017-04-04T00:00:00+00:00http://herbert.id/2017/04/an-interesting-problem-from-indonesias-national-math-olympiad<p>Last year there was a very interesting problem from Indonesia’s National Olympiad:</p>
<blockquote>
<p>Problem (Indonesia’s National Math Olympiad 2016 no. 4). <script type="math/tex">ABC</script> is a triangle such that the angles satisfy the equation</p>
<script type="math/tex; mode=display">\frac{\cos A}{20} + \frac{\cos B}{21} + \frac{\cos C}{29} = \frac{29}{420}.</script>
<p>Prove that <script type="math/tex">ABC</script> is a right-angled triangle.</p>
</blockquote>
<p>My first reaction was “Huh? How the heck?”</p>
<p>An intriguing problem indeed. Firstly, the equations are not symmetrical, and secondly, how the heck does the numbers 20, 21, 29, and 420 appear?</p>
<p>Some people might argue that this problem is ugly. Indeed, putting these “magic numbers” doesn’t look really nice; furthermore, trigonometric functions usually never appear in a problem explicitly. Nevertheless, whether it’s pretty or ugly, I think we can agree that this problem is interesting and can throw some of the most experienced problem solvers even.</p>
<p>Note that the way to solve this is not by plugging one of <script type="math/tex">A</script>, <script type="math/tex">B</script>, or <script type="math/tex">C</script> with 90 degrees. This attempt will only solve the converse (<script type="math/tex">ABC</script> right-angled implies the equation).</p>
<p>One easy way would be to substitute <script type="math/tex">C</script> with <script type="math/tex">180^{\circ} - A - B</script> and use trigonometric identities to obtain <script type="math/tex">\cos C = -\cos (A + B) = \sin A \sin B - \cos A \cos B</script>. From there who knows if we can obtain something like <script type="math/tex">\cos A = 0</script> (which means that <script type="math/tex">A = 90^{\circ})</script>.</p>
<p>Bashing (like this approach) is an OK solution, and if you’re adept enough you might be able to solve this in around an hour (which is nice - the olympiad runs in 2 days, each day containing 4 problems in 4 hours, and usually problem 4 is the hardest of the day, which is indeed the case here).</p>
<p>My friend came with another approach that can involve even more bashing - move <script type="math/tex">\frac{\cos C}{29}</script> to the right and square both sides! He got</p>
<script type="math/tex; mode=display">\left( \frac{\sin A}{20} - \frac{\sin B}{21} \right)^2 = -\frac{\cos^2 C}{29^2}</script>
<p>LHS is nonnegative and RHS is nonpositive, hence both must be zero and <script type="math/tex">\cos C = 0 \implies C = 90^{\circ}</script>. Let’s come up with a less bashing solution, though!</p>
<p>Now, from the numbers 20, 21, 29, and 420, we can notice two things:</p>
<ul>
<li><script type="math/tex">20 \times 21 = 420</script>. More importantly,</li>
<li><script type="math/tex">20^2 + 21^2 = 29^2</script>, meaning that there is a right-angled triangle with sides 20, 21, and 29.</li>
</ul>
<p>The last fact above might not be easy to catch. Intuitions like this separate the best problem solvers from others, and is the reason why practicing is important in olympiad mathematics. It sharpens your intuition.</p>
<p>Building from the two facts above, we can change the equation to:</p>
<script type="math/tex; mode=display">\frac{21}{29} \cos A + \frac{20}{29} \cos B + \frac{20 \cdot 21}{29 \cdot 29} \cos C = 1</script>
<p>Now if we take <script type="math/tex">\theta</script> an angle between 0 and <script type="math/tex">90^{\circ}</script> such that <script type="math/tex">\sin \theta = \frac{20}{29}</script>, we have <script type="math/tex">\cos \theta = \frac{21}{29}</script> and the equation can be simplified further into</p>
<script type="math/tex; mode=display">\cos \theta \cos A + \sin \theta \cos B + \sin \theta \cos \theta \cos C = 1.</script>
<p>Aha! This looks much more managable isn’t it? Even though we introduced a new variable, there are no more these so-called “magic numbers”!</p>
<p>We’re still stuck here though, so no harm in plugging <script type="math/tex">C = 180^{\circ} - A - B</script> and expanding <script type="math/tex">\cos C</script> out, which we know equals <script type="math/tex">\sin A \sin B - \cos A \cos B</script> (we discussed this). Hence the equation becomes</p>
<script type="math/tex; mode=display">\cos \theta \cos A + \sin \theta \cos B + \sin \theta \cos \theta \sin A \sin B - \sin \theta \cos \theta \cos A \cos B = 1.</script>
<p>Looks messy? Not at all - this can be simplified to</p>
<script type="math/tex; mode=display">(1 - \cos \theta \cos A)(1 - \sin \theta \cos B) = \sin \theta \cos \theta \sin A \sin B.</script>
<p>And from here we can finish this with a neat trick. Notice that <script type="math/tex">1 \ge \cos (A - \theta) = \cos A \cos \theta + \sin A \sin \theta</script>, meaning <script type="math/tex">1 - \cos \theta \cos A \ge \sin \theta \sin A</script>. Similarly, by using <script type="math/tex">1 \ge \sin (B + \theta)</script> we have <script type="math/tex">1 - \sin \theta \cos B \ge \cos \theta \sin B</script>. Multiplying these inequalities, we have:</p>
<script type="math/tex; mode=display">(1 - \cos \theta \cos A)(1 - \sin \theta \cos B) \ge \sin \theta \cos \theta \sin A \sin B.</script>
<p>However, we know that they are actually the same! Hence the two inequalities that we started off must be equalities after all, and hence, <script type="math/tex">1 = \cos (A - \theta)</script> and <script type="math/tex">1 = \sin (B + \theta)</script>. Since we assumed <script type="math/tex">% <![CDATA[
0 < \theta < 90^{\circ} %]]></script> and <script type="math/tex">A</script> and <script type="math/tex">B</script> are angles of a triangle, we have <script type="math/tex">A - \theta = 0</script> and <script type="math/tex">B + \theta = 90^{\circ}</script>, making <script type="math/tex">A + B = 90^{\circ}</script>.</p>
<p>That means <script type="math/tex">C = 90^{\circ}</script>, and we are done!</p>
<p>To summarize, our solution has two main points:</p>
<ol>
<li>Setting <script type="math/tex">\sin \theta = \frac{20}{29}</script> to remove all the magic numbers</li>
<li>Removing <script type="math/tex">C</script> from the equation and factoring it out nicely</li>
<li>Using inequalities and showing that the inequalities must be equalities after all</li>
</ol>
<p>Hopefully these shows how cool math olympiad problems can be :) intuition is very important and can only be sharpened by solving more and more problems.</p>Herbert Ilhan TanujayaLast year there was a very interesting problem from Indonesia’s National Olympiad:A Generalized Version of the Fundamental Theorem of Calculus2017-03-31T00:00:00+00:002017-03-31T00:00:00+00:00http://herbert.id/2017/03/a-generalized-version-of-the-fundamental-theorem-of-calculus<p>Last semester (AY16/17 sem 1) I had a chance to take <a href="https://nusmods.com/modules/MA3110S">MA3110S</a> under Prof. Chua Seng Kee. <a href="http://ww1.math.nus.edu.sg/undergraduates.aspx?f=UP-SPM">SPM (Special Programme in Math)</a> modules are scary, but Prof. CSK has managed to up it into another level, giving things like generalized versions of theorems in the textbooks as well as his scary true/false questions! I really learned a lot under his guidance. (Your life in SPM won’t be complete if you haven’t taken any of his modules haha)</p>
<p>In this post, I will present one of the most memorable theorems from him. Before that, we would need to define some more definitions:</p>
<blockquote>
<p><em>Definition.</em> The left-hand derivative <script type="math/tex">f'_-(x)</script> is defined as <script type="math/tex">\lim_{h \to 0^-} \frac{f(x + h) - f(x)}{h}</script>. Similarly, the right-hand derivative <script type="math/tex">f'_+(x)</script> is defined as <script type="math/tex">\lim_{h \to 0^+} \frac{f(x + h) - f(x)}{h}</script>.</p>
</blockquote>
<p>For example, for the function <script type="math/tex">f: \mathbb{R} \to \mathbb{R}</script> defined as <script type="math/tex">f(x) = \vert x \vert</script>, the value <script type="math/tex">f'(0)</script> does not exist. However, <script type="math/tex">f'_-(0) = -1</script> and <script type="math/tex">f'_-(0) = 1</script>.</p>
<p>It should be clear that if <script type="math/tex">f'(c)</script> exists for some <script type="math/tex">c \in \mathbb{R}</script>, we have <script type="math/tex">f'_-(c) = f'_+(c) = f'(c)</script>. At the same time, if <script type="math/tex">f'_-(c) \ne f'_+(c)</script>, the derivative is not defined at <script type="math/tex">c</script> (just like the case of <script type="math/tex">\vert x \vert</script> in 0).</p>
<p>Now, we can proceed to our main result, which can be considered as the one-sided version of the Fundamental Theorem of Calculus:</p>
<blockquote>
<p><em>Theorem (CSK’s FTC).</em> Let <script type="math/tex">f</script> be a (Riemann) integrable function on <script type="math/tex">[a, b]</script>. Define <script type="math/tex">F: [a, b] \to \mathbb{R}</script> such that <script type="math/tex">F(x) = \int_a^x f(t) \, dt</script>. Then, if <script type="math/tex">\lim_{x \to c^-} f(x)</script> exists, we have <script type="math/tex">F'_-(c) = \lim_{x \to c^-} f(x)</script>.</p>
<p>Similarly, if <script type="math/tex">\lim_{x \to c^+} f(x)</script> exists, we have <script type="math/tex">F'_+(c) = \lim_{x \to c^+} f(x)</script>.</p>
</blockquote>
<p>There are some interesting results that can be obtained. For example, it can be proven directly that <script type="math/tex">f</script> is continuous at <script type="math/tex">x_0</script> if and only if <script type="math/tex">F</script> is differentiable at <script type="math/tex">x_0</script>. Hence, we know that if <script type="math/tex">g: \mathbb{R}^+ \to \mathbb{R}</script> such that <script type="math/tex">g(x) = \int_0^x \lfloor x \rfloor \, dx</script>, <script type="math/tex">g</script> is not differentiable at <script type="math/tex">\mathbb{N}</script>. In fact, <script type="math/tex">g'_-(x) = \lceil x \rceil - 1</script> and <script type="math/tex">g'_+(x) = \lfloor x \rfloor</script>.</p>
<p>Also, note that it is required for <script type="math/tex">\lim_{x \to c^-} f(x)</script> exists (similarly for the right-hand version). For example, one can take <script type="math/tex">f: [0, 1] \to \mathbb{R}</script> such that <script type="math/tex">f(x)</script> equals 1 if <script type="math/tex">x</script> is in the form of <script type="math/tex">\frac{1}{2^k}</script> for <script type="math/tex">k \in \mathbb{N}</script>, and 0 otherwise. By Riemann sums one can prove that <script type="math/tex">\int_0^1 f(x)\, dx = 0</script>, yet <script type="math/tex">\lim_{x \to 0^+} f(x)</script> is undefined.</p>
<p>It is also interesting to note that this is also slightly different from the Fundamental Theorem of Calculus usually taught in high schools. Normally, the Fundamental Theorem of Calculus refers to this theorem:</p>
<script type="math/tex; mode=display">\int_a^b f'(x) \, dx = f(b) - f(a).</script>
<p>By CSK’s FTC proving this is obvious, since <script type="math/tex">F(b) = \int_a^b f(t) \, dt</script> and <script type="math/tex">F(a) = 0</script> in CSK’s FTC notations.</p>
<p>Anyways, let’s prove CSK’s FTC! Here I will only prove the right-sided version, since the left-sided version is very similar. Let <script type="math/tex">\lim_{x \to c^+} f(x) = L</script> exist. Notice that</p>
<script type="math/tex; mode=display">% <![CDATA[
\begin{align*}
\left\vert F'_+(c) - L \right\vert &= \lim_{h \to 0^+} \left\vert \frac{F(c + h) - F(c)}{h} - L \right\vert \\
&= \lim_{h \to 0^+} \left\vert \frac{\int_a^{c + h} f(t) \, dt - \int_a^{c} f(t) \, dt}{h} - L \right\vert \\
&= \lim_{h \to 0^+} \left\vert \frac{\int_c^{c + h} f(t) \, dt}{h} - L \right\vert \\
&= \lim_{h \to 0^+} \left\vert \frac{\int_c^{c + h} f(t) \, dt - hL}{h} \right\vert \\
&= \lim_{h \to 0^+} \left\vert \frac{\int_c^{c + h} \left( f(t) - L \right) \, dt}{h} \right\vert \\
&\le \lim_{h \to 0^+} \frac{\int_c^{c + h} \left\vert f(t) - L \right\vert \, dt}{h} \\
\end{align*} %]]></script>
<p>Now if <script type="math/tex">\lim_{x \to c^+} f(x) = L</script> exists, that means that for all <script type="math/tex">\varepsilon > 0</script> there exists <script type="math/tex">\delta > 0</script> such that <script type="math/tex">% <![CDATA[
c < x < c + \delta %]]></script> implies <script type="math/tex">% <![CDATA[
\vert f(x) - L \vert < \varepsilon %]]></script>. This would mean that if <script type="math/tex">% <![CDATA[
0 < h < \delta %]]></script>, we have <script type="math/tex">\int_c^{c + h} \left\vert f(t) - L \right\vert \, dt \le h\varepsilon</script>. Taking <script type="math/tex">h \to 0^+</script>, we have that <script type="math/tex">\left\vert F'_+(c) - L \right\vert</script> goes to 0 as well, and hence, <script type="math/tex">F'_+(c) = L = \lim_{x \to c^+} f(x)</script>. We are done.</p>Herbert Ilhan TanujayaLast semester (AY16/17 sem 1) I had a chance to take MA3110S under Prof. Chua Seng Kee. SPM (Special Programme in Math) modules are scary, but Prof. CSK has managed to up it into another level, giving things like generalized versions of theorems in the textbooks as well as his scary true/false questions! I really learned a lot under his guidance. (Your life in SPM won’t be complete if you haven’t taken any of his modules haha)