Jekyll2023-12-28T10:48:52-05:00http://horatiulazu.ca/blog//Horatiu’s BlogMy blog, discussing various topics from my life.
A week cycling in Bormio and the Alps2023-07-31T19:49:29-04:002023-07-31T19:49:29-04:00http://horatiulazu.ca/blog/general/2023/07/31/alps-cycling<p><img src="/blog/assets/stelviosummit.jpg" /></p>
<p>In July 2023 I had the opportunity to spend a week in <a href="https://en.wikipedia.org/wiki/Bormio" target="_blank">Bormio, Italy</a> and go road biking in the Alps. The northern region of Lombardy is filled with iconic climbs and history.</p>
<p>For reference, road cycling climbs are often characterized by several properties:</p>
<ol>
<li><strong>Altitude</strong> - the difficulty in riding increases as altitude increases due to the lack of oxygen. For example, beyond 2500m there is ~25% less oxygen available compared to sea level.</li>
<li><strong>Gradient</strong> - the average slope of the climb. Inclines ranging from 3-5% are considered mild, 5-7% moderate and 7%+ is generally considered difficult.</li>
<li><strong>Distance</strong> - the majority of climbs are between 10-40km from beginning to summit.</li>
</ol>
<p>The Tour de France provides a classification for climbs, going from Category 4 (easiest) to Category 1 (hardest), with a special category <a href="https://en.wikipedia.org/wiki/Hors_cat%C3%A9gorie">HC (Hors Catégorie)</a> reserved for climbs originally “unpassable by car”.</p>
<h2 id="day-1-lago-di-cancano">Day 1: Lago di Cancano</h2>
<p><strong>28.32km, 762m elevation gain (Category 2)</strong></p>
<p>To warm-up and test the rental bike, a local ride up to Lago di Cancano was worthwhile. The ride was relatively short but it featured a Category 2 climb reaching 1938m elevation with a 6.9% grade. Notably, even the easiest ride around Bormio is steeper and nearly twice the elevation gain of Bear Mountain in NY.</p>
<p><img src="/blog/assets/cancano.jpg" />
<br /></p>
<p><img src="/blog/assets/alpine-dam-solo.jpg" /></p>
<h2 id="day-2-stelvio-pass--umbrail-pass">Day 2: Stelvio Pass + Umbrail Pass</h2>
<p><strong>105.14km, 3163m elevation gain (2x HC climbs)</strong></p>
<p>The Stelvio Pass is the highest elevation mountain pass in the Eastern Alps (2757m tall), and the 2nd overall standing 7m below France’s Col de l’Iseran. It was built in the 1800s by the Austrian empire to connect Lombardy with the rest of Austria. It is featured regularly as the Cima Coppi of the Giro D’Italia.</p>
<p>There are three ways to get to the summit - riding the Stelvio from Bormio (Italy), Prato (Italy), or Umbrail Pass from Val Müstair (Switzerland).</p>
<p><img src="https://i.imgur.com/JEZPwQR.png" /></p>
<blockquote>
<p>Figure 2.1: In this ride, I climbed from Bormio, descended Umbrail Pass into Switzerland, then went back up from Prato to the summit and then descended back to Bormio.</p>
</blockquote>
<p>Aside from a short section with 13-14% slope it was fairly managable. I managed to do it without stopping except for a small photo break.</p>
<p><img src="/blog/assets/bormio.jpg" /></p>
<p>After doing 40 hairpin turns, taking a left-turn on the SS38 will lead to the Umbrail Pass which is the highest elevation mountain pass in Switzerland descending down to the region of Val Müstair. After arriving at Val Müstair it was flat for a short period, I ate a quick sandwhich and then prepared for the final monster which is the 1893m climb to the Stelvio summit from Prato.</p>
<p>In combination with fatigue from the earlier climb, I found this HC climb absolutely brutal.</p>
<p><img src="/blog/assets/stelvio-profile.png" /></p>
<p>Things took a turn for the worse at around 14km into the climb once reaching the forest area where gradients hovered around 8-10% bend after bend. After passing 2000m elevation the altitude really sunk in making it even more miserable, and my right knee was hurting on each pedal stroke. Unfortunately, I had no option but to continue because my hotel was on the other side of the mountain…</p>
<p>Getting out of the forest and seeing the hairpin turns spiraling up to the Stelvio peak made summitting seem like an unachievable feat in the moment. Even pushing 150 watts (a relatively light power output for me) was difficult due to fatigue and lack of oxygen. Though at the end, after completing the 48th bend and 3.5 hours of suffering to reach the summit, words cannot describe the experience of joy (and relief).</p>
<p><img src="/blog/assets/stelvio.jpg" /></p>
<p><br /></p>
<p><img src="/blog/assets/stelvio-east.jpg" /></p>
<h2 id="day-3-sondalo">Day 3: Sondalo</h2>
<p><strong>42.46km, 550m elevation gain</strong></p>
<p>This was a recovery day, featuring an easy ride to Sondalo and back through the country roads. It was relatively flat, with one Category 4 climb (7.8% with 101m elevation gain) but otherwise easy.</p>
<h2 id="day-4-bormio-2000">Day 4: Bormio 2000</h2>
<p><strong>17.3km, 577m elevation gain (HC -> Category 2)</strong></p>
<p>Originally Day 4 was supposed to be Gavia pass, but unfortunately there was a thunderstorm scheduled at noon. The Bormio 2000 climb was nearby, which was also an HC climb with an 8.7% average gradient and 817m elevation gain, so that was promising. Sadly, the weather took a turn for the worse even earlier so I had to turn around a couple of minutes before reaching the summit.</p>
<p>Luckily I got back to the hotel minutes before the storm started, so that was good.</p>
<p><img src="/blog/assets/bormio2000.jpg" /></p>
<h2 id="day-5-passo-di-gavia">Day 5: Passo di Gavia</h2>
<p><strong>51.82km, 1422m elevation gain (HC climb)</strong></p>
<p>The weather improved in Day 5 so took an attempt at Gavia Pass - the 10th highest elevation pass in the Alps reaching 2621m elevation. It connects the Sondrio in the north with Ponte di Legno in the south.
<img src="/blog/assets/gavia-profile.png" /></p>
<p>The climb wasn’t overly difficult, starting off rather mild until Santa Caterina. Afterwards, it became more challenging because of short pitches that were often 10-14%. As a prize there was a nice view at the end :)</p>
<p><img src="/blog/assets/gavia.jpg" /></p>
<h2 id="recap--forward-look">Recap & Forward Look</h2>
<p>These were an enjoyable set of rides, and the views were worth the pain. If I were to do it again, I would take into consideration:</p>
<ol>
<li><strong>Bringing my own road bike</strong> - firstly the rental was expensive and also worse quality than my personal road bike (Cervélo S5). Second, it put a lot of pressure to ride everyday to make the most of the rental when I could have hiked on off days to recover better from rides.</li>
<li><strong>Attempt Mortirolo</strong> - while not as high elevation as the other climbs, Mortirolo is another one of the famous climbs that is often considered extremely difficult due to its ~10.5% steep gradient. Lance Armstrong said in an interview that it was the hardest climb he ever did.</li>
<li><strong>Study the descents beforehand</strong> - during the descent from Bormio 2000 I went into a hidden hairpin turn at ~52km/h and skidded off the road (luckily didn’t fall off a cliff…) - it is difficult to memorize the map but knowing key corners can help to avoid excessive or lack of braking before corners.</li>
</ol>
<p>Next year the plan is to attempt Mount Evans and Pike’s Peak in Colorado. They are the highest (and 2nd highest) elevation paved mountain peaks in North America surpassing 4000m altitude, so it is an exciting challenge to train and look forward to!</p>
<h2 id="youtube-descents">YouTube Descents</h2>
<p>I recorded descents for all the climbs and overlayed the Strava data - enjoy.</p>
<h4 id="day-2-descents-stelvio--umbrail-pass">Day 2 Descents (Stelvio + Umbrail Pass)</h4>
<iframe width="560" height="315" src="https://www.youtube.com/embed/XTSSt4Z2_hg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<p><br /></p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/V9bMEa7RDe4" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>
<p><br /></p>
<h4 id="day-5-descent-gavia-pass">Day 5 Descent (Gavia Pass)</h4>
<iframe width="560" height="315" src="https://www.youtube.com/embed/Y3dTcvyWmYs" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share" allowfullscreen=""></iframe>Visiting Grand Canyon, Death Valley, and Zion National Parks2023-04-28T19:49:29-04:002023-04-28T19:49:29-04:00http://horatiulazu.ca/blog/travel/2023/04/28/canyon-np<p>In April 2023 I flew out with friends to spend a week visiting several National Parks. We started in Pheonix and visited the Grand Canyon, Las Vegas, Death Valley, and Zion.</p>
<h2 id="grand-canyon">Grand Canyon</h2>
<p>First stop was the Grand Canyon, where we planned to hike down to the Colorado River and back in one day, otherwise known as the “Rim to River to Rim” hike. The hike itself is ~14.83mi and 4675ft elevation gain, starting from South Kaibab and going to Bright Angel along the Southern Rim of the Grand Canyon.</p>
<p><img src="/blog/assets/gc-warning-sign.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 1.1: Warning sign at the trailhead of South Kaibab, stating to not hike down to the Colorado River and back in one day, but we chose to ignore it. More on this later…</p>
</blockquote>
<p>We arrived at the trailhead by around 5:00am local time, before sunrise, where we patiently waited to start the hike. The trail was nearly empty at the time!</p>
<p><img src="/blog/assets/gc-kaibab-2.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 1.2: View from near the Ooh Aah Point (6660 ft elevation)</p>
</blockquote>
<p>Descending down the South Kaibab trail was enjoyable because it is a fairly gradual slope, so it is not as hard on the knees. Different sections of the descent brought unique views and perspective.</p>
<p><img src="/blog/assets/gc-kaibab-1.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 1.3: View around 2 hours into the descent on the South Kaibab trail, near Cedar Ridge</p>
</blockquote>
<p><img src="/blog/assets/gc-colorado-river.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 1.4: The Colorado River at the bottom of the Grand Canyon. The water wasn’t the cleanest!</p>
</blockquote>
<p>Unlike a mountain summit where if the difficulty climbing presents to be too difficult you could elect to turn around early and descend, here that isn’t an option. Furthermore, it is hot at the bottom of the canyon.</p>
<p>I didn’t have any issues going back up, but my friends found it to be a streneous effort. I went ahead and left two of my friends to go up together behind me. Unfortunately, one of them <a href="https://www.trainerroad.com/blog/what-is-bonking-causes-dangers-and-prevention/" target="_blank">bonked</a> (ran out of glycogen), resulting in dizziness, nauseau, and fatigue.</p>
<p>This quickly became a pressing issue because as I was waiting at the top, it was getting dark and my friends were nowhere to be found! One of them eventually made their way up, but my friend who bonked was still in the Canyon. Luckily, he eventually got cellular signal closer to the top of the Canyon and we were able to get into contact with him. He made it right as it was getting dark.</p>
<p><img src="/blog/assets/gc-bright-angel-summit.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 1.5: Near the summit after climbing up Bright Angel. To my surprise, there were still snow and ice patches near the top (which would make sense considering it’s at nearly 6750ft elevation)</p>
</blockquote>
<div class="strava-embed-placeholder" data-embed-type="activity" data-embed-id="8909537913"></div>
<script src="https://strava-embeds.com/embed.js"></script>
<p>Overall, the Grand Canyon was a great experience and doing the “Rim to River to Rim” hike showcases a wide array of scenic views and sceneries that make it worthwhile effort!</p>
<h2 id="las-vegas">Las Vegas</h2>
<p>As an intermediate to the next leg of the trip, we drove to Las Vegas as our new “headquarters”. Personally, I did not find Las Vegas very enjoyable, but some of the shows were moderately interesting. The main turnoff for me was that smoking was common everywhere, and to get into our hotel we passed through a casino that was smoke-filled…</p>
<p><img src="/blog/assets/vegas.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 2.1: The Welcome to Fabulous Las Vegas sign. It was a bit out of the way, though naturally it was on the edge of the city.</p>
</blockquote>
<p><img src="/blog/assets/hoover-dam.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 2.2: Quick stop by Hoover Dam which was along the way to Las Vegas</p>
</blockquote>
<h2 id="death-valley">Death Valley</h2>
<p>We then ventured over to California to checkout Death Valley. This day trip showed some interesting view-points. It wasn’t that warm in California as advertised.</p>
<p><img src="/blog/assets/death-valley-2.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 3.1: Zabriskie Point</p>
</blockquote>
<p><img src="/blog/assets/death-valley-3.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 3.2: Badwater Basin - the lowest point in North America (282ft below sea level) comprised mainly of sodium chloride (table salt)</p>
</blockquote>
<p><img src="/blog/assets/death-valley-6.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 3.3: Sand Dunes</p>
</blockquote>
<p><img src="/blog/assets/death-valley-5.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 3.4: Dante’s Point overlooking the Badwater Basin (5575ft)</p>
</blockquote>
<h2 id="zion">Zion</h2>
<p>Zion has a series of flagship hikes including “Angel’s Landing” and “The Narrows”. Unfortunately, we didn’t have a reservation for Angel’s Landing and the water was not in good condition for “The Narrows”. This wasn’t an issue, as there was still plenty to visit in our two days at the park!</p>
<p>First, we did the Observation Point via East Mesa Trail hike. This is arguably the best viewpoint at the park, though due a rockslide you cannot hike from the base of Zion but go through the East Mesa trail instead.</p>
<p><img src="/blog/assets/zion-1.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 4.1: View from observation point. You can see Angel’s Landing</p>
</blockquote>
<p>The actual hike was not very enjoyable. There was a lot of snow melt, which made it impassible by car to the trail-head so we endured an extra 4.3 mile on the hike because we had to park at the Zion Ponderosa ranch, while making the actual traversal tricky. It was relatively flat with no view-points until the very end.</p>
<div class="strava-embed-placeholder" data-embed-type="activity" data-embed-id="8928090388" data-style="standard"></div>
<script src="https://strava-embeds.com/embed.js"></script>
<p><img src="/blog/assets/observation-point-mud.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 4.2: One of many cars that failed to make it to the trail-head. There was a truck waiting on the private road that charges $1000 to tow any stuck cars (the day we went was quite profitable)</p>
</blockquote>
<p>The rest was a constellation of smaller hikes we chained together, including Pa’arus, Emerald Pools (low/mid/upper), and Riverside Walk.</p>
<p><img src="/blog/assets/zion-2.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 4.3: Sunrise overlooking the Virgin River after running ~5K from Springdale (and the Pa’arus trail). There were many wild animals before sunrise, running in the dark was kind of sketchy</p>
</blockquote>
<p><img src="/blog/assets/zion-waterfall.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 4.4: Lower Emerald Pools</p>
</blockquote>
<p><img src="/blog/assets/zion-emerald.jpg" alt="I'm an inline-style link" /></p>
<blockquote>
<p>Figure 4.5: Upper Emerald Pools. The hike was light but the views paled in comparison to Observation Point, and it was very crowded. Some parts of the hike had a queue…</p>
</blockquote>
<h2 id="conclusion">Conclusion</h2>
<p>The trip was an exciting turn of hikes across many picturesque canyons and vistas. However, this only scratched the surface, with plenty of trails and parks left to visit in the area such as Bryce Canyon and Antelope Canyon … so a lot to look forward to!</p>In April 2023 I flew out with friends to spend a week visiting several National Parks. We started in Pheonix and visited the Grand Canyon, Las Vegas, Death Valley, and Zion.Living in Toronto for Eight Months (COVID-19 Edition)2020-12-31T18:49:29-05:002020-12-31T18:49:29-05:00http://horatiulazu.ca/blog/general/2020/12/31/review-2020<h3 id="what-happened">What happened?</h3>
<p>If you asked me a year ago I would be pretty shocked to see this title on my blog, knowing that I was not supposed to be in Toronto for even a single month in 2020. As you are all probably aware, COVID-19 led to many internships and school terms turning virtual and remote, and I followed suit with this.</p>
<p><img src="https://i.imgur.com/6opeWEI.jpg" alt="" /></p>
<p>Luckily, aside from location and being remote I wasn’t very impacted, and I am fortunate to have a safe place to live and continue my studies along with my parents at home.</p>
<p><img src="https://i.imgur.com/O42ktcy.jpg" alt="" /></p>
<h3 id="april---may-2020">April - May 2020</h3>
<p>This period marked the gap between my Facebook and Google internships. During this time, I received my first full-time return offer from Facebook which was very exciting, and I could be rest assured that despite the pandemic taking place, I could still secure full-time employment at a top choice company for me.</p>
<p>I also had some spare time to explore numerous side-projects. Several notable projects include:</p>
<p>1) Exploring the field of Real-Time Communication at large, which originated from a report I wrote for school in relation to my term at Facebook, and later I greatly expanded it out of personal interest. Feel free to <a href="http://www.horatiulazu.ca/blog/assets/blog-report.pdf" target="_blank"> download</a> the paper.</p>
<p>2) Further learning of React and Selenium web-scraping to create a data visualization site for WaterlooWorks statistics. The project involved scraping data off of Waterloo’s job board, cleaning the data and placing it on a CDN along with making the responsive site. This is my first mobile responsive site and I’m fairly proud of it. Check it out <a href="http://horatiulazu.ca/coop" target="_blank">here</a>!</p>
<p><img src="https://i.imgur.com/TojfM67.png" alt="" /></p>
<p>3) Creating the initial services for a major overhaul of my personal site and side-projects.</p>
<p>I will not go into details here yet (stay tuned in 2021!) but the essence is that I am working on creating my own Raspberry Pi farm to host all my side-projects, while having failover onto AWS. This distributed cross-cloud and micro-services architecture was motivated to allow me to reduce costs on AWS EC2 instances, while maintaining reliability and flexibility. It is an ambitious project that I plan to continue for fun on weekends once I start full-time.</p>
<p>During this time period, I focused on creating the low-level C++ server that leverages Boost.asio to create a multi-threaded HTTP reverse-proxy and handler for different types of requests, allowing for rerouting and redirection capabilities similar to Apache or NGINX. The multi-threaded server is capable of handling around 1350 requests per second on my 2012 MacBook Pro, and is configurable to most of my use-cases. The bulk implementation was completed, alongside a basic telemetry system with heart-beats that provides analytics on particular service health.</p>
<h3 id="may---august-2020">May - August 2020</h3>
<!-- ![](http://www.horatiulazu.ca/assets/images/GoogleLogoLarge2.png) ![](http://www.horatiulazu.ca/assets/images/firebaselogolarge.png) -->
<h4 id="google-internship">Google Internship</h4>
<p>This period defines my Google internship on Firebase SDKs. While my last co-op was unfortunately virtual, the Google and Firebase culture certainly bled through and it was an absolute joy to work at such an extremely Engineering focused company. Special shoutout to my host and everyone else at Firebase!</p>
<p><img src="https://i.imgur.com/bZliosf.png" alt="" /></p>
<p><img src="https://i.imgur.com/QUUOpMG.png" alt="" /></p>
<p><img src="https://i.imgur.com/e5YoyGH.png" alt="" /></p>
<p>During my term at Firebase I focused on the server-sided Admin SDKs. In my first month I <a href="https://github.com/firebase/firebase-admin-dotnet/commit/d5a7a3f1b34019f5656331b81e5a6f832f7f7a48">built</a> and <a href="https://twitter.com/FirebaseRelease/status/1273362463627579394">shipped</a> the ability to bulk import users into Firebase using the C# SDK. The remainder of my term focused on designing and prototyping the next-generation Firebase Node.js Admin SDK.</p>
<p>In particular, I researched best-practices on TypeScript (such as ECMAScript module export edge-cases) along with lessons learned from other SDKs to paint a picture on how the future SDK could look like. After careful evaluation of several designs, with heavy support of my host and members of adjacent teams, alongside 40+ pages of design documentation and several prototypes, I am proud to confirm that the formal proposal was approved (alongside some minor changes) and sets the future overarching design for the most popular Firebase server-sided SDK.</p>
<p>Alongside writing lots and lots of documents and prototypes, I started a new initiative to auto-generate typings from source. Previously, the Node.js SDK would have a separate <code class="highlighter-rouge">d.ts</code> typings file for all TypeScript types manually annotated, which was required due to complex monkey-patching cache logic used to lazily load modules. This process is error-prone (I ended up finding and fixing >20 bugs related to this), and keeping them in sync can be non-trivial.</p>
<p>However, with some careful implementation, I refactored the majority of the SDK to auto-generate typings along with adopting several new open-source tools such as <a href="https://www.npmjs.com/package/@microsoft/api-extractor">API extractor</a> and reviewing the documentation generators to play nicely with the new typing scheme. Shortly after I left, the remaining work was completed and as of today all typings are auto-generated in the Node.js SDK.</p>
<h4 id="road-biking--tennis">Road Biking + Tennis</h4>
<p>During the 2020 cycling season, I managed to bike 3,786.3km across 60 rides, making it furthest distance I biked thus far in a year. Nearly every weekend, I’d ride 115-140km with my dad to Oakville and back, which was a significant contributor to this. Special shoutout to my friends as well who I went on group rides with!</p>
<!-- <iframe height='405' width='590' frameborder='0' allowtransparency='true' scrolling='no' src='https://www.strava.com/activities/3850798877/embed/b01315ac5386fdd4fc044ae071a7c155abd4f4fe'></iframe> -->
<p>My previous cycling distance record was 209km set the day after graduating high-school. This year, I broke that record by riding 245.6km with an average speed of 28.0km/h from Toronto to St. Catherines and back.</p>
<iframe height="405" width="590" frameborder="0" allowtransparency="true" scrolling="no" src="https://www.strava.com/activities/3918950301/embed/397f769f210f413cc042100a6215b946218b31db"></iframe>
<!-- ![](https://i.imgur.com/IVSRatG.png) -->
<p>Of course, I could not use that as an excuse to not do my annual Niagara Falls bike ride, so here it is! The following ride was pretty cute as well, thought it’d mention a shoutout. If you want to do that route, keep in mind the east side is 10x more photogenic than the west.</p>
<iframe height="405" width="590" frameborder="0" allowtransparency="true" scrolling="no" src="https://www.strava.com/activities/3686058913/embed/32babc0a122a1cc1a04bb84344ae03731b7e9dd3"></iframe>
<iframe height="405" width="590" frameborder="0" allowtransparency="true" scrolling="no" src="https://www.strava.com/activities/4017632194/embed/23234310b9e3cc1414352748115fe4e775ea64a6"></iframe>
<p>In terms of other sports, I played tennis twice per week with several friends, and just went on regular walks.</p>
<h4 id="youtube-learn-in-5-minutes-series">YouTube (Learn in 5 Minutes Series)</h4>
<p>If you subscribe to me on YouTube, you would likely be aquainted with my “Learn in 5 Minutes” YouTube series. The objective is/was to provide short clips introducing viewers to a topic sufficiently for obtaining background in it. Every week I would post a video either related to algorithms or software engineering.</p>
<p>During the summer (and a bit into fall) I created 27 videos spanning numerous topics including number theory algorithms (sieves, modular combinatorics) and in-depth view at software engineering concepts such as HTTP header compression (HPACK/QPACK). Feel free to visit the repository <a href="https://github.com/SoftwareEngenius/five-minute-tutorials"> repository</a> or <a href="https://softwareengenius.github.io/">website</a>.</p>
<iframe width="560" height="315" src="https://www.youtube.com/embed/R9mnjPgDCQk" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<iframe width="560" height="315" src="https://www.youtube.com/embed/ahYgoV8MDtg" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen=""></iframe>
<p>Given my experience with YouTube, I expected two things:</p>
<ol>
<li>
<p>I will receive very few views at the beginning, particularily because my videos were traditionally scattered and my subscribers are saturated across various interests</p>
</li>
<li>
<p>The occasional video will rank well, and consequently consistently receive views</p>
</li>
</ol>
<p>And turns out my hypothesis served correct so far. Broadly speaking, I have two types of videos: either it is consistently growing in views or it is dormant. Unfortunately, the vast majority of videos are in the second camp, but some still receive views and I am optimistic that they will receive more views later on.</p>
<p><img src="https://i.imgur.com/XEQuNp8.png" alt="" /></p>
<p><img src="https://i.imgur.com/quIkCpO.png" alt="" /></p>
<p>This entire project was extremely time consuming, and occasionally I wondered “is it worth it?”. For many of the topics chosen, I did not have expertise before the videos, so I would need to familiarize myself with the concepts to deliver a presentation, create the presentation, record, edit and upload. To upload at a cadence of one video per week, I would need to spend at least 6hrs/week creating videos.</p>
<p>I will not hide the fact that delivering content every week became a stressful ordeal at times.</p>
<p>However, the answer to this question is still yes. Even in the case of nobody watching the videos, at the very least I learned something new. All the topics in my videos are topics that I was always interested in but did not have the time to learn them comprehensively, and through the videos it motivated me to take a deeper look. Reading through the QPACK draft because there is very little online documentation on the bleeding edge of the standard takes some level of dedication. Secondly, the views are bonus, and I am glad when my viewers learn something new from my videos :)</p>
<p>I’m a fairly firm believer that building fundamentals is important, and this was an example of this. I am glad to confirm for more than half the videos, I already applied the particular knowledge gained in some area (Google conversion interview, univeristy course or casual conversation). My LZ77 compression video helped a friend with her interview (coincidentally that popped up!), HTTP XSS attacks were a core focus in CS 458 (security at Waterloo) and more. Moral of the story: learning is never a waste, and sometimes wish more people took this mindset instead of simply cramming and forgetting (i.e., Leetcode or school courses).</p>
<!--
#### Minecraft
You may or may not know that throughout high-school I built out a fairly large Minecraft map of a giant house, alongside a city in the "backyard". Throughout the years I continued to contribute to it here and there, but after my friend purchased Minecraft the idea of both working together on an island sounded interesting.
Kind of nostalgic but here's a video from 7 years ago with the past state (feel strange watching it now):
<iframe width="560" height="315" src="https://www.youtube.com/embed/0pnZlDTMflQ" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe>
The new world was situated on an island, originally we started in survival mode but over time got lazy and used creative mode more and more.
![](https://i.imgur.com/8g9sXOJ.png)
A core focus was automation, which I didn't really explore prior. Several examples include automated iron ores collection, a storage facility and farming. The options really do feel limitless in Minecraft.
![](https://i.imgur.com/FlVvGKR.png)
-->
<h3 id="september---december-2020">September - December 2020</h3>
<p>This period marks my 4A CS term. There is not much to outline for the academic term. As with many academic terms at Waterloo, my schedule quickly got busy as I had to conquer assignments from multiple classes with occasionally tricky deadlines. The school term for me was not too difficult, although it was time consuming because I took two “bird” courses that ended up having lots of essays to write.</p>
<p>I will outline one strategy that I used to keep myself at ease during the term, which was creating a structure during my days and trying to stick with it. Every night (morning?) at 12:00am I would make a TODO list for the next day, broken down into hourly chunks with goals. At the beginning of the term, I outlined hotspots with heavy assignment loads and ensured that I would start working on those assignments ahead of time.</p>
<h3 id="forward-looking-at-2021">Forward Looking at 2021</h3>
<p>Without a doubt, 2020 was a strange year. Next year will be significant for me as it marks the ending of my undergraduate time at the University of Waterloo, and the beginning of my full-time career as a Software Engineer. I am hoping to end off strong at Waterloo, have some rest, and start strong at Facebook in a team that will align with my interests.</p>
<p>As with any major transition period, while it comes with excitement for new beginnings it is critical to keep priorities aligned, goals set and move with an open-mind for potential changes and challenges.</p>
<p>I wish all the best for everyone in the New Year with lots of success, health and happiness.</p>What happened?RTC (Real-Time Communication) at scale2020-05-02T19:49:29-04:002020-05-02T19:49:29-04:00http://horatiulazu.ca/blog/coop/2020/05/02/rt-calling<p>I recently did an internship at Facebook where I was part of the RTC (Real-Time Communications) org, responsible for the products and infrastructure powering voice/video calling across Facebook’s family of apps (such as Messenger and Instagram). My team was responsible for the P2P and multiway calling protocols, including lower-level services written in C++ and platform integrating with partner teams in PHP.</p>
<p>This post will dedicate itself to discussing some of the protocols that go into audio and video calling.</p>
<p><strong>It is not related to Facebook itself in any way.</strong> <a href="/blog/assets/blog-report.pdf">You can view the PDF version of this post here. </a></p>
<hr />
<h2 id="table-of-contents">Table of Contents</h2>
<p><a href="#summary">Executive Summary</a></p>
<ol>
<li><a href="#introduction">Introduction</a></li>
<li><a href="#analysis">Analysis</a>
<ul>
<li>2.1 <a href="#overview">Overview of Signaling and Media Streaming</a></li>
<li>2.2 <a href="#history">History of Signaling and Media Streaming</a></li>
<li>2.3 <a href="#h323">ITU-T H.323 Protocols</a></li>
<li>2.4 <a href="#sip">Session Initiation Protocol (SIP)</a></li>
<li>2.5 <a href="#signal">Signal Protocol (formerly TextSecure Protocol)</a></li>
<li>2.6 <a href="#webrtc">WebRTC Framework</a></li>
<li>2.7 <a href="#rtsp">Real-Time Streaming Protocol (RTSP)</a></li>
<li>2.8 <a href="#rtcp">Real-Time Control Protocol (RTSP)</a></li>
<li>2.9 <a href="#rtp">Real-Time Transport Protocol (RTP)</a></li>
<li>2.10 <a href="#audiostreaming">Audio Streaming Codecs</a></li>
<li>2.11 <a href="#videostreaming">Video Streaming Codecs</a></li>
<li>2.12 <a href="#dash">Dynamic Adaptive Streaming over HTTP (DASH)</a></li>
<li>2.13 <a href="#multiway">Multiway Calling Architectures</a></li>
</ul>
</li>
<li><a href="#casestudies">Case Studies</a>
<ul>
<li>3.1 <a href="#skype">Skype’s P2P Signaling Protocol (2003)</a></li>
</ul>
</li>
<li><a href="#conclusion">Conclusion</a>
<ul>
<li>4.1 <a href="#result">Resulting architecture of RTC systems</a></li>
</ul>
</li>
</ol>
<h3 id="executive-summary">Executive Summary</h3>
<p>This post discusses and analyzes different protocols and technologies used in real-time communication systems by social media platforms. Real-time communication typically involves both a signaling and media stack. During signaling, actions are performed, such as ringing and hanging up, which often need to be relayed to users in an efficient manner. Similarly, media involves the encoding, relay and encryption of a stream of data, which includes audio and/or video. This post primarily focuses on a high-level overview of the different stacks, along with open-source protocols defined for use in this context. In addition, this post discusses different algorithms that are applied for media streaming, in the context of codecs and the perspective of performance.</p>
<p>The objective of this post it to raise familiarity with the challenges associated with real-time communications, and how different companies tackled problems related to it. In particular, this post focuses on the signaling and media streaming aspects, and not on underlying infrastructure details such as ensuring durability or reliability of hardware.</p>
<h3 id="introduction">Introduction</h3>
<p>The objective of real-time communication in the context of social media is to connect people from different avenues of the world together in a lifelike fashion. By allowing peer-to-peer or multiway communication through audio and/or video, users can connect in a personable way despite being geographically disperced. Billions of calls are performed everyday over the internet, including those from popular sites such as WhatsApp which support over 1 billion daily users (Mansoor Iqbal, 2020). During crisis and major world-events (such as COVID-19), real-time communication and social media can truly bring the world together in a safe manner.</p>
<p>Numerous technical challenges arise from designing scalable systems that handle users from different regions. First includes how to initiate the call and inform users that another user wishes to connect. Afterwards, there needs to be a direct connection established to allow for a reliable stream of data to be sent to the recipient(s). Moreover, there are variable network conditions that need to be accounted for such as low bandwith, unreliable connections or firewalls. In addition, there is the consideration of encrypting data sent over the network, and ensuring privacy of users by designing robust systems. This is just a small sample of the set of challenges faced by software engineers, architects and site reliability engineers in designing the world-class systems powering the large-scale social media websites of today.</p>
<h3 id="analysis">Analysis</h3>
<h4 id="overview-of-signaling-and-media-streaming">Overview of Signaling and Media Streaming</h4>
<p>The high-level goal of real-time communication is to connect a party of people together such that they can communicate with low latency. To achieve this, two connections are established: signaling and media. Signaling is the process of exchange information between different parties, typically associated with events or actions. Media is the actual media being sent, which includes audio and/or video.</p>
<p><img src="https://i.imgur.com/vm5jGcT.png" /></p>
<blockquote>
<p>Figure 1: The simplest form of peer-to-peer communication.</p>
</blockquote>
<p>Signaling is primarily used for session control, network data, media metadata and key material. Session control involves using messages to open, modify and close communication channels. Network data consists of any information for connecting the endpoints, such as obstructions in access. Media metadata is used to transmit metadata such as media types supported, codecs, along with encryption schemes used. Lastly, key material is used for sending encryption keys used to decode sensitive data sent over the network.</p>
<p>Media streams are used to send the data that users want to experience, such as audio and/or video. Audio and video media is typically encoded using different codecs and can be fine tuned depending on network conditions. Moreover, audio and video media is often susceptible to packet loss, as performance is more critical than losing individual packets of information. Various techniques have been researched and developed over the years to allow for more effective lossy/loseless compression, improved sound quality (echo cancellation, jitter avoidance, etc.) and remidiation of intermittent packet loss.</p>
<h4 id="history-of-signaling-and-media-streaming">History of Signaling and Media Streaming</h4>
<p>Signaling is not a modern concept - in fact, it was first introduced in the 1800’s as a major component of telephone systems. Electric pulses and audible tones are used to transmit information such as busy signalings, addressing, dial tones and to request service. The rotary dial in 1896, which was later formalized by 1910, introduced the first design of performing such requests which relayed electrical pulses in current flow (“Development of long-distance transmission”, n.d.). Over time, the standards incrementally envolved to take into consideration overseas transmission, newer transmission mediums such as coaxial cabling and applied techniques such as FDM (frequency division multiplexing) to best utilize existing infrastructure.</p>
<p>In contrast to telephone communication, real-time communication used by social media websites typically uses VoIP (Voice over Internet Protocol) versus PSTN (public switched telephone network). While general concepts remain consistent (such as signaling, connection setup, digitization of analog signals via quantization), data is transmitted over a packet-switched network versus circuit-switched network.</p>
<p>The first providers of VoIP mirrored solutions of the legacy telephone sys- tems. Second generation providers including Skype adopted large-scale closed networks providing free calling capability with the ability to tap into PSTN
networks if preferred at cost. The latest generation leverages federated VoIP, which is a form of voice telephony using packets across autonomous domains without requiring switching centers or centralized virtual exchange points. An example would include Google Talk which uses ENUM (E.164 Electronic Number to URI Mapping standard), acting similarly to DNS record types where a telephone number would map to a URI or IP address.</p>
<p>VoIP calling is implemented using a combination of propriertary and open-source protocols. The foundation was formed in 1928 by Homer Dudley at Bells Labs creating the first electronic voice synthesizer (known as Vocoder). Later, ARPANET (Advanced Research Project Agency) built the first packet-switched network in 1969. By 1973 the first voice data packet was transmitted by MIT, followingly the first audio codec was approved by 1988 (G.722), and then finally by 1991 Autodesk released the first VoIP application (known as Netfone) to the public domain. Development increased substantially after that point, leading to Free World Dialup in 1994 and the first-for-profit VoIP application VocalTec in 1995 (Robert Pepper, 2014).</p>
<p>The H.323 system specification, instated in 1996, is recommended by the ITU Telecommunication Standardization Sector (ITU-T) as a stack of proto- cols used to provide audio/video communication on packet-switched networks (Margaret Rouse, n.d.). The standard addresses call signaling, control, mul- timedia transport and control, along with bandwith control for peer to peer and multiway conferences. As part of the H.323 family of telecommunication protocols includes H.225.0 RAS (Registration, Admission, Status), H.225.0 Calling Signaling, H.245 control protocol and Real-Time Transport Protocol (RTP) for delivering media.</p>
<p>SIP (Session Initiation Protocol) was originally standardized in 1999, and is designed to dictate signaling and call setup protocols for IP-based communications (Tien-Thinh Nguyen, Christian Bonnet, 2016). Unlike H.323, which is was standardized by the International Telecommunication Union (ITU), SIP was formalized by the Internet Engineering Task Force (IETF), hence distin- guishing its roots in the internet community rather than telecommunications industry. SIP works in conjunction with other protocols including SDP (Ses- sion Description Protocol) to negotiate and establish call state.</p>
<p>With respect to streaming media, MPEG-DASH (Dynamic Adaptive Streaming under HTTP, also known simply as DASH) was published in 2012 as the only adaptive bit-rate HTTP streaming solution recognized internation- ally for delivering variable bitrate content. This codec agnostic solution is used by YouTube, Netflix and more. Popular codecs used today for encod- ing video streams in real-time communication includes H.264 (MPEG-4 AVC) introduced in 2003 constituting the majority of tra c, and H.265 (MPEG-H HEVC) introduced in 2013.</p>
<h4 id="itu-t-h323-protocols">ITU-T H.323 Protocols</h4>
<p>H.323 is a standard that defines a series of protocols to provide audio/video communication sessions across packet-switched networks. The standard, created in 1996 by ITU-T (ITU Telecommunication Standardization Sector) addresses call control and signaling, multimedia control and transport, along with bandwith control for both peer-to-peer and multiway conferences (Margaret Rouse, n.d.). While originally created for use in LAN networks, it was quickly adopted for a variety of IP networks including WANs and the greater internet. The standard has since been iterated on, last updated in 2009, while remaining entirely backwards compatible since its first version. It resides on TCP port 1720.</p>
<p>Firstly, several definitions will be addressed.</p>
<p><strong>Definition. 2.1.</strong> Terminal The fundamental element in a H.323 system representing a device users would typically use (such as a phone).</p>
<p><strong>Definition. 2.2.</strong> Multipoint Control Unit (MCU) Device responsible to acting as a conference bridge and allows mixing both video/audio.</p>
<p><strong>Definition. 2.3.</strong> Gateway Device responsible for enabling communication between H.323 networks and other network types (such as PSTN). An example of using gateways includes allowing enterprise IP phones to communicate with external users via PSTN.</p>
<p><strong>Definition. 2.4.</strong> Gatekeeper An optional component across the H.323 network which provides a series of services to terminals and gateways. Examples include endpoint registration, admission control, and user authentication.</p>
<p>Gatekeepers use one of two signaling modes: direct and gatekeeper routed. When direct, endpoints use RAS protocol to learn the endpoint of the remote endpoint and a call is established directly with the remote device. In contrast, when using gatekeeper routed the gatekeeper is used as the intermediary and all signaling passes through it.
Definition. 2.5. Zone The set of endpoints registered to a single Gatekeeper in a H.323 system.</p>
<p>After the address of the remote terminal is determined the initiating endpoint will leverage H.225.0 call signaling to establish communication with the remote terminal. Fast-connect procedures are defined in H.323 to establish calls using only 2-3 messages, reducing latency. Endpoints generally use RAS protocol to communite with a gatekeeper, while gatekeepers also use RAS protocol to connect with other gatekeepers.</p>
<p>A terminal would typically invoke a GRQ (Gatekeeper Request) message for discovering gatekeepers that are willing to accept messages. After, applicable gatekeepers would confirm (GCF) and the terminal node would pick its most desirable gatekeeper.</p>
<p>Upon sending a call initiation signal, the terminal node would perform an admission request (ARQ) to its preferred gatekeeper. In response, the resolves address is returned in the form of an admission confirm message (ACF). Now, the initiating terminal node will send the ARQ to the remote endpoint. The remote endpoint will then likewise send an ARQ and receive an ACF from its preferred gatekeeper. This step is required for ensuring the device is properly authenticated to call the other recipient, and network conditions are capable of sustaining a call.</p>
<p><img src="https://i.imgur.com/KxzwQdK.png" alt="" /></p>
<blockquote>
<p>Figure 2: Signaling between two endpoints with gatekeepers using H.225.0.</p>
</blockquote>
<p>After the call was initiated, the endpoints may further invoke H.245 call control signaling to provide more control. H.245 is capable of sending information required such as encryption, jitter management and preferences (disabling audio and/or video). However, one disadvantage of H.245 is the lengthy four-way protocol handshake required when opening logical channels of a conference. This was alleviated using fastStart in a H.225.0 message (“H.323 Fast Start”, 2015). Later, H.460.6 introduced Extended Fast Connect Feature, providing a one-way handshake.</p>
<h4 id="session-initiation-protocol-sip">Session Initiation Protocol (SIP)</h4>
<p>Session Initiation Protocol, standardized in 1999, is a signaling protocol used for the initiation, maintenance and termination of real-time voice/video applications (Tien-Thinh Nguyen, Christian Bonnet, 2016). The protocol is text-based, and borrows its structure from Simple Mail Transfer Protocol (SMTP) and Hypertext Transfer Protocol (HTTP). SIP is used alongside Session Description Protocol (SDP) as payload containing media metadata, and was designed to be independent of the transport layer. Therefore, SIP can be transfered over Transmission Control Protocol (TCP), User Datagram Protocol (UDP) or Stream Control Transmission Protocol (SCTP). In addition, SIP transmission is accompanied by encryption using Transport Layer Security (TLS). In contrast with H.323, SIP has been standardized by the IETF rather than the ITU, hence has a stronger connection to the internet community than telecommunications.</p>
<p>First, several definitions will be addressed.</p>
<p><strong>Definition. 2.6.</strong> User Agent Client (UAC) User Agent capable of receiving responses and sending requests.</p>
<p><strong>Definition. 2.7.</strong> User Agent Server (UAS) User Agent capable of receiving requests and sending responses.</p>
<p><strong>Definition. 2.8.</strong> User Agent Network endpoint that receives SIP messages and is capable of managing SIP sessions. Unlike other protocols, SIP requires all user agents to implement both client and server roles, which includes ability to handle both requests and responses.</p>
<p><strong>Definition. 2.9.</strong> Proxy Server Network endpoint that receives SIP messages and is capable of managing SIP sessions. Unlike other protocols, SIP requires all user agents to implement both client and server roles, which includes ability to handle both requests and responses.</p>
<p><strong>Definition. 2.10.</strong> Registrar SIP endpoint that accepts REGISTER requests, and records the address from the requesting user agent. Registrars are often used to locate other user agents.</p>
<p>SIP can be used for both peer-to-peer and multiway conferences. Typically, both user agents will register with the registrar by providing REGISTER requests along with any authentication. Afterwards, the UAC will signal requests (such as an INVITE) to the UAS.</p>
<p>A selection of common requests/responses is included in Table 1 and 2.</p>
<table>
<thead>
<tr>
<th style="text-align: center">Request Name</th>
<th style="text-align: center">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">REGISTER</td>
<td style="text-align: center">Registers the caller’s URI to the registar, used by “To” field</td>
</tr>
<tr>
<td style="text-align: center">ACK</td>
<td style="text-align: center">Confirms user agent receiving final response to INVITE</td>
</tr>
<tr>
<td style="text-align: center">BYE</td>
<td style="text-align: center">Signal that the call should be terminated</td>
</tr>
<tr>
<td style="text-align: center">INVITE</td>
<td style="text-align: center">Initiate dialog from UAC to UAS to begin call</td>
</tr>
<tr>
<td style="text-align: center">CANCEL</td>
<td style="text-align: center">Cancel any pending requests (such as pre-existing rings)</td>
</tr>
<tr>
<td style="text-align: center">REFER</td>
<td style="text-align: center">Ask recipient to takeover existing call (and issue request)</td>
</tr>
<tr>
<td style="text-align: center">INFO</td>
<td style="text-align: center">Send any conference metadata while not modifying session state</td>
</tr>
<tr>
<td style="text-align: center">MESSAGE</td>
<td style="text-align: center">Deliver a text message, typically used by IM clients</td>
</tr>
<tr>
<td style="text-align: center">PUBLISH</td>
<td style="text-align: center">Deliver an event to a notification server</td>
</tr>
<tr>
<td style="text-align: center">OPTIONS</td>
<td style="text-align: center">Retrieve the capabilities of an endpoint/user agent</td>
</tr>
</tbody>
</table>
<blockquote>
<p>Table 1: Session Initiation Protocol Requests</p>
</blockquote>
<table>
<thead>
<tr>
<th style="text-align: center">Response Code</th>
<th style="text-align: center">Description</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">1xx</td>
<td style="text-align: center">Request was valid and is being processed (asynchronously)</td>
</tr>
<tr>
<td style="text-align: center">2xx</td>
<td style="text-align: center">Successful completion (for INVITE, it indicates an accept)</td>
</tr>
<tr>
<td style="text-align: center">3xx</td>
<td style="text-align: center">Call redirection required (INVITE to another user agent)</td>
</tr>
<tr>
<td style="text-align: center">4xx</td>
<td style="text-align: center">Invalid request (potentialy due to incorrect request syntax)</td>
</tr>
<tr>
<td style="text-align: center">5xx</td>
<td style="text-align: center">Server failure (potentially internal errors)</td>
</tr>
<tr>
<td style="text-align: center">6xx</td>
<td style="text-align: center">Global failure (not limited to one server, can be invalid destination)</td>
</tr>
</tbody>
</table>
<blockquote>
<p>Table 2: Session Initiation Protocol Responses</p>
</blockquote>
<p>Lastly, an extension of SIP is the Session Initiation Protocol for Instant Messaging and Presence Leveraging Extensions (SIMPLE), which can be used for features such as the <code class="highlighter-rouge">...</code> typing indicator, and adds more specialized request/response headers for instant messaging. The first line of requests contains the method along with the request URI.</p>
<p>On the other hand, the first line of responses is the response code.</p>
<h4 id="signal-protocol">Signal Protocol</h4>
<p>The Signal Protocol, started in 2013 by Open Whisper Systems, is a non-federated cryptographic protocol allowing for end-to-end encryption of voice/video calls and instant messaging conversations. Originally it was used in the popular messaging app Signal, but later was adopted by other apps including WhatsApp and Messenger/Skype in their optional private conversation modes.</p>
<p>By implementing end-to-end encryption, the contents of messages and calling signals/media are only available to the sender and recipient(s), and not the intermittent servers and devices. First, a set of long-term identity key pairs, medium-term signed prekey pair and several ephemeral prekey pairs are gen- erated locally and stored securely. After, all the public keys and registration ids are stored in a “key bundle” and registered with the Key Distribution Center. For the sender to send messages, the sender has to be able to get the registration ID and public keys from the receiver, which can be found in the publically accessible key bundle.</p>
<p>To start a session, the sender uses their identity and medium-term private keys along with the recipient’s set of private keys to determine the master shared secret. The recipient can then receive the master shared secret, decipher and validate it. Afterwards, the two users can send messages to each other.</p>
<p>While the session is alive, the sender uses the master shared key and the recipient’s ephemeral keys to create a root key, chain key and message chain. This results in a new set of single-use epehemeral keys to encrypt/decrypt future messages. In total, the protocol combines the Double Ratchet algorithm, prekeys, an extended triple Elliptic-curve Diffie-Hellman (X3DH) key agreement protocol, along with using Curve25519, AES-256 and HMAC-SHA256 as cryptographic primitives (Ksenia Kozhukhovskaya, 2017).</p>
<p>X3DH is used for generating all the required keys for the sender/receiver to communicate, including the shared secret key between the two callers, by registering identity and prekeys to a server. Using this architecture, the caller can retrieve the “prekey bundle” of the callee even when the callee is online.</p>
<p>Double Ratchet Algorithm is used to provide end-to-end encryption based on the shared secret key retrieved by X3DH. From the shared secret key produced, a “root key” and “sending chain key” are generated. Using a key derivation chain (KDF) from the sending chain key, each subsequent message has a differ- ent epehemeral key by advancing down the chain, and it makes it impossible to decrypt out-of-order messages. In short, when encrypting messages, the sender always forwards the sending chain by one, and generates a new sending chain key and messaging encryption key.</p>
<p>Finally, AES-256 and HMAC-SHA256 are both 256-bit length encryption functions/block ciphers to protect and encrypt sensitive data. Using the master private keys that are shared across caller/callee, the schemes are used to ensure that the data is essentially non-recoverable without the proper credentials.</p>
<h4 id="webrtc-framework">WebRTC Framework</h4>
<p>WebRTC (Web Real-Time Communication) is an open-source project providing mobile applications and web-browsers with RTC capabilities via simple APIs. The mission of WebRTC is to “enable rich, high-quality RTC applications to be developed for the browser, mobile platforms, and IoT devices, and allow them all to communicate via a common set of protocols”.</p>
<p>First, several definitions will be addressed.</p>
<p><strong>Definition. 2.11.</strong> Session Description Protocol (SDP) Format used in describing and negotiating a session’s profile (which includes properties and parameters such as media types and encryption keys). SDP can use attributes extending the protocol’s capabilitiesas key/value pairs, and is otherwise a text- based format with one field per line.</p>
<p><strong>Definition. 2.12.</strong> NAT (Network Address Translation) Process for translating local IP addresses to public IP addresses, often used for security reasons and conserves legally registered IP addresses (due to limitations with IPv4 bit count).</p>
<p>NATs typically work in four ways: full cone (internal address maps to external address and any external host can send requests), address-restricted cone (any host can send to an externally mapped host/port if and only if that host/port previously sent to the host), port-restricted cone (similar to address-restricted cone, except not only did the receiver need to have previously sent to the same host, but also the matching sender port), and lastly symmetric (where a new mapping is used for each request).</p>
<p><strong>Definition. 2.13.</strong> STUN (Session Traversal Utilities for NAT) Lightweight and simple service which provides the public IP address of its caller, used to reply back its IP address to the original caller to establish an IP address; typically used in non-symmetric NATs (full-cone, address-restricted, port-restricted).</p>
<p><strong>Definition. 2.14.</strong> TURN (Traversal using Relays around NAT) Computationally expensive relay for sending messages/media to a destination, typically used in symmetric NATs where the public IP cannot be discoverable and instead requires a statically defined external service to send information to the requesting caller.</p>
<p><strong>Definition. 2.15.</strong> ICE (Interactive Connectivity Establishment) Standard used for performing NAT (Network Address Translation) traversals. ICE deals with returning candidate agent addresses (local, reflexive such as STUN and relayed such as TURN).</p>
<p><strong>Definition. 2.16.</strong> Trickle ICE Optimization of ICE specification providing parallelized connectivity checks across the candidate addresses, reducing the overall initiation time.</p>
<p>There are several core components of the WebRTC framework that is exposed through the JavaScript APIs. Some examples include “getUserMedia” that is used to acquire the audio/video media. In addition, “RTCPeerConnection” is used for performing signal processing, security, peer-to-peer communication and bandwith management. Another core capability is the “RTCDataChannel” for allowing bidirectional communication among peers for sending media using a system based on WebSockets (MDN Contributors, 2020).</p>
<p>The first task in establishing a WebRTC connection is to use a signaling server for resolving the connection. The role of the signaling server is to act as an intermediary and allow the peers to establish a connection with minimizing pri- vate information exposure. WebRTC does not mandate any specific transport mechanism. Instead, WebRTC leverages SDP payloads and ICE to determine candidates for establishing the connection.</p>
<p>For exchanging media WebRTC uses SDP to execute the offer and answer mechanism across peers. Unfortunately, firewalls and NATs are often used in the real-world to both protect private IPs and limit the number of registered IP addreses due to limited IPv4 addressing. However, NAT mappings would occur at the network layer, and hence will change the TCP/UDP packet headers but leave SDP payloads unchanged hence leaving them unaware of handling external NAT IP addresses and port restrictions.</p>
<p><img src="https://i.imgur.com/Cq1qLlL.png" alt="" /></p>
<blockquote>
<p>Figure 3: Example of simple signaling and media transfer without NATs.</p>
</blockquote>
<p>Depending on the type of NAT (full cone, address-restricted/port-restricted cone or symmetric), a different configuration of services is required. STUN is used in the case of asymmetric NATs, where it is simply invoked to determine the public IP address and then that can be used for media transfer. This is a lightweight solution and is common in practice. Unfortunately, for symmetric cones a relay solution is required where TURN servers will be used as the intermediary endpoints. The disadvantage of this approach is that TURN servers are expensive, add additional latency and will be another source of failure among the media streaming path.</p>
<p><img src="https://i.imgur.com/PEEHjtJ.png" alt="" /></p>
<blockquote>
<p>Figure 4: Example of signaling and media transfer between coned/reflexive NATs.</p>
</blockquote>
<p>Popular users of the WebRTC users includes Facebook with over 300 million monthly active users, Discord and Skype (Chad Hart, 2017).</p>
<h4 id="real-time-streaming-protocol-rtsp">Real-Time Streaming Protocol (RTSP)</h4>
<p>Real-Time Streaming Protocol (RTSP), standardized in 1998 by the IETF, is a network control protocol for controlling streaming media servers (Ivn Santos- Gonzlez, Alexandra Rivero-Garca, Jezabel Molina-Gil, Pino Caballero-Gil, 2017). RTSP is not responsible for the streamed data itself, which is often handled by the Real-time Transport Protocol (RTP).</p>
<p>Instead, it facilitates capabilities such as play and pause. The protocol is similar to HTTP, with several exceptions such as RTSP being stateful and resides on both TCP/UDP ports 554 (although UDP is rarely ever used).</p>
<p>RTSP supports several types of requests, including several overlapping with HTTP. “OPTIONS”, similar to HTTP, returns the accepted request types by the server. “DESCRIBE” requests are used for obtaining SDP presentation descriptions, which contain metadta such as the list of media streams that are controlled by the aggregate URL, bitrate, mime type, etc. Lastly, there are “PLAY”, “PAUSE” and “TEARDOWN” requests.</p>
<p>Popular client implementations of this protocol include VLC media player, Skype, Spotify, QuickTime and Windows Media Player. In addition, YouTube uses RTSP as an available streaming option when the mobile HTTPS site is viewed on desktop.</p>
<h4 id="real-time-control-protocol-rtcp">Real-Time Control Protocol (RTCP)</h4>
<p>Real-Time Control Protocol (RTCP) is often used in conjunction with RTP to provide out-of-band (signaling running on a dedicated channel away from media channels in PTSN) control information and statistics for an RTP session. RTCP provides feedback on the quality of service including roundtrip delay time, packet delay variation, packet loss and packet counts. This allows for adaptive capabilities such as Dynamic Adaptive Streaming over HTTP (DASH) to be implemented (“RTP Control Protocol (RTCP)”, 2017). RTP is generally executed on an even-numbered UDP port, and RTCP on the next odd-numbered port. While RTCP does not provide any encryption/authentication methods, Secure Real-time Transport Protocol (SRTP) provides such capabilities.</p>
<p><img src="https://i.imgur.com/84TF2Oa.png" alt="" /></p>
<blockquote>
<p>Figure 5: Example of signaling between two peers in WebRTC.</p>
</blockquote>
<p><img src="https://i.imgur.com/MJBciLm.png" alt="" /></p>
<blockquote>
<p>Figure 6: Example of ICE negotiation in WebRTC.</p>
</blockquote>
<table>
<thead>
<tr>
<th style="text-align: center">Web-Browser</th>
<th style="text-align: center">Supported Versions</th>
</tr>
</thead>
<tbody>
<tr>
<td style="text-align: center">Microsoft Edge</td>
<td style="text-align: center">12+</td>
</tr>
<tr>
<td style="text-align: center">Google Chrome (Desktop)</td>
<td style="text-align: center">28+</td>
</tr>
<tr>
<td style="text-align: center">Mozilla Firefox</td>
<td style="text-align: center">22+</td>
</tr>
<tr>
<td style="text-align: center">Opera</td>
<td style="text-align: center">18+</td>
</tr>
<tr>
<td style="text-align: center">Safari</td>
<td style="text-align: center">11+</td>
</tr>
<tr>
<td style="text-align: center">Google Chrome (Android)</td>
<td style="text-align: center">28+</td>
</tr>
<tr>
<td style="text-align: center">Mozilla Firefox (Android)</td>
<td style="text-align: center">24+</td>
</tr>
<tr>
<td style="text-align: center">Opera (Mobile)</td>
<td style="text-align: center">12+</td>
</tr>
<tr>
<td style="text-align: center">iOS MobileSafari/WebKit</td>
<td style="text-align: center">iOS11+</td>
</tr>
</tbody>
</table>
<blockquote>
<p>Table 3: Browsers with WebRTC Support</p>
</blockquote>
<p>RTCP provides three functionalities. First, RTCP provides statistics on qual- ity of service during media distribution. This can be used, as mentioned earlier, for the implementation of DASH alorithms and paging Engineers to look into potential network disruptions. Secondly, RTCP provides Canonical Name Records (CNAME) to conference participants and allows for effective third-party monitoring. Lastly, RTCP is often used for reaching all conference participants, as RTP only transmits via media source. RTCP reporting is applied to all participants of a conference, and is randomized in reporting time to avoid unintended synchronization of reporting (typically intervals around five seconds are used). As a best practice to avoid network congestion, RTCP bandwith usage should not be higher than 5% of overall session bandwith to prevent causing degraded performance.</p>
<p>Several types of messages are supported by RTCP, and can be extended further to include custom packets. Sender report (SR) is periodically sent by active senders in conferences to report reception and transmission statistics. Receiver report (RR) is for receivers of RTP packets, and likewise sends quality reports to senders. Source description (SDES) provides CNAMEs to session participants for use in third-party monitoring solutions. Goodbye (BYE) is used to shut down a tream of data.</p>
<h4 id="real-time-transport-protocol-rtp">Real-Time Transport Protocol (RTP)</h4>
<p>Real-Time Transport Protocol (RTP), standardized by the IETF in 1996, is a networking protocol for audio and video delivery over IP networks. It is often used in conjunction with SIP and RTCP to create and ensure quality transmission of media. RTP is designed for real-time transfer of media streams and provides capabilities such as detection of packet loss, out-of-order delivery, jitter compensation and IP multicast support. RTP is based on application- layer framing, where the protocol functions are implemented in the application rather than operating system.</p>
<p>RTP supports a range of multimedia formats and is extensible by design. By providing profiles and one or more payload formats for each class of application (such as audio or video), RTP supports definining mapping codecs to payload format codes inside profiles, which then describes the transport method for the encoded data. For example, Secure Real-Time Transport Protocol (SRTP) is an RTP profile for providing cryptographic services for transferring payload data. RTP senders would capture the media, encode it, then transmits it as an RTP packet with the appropriate timestamps and increasing sequence numbers. RTP receivers then detect missing packets and may reorder packets.</p>
<p>Afterwards, the stream can be decoded and presented to the user.
Packet headers for RTP support various mandatory fields, as shown in the figure below. Version is used to indicate the current version of the protocol used (2 bits, where the current version is 102). P (Padding, 1 bit) may indicate extra padded bytes at the end of a header, typically used due to size requirements imposed by encryption algorithms. X (Extension, 1 bit) is used to indicate presence of an extension header between header and the payload. CC (CSRC count, 4 bits) indicates the number of CSRC identifiers fllowing SSRC. Sequence numbers are incremented for each RTP data packet sent, used to implement out-of-order delivery and detect lost packets. SSRC (synchronization source identifier, 32 bits) is used to identify the source of the stream, while CSRC (contributing source ids, 32 bits each) are used to enumerate contributing sources to the stream. Lastly, the optional header extension would include a 16 bit profile-specific identifier and a 16 bit length specifier, with the custom extension header data following.</p>
<p><img src="https://i.imgur.com/5Expq9c.png" alt="" /></p>
<blockquote>
<p>Figure 8: Example of an RTP packet header.</p>
</blockquote>
<h4 id="audio-streaming-codecs">Audio Streaming Codecs</h4>
<p>The process of encoding and decoding audio data is handled by an audio codec (coder/decoder). In real-time communication, the balance between having high-quality audio and low-bandwith requirements is a power struggle that often requires compromises from either side. The two general categories of factors affecting encoded audio is the codec chosen and the details about the source audio’s contents and format.</p>
<p>Different codecs provide a variety of parameters to tune allowing for vari- able bit rate, audio frequency bandwith and many other custom fields. Audio codecs are based on advanced numerical computational algorithms and can provide either lossless or lossy compression. AAC (Advanced Audio Coding) is a common codec defined by the MPEG-4 (H.264) standard, used by Blue-ray disks, HDTV, and songs purchased from iTunes (MDN contributors, 2019). However, the format is protected by numerous patents making distribution less predictable. On the other hand, G.722 codec is built with voice compression in mind, is low-latency and uses Adaptive Differential Pulse Code Modulation (ADPCM) to reduce recording size. G.722 is mandated by the WebRTC specification and is typically used for WebRTC connections. Lastly, MP3 (MPEG-1 Audio Layer III) is one of the most common codecs, where MPEG-1 MP3 is generally used for music and MPEG-2 MP3 is used for simpler sounds and requires less space. Moreover, MP3 patents expired as of 2017 in the United States making it supported by all popular web-browsers.</p>
<p>The WebRTC API does not mandate if a particular can be used in a track, but instead requires support VP8 and H.264’s constrained baseline profiles for video and G.711 PCM (A-law) and G.711 PCM (u-Law) audio codecs for Chrome, Firefox and Safari browsers.</p>
<h4 id="video-streaming-codecs">Video Streaming Codecs</h4>
<p>Video encoding is the process of turning raw video into a digital format to be viewed on different devices. Similar to audio codecs, video codecs vary in behavior and specialization depending on use-case, and are often either lossless or lossy in nature. Encoded videos are wrapped into a “video container” (such a .mp4, mov), which contains the video codec, audio codec and associated metadata. The key difference between a container format and a codec is that the codec is at the source and playback to compress and decompress respectively, while the container can be used to determine which programs accept the stream and holds the different components (audio, video, closed captioning) together.</p>
<p>The most common encoding for streaming is MPEG-4 H.264/AVC (Advanced Video Coding), developed by the International Telecommunications Union (ITU) and International Organization for Standardization/International Electrotechnical Commission (ISO/IEC) Moving Picture Experts Group. H.264/AVC has capabilities of being packaged into a variety of container types including .mov, .mp4, .3GP and more.</p>
<p>MPEG-5 H.265/HVEC is a newer codec, which has improved compression efficiency and supports 8k resolution. However, the market-share is only around 10% due to royalty complications (Traci Ruether, 2019). To address this, AV1 was created in a partnership with several leading tech companies (Google, Mi- crosoft, Amazon, Netflix, and more) to create a performant royalty-free codec. Google created the VP9 codec as a royalty-free and more performant version of HVEC, however it is not supported on Apple devices and is often considered to be “AV0”.</p>
<h4 id="dynamic-adapative-streaming-over-http-dash">Dynamic Adapative Streaming over HTTP (DASH)</h4>
<p>Dynamic Adaptive Streaming over HTTP (MPEG-DASH, otherwise known simply as DASH) was standardized in 2012 to provide adaptive bitrate streaming for media over HTTP web servers. DASH works by taking content and breaking it down into smaller HTTP-based file segments, where each segment contains a small interval of content. Each individual segment is then broken down into different bit rates. While the media is streamed, the client applies a bit rate adaption (ABR) algorithm that selects the appropriate segment such that the content can be downloaded in time without causing rebuffering/stalls and maximizes quality.</p>
<p>DASH uses TCP as the chosen transport protocol, and remains agnostic to codecs (meaning it supports various formats including H.264, H.265, VP9), ABR logic and the underlying application layer protocol. Alongside the individual segments containing intervals of streamed data, DASH uses a media presentation description (MPD) to describe segment information (URL, timing, bit rates, etc.), and is presented using a variety of data-structures such as timelines or lists. While there is no restrictions on the type of media data, the specification provides recommendations on using two types of containers: MPEG-2 Transport stream or an ISO base media file format (such as MP4).</p>
<p>Adoption is widespread for DASH, including support by YouTube and Netflix and VLC. While not directly supported in HTML5, there are open-source JavaScript implementations for DASH adding the functionality. Moreover, when combined with WebGL, HTML5-based DASH allows for streaming of 360 degrees content. Alongside developer and product support, content distribution networks (CDN) support for DASH is vast, including Akamai, Amazon CloudFront, CloudFlare and Azure Media Services.</p>
<p>HTTP Live Streaming (HLS), released in 2009, is an alternative HTTP-based adaptive bitrate streaming protocol developed by Apple (Max Wilbert, 2020). Similar to DASH, HLS breaks the stream into a sequence of smaller HTTP-based file downloads in a variety of different bitrates. After, the list of available streams encoded at different bitrates is sent as an extended M3U playlist to the client. Unlike RTP, HLS allows for traversing firewalls and NATs that would otherwise allow regular HTTP tra c to travel through. Later versions of the protocol also introduced subtitle support.
The architecture of HLS is comprised of three parts: server, distributor and client. The server is responsible for preparing the video for delivery, encoding encoding the video files in H.264 format in variable bitrates as MP3, AAC, AC-3 or EC-3, and then encapsulated in MPEG-2 Transport Stream. After, the segmenter will divide the MPEG-2 TS file into equally sized segments, and then create the index file for fragmented files as an m3u8 file. The distributor will then act as a regular HTTP web server, and deliver the required m3u8 playlist file and ts segment files required to stream the content. Lastly, the client will retrieve the m3u8 file containing the index of segments, and then retrieve the necessary segments from the distributor.</p>
<h4 id="multiway-calling-architecture">Multiway Calling Architecture</h4>
<p>WebRTC media streaming natively supports communication across two different peers, but often real-time communication applications support multiway calling involving numerous peers. Multiway conferences for voice and video can be supported using three architectures: mesh, mixing and routing (Tsahi Levent-Levi, 2019). In addition, signaling often uses an intermediary centralized server acting as an anonymous “peer”, and relays signaling using ICE candidates (such as STUN or TURN).</p>
<p>Mesh routing involves an <code class="highlighter-rouge">n^2</code> amount of linkages, where n is the number of peers, which is not scalable to many users and requires substantial amounts of bandwith. Mixing leverages an MCU (Multipoint Conferencing Unit) which acts as a centralized point where multiple streams from individual peers are combined into a single unified stream. However, despite the simplicity in design, it comes without flexibility such as client-specific processing because the streams are combined at the relay source.</p>
<p>Routing leverages an SFU (Selective Forwarding Unit), which instead acts as the router of the media (Tsahi Levent-Levi, 2019). In contrast to mixing, routing will send individual streams directly to other peers allowing for more client-sided flexibility in terms of processing. Three approaches are applied for routing: multi-unicast, simulcast and SVC. Multi-unicast is the trivial approach, where users would send streams to the SFU and the SFU would decide where to route, not performing any bit rate adaption. On the other</p>
<p><img src="https://i.imgur.com/PZrSudz.png" alt="" /></p>
<blockquote>
<p>Figure 8: Example of multiway mesh streams.</p>
</blockquote>
<p><img src="https://i.imgur.com/Iz27xCu.png" alt="" /></p>
<blockquote>
<p>Figure 9: Example of multiway mixing streams.</p>
</blockquote>
<p><img src="https://i.imgur.com/IHfZFTV.png" alt="" /></p>
<blockquote>
<p>Figure 10: Example of multiway routing streams.</p>
</blockquote>
<p>hand, simulcast will accept multiple streams of varying qualities and then will send the supported ones (based on network conditions and device capabilities) to the other peers.</p>
<p>Lastly, SVC (scalable video coding) uses a similar methodology as simulcast, but instead of sending disjoint streams of varying bitrates, SVC sends a layered stream of increasing quality where particular layers can be “peeled” off the top to reduce quality. This is an enhancement over simulcast because it reduces computation time and allows error corrections to occur only at base levels; it was introduced to WebRTC in the VP9 video codec.</p>
<h3 id="case-studies">Case Studies</h3>
<h4 id="skypes-p2p-signaling-protcol-2003">Skype’s P2P Signaling Protcol (2003)</h4>
<p>Skype is a VoIP client developed by Microsoft allowing users to place audio and video calls over the internet. Skype uses an overlayed peer-to-peer network, similar to its file sharing predecessor Kazaa (Salman A. Baset, Henning G. Schulzrinne, 2004). There are two types of nodes in the architecture: ordinary hosts and super nodes (SN). An ordinary node is any Skype client that is used for issuing voice and video calls. Super nodes are also Skype clients, however they are promoted to SN once it is identified that they have a public IP address, performant hardware (RAM, CPU) and network bandwith. In addition, the login server is a critical piece of Skype infrastructure as it handles the login credentials of users and in later Skype versions the friends lists of users. Moreover, SkypeOut and SkypeIn are servers used to bridge VoIP with PSTN, however it is not used in pure VoIP calls.</p>
<p>Every Skype node uses a variant of the STUN/TURN protocol to identify NAT and firewall restrictions of users. When the SC is loading, it first sends an HTTP request to determine if there should be a software update. After, a connection to a SN is established. A critical component of Skype clients is the host cache (HC), which is built and refreshed regularly to contain a list of SN capped at a length of 200. If none of the SN inside the HC are reachable, Skype resorts to establishing a TCP connection to a bootstrapped list of 8 hardcoded SN addresses - if that does not work either then Skype fails to login. After SC is connect to a SN, the client applies the username/password to authenticate with the Skype login server.</p>
<p>Skype user search leverages a Global Index (GI) technology (Salman A. Baset, Henning G. Schulzrinne, 2004). First, the SC will ask the SN over UDP if it knows any users matching the regular expression. If an SN does not know, then it will provide 8 addresses of SN nodes over TCP to the SC to further query. This process repeats with 16, 32, and exponentially many more nodes. After an abritrary cutoff, if none are found, the login server is requested (which is always invoked in the case of no matching usernames). Repeated queries are cached locally at the client.</p>
<p>The Skype protocol has no silence suppression, meaning it still sends UDP packets containing null noise even when muted. There are two benefits to this design choice, namely preventing reapplying UDP bindings and in the case of TCP being used it prevents drops in the TCP congestion window which would reduce preliminary RTT time until it ramps back up. The codec of choice used by Skype is iCodec, with a minimum bandwith requirement of 2Kb/second (Salman A. Baset, Henning G. Schulzrinne, 2004). Lastly, it was observed that in conferences, the media is not fully meshed, meaning the most powerful machine is elected to be the host and collect and later distribute the streams to the remaining hosts, as show in the figure below.</p>
<p><img src="https://i.imgur.com/FDQAqyr.png" alt="" /></p>
<blockquote>
<p>Figure 11: Example conference signaling without full mesh.</p>
</blockquote>
<h3 id="conclusions">Conclusions</h3>
<h4 id="resulting-architecture-of-rtc-systems">Resulting architecture of RTC systems</h4>
<p>This post provides a high-level overview of different protcols, standards and codecs used to implement scalable real-time communication services such as video calling. Various protocols and technologies are involved in the signaling, encoding, and delivery of media over networks. Over time, numerous opti- mizations have been applied to reduce the number of handshakes required to establish connections, save bandwith and improve call quality. By applying the techniques applied in this post, it becomes possible to get a high-level view of designing a large-scale system supporting numerous concurrent users across ranging network setups such as private IP addresses guarded by fire- walls. In the future, frameworks such as WebRTC will become more mature, codecs such as H.265 will have widespread adoption and protocols will be added making it easier than ever to keep the world connected and together by using real-time communication.</p>
<h3 id="references">References</h3>
<p>[1] Mansoor Iqbal (2020). WhatsApp Revenue and Usage Statistics. Business of Apps (2020). Retrieved April 12, 2020, from https://www.businessofapps.com/data/whatsapp-statistics/</p>
<p>[2] (n.d.) Development of long-distance transmission. ENCYCLOPDIA BRITANNICA. Retrieved April 12, 2020 from https://www.britannica.com/technology/telephone/Transmission</p>
<p>[3]Robert Pepper (2014). The History of VoIP and Inter- net Telephones GetVoIP. Retrieved April 12, 2020, from https://getvoip.com/blog/2014/01/27/history-of-voip-and-internet- telephones/</p>
<p>[4] Jan Ozer (2011). What Is MPEG DASH? Streaming Media. Retrieved April 12, 2020, from https://www.streamingmedia.com/Articles/ReadArticle.aspx?ArticleID=79041</p>
<p>[5] Margaret Rouse (n.d.). H.323 Search Networking. Retrieved April 12, 2020, from https://searchnetworking.techtarget.com/definition/H323</p>
<p>[6] Tien-Thinh Nguyen, Christian Bonnet (2016). IP Mobility Management for Future Public Safety Networks Wireless Public Safety Networks 2. Retrieved April 12, 2020, from https://www.sciencedirect.com/topics/computer-science/session- initiation-protocol</p>
<p>[7] (2015). H.323 Fast Start. Dialogic.. Retrieved April 12, 2020, from https://www.dialogic.com/webhelp/BorderNet2020/1.1.0/WebHelp/h323 faststart.htm
31</p>
<p>[8] Ksenia Kozhukhovskaya (2017). Demystifying the Signal Protocol for End-to-End Encryption (E2EE) Cloudboost. Retrieved April 12, 2020, from https://blog.cloudboost.io/demystifying-the-signal-protocol-for-end- to-end-encryption-e2ee-3e31830c456f</p>
<p>[9] MDN Contributors (2020). WebRTC API. Mozilla Developer Network. Retrieved April 12, 2020, from https://developer.mozilla.org/en- US/docs/Web/API/WebRTC API</p>
<p>[10] Chad Hart (2017). WebRTC: One of 2016’s Biggest Technologies No One Has Heard Of Web RTC World. Retrieved April 12, 2020, from http://www.webrtcworld.com/topics/webrtc-world/articles/428444- webrtc-one-2016s-biggest-technologies-no-one-has.htm</p>
<p>[11] Ivn Santos-Gonzlez, Alexandra Rivero-Garca, Jezabel Molina-Gil, Pino Caballero-Gil (2017). Implementation and Analysis of Real-Time Streaming Protocols US National Library of Medicine. Retrieved April 12, 2020, from https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5424723/</p>
<p>[12] (2017). RTP Control Protocol (RTCP) SDX Central. Retrieved April 12, 2020, from https://www.sdxcentral.com/resources/glossary/rtp-control-protocol-rtcp/</p>
<p>[13] MDN contributors (2019). Web audio codec guide Mozilla Developer Network. Retrieved April 12, 2020, from https://developer.mozilla.org/en- US/docs/Web/Media/Formats/Audio codecs</p>
<p>[14] Traci Ruether (2019). Video Codecs and Encoding: Everything You Should Know (Update) Wowza Media Streams. Retrieved April 12, 2020, from https://www.wowza.com/blog/video-codecs-encoding
32</p>
<p>[15] Max Wilbert (2020). What is HLS streaming and when should you use it? dacast. Retrieved April 12, 2020, from https://www.dacast.com/blog/hls-streaming-protocol/</p>
<p>[16] Tsahi Levent-Levi (2019). WebRTC Multiparty Architectures BlogGeek. Retrieved April 12, 2020, from https://bloggeek.me/webrtc-multiparty- architectures/</p>
<p>[17] Salman A. Baset and Henning G. Schulzrinne (2004). An Analysis of the Skype Peer-to-Peer Internet Telephony Protocol Columbia University. Retrieved April 12, 2020, from http://www1.cs.columbia.edu/ salman/publications/skype1 4.pdf</p>I recently did an internship at Facebook where I was part of the RTC (Real-Time Communications) org, responsible for the products and infrastructure powering voice/video calling across Facebook’s family of apps (such as Messenger and Instagram). My team was responsible for the P2P and multiway calling protocols, including lower-level services written in C++ and platform integrating with partner teams in PHP.Living in Seattle for Four Months2020-03-26T19:49:29-04:002020-03-26T19:49:29-04:00http://horatiulazu.ca/blog/coop/2020/03/26/seattle<p>I am fortunate to have spent three (not four) months in Seattle, Washington at Facebook! Here is an overview of some activities and sightseeing accomplished during the term.</p>
<p>Overall, the Seattle winter weather didn’t really play in the favor of outdoor activities (fact: January had 28 days of rain), but nonetheless it was a fun term with friends and nice co-workers.</p>
<p><img src="https://i.imgur.com/GhX7YVj.png" /></p>
<h3 id="why-end-early">Why end early?</h3>
<p>The Coronavirus (COVID-19) essentially sprung from being ever so distant in Wuhan, China but eventually landed at the doorsteps in Washington. After border restrictions were put in place, the University of Waterloo, Canadian Government and Visa Sponsorer (Cultural Vistas) all recommended to return.</p>
<p>While I originally refused (as I wanted to spend more time at Facebook), eventually CECA (UW’s co-op department) told Facebook to make co-ops return. In retrospect, it was the right decision for safety.</p>
<h3 id="living-situation">Living Situation</h3>
<p>The corporate housing (<a href="https://www.624yale.com/" target="_blank">624 Yale</a>) was quite good - I’d place it between Bloomberg (<a href="https://www.92y.org/residence" target="_blank">92y</a>) and Citadel (<a href="https://www.marqueeblock37.com/" target="_blank">Block 37 Marquee</a>) in terms of desirability. Fully furnished 2 bedroom 2 bathroom in South Lake Union made it easy to get to Facebook Dexter, and it was in a safe and quiet neighborhood. The bathrooms and kitchen were fully equipt with just about anything you’d need.</p>
<p>The bi-weekly cleaners were also a plus, in addition to in-suite laundary. The building’s gym was quite poor compared to Citadel’s rooftop gym, or even Bloomberg’s 92y gym. There weren’t any other amenities offered in the building, such as a swimming pool, and the lobby felt small. In addition, there was no 24/7 reception. However, it was substantially quieter than both Citadel and Bloomberg’s, so I enjoyed it.</p>
<p><img src="https://i.imgur.com/ydZDcHM.jpg" /></p>
<p>Seattle’s already dark and gloomy, the lack of lighting was a major issue during the winter rainy months.</p>
<p><img src="https://i.imgur.com/hN3qWfH.png" /></p>
<h3 id="swag">SWAG</h3>
<p>Of course, not an accomplishment or venture but the most important part of any internship; during my internship at Facebook I was on the Messenger RTC (Real-Time Communications) organization, more specifically focusing on call signaling. Not pictured here is a free Facebook Portal Mini.</p>
<p><img src="https://i.imgur.com/UZE1x9E.jpg" /></p>
<h3 id="mount-si">Mount Si</h3>
<p>On the second weekend, I ventured out and hiked up Mount Si. I wasn’t entirely certain what to expect, but it turned out to be a fairly involved climb. The <a href="https://www.strava.com/activities/3026758376" target="_blank">route</a> can be seen below.</p>
<p><img src="https://i.imgur.com/6l3QX0Z.png" /></p>
<p>Coming in at around 1000m elevation gain (3300ft) over 14km roundtrip (8.7 miles), the route was certainly more involved than the Muir Woods trail in San Francisco several terms ago. The biggest difficulty wasn’t at all the actual route, but the lack of preparation on my part.</p>
<p>More specifically, I underestimated the amount of snow there was and I grossly missed on equipment such as spikes and poles. On several occasions I slipped on the ascent/descent, but slow and steady went ok in the end… in the future: gloves, spikes and poles.</p>
<p><img src="https://i.imgur.com/n5dEGnX.jpg" /></p>
<p><img src="https://i.imgur.com/HRo4oK7.jpg" /></p>
<p><img src="https://i.imgur.com/QoIdwdA.jpg" /></p>
<h3 id="mount-squak">Mount Squak</h3>
<p>Another smaller hike followed, which wasn’t anywhere near as challenging and the view was just fog. Nonetheless, some statistics for that hike here.</p>
<p><img src="https://i.imgur.com/YcRUsw3.png" /></p>
<h3 id="mount-snoqualmie">Mount Snoqualmie</h3>
<p>“Work from home Wednesdays”, aka paid ski trip for the entire Bellevue office.</p>
<p><img src="https://i.imgur.com/KQ4Ytdu.jpg" /></p>
<h3 id="vancouver">Vancouver</h3>
<p>Not the best weather, nor went up Whistler mountain, but nonetheless touring Stanley park was enjoyable.</p>
<p><img src="https://i.imgur.com/v8UvEur.jpg" />
<br /><br />
<img src="https://i.imgur.com/t1I7XZR.png" /></p>
<p><img src="https://i.imgur.com/dG3FAcy.jpg" /></p>
<h3 id="ride-to-kenmore">Ride to Kenmore</h3>
<p>One of my friends from high-school had just started full-time at Facebook Seattle and lent me his hybrid Giant bike. The Burke Gilman trail was particularily pleasant to ride on.
<img src="https://i.imgur.com/TBrJ0AY.png" /></p>
<p><img src="https://i.imgur.com/4bhbE5q.jpg" /></p>I am fortunate to have spent three (not four) months in Seattle, Washington at Facebook! Here is an overview of some activities and sightseeing accomplished during the term.Living in Chicago for Four Months2019-07-16T19:49:29-04:002019-07-16T19:49:29-04:00http://horatiulazu.ca/blog/coop/2019/07/16/chicago<p>I am fortunate to have spent an internship at Citadel Securities in Chicago, Illinois.</p>
<p>Citadel Securities is the largest market maker in the United States responsible for executing over 20% of US equity trades, including over 39.5% of retail shares traded. You can watch <a href="https://www.youtube.com/watch?v=2u007Msq1qo" target="_blank">this video</a> outlining high frequency trading (plus you get to see my co-workers, it was filmed behind my desk!).</p>
<p><img src="/blog/assets/chicago-plane.jpg" /></p>
<p>The provided corporate housing is a 2 bedroom + 2 bath luxury condo suite shared with 1 roommate being around 5 minutes walk from the office in the heart of the loop.</p>
<p>This has been by far the best corporate housing so far, we even had a weekly cleaner and lots of amenities including swimming pool, rooftop terrace/gym, billiards, and more!</p>
<p><img src="/blog/assets/chicago-living.jpg" /></p>
<p><img src="/blog/assets/chicago-kitchen.jpg" /></p>
<p><img src="/blog/assets/chicago-bedroom.jpg" /></p>
<p>The views from the corporate housing were great too.</p>
<p><img src="/blog/assets/chicago-view1.jpg" /></p>
<p><img src="/blog/assets/chicago-view2.jpg" /></p>
<!-- <img src="/blog/assets/chicago-lightning-1.jpg"/> -->
<p><img src="/blog/assets/chicago-lightning.gif" /></p>
<p>This is the view of Chicago from the Planetarium.</p>
<p><img src="/blog/assets/chicago-planetarium-view.jpg" /></p>
<p>The following are pictures of River North.</p>
<p><img src="/blog/assets/chicago-river-north-1.jpg" /></p>
<p><img src="/blog/assets/chicago-river-north-3.jpg" /></p>
<p><img src="/blog/assets/chicago-river-north-2.jpg" /></p>
<p>View of Chicago from the Botanical Garden.</p>
<p><img src="/blog/assets/chicago-botanical.jpg" /></p>
<p>John Hancock building on a foggy day.</p>
<p><img src="/blog/assets/chicago-john-hancock.jpg" /></p>
<p><img src="/blog/assets/chicago-john-beach.jpg" /></p>
<p>Views from John Hancock Tower 96th floor</p>
<p><img src="/blog/assets/chicago-john-north.jpg" /></p>
<p><img src="/blog/assets/chicago-john-east.jpg" /></p>
<p><img src="/blog/assets/chicago-john-south-east.jpg" /></p>
<p><img src="/blog/assets/chicago-john-south.jpg" /></p>
<p>Ken Griffin, the founder of Citadel, paid over $300 million for this painting.</p>
<p><img src="/blog/assets/chicago-painting.jpg" /></p>
<p>Citadel HQ (outside)</p>
<p><img src="/blog/assets/chicago-citadel.jpg" /></p>
<p>Chicago Botanical Garden</p>
<p><img src="/blog/assets/chicago-cascade.jpg" /></p>
<p><img src="/blog/assets/chicago-garden1.jpg" /></p>
<p><img src="/blog/assets/chicago-garden2.jpg" /></p>
<p><img src="/blog/assets/chicago-garden3.jpg" /></p>
<p><img src="/blog/assets/chicago-garden4.jpg" /></p>
<p><img src="/blog/assets/chicago-garden5.jpg" /></p>
<p>Lastly … <a href="https://medium.com/@LumpyBatter/what-is-kobe-beef-and-why-is-it-so-expensive-4b76674ed2c1" target="_blank">3oz Wagyu Kobe Beef</a> :)
<img src="/blog/assets/chicago-steak.jpg" /></p>I am fortunate to have spent an internship at Citadel Securities in Chicago, Illinois.Living in San Francisco for Four Months2019-04-20T19:49:29-04:002019-04-20T19:49:29-04:00http://horatiulazu.ca/blog/coop/2019/04/20/cali-round-1<p>Starting off, I’m roughly four months late! I decided to do one of these posts for each internship, and since I’m starting my next co-op in Chicago at Citadel Securities in less than 2 weeks, and I have an exam in 3 days, there’s no better time to procrastinate.</p>
<p>In my opinion, road biking let me explore the regions around San Francisco much better than I would be able to otherwise see (with the exception of having a car), so I hope you enjoy the views :-)</p>
<p><img src="/blog/assets/sf-11.JPG" /></p>
<blockquote>
<p>Golden Gate Bridge, fun fact is that it actually has an incline which is noticable while cycling</p>
</blockquote>
<p><img src="/blog/assets/sf-20.JPG" /> <!-- remove 7 --></p>
<blockquote>
<p>View from the Golden Gate Bridge</p>
</blockquote>
<p><img src="/blog/assets/sf-2.jpg" /></p>
<blockquote>
<p>Alongside Highway 1, around 2/3rds the way towards San Francisco form Santa Cruz</p>
</blockquote>
<p><img src="/blog/assets/sf-4.JPG" /></p>
<blockquote>
<p>Alongside Highway 1, halfway between San Francisco and Santa Cruz</p>
</blockquote>
<p><img src="/blog/assets/sf-3.JPG" /></p>
<blockquote>
<p>View of downtown San Francisco at sunset from Twin Peaks</p>
</blockquote>
<p><img src="/blog/assets/sf-5.JPG" /></p>
<blockquote>
<p>Alpine Dam, south of Fairfax</p>
</blockquote>
<p><img src="/blog/assets/sf-10.jpg" /></p>
<blockquote>
<p>Overlooking Mill Valley</p>
</blockquote>
<p><img src="/blog/assets/sf-8.jpg" /></p>
<blockquote>
<p>Mount Tamalpais West Peak</p>
</blockquote>
<p><img src="/blog/assets/sf-18.jpg" /></p>
<blockquote>
<p>Mount Tamalpais East Peak</p>
</blockquote>
<p><img src="/blog/assets/sf-14.jpg" /></p>
<blockquote>
<p>Overlooking Muir Woods</p>
</blockquote>
<p><img src="/blog/assets/sf-15.jpg" /></p>
<blockquote>
<p>More views of Muir Woods</p>
</blockquote>
<p><img src="/blog/assets/sf-9.jpg" /></p>
<blockquote>
<p>Overlooking Stinson Beach</p>
</blockquote>
<p><img src="/blog/assets/sf-13.jpg" /></p>
<blockquote>
<p>Clearer picture of Stinson Beach</p>
</blockquote>
<p><img src="/blog/assets/sf-16.jpg" /></p>
<blockquote>
<p>Overlooking San Rafael</p>
</blockquote>
<p><img src="/blog/assets/sf-17.jpg" /></p>
<blockquote>
<p>Sunset view of Battery Spencer</p>
</blockquote>
<p><img src="/blog/assets/sf-22.jpg" /></p>
<blockquote>
<p>Marin Headlands, the descent reaches -18% slope (there’s a sign)</p>
</blockquote>
<p><img src="/blog/assets/sf-21.jpg" /></p>
<blockquote>
<p>Ocean Beach, just south of Lands’ End</p>
</blockquote>
<p><img src="/blog/assets/sf-24.jpg" /></p>
<blockquote>
<p>Lake Merced Park</p>
</blockquote>
<p><img src="/blog/assets/sf-26.jpg" /></p>
<blockquote>
<p>Foggy Golden Gate Bridge</p>
</blockquote>
<p><img src="/blog/assets/sf-27.jpg" /></p>
<blockquote>
<p>View from just past the Seven Sisters</p>
</blockquote>
<p><img src="/blog/assets/sf-31.jpg" /></p>
<blockquote>
<p>Amy’s Island Peak</p>
</blockquote>
<p><img src="/blog/assets/sf-23.jpg" /></p>
<blockquote>
<p>Alcatraz Island</p>
</blockquote>
<p><img src="/blog/assets/sf-28.jpg" /></p>
<blockquote>
<p>View of a cell at Alcatraz</p>
</blockquote>
<p><img src="/blog/assets/sf-25.jpg" /></p>
<blockquote>
<p>San Francisco’s Cable Car Museum</p>
</blockquote>Starting off, I’m roughly four months late! I decided to do one of these posts for each internship, and since I’m starting my next co-op in Chicago at Citadel Securities in less than 2 weeks, and I have an exam in 3 days, there’s no better time to procrastinate.Software Engineering Internship: Yelp Inc2018-11-25T18:49:29-05:002018-11-25T18:49:29-05:00http://horatiulazu.ca/blog/coop/2018/11/25/yelp<p><img src="/blog/assets/yelp-logo-2.png" alt="" /></p>
<p><em>Anything I say here is my own words, and does not represent Yelp.</em></p>
<p><b> Table of contents </b></p>
<ol>
<li><a href="#intro"> Introduction </a></li>
<li><a href="#application"> Application Process </a>
<ul>
<li><a href="#application"> Applying</a></li>
<li><a href="#interviews"> Interviews </a></li>
<li><a href="#compensation"> Compensation</a></li>
<li><a href="#projectselection"> Project Selection</a></li>
</ul>
</li>
<li><a href="#internevents"> Internship Events </a></li>
<li><a href="#office"> 140 New Montgomery Office </a></li>
</ol>
<h3 id="intro"> Introduction </h3>
<p>I did a co-op term at Yelp in Fall 2018 on the Production Engineering team in San Francisco, California. It went great with a lot of learning and an excellent team. Learn more about what I did at Yelp <a href="/blog/coop/2018/11/24/thirdcoop.html" target="_blank"> here</a>.</p>
<p>My favourite part was the amazing mentorship and getting to see my project rolled out to production. The experience overall felt “chill”, with the culture being more <em>startup-y</em>, making room for a unique experience.</p>
<h3 id="application"> Applying </h3>
<p>Since I attend the University of Waterloo, I went through our job board. I submitted my resume and grade report. I was then contacted to do a HackerRank challenge. The position was reasonably competitive, over 500 applicants for around 10-20 spots.</p>
<h3 id="interviews"> Interviews </h3>
<p>The first step is a HackerRank challenge, which was one problem and you had 15 minutes to do it. It’s a very simple problem, mostly used to take out people who would be unlikely to pass the following rounds.</p>
<p>The second round is a 45 minute Skype call with another engineer, where I had a medium difficulty problem (dynamic programming), and talked about my previous work at Bloomberg.</p>
<p>The final round is two back-to-back 45 minute calls, including system design and another easy algorithms round along with some networking and operating system trivia.</p>
<h3 id="compensation"> Compensation </h3>
<p>The compensation is ok, comparable to other Bay Area companies (similar to Google, lower than Facebook) but it was my lowest paying term. The issue with the compensation is the lack of corporate housing, since the housing stipend is heavily taxed and less than initially perceived.</p>
<h3 id="projectselection"> Project Selection </h3>
<p>Since the roles are fairly specific (infrastructure, backend, fullstack, etc), you don’t know your team placement until a few weeks before the term begins.</p>
<p>For me specifically, I had no prior experience with production engineering and received that team placement which in my opinion seemed like a risky maneuver since I hadn’t previously shown any interest in the field.</p>
<p>However, in the end it turned out fine and got to learn many new topics including general infrastructure, cloud technologies and networking.</p>
<h3 id="internevents"> Internship Events </h3>
<p>We had an impressive amount of events for an off-season internship including (but not limited to):</p>
<ul>
<li>Exploratorium</li>
<li>Bowling</li>
<li>Mini-golf putt</li>
<li>Escape room</li>
<li>Hike + visit to Amy’s Island</li>
<li>Magic show</li>
<li>Ice skating trip</li>
<li>Salsa dancing</li>
<li>Pumpkin carving</li>
<li>Lunch with executives</li>
</ul>
<p>And of course lots of food events!</p>
<h3 id="office">140 New Montgomery Office</h3>
<p>The office is very cute, and is actually a historic landmark and was formerly one of the tallest in SF!</p>
<p><img src="/blog/assets/yelp-office.jpg" alt="" /></p>
<p>The office was decorated and full of Christmas spirit :)
<img src="/blog/assets/yelp-office7.jpg" alt="" /></p>
<p>Each floor had a pantry, so over 13 pantries which all look different!
<img src="/blog/assets/yelp-office6.jpg" alt="" /></p>
<p><img src="/blog/assets/yelp-office5.jpg" alt="" /></p>
<p><img src="/blog/assets/yelp-office2.jpg" alt="" /></p>
<p>Another seasonal decoration, but for Halloween …
<img src="/blog/assets/yelp-office4.jpg" alt="" /></p>
<p>Guess who did this? We had a team event :-)
<img src="/blog/assets/sf-30.jpg" /></p>
<p>I worked on Floor 11. Just as a fun fact, Bloomberg actually has their San Francisco Engineering office in the same building as Yelp, they are just several floors above!</p>Work Term #3: Scalable cloud metadata search2018-11-24T18:49:29-05:002018-11-24T18:49:29-05:00http://horatiulazu.ca/blog/coop/2018/11/24/thirdcoop<p><img src="/blog/assets/yelp-logo-2.png" alt="" /></p>
<p>During my third co-op, I developed and shipped a scalable cloud metadata search service at Yelp. The core service is written in Python, and I used Terraform and Puppet to provision and configure the infrastructure promoting reproducability and scalability. This is hosted on AWS, more specifically using a serverless architecture leveraging API Gateway and Lambda.</p>
<p>What is a <em>cloud metadata search service</em>? Put simply, it keeps an inventory of all the servers that Yelp has and any facts about them. Some examples of facts include software that is provisioned on that machine, operating system information, memory available, geographical regions and hundreds of other fields.</p>
<p>This service presents the following features:</p>
<ul>
<li>Expressive discovery using custom DSL querying capabilities in 2 languages (JSON and query string)</li>
<li>Significantly faster querying (<100ms per query, real-time event-based invalidation, 5 minute refreshes)</li>
<li>Highly scalable serverless architecture which is reproducable and configured using code</li>
<li>Deployment process using S3 and Jenkins, unit tests and end-to-end tests using Docker</li>
</ul>
<p>My favourite part of this project is that I went through all stages of the development process, and got to see something created from scratch go through to production!</p>
<p>Formerly, there was a solution using MCollective that would distribute the request to all known hosts and fetch information then. The issue with that solution is that the query time is essentially a function of the slowest host to respond - since it would wait for all hosts to respond and filter afterwards.</p>
<p>Meanwhile, the inventory service works in the other direction. Instead, each host contains a cron-job that emits metadata on that machine to the inventory service.</p>
<p>The service has its own DSL query language, which can be provided in JSON:</p>
<div class="language-json highlighter-rouge"><pre class="highlight"><code><span class="p">{</span><span class="w">
</span><span class="nt">"filter"</span><span class="w"> </span><span class="p">:</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"AND"</span><span class="p">,</span><span class="w">
</span><span class="nt">"args"</span><span class="p">:</span><span class="w"> </span><span class="p">[{</span><span class="w">
</span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"matching"</span><span class="p">,</span><span class="w">
</span><span class="nt">"arg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"hostname: interndev1-us[east,west]1dev*"</span><span class="w">
</span><span class="p">},</span><span class="w"> </span><span class="p">{</span><span class="w">
</span><span class="nt">"type"</span><span class="p">:</span><span class="w"> </span><span class="s2">"matching"</span><span class="p">,</span><span class="w">
</span><span class="nt">"arg"</span><span class="p">:</span><span class="w"> </span><span class="s2">"cpu_count >= 16"</span><span class="w">
</span><span class="p">}]</span><span class="w">
</span><span class="p">},</span><span class="w">
</span><span class="nt">"sort"</span><span class="p">:</span><span class="w"> </span><span class="s2">"uptime asc, hostname desc"</span><span class="p">,</span><span class="w">
</span><span class="nt">"limit"</span><span class="p">:</span><span class="w"> </span><span class="mi">1000</span><span class="p">,</span><span class="w">
</span><span class="nt">"offset"</span><span class="p">:</span><span class="w"> </span><span class="mi">10</span><span class="p">,</span><span class="w">
</span><span class="nt">"index_epoch_threshold"</span><span class="p">:</span><span class="w"> </span><span class="mi">2000</span><span class="w">
</span><span class="p">}</span><span class="w">
</span></code></pre>
</div>
<p>Alternatively using the string query syntax (inspired by Apache Solr’s streaming syntax):</p>
<div class="highlighter-rouge"><pre class="highlight"><code>$ search(filter(AND("hostname: interndev1-us[east,west]1dev*", "cpu_count >= 16")), sort("uptime asc, hostname desc"), limit(1000), offset(10), index_epoch_threshold(2000))
</code></pre>
</div>
<p>As part of this project, I also implemented query parsers for both that convert the associated query into an Elasticsearch URI query. This included making some interesting optimizations along with a basic n-ary preorder tree traversal to assemble the query string.</p>
<p>One of the greatest challenges was dealing with a schemaless design. Since there’s hundreds of fields, it isn’t managable to manually introduce every field in a fixed schema since teams are constantly adding or removing fields. Luckily, Elasticsearch provides the ability to give a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-mapping.html">dynamic schema</a>, where it infers type. However, this was quickly proven unreliable because we experienced type collisions (ex: initially a field was <code class="highlighter-rouge">bool</code>, later it would be indexed as <code class="highlighter-rouge">int</code>). Such a type collision would yield errors when ingesting metadata - later invalidating that instance from search results and returning incomplete data!</p>
<p>The solution to this was leveraging a <a href="https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html">dynamic templated mapping</a>. We would have four types: <code class="highlighter-rouge">*_str</code>, <code class="highlighter-rouge">*_bool</code>, <code class="highlighter-rouge">*_long</code>, <code class="highlighter-rouge">*_float</code>. The service would append to each field being ingested the type it thinks would be the type. This type would always conform with the equivalent Elasticsearch type, hence all ingestion requests would go through. The only concern now becomes - fields can sometimes be more sparse and scattered across multiple fields - at worst 4x worse performance! However, this rarely the case, and providing the guaranteed that documents will be ingested is more critical than <em>slightly</em> slower retrieval.</p>
<p>How does this work for retrievals? All the retrieval queries would need to be appropriately tokenized, have type introspection occur and inspect the mapping of the index, and then adjusted accordingly. This includes expanding any numerical field queries to be <code class="highlighter-rouge">(field_long:val OR field_float:val)</code>. While a tricky process for all the arguments that can be passed to the inventory service, the strict DSL and with a careful implementation and lots of unit tests it worked well.</p>
<p>The final benchmarks using Apache Bench and AWS CloudWatch yield <150ms p99 query times for indexing and <90ms p99 search queries while exceeding one million API requests daily.</p>
<p>To conclude, I learned a lot this term with an amazing mentor who was extremely helpful throughout the entire process - something which I think is invaluable and definitely made this one of my favourite terms.</p>YouTube: Analytics on 1.2 million views2018-11-12T18:49:29-05:002018-11-12T18:49:29-05:00http://horatiulazu.ca/blog/programming/2018/11/12/youtube<p><img src="/blog/assets/youtube-cover.png" alt="link" /></p>
<p>I’ve been running my YouTube channel <a href="https://www.youtube.com/user/ComputerBunnyMath123" target="_blank">SoftwareEngenius</a> for the past six years. Initially, it started as a place to post videos regarding my hobbies (mostly computer tutorials and programming videos). It has mostly stayed that way, with more recent videos being very focused on software development.</p>
<p><strong>Note</strong>: Sometimes it may count under 1.2 million - it’s either statisics not updated or unlisted videos.</p>
<h4> Motivation</h4>
<p>I had a fantastic experience building out my YouTube channel, and I don’t regret the many hours spent on this channel. The following, however, is worth noting:</p>
<blockquote>
<p>The goal of the channel isn’t about “becoming popular” or “getting rich”, rather it was more for me to genuinely produce content that I thought had some value. Unfortunately, this also means I don’t have much interest in production quality, and felt like getting my thoughts out are more important than spending time editing.</p>
</blockquote>
<p>I believe I was successful in reaching my goal. I was able to make a clear amount of videos covering a wide span of topics, though it clearly shows as some of my videos aren’t of the best quality. From 2013-2015 I gained momentum and my channel was growing, and to keep that going I was heavily invested in producing more quantity, which at the time led to saturation in my video content.</p>
<h4> Beginning: 2012/2013</h4>
<p>This was the starting phase of the channel, back when it was called <code class="highlighter-rouge">ComputerBunnyMath123</code>. Clearly, I didn’t have any niche in mind, so I began by posting completely random videos including programming tutorials, software videos and some games including Minecraft, Chess and Brain Age 2. I didn’t get many views at the start, only after adding more content (~20 videos) did I start getting over 1,000 views per month.</p>
<p><img src="/blog/assets/youtube-growth.png" alt="" /></p>
<h4> Growing Content and Saturation: 2013/2014/2015</h4>
<p>This period I would characterize as gaining many more views, which led me to wanting more and produce vasts amounts of videos. It was mostly sparked by my <a href="https://www.youtube.com/watch?v=MEvY27PB6ZU" target="_blank">Ripley’s Aquarium video</a>, which got over 20,000 views early on and motivated me to continue producing random content. This continued for some time, and I spent most of the Summer of 2014 reaching out to companies to review their products.</p>
<p>I wouldn’t say this period of time was used effectively - I ended up greatly saturating my channel with a wide array of random topics which led to very poor user retention.</p>
<p><img src="/blog/assets/youtube-growing.png" alt="" /></p>
<h4> Focused and Slowing Growth: 2016/2017</h4>
<p>During this time period I didn’t have a lot of time to devote to making videos (I was busy applying to university or in my freshmen year), so I had to focus my efforts. By this time, I gained some exposure to various Computer Science concepts more in-depth, and made the decision to change the focus of the channel to be <em>only software related</em>.</p>
<p><img src="/blog/assets/youtube-maturing.png" alt="" /></p>
<h4> Matured and Rarely Updated: 2017 onwards</h4>
<p>I hadn’t uploaded many videos recently, occasionally only when I visit an interesting concept or learn something more niche I would bother to make a new video. I don’t actively pursuit creating videos for the views or fame. Instead, if I find something cool I’ll share it! Some notable videos during this time period includes my functional programming channel or an <em>upcoming series I am working on</em>!</p>
<p>More interestingly though, is despite the lack of posting videos I didn’t drop much at all in terms of revenue or views. Personally, I find this surprising because many of my videos tags included <code class="highlighter-rouge">2014</code> or <code class="highlighter-rouge">2015</code>, but seems despite this I’m still getting traffic.</p>
<p><img src="/blog/assets/youtube-current.png" alt="" /></p>
<h4> Future Plans</h4>
<p>I have no plans for this channel. If I find something cool - I will make a video and post it :)</p>
<h4> More Analytics!</h4>
<p>Who doesn’t enjoy numbers and charts? Below are standard analytics provided by YouTube.</p>
<p><img src="/blog/assets/youtube-views.png" alt="link" /></p>
<p>Unfortunately, my most popular videos aren’t representative at all of my channel as a whole - but this goes to show how luck with tags and spotting a niche can help bring in more viewers.</p>
<p><img src="/blog/assets/youtube-device.png" alt="link" /></p>
<p>I find it interesting how the time spent on mobile is almost as much as on the computer - and that such a large proportion of viewers watch on phone or tablet.</p>
<p><img src="/blog/assets/youtube-traffic.png" alt="link" /></p>
<p>Interestingly, it looks like YouTube’s recommender systems worked in my favour from August 2014 to February 2015 and then rarely suggested my videos until late 2016….</p>
<p><img src="/blog/assets/youtube-age.png" alt="link" /></p>
<p>There are no surprises here, the vast majority of viewers will be “tech savvy” and likely near college age.</p>
<p>All in all, hope you enjoyed these facts and numbers :)</p>
<!-- ![link](/blog/assets/youtube-os.png)
Interestingly, there were more Macintosh users than Windows until around late 2015. This is likely because I used to have more content for Mac users. More interestingly, is that Mac and iOS users tend to watch my videos for longer!
![link](/blog/assets/youtube-functional.png)
-->