Very frustrating day of coding. Had to fix a huge amount of random routing issues and bugs. Implemented If-Range header and options for Router(). There wasn’t that much progress today because of these random bugs that took hours to fix. ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Brother visited me today, we talked a little, but there wasn’t really anything interesting in that. Cooked khinkali:
Coded a tiny bit of uExpress, fixed some routing issues and implemented most of options for res.sendFile.
Been feeling kinda bad and anxious all day. Guh set an alarm clock to nap for a hour and woke up 3 hours later in middle of the night…
me fr
Brain started working after 30 minutes so I got up. Spent rest of the night and morning (until 7 AM) implementing HEAD requests, ETag support, multiple FS threads, Range, If-Match, If-Modified-Since, If-Unmodified-Since headers. It’s approaching full completion. Might be fully done tomorrow.
The good thing about naps before night is that I’m super productive on autopilot. I barely remember what I was doing, yet I did a ton. And it’s always like this, I sometimes nap specifically to get in this phase to do a lot of boring work.
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Spent the day going through Express middlewares and testing if they work with uExpress. While checking cookie-session middleware, it didn’t set the header for some reason. After a long debugging, I saw that this code was causing the issue:
It checks for res.set (which I have implemented) and then uses node’s class setHeader function for some reason. Since I don’t implement that class, I started to think what can I do to fix this.
At first I tried to implement the class, but it caused random issues and almost definitely caused slowdowns, which I really didn’t want. I started thinking of some dumb things like removing res.set if usage of cookie-session was detected, but it’d be extremely terrible and could cause random bugs. I almost gave up and marked it as incompatible (which I had to do with compression middleware due to similar random issues), but then I’ve decided to check Node.js’s code and check what that function does:
So it did set the header, but used something called kOutHeaders as key. I scrolled above and this is what I saw:
Which was:
I don’t have a lot of experience working with Symbols, so I googled and apparently it’s a way to create unique identifier. So I couldn’t just call my headers object kOutHeaders, I had to actually get out that symbol from there.
At first, the most obvious attempt, I just tried to require it from internals like const { kOutHeaders } = require('internal/http'), but Node said that internal/http doesn’t exist. Apparently you can’t access internal modules without a flag, so this definitely wouldn’t work as a solution.
Then I went with a second attempt, I got out http.OutgoingMessage class, created an object from it, and tried to use Object.keys() on it, which sadly did not show any Symbols, only strings. But to my blessing I found out about Object.getOwnPropertySymbols(), and it worked. I got out kOutHeaders from internal module!
After I added this[kOutHeaders] = this.headers;, it resulted in a bit weird format that Node.js internally uses, so I had to write a proxy that would actually use correct format for my object:
And everything started to work!
All middlewares I use in my servers work fine, yay!
More Yume 2kki at the end of the day. Today’s worlds were definitely more high quality than usual but I still haven’t found something to be really memorable. Visited Secret Society again to appreciate it’s beauty.
Implemented view rendering today for uExpress. Added a bunch of tests for popular view engines.
Fixed next() for .param and bunch of other routing issues.
Now only express.static() and res.sendFile options left and it’s done!
Finally played Yume 2kki again. We’ve decided to go on Expedition. Expedition went through Amoeba Woods, so we’ve got to visit this beautiful place again.
Expedition destination was a pretty boring place, so we just continued going into random places. We’ve found some pillar world and some person chilling in front of entrance to a new place.
A new place was Downpour (Rain World!!!), the only thing that world had was 2 powerlines and endless rain and lightning. The lightning effect was really nicely done. Probably the best I’ve seen out of all games I’ve ever played.
After that we found our Final Destination: Rainy Sundial.
Looking up, you can see beautiful cherry blossom tree. Using Rainbow effect, you can stop the rain and Blossom will become more colorful, and on the right side a new Main Menu theme unlocks. It’s probably our deepest world so far, being more than 10 worlds deep. It feels rewarding to find something at the end of the journey.
Windy weather is so rare for some reason lately. I woke up and it was a really nice wind, so I went outside for a walk. I went to a park, sat down in front of a river and ████████████████████████████████████████████████████
████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
██████████████████████
Spent the day optimizing uExpress even more. Few things I did:
In general, on simplest requests speed went from 55k to 70k. Also implemented req.fresh.
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Huge day for uExpress:
Implemented almost all app settings, refactored routing, fixed baseUrl and implemented body reading. I didn’t even really had to code body parsers, just made Request extend Readable stream and it worked with Express’s middlewares. Very nice feeling. Cookie parser also worked with no changes except for signing cookies.
██████████████████████████████████████████████████████████████████████████████████████████████████████
█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
██████████████████████████████████████████████████████████████████████████████████████████████████████████████████ Implemented piping into res and res.sendFile.
At first I implemented sendFile using only streams, and speed was about 13k/sec. In comparison Express has 4k/sec. Then I’ve decided to split the mechanism into 2 parts:
This made it go to ~18k/sec. While playing around with readFileSync I realized that it’s much faster - 25k/sec, but I really didn’t work to play around with sync on main thread, so I’ve decided to experiment a bit with workers, and it worked out better than I expected! Speed is now 30k/sec, I finally found a good use for workers.
Basically finished implementing res.
Spent the day implementing APIs. Finally no more routing… For now.
Getting quite green now. Hopefully it’ll be in actually usable state soon. While coding this, I’ve noticed that creating Buffer and then checking it’s underlying ArrayBuffer with Buffer.buffer showed completely different content and size.
It made me extremely confused, and I actually thought I found a Node.js vulnerability, but apparently it’s expected behavior. But it’s VERY weird. You can read and write into completely random Buffers you usually shouldn’t have any access to.
█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Quite a lot of progress on uExpress today. Added a lot of Request APIs, and then spent the night trying to optimize routing.
If path is simple (just a string), it’s possible to precalculate the path request would go through. This is what I did initially, and it raised speed from 20k/sec to 27k/sec at 1000 routes. But then I realized, I can then create actual uWS routes when route is simple and just use that precalculated path. And that significantly increased speed. It’s now 61.66k with 1000 routes and 90k/sec with 1 route. And 1000 routes + 1000 middlewares is now 26k/sec instead of 3k/sec. Great progress so far. Tomorrow I’ll have to also make it work with routers too, as right now it only optimizes paths for top application only.
My throat has been hurting for past 3 days, and today I felt kinda unenergetic and weird, but not really sensibly sick. Not sure what’s going on here.
Went to buy tasty treats and other food today. Then spent rest of the day optimizing uExpress. I got curious, and created test with 1000 routes, and suddenly speed went down to 2k/sec, that was even slower than normal Express at 4k/sec! It made me actually scared that it was all useless and my bad code would destroy any gain from using uWS.
But after a round of different optimizations, it’s now 30k/sec at 1000 routes, while Express is at 4k. Back to huge speed improvement.
Played more Yume 2kki with friends. Found a nice place with very good music.
Looks like I can only remember dreams if I go to sleep at about 2 am, since I actually sleep full 8 hours then. It was a bit nonsensical, too much random stuff going on to really make a good story, but at the end I was shown a very cozy bridge over the river by Aira. They then jumped from it and swam to the other side, while I had to go celebrate someone’s birthday for some reason.
Actually it seems that I slept for 9 hours today, so that’s why I’m feeling rested and had a dream. I’ve noticed a long time ago that no matter when I go to sleep, I always wake up at 11. It has been a long running curse for me, as it doesn’t allow me to rest properly all the time, since I often don’t go to sleep until 4-5 am.
As I’m writing this, I just realized the possible reason for why is this happening. About 4 years ago, my brother kept waking me up at about 10-11 am every single day no matter if I had to do anything or not. I kept telling him that I don’t need to do anything today but he never listened, we kept arguing every morning, sometimes we had mini fights, he would take my blanket, would keep coming in my room to make sure I didn’t fall back also again.
No wonder my brain keeps waking up on it’s own now, somehow always perfectly at 11. And I have no idea how to ever recover from this, it’s my dream to just be able to sleep through brightest and hottest part of day.
I looked up some messages and sure enough:
Man.
Continued coding uExpress today. More progress:
Supporting multiple mountpaths took the longest time. Got to use a stack for the first time in my programmer life.
Continued coding uExpress today. Managed to implement Routers, which were really hard. Not gonna lie, I almost gave up but somehow did it (though idk if they really work properly in all cases…)
They’re pretty slow though. Nested router brings down speed to like 55k/sec. Normal request rate is about 85k/sec.
Finally played Yume 2kki again. Explored computer-y parts of it today, one of them was a maze (Data Stream) with a secret and cool music.
█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████
Recently I’ve realized how line between sleeping and being awake is very blurry for me. I can be extremely energetic and then in 5 minutes already falling asleep.
Today I’ve started coding µExpress - hopefully a drop-in replacement for Express, with much better performance.
So far, it’s going unexpectedly well. I did a benchmark, and it’s 9X faster than normal Express.
Express: 10k/sec µExpress: 92k/sec µWS: 106k/sec
Along the way making tests, and so far implemented following features:
Apparently today there was a foodfest in the city, so I’ve decided to go and check it out. There was a big line for place where you buy tickets, so I decided to buy a ticket online. After doing it, I realized I need a paper thingy to pass, so I went in line anyway… But then after standing the entire time in line, apparently I could’ve just walked in as people with online tickets pass without immediately. Amazing.
There was quite a lot of different food there, it was quite hard to decide what to buy.
I found bubble tea stand, and bought it to try for the first time ever:
Then I got this:
Everything was nice, but I didn’t like shrimp that much because I didn’t really know how to properly clean it.
I walked around some more and listened to the music. Bought corn and some meat.
Then bought and tried churros for the first time too:
And then decided to leave. I had mixed thoughts about going there, in general it was fine, but I was alone and I don’t like crowd and loud places thaaat much, so I didn’t really enjoy it. But I got to try some cool stuff for the first time, so it was definitely worth it.
Added uploading videos and audios, gallery view for photos (that you can see in this entry).
So tired. ████████████████████████████████████████████████████████████████████████████████████████████████████████
Tried to find exact version of song from this video but couldn’t. So I recreated it as much as possible and created a longer version of this video:
