Yay, fixed issue with hanging! Apparently it was caused by pushing too much data into Writable stream. uWS returns about 512 KB chunks, while Writable disk stream can only handle at most 128 KB (and by default Express sends 64 KB). After I made it save data in buffer and only send 64 KB chunks instead it works! No more hanging and empty files.

I think I’m getting sick of coding this thing now. Now coding it is not fun but constantly frustrating. I have finally finished the last thing in the compatibility list and started testing it on my real projects, with first one being Diary. It couldn’t start at all for some reason, and apparently I needed to make sure port is a number. Then for some reason assets kept sending 404’s, and that was due to bad handling of conditionals. After I implemented If-Modified-Since properly it fully worked with no issues! Diary can use uExpress fully.

Then moved onto Nekoweb. Immediately there was some issue with routing, request was replied twice. After some terrible debugging session (uWS fucking removes stack trace!!!) I managed to fix it. But now I’m stuck at weird issue with express-fileupload, it just doesn’t write anything into file and often hangs forever, and I’m not sure why it’s happening. So frustrating. I also realized that my solution with skip keys is actually terrible, Nekoweb has a ton of middlewares on each endpoint, but with my way it has to go through every single one when it could only check pattern once, so I’ll have to recode that as well.

Very frustrating day of coding. Had to fix a huge amount of random routing issues and bugs. Implemented If-Range header and options for Router(). There wasn’t that much progress today because of these random bugs that took hours to fix. ████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Brother visited me today, we talked a little, but there wasn’t really anything interesting in that. Cooked khinkali:

██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Spent the day going through Express middlewares and testing if they work with uExpress. While checking cookie-session middleware, it didn’t set the header for some reason. After a long debugging, I saw that this code was causing the issue:

It checks for res.set (which I have implemented) and then uses node’s class setHeader function for some reason. Since I don’t implement that class, I started to think what can I do to fix this.

At first I tried to implement the class, but it caused random issues and almost definitely caused slowdowns, which I really didn’t want. I started thinking of some dumb things like removing res.set if usage of cookie-session was detected, but it’d be extremely terrible and could cause random bugs. I almost gave up and marked it as incompatible (which I had to do with compression middleware due to similar random issues), but then I’ve decided to check Node.js’s code and check what that function does:

So it did set the header, but used something called kOutHeaders as key. I scrolled above and this is what I saw:

Which was:

I don’t have a lot of experience working with Symbols, so I googled and apparently it’s a way to create unique identifier. So I couldn’t just call my headers object kOutHeaders, I had to actually get out that symbol from there.

At first, the most obvious attempt, I just tried to require it from internals like const { kOutHeaders } = require('internal/http'), but Node said that internal/http doesn’t exist. Apparently you can’t access internal modules without a flag, so this definitely wouldn’t work as a solution.

Then I went with a second attempt, I got out http.OutgoingMessage class, created an object from it, and tried to use Object.keys() on it, which sadly did not show any Symbols, only strings. But to my blessing I found out about Object.getOwnPropertySymbols(), and it worked. I got out kOutHeaders from internal module!

After I added this[kOutHeaders] = this.headers;, it resulted in a bit weird format that Node.js internally uses, so I had to write a proxy that would actually use correct format for my object:

And everything started to work!

All middlewares I use in my servers work fine, yay!

More Yume 2kki at the end of the day. Today’s worlds were definitely more high quality than usual but I still haven’t found something to be really memorable. Visited Secret Society again to appreciate it’s beauty.

Implemented view rendering today for uExpress. Added a bunch of tests for popular view engines.

Fixed next() for .param and bunch of other routing issues.

Now only express.static() and res.sendFile options left and it’s done!

Finally played Yume 2kki again. We’ve decided to go on Expedition. Expedition went through Amoeba Woods, so we’ve got to visit this beautiful place again.

Expedition destination was a pretty boring place, so we just continued going into random places. We’ve found some pillar world and some person chilling in front of entrance to a new place.

A new place was Downpour (Rain World!!!), the only thing that world had was 2 powerlines and endless rain and lightning. The lightning effect was really nicely done. Probably the best I’ve seen out of all games I’ve ever played.

After that we found our Final Destination: Rainy Sundial.

Looking up, you can see beautiful cherry blossom tree. Using Rainbow effect, you can stop the rain and Blossom will become more colorful, and on the right side a new Main Menu theme unlocks. It’s probably our deepest world so far, being more than 10 worlds deep. It feels rewarding to find something at the end of the journey.

Windy weather is so rare for some reason lately. I woke up and it was a really nice wind, so I went outside for a walk. I went to a park, sat down in front of a river and ████████████████████████████████████████████████████

████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

██████████████████████

Spent the day optimizing uExpress even more. Few things I did:

• switched from PassThrough to Writable (+2.5k)
• wrote a tiny pipe for Request instead of Readable (+2.5k)
• now only reading body on non-GET requests (+10k)

In general, on simplest requests speed went from 55k to 70k. Also implemented req.fresh.

██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Huge day for uExpress:

Implemented almost all app settings, refactored routing, fixed baseUrl and implemented body reading. I didn’t even really had to code body parsers, just made Request extend Readable stream and it worked with Express’s middlewares. Very nice feeling. Cookie parser also worked with no changes except for signing cookies.

██████████████████████████████████████████████████████████████████████████████████████████████████████

█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

██████████████████████████████████████████████████████████████████████████████████████████████████████████████████ Implemented piping into res and res.sendFile.

At first I implemented sendFile using only streams, and speed was about 13k/sec. In comparison Express has 4k/sec. Then I’ve decided to split the mechanism into 2 parts:

1. If file is less than 64 KB in size, just read it with readFile and send (since streams read file in 64 KB chunks anyway, useless overhead).
2. If file is over 64 KB create a read stream and pipe into response, like before

This made it go to ~18k/sec. While playing around with readFileSync I realized that it’s much faster - 25k/sec, but I really didn’t work to play around with sync on main thread, so I’ve decided to experiment a bit with workers, and it worked out better than I expected! Speed is now 30k/sec, I finally found a good use for workers.

Basically finished implementing res.

Spent the day implementing APIs. Finally no more routing… For now.

Getting quite green now. Hopefully it’ll be in actually usable state soon. While coding this, I’ve noticed that creating Buffer and then checking it’s underlying ArrayBuffer with Buffer.buffer showed completely different content and size.

It made me extremely confused, and I actually thought I found a Node.js vulnerability, but apparently it’s expected behavior. But it’s VERY weird. You can read and write into completely random Buffers you usually shouldn’t have any access to.

█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Quite a lot of progress on uExpress today. Added a lot of Request APIs, and then spent the night trying to optimize routing.

If path is simple (just a string), it’s possible to precalculate the path request would go through. This is what I did initially, and it raised speed from 20k/sec to 27k/sec at 1000 routes. But then I realized, I can then create actual uWS routes when route is simple and just use that precalculated path. And that significantly increased speed. It’s now 61.66k with 1000 routes and 90k/sec with 1 route. And 1000 routes + 1000 middlewares is now 26k/sec instead of 3k/sec. Great progress so far. Tomorrow I’ll have to also make it work with routers too, as right now it only optimizes paths for top application only.

Continued coding uExpress today. Managed to implement Routers, which were really hard. Not gonna lie, I almost gave up but somehow did it (though idk if they really work properly in all cases…)

They’re pretty slow though. Nested router brings down speed to like 55k/sec. Normal request rate is about 85k/sec.

Finally played Yume 2kki again. Explored computer-y parts of it today, one of them was a maze (Data Stream) with a secret and cool music.

█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Recently I’ve realized how line between sleeping and being awake is very blurry for me. I can be extremely energetic and then in 5 minutes already falling asleep.

Today I’ve started coding µExpress - hopefully a drop-in replacement for Express, with much better performance.

So far, it’s going unexpectedly well. I did a benchmark, and it’s 9X faster than normal Express.

Express: 10k/sec µExpress: 92k/sec µWS: 106k/sec

Along the way making tests, and so far implemented following features:

Day started from me seeing that my website, services, Nekoweb and everything else was down. Apparently Contabo was having some issues in their datacenter, and I think this was the last stretch for me. While my VPS was usually fine and rarely had any issues, Nekoweb’s one constantly gets random freezes from time to time, and Nekoweb being down for 10 hours is just unacceptable.

I was meaning to do it for a long time, and this has finally made me get an actual dedicated server for Nekoweb. I browse Hetzner auction very often, but it seems to keep having same-ish servers with not very good specs, at the minimal range of price.

So I’ve decided to look for more expensive options, and found a pretty good server for it’s price. Quite modern CPU (2019), 6x2 TB of storage, 64 GB of RAM.

As I was ordering it, something unbelievable happened! Price of the server changed in my eyes, and now it had the minimal price. I was checking the auction (it’s Dutch style) quite often throughout the months, and I’ve never seen server this good to be sold at this price, so I hurried and ordered it. Now I have this server for same price as all other, much worse servers. I plan to move all my sites and services to it, not just Nekoweb, since it has more than enough specs for everything.

This is so beautiful, I’ve never seen htop so big…

I then installed Ubuntu on the server. And then realized I need to get RAID1, so installed it again with RAID1. And then realized I did it with wrong partitions, so installed it again…

RAID1 sync is taking a very long time, so I can’t really do anything with server while it’s going, since it’s just too slow.

In mean time, my brother visited me, and we talked about what’s going on in our life. █████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████

Played more Yume 2kki with friends. Got all of the effects now, so I’m finally truly free to explore the world. At the end I have rested in my favorite place so far - Sewers.

It’s my element.