Sometimes, it's hard for me to explain (even to myself) what I do all day at my job. I maintain some pretty simple, static web pages at my job that should be low effort. But recently, I created and fixed a bug that involved so many unexpected things, and manifested in a weird enough way that I thought it would be useful to demonstrate the complexity of a frontend engineer's job.
Over the past couple months, my team has been merging two codebases into the same
git repository. The work was fairly simple: two Ember.js apps
that can be merged into one. The apps have a few differences that can be handled
if statements or by consolidating behaviors. For example, if they
use different versions of a dependency, it's easy enough to pick the more recent
one, test and move on.
When testing the build from the merged codebase in a QA environment, someone noticed and filed a bug that the top navigation bar appears unstyled for a second.
Debugging: The Hunch
Since this is, as they say, NotMyFirstRodeo, my first hunch is that server side
rendering is broken, skipping a
<link rel=stylesheet> tag injection at runtime.
I've seen this happen before, and I've just fixed a JS exception in SSR the day before.
Because I'm unable to cursorily reproduce the bug, I send it back to the originator
and move on with my day. The next day, I get the same bug back. Not Fixed.
This time I reach out to the originator and they point out that the bug only reproduces in Firefox. I had missed that in the report the day before!
This time, I put in a little more effort. I build the production server locally.
FastBoot (Ember.js's SSR solution) has a different app server for local builds
(using a middleware that injects into the regular development
and a different server for prodution (
fastboot-app-server). They're both roughly
the same, and the meat of the SSR is the same
fastboot package, but as luck
would have it, The
fastboot package in our production server is using v3 and
and the one in
ember-cli-fastboot is v2. It hasn't happened yet, but I'm always
paranoid that some day this will cause that special flavor of "works on my machine bug"
that we love to see as developers. eye roll
So I build the production server and spin it up.
DOES. NOT. REPRODUCE.
At this moment, if I had sleeves I would have rolled them up. It's time to dig into the QA environment. See the belly of the beast.
ssh into one of the hosts and tail the server logs. These logs should be
forwarded to Splunk, but I want to see them in realtime and I have no idea how
the Splunk team's synchronization works, or how long it takes. (There's a whole
aside here about how long it took me to get access to and setup the credentials
to be able to
ssh into these remote machines. Long story short, it took a long
time when I first got it, and if I ever lost access, I don't know how long it
would take to get again.)
The issue is apparent.
FastBoot.require('some-package') is throwing an exception,
and the stack trace is all over the logs. That's a red flag if I ever saw one!
That would definitely break the server side render and cause an flash of unstyled text.
I still have two remaining mysteries: why can't I reproduce this locally? and
why only Firefox? A server side exception has no bearing on the browser.
Looking at the code, it's clear that
some-package is in the
fastbootDependencies keys. For it to be part of the available dependencies
in the production server, it needs to be in
- "some-package": "1.3.0",
+ "some-package": "1.3.0",
I also confirm that before the repositories merged, this was configured correctly.
An Aside on FastBoot
ember build, the
ember-cli-fastboot addon programmatically generates
package.json based on the actual source
package.json file. But it needs
packages to be in both the
dependencies section (to get the version) and the
fastbootDependencies (to allow list it in the sandbox). There are a couple issues
with this approach:
- It's impossible to deterministically install dependencies for production, as there is no lockfile.
- It's very easy to get it wrong (which is still a sensible tradeoff for the extra security!)
Back to Debugging
So why doesn't it reproduce locally?
It turns out that
FastBoot.require() wraps Node's
require, which can
snake up your directory tree to find modules in
node_modules/ in parent directories.
Try this in an empty directory
## create a directory tree
mkdir -p parent/middle/leaf
## create a pseudo package in the top level of that tree
mkdir -p parent/node-modules/bliggity-bloo
echo "module.exports.default = 'what up council!'" >> parent/node-modules/bliggity-bloo/index.js`
## go into the leaf directory
## evaluate some code to exercise the pseudo package
## The -e flag is to evaluate JS without entering the Node repl.
## You can also enter the Node repl and require('bliggity-bloo') for the
## same effect.
node -e "console.log(require('bliggity-bloo').default)"
Notice that the
bliggity-bloo package is two directories above, but it's
still available to require!
So that explains why I couldn't reproduce the bug locally. My
was nested inside my app code directory, which has its own
include development dependencies. So when running the production server using
FastBoot.require() worked just fine on my machine even
with the production server!
But if I move my production
dist somewhere else, say:
# move the dist direct to my hom directory
mv my-app/dist ~/
# go there
# start the server
I can reproduce the SSR failure with the same stack trace as I saw in the QA server logs!
Debugging: Firefox vs Chrome
Why does the bug only reproduce in Firefox?
I haven't figured this out, but what I did notice is that I could reproduce
in Chrome by simulating a slower CPU. My hunch here is that the Firefox
<link> injection and then the resource download slightly differently
than Chrome. Without CPU throttling, the flash of unstyled text was not observable
to the human eye.
I would love to know if someone has any ideas!
So why do we inject a
<link> stylesheet at runtime in the first place? Why isn't
it just part of
index.html? The navigation on our website is built by a different
team to be applied across many disparate domains. They provide JS and CSS assets
as a package. To render it, we have to find the right version for the right locale
at runtime and inject that into the DOM. We can usually do this during SSR, but
when that fails, the same functionality runs on the client (the browser) as well.
This isn't the most convenient architecture to maintain, but it's a function of how different teams work together. Conway's Law in full effect.
Sometimes things are more complicated than they seem! Sometimes we make them more complicated than they need to be. Sometimes the truth is somewhere in the middle and complications arise from constraints that we can't control or are too not worth changing.
I don't think this was a particularly hard problem to solve. I knew where to look and I knew next steps for debugging as each step before left me with more questions. But at the very least, it took two cycles of communicaton with the bug originator, two rounds of reproducing the bug, and two rounds of switching contexts. This is the type of thing that eats up time in large organizations. This is the type of thing to be vigilant about. No bugs made it to production, but how often do our processes (or more generally: the status quo) have an opportunity cost that we don't capture?