Programming – GusBook

Future-Driven Development

June 19, 2017June 20, 2017 guscost1 Comment

Have you used Yarn? Yarn is like NPM but it has a “lock file” to prevent conflicting versions in sub-dependencies and sub-sub-dependencies from breaking your bundles. It is blowing up for good reason – the lock file is a great feature.

But we all know lots of people are going to stick with NPM, and it will turn into a whole schism until the mainstream switches, or until NPM absorbs the feature (my prediction). Why? Because everyone gets so worn down chasing the de-facto official JavaScript best practices year after year, that new (maybe worthwhile) ideas get ignored with the rest of the cutting-edge.

This is the sad result of too much, let’s call it “future-driven development”, a pernicious malady affecting software projects near you. It comes in many forms, here are a few:

Building and re-building unit tests while a prototype is changing rapidly
Over-normalizing your database, adding abstractions and joins everywhere
Using ES2018 in 2017 with bloated polyfills because “someday they’ll be native”

In academia, you are expected to act like this. Researchers try zany architectures and plan for uncertain future scenarios because that is the whole point of research. If you’re doing R&D in any capacity this is somewhat true – it is dangerous to ignore anything promising and new for too long.

However, people also act like this way too much in the professional world. When building a dashboard for a customer, you are not trying to win Architecture of the Year, you are not building a reference implementation of All Current Best Practices, your dashboard might not always need to run for a hundred years, and you are not building the dashboard for yourself. The tools only matter so far as they continue to help you deliver an effective product.

Objection #1: How do you know what you’re going to need until it’s too late?

You don’t, and you never will. That’s where experience comes into play. You will never arrive at an ideal implementation for any particular set of requirements by following every last one of today’s best practices (most of which were developed to solve a Very Serious Problem that you might never have).

Objection #2: Doesn’t this make it harder to improve future standards?

This objection was originally a sarcastic caricature, which sums up my feelings.

How to do Programming

July 3, 2016June 19, 2017 guscost1 Comment

TL;DR: Identify assumptions and write them down.

There’s a common (I think) misconception that the programming trade is all about knowing obscure facts and doing hard math. From this it follows that a person has to get really good at math or know a huge number of things before doing programming.

Unfortunately people who have this misconception can be discouraged from trying in the first place. It might be unrealistic to think that every single person not only can but also will want to do it, but I think that lots of people with this misconception could do very well at programming, once they understand what it is actually like, and if they put some effort into learning it.

In reality, being good at the math or knowing all about compilers and language features does not help with a large percentage of the day-to-day work that most programmers do. Instead, their effort goes into writing down all the assumptions made by the high-level description of a feature. In other words, the requirements will say “display a count of the current total” and the programming work is about finding the assumptions implied by that description (“what does ‘current’ actually mean?” etc), then writing them down explicitly. Once you write down the assumptions in the right way, the explicit representation of the assumptions is your code and you are done. Getting everything written down correctly used to be much harder, but with modern programming tools it isn’t even possible to make some of the most problematic mistakes anymore, and the tools can catch lots of other mistakes automatically. For everything else you would want to get help from a more experienced colleague, or just ask strangers on Stack Overflow.

There are programmers who would take issue with this description. I’m using the terms “doing programming” and “day-to-day programming” strategically, really to mean commercial or hobbyist programming for which there are mature high-quality tools and well-understood best practices. On the cutting edge of academia, and within programming projects that have very demanding requirements, advanced math and knowing lots of obscure facts can be much more important.

Basically, what I’m saying is that people tend to confuse the larger industry with that super-difficult hacker nerd work they see in movies and TV shows. In fact, the vast majority of programming work going on right now is the other kind. There are huge numbers of those jobs to be done, because it is where all the theory and advanced knowledge finally can be applied to a real-world problem. Many large software teams have so much of this kind of work that they split out a whole department for “requirements engineering”, which is taking the really high-level descriptions with tons of assumptions, and breaking those out into statements with fewer or no assumptions left in each statement, so that the code writers can focus on making the final code work. The best requirements are harder to write than all the coding work that comes after!

Maybe someone has told you before that programming is all about breaking problems down into smaller problems. It’s another way of saying the same thing.

ReactJS Techniques: Version Tags

December 13, 2015June 19, 2017 guscostLeave a comment

Version tags are a pretty simple technique I’ve used with ReactJS to optimize rendering larger apps. This optimization does not add any dependency on an immutable data library. Right away I have to say that this is not something you want to be choosing all of the time. If you want to be able to use shouldComponentUpdate optimizations easily by default, or if you have any reason to be saving and replaying undo history, then you should seriously consider using an immutable data structure. But there are a couple of common reasons to skip that piece of architecture. First, If you simply cannot afford to load any more JavaScript after ReactJS and the other libraries you might already be using, then the convenience of ImmutableJS or a similar library might not justify the overhead to load it. And second, if it would be too difficult to learn and/or integrate ImmutableJS with your existing architecture, this is one alternative. In some ways it is much simpler, but in one important way it is more difficult to use.

If you want to do a very clean implementation of a flux-type architecture, your data might propagate down from a top-level “controller-view” through a whole big tree of React components. Each component in the tree will potentially render on every state change, so to avoid this you want certain nodes to be implementing shouldComponentUpdate and checking if their particular data has changed. The simplest way to do this is to flatten all props and check them one by one. While this can give a performance boost, the boilerplate required to do so can get ridiculous. ImmutableJS and similar libraries eliminate this problem by making the state an immutable data structure. Basically this means that every time a node in the state updates, that node is copied and a new reference is made to hold the new node, along with every node directly above it in the state tree. Then any component which only needs a particular state node to render can check if that reference has changed, and superfluous rendering can be easily skipped. Having the old nodes around also makes undo features trivial to add.

Here’s how version tags work. Instead of copying the node(s) that you need to be updating, you just change a version tag on those nodes. Then instead of checking an immutable reference, the components check if the version has changed to see whether to render. That’s it.

This means you’ll have to add a version tag to the same scope as each node of data you want to be able to optimize for, and it means you will have to manually update the version tag every time something in that node changes. This approach is a sort of middle ground optimization between checking every individual value type at one extreme, and working with and checking an immutable data structure at the other. However in some situations it has the potential to perform better than both.

As a simple example you might have an application that displays a list of clients and a list of appointments. You could add a version tag to each object in your state:

var _state = { clients: [...], clientsVersion: 1, appointments: [...], appointmentsVersion: 1 };

And then if you ever update something in appointments you’d do appointmentsVersion++; as well. The component that renders the appointments implements shouldComponentUpdate which just has to compare appointmentsVersion. Versions can be added to as many nodes of the tree as you like, to get the degree of optimization you need at the time.

Will updating versions all the time be in several ways less advantageous than using ImmutableJS? Absolutely. But it is a useful technique.

React.JS and Flux ideas for practical applications

May 27, 2015June 15, 2015 guscost1 Comment

Here’s a list of ideas and techniques I’ve used when working with ReactJS and Flux. There are two sections: general suggestions of how to get good results, and notes from optimizing for slower hardware. Hopefully they provide a little bit of insight for when you’re looking at all these new concepts and going:

Ah yes interesting, why not continue reading the web page?

It’s difficult to say which of these items if any would definitely be “best practices” but they are available here for reference. Use at your own discretion and happy coding. Several ideas are from [0]react primer draft and [1]tips and best practices, both good resources to introduce and guide learning on the subject.

General suggestions of how to get good results

This has been said elsewhere, but get as much state as possible into one place in your application. With Flux this is the store(s). The first diagram from [1] with one-way propagation of data through the tree of components is the ideal, meaning no this.state anywhere. However, there will inevitably be integrated components which have other state, and this should be as contained as possible using this.state and callbacks. Also, any state which is definitely only significant to the one component (such as the open/closed state of a dropdown menu in many cases) can be kept internally without a problem. With Flux this leads toward fewer or even no meaningful state updates happening outside the standard data flow, which is very nice.
Some action creators get data from AJAX and pass it along by dispatching an action. There is a way to do this with one REQUEST action and then one RECEIVE action, which is kinda nice because they can be tested and implemented separately, and the REQUEST action doesn’t have to have any callbacks. Another option could be for the action creators to attach promises with each asynchronous action. Then the stores or a service could unpack the data before updating the state. I haven’t actually used that method and there are cases where it doesn’t provide as much flexibility (simultaneous AJAX actions may cause a “cannot dispatch in the middle of a dispatch” error), so try at your own risk.

Notes from optimizing for slower hardware

Another point from [1], but definitely use the pureRenderMixin and/or implement the same kind of behavior with shouldComponentUpdate (more on that below).
I didn’t realize for a while that you shouldn’t just run the dev source through a minifier for production. With so much relying on pure JS interpreter speed, all those great log messages and debug exceptions make quite a difference. Make sure to build or copy the actual production script when using React in production.
There are two basic ways to show or hide sub-components based on UI state. One is to leave them out of the render (I’ve used a ternary expression that returns null if the sub-component is invisible). The other is to render them with a “visibility” prop and have the sub-component set a display:none style internally. The first method keeps the DOM small and clean, but it means more DOM manipulation each time you show or hide the sub-component. The second is more efficient if the sub-component visibility changes often, but it means more DOM elements in the page at any one time (although some are invisible). When optimizing, use the strategy that is more appropriate for the situation.
Don’t put complex literal props inline. These cannot be used with shouldComponentUpdate since === will return false every time. Maybe you shouldn’t use literal props at all, but that is if you want to set up constants for everything (usually not a bad idea).
Use shouldComponentUpdate. As far as I know there are four basic optimization strategies with shouldComponentUpdate:
- One is to split up complex props into lots of flat value props, and just check through each one. I would think this can get verbose and maybe a little less efficient, but doing a bunch of === checks should get pretty good performance with little additional code complexity.
- The second is to use a library like ImmutableJS or Mori to build an immutable state tree and update it through controlled methods. This adds a dependency and some not-very-serious overhead but enables pleasant management of an immutable state object that can be compared before re-rendering quickly and easily. The argument to use an immutable state object is much stronger if you have to implement some kind of undo/redo functionality, but either way it’s a great idea in many situations.
- The third is to hand-roll some kind of version checking in each shouldComponentUpdate. I feel like this has the potential to be a very lean strategy CPU-wise but it is also a bit more complicated to maintain. Basically you add a UUID version tag to each node in the state tree you want to optimize, and whenever updating that node you also update the version tag. Then shouldComponentUpdate is just a matter of looking at the version tag for the node.
- Finally there is the strategy of running shouldComponentUpdate fewer times. This is where you might use internal state in lower-level components, and refs to skip typing in a textbox. Another option is to set shouldComponentUpdate to always return false, and use componentDidMount and componentWillUnmount to set up and clean up DOM elements that are directly manipulated from componentWillReceiveProps. These are mentioned at the end of the list because for all but trivial cases moving state out of the standard flow will make code more complicated since synchronization between the two state repositories is now a requirement. Before using them take some time looking into alternative solutions.

Fun with Tailgaters

February 26, 2015June 19, 2017 guscost2 Comments

Commuting in the car means a lot of time spent accidentally thinking about how it could be improved. I’ve come up with several ideas for accessories, and this first one is useless but very fun. For some time I was planning a system where buttons on the dashboard displayed messages on the rear bumper. Stuff like “C’MON, REALLY?” to display to tailgaters, and maybe amusing stuff like “WHAT’S WRONG WITH THIS GUY?” when someone up ahead is driving poorly. It would be a pretty straightforward bank of switches plugged into an Arduino, and either a screen with pictures or (better) a train-schedule type letter sign, if those can still be found.

A few weeks back, however, I remembered that dog picture from the Internet that says “deal with it” when the sunglasses drop:

I realized there couldn’t be a classier way to respond to tailgaters than with a live-action version, so I decided to make one. It would be a simple Arduino project, with a servo that lowers the sunglasses over the eyes of a stuffed dog. Then a relay would light up the “Deal with it” text on cue.

Setting up the Arduino code and wiring the components didn’t take more than a few hours:

Then there was the simple matter of printing out some sunglasses and attaching them to a servo (cardboard backing, super glue, and a zip tie for the arm):

Finally the sign had to be prepared. I decided to go all out and buy a real neon sign, since that is totally fantastic and you can get them custom-built. The sign arrived with this nice label:

I also opted to buy a pre-packaged relay to switch the sign, since I’m not a trained electrician and you don’t want to trifle with AC power from the wall outlet. The PowerSwitch Tail II is great, you just plug in 5V and ground wires to the side and it works like an extension cord with a switch. The rest of the wiring was just a couple of leads going to 5V and ground, and one pull-down resistor for the button. I also got a 300 watt inverter to provide power from the car battery, and a big red button to activate the sign. Wiring it all together for a test run, it looked pretty good:

The sign turned out to be bigger than I had figured, and it takes up the whole back window of the car. Luckily it has a clear backing so my view isn’t obstructed. There’s still some polishing to go, but it’s working very well.

Nobody has tailgated me anywhere near the threshold level for sign-activation yet (perhaps this is rarer than I thought) but it’s bound to happen eventually. You know when you’re waiting in a line of cars to pass a slow-moving truck, and some chucklehead decides to tailgate you, so that maybe you’ll do the same to the car in front and so on (I assume)? The next time that happens to me, I’ll press this button on the dashboard:

And I’ll take my sweet time to finish the pass. Meanwhile the offending driver will see this:

Arduino code is on Github if you’re interested.

React: Why use it?

February 21, 2015June 19, 2017 guscostLeave a comment

The latest JavaScript brand to hit critical mass is React. It’s a few years old, and still not at 1.0, but that hasn’t stopped it from making large waves in the community. If you’re just now getting proficient with Angular after switching from Knockout or Backbone, this might feel annoyingly familiar (and you might want to ignore React for a few years while the dust settles). But if you’re interested in where JavaScript at scale might be headed in the future, React deserves your attention for a few reasons.

The first is non-technical: teams at Facebook are using it internally to build very large systems like the chat application. The fact that the React brand of dog food makes up a large part of their diet means that it merits serious consideration. Other libraries like Knockout or Angular are not deployed at nearly the same scale within Microsoft or Google.

Of course, plenty of software tools see wide adoption because of fashion, so with React (and the proposed Flux architecture that goes along with it) we also have to look at the technical situation. Facebook has distributed some good media on this subject (see this video and the Flux documentation) but I get the sense that some of the ideas can be obscured by the vocabulary. I’ve seen “declarative” and “virtual DOM” and “immutable” too many times to count, but these might not communicate the relevant ideas to everyone. You might wonder: What makes React different if both Knockout and Angular use “declarative” markup too?

I think the central idea percolating through the community is stateless UI. React is designed to make it practical to move all state out of your UI components and into a separate, single place for each part of your problem domain (Flux calls these “Stores”). If you’ve ever encountered bugs with two-way view-model type stuff, eliminating these bugs while still delivering good performance is what differentiates this approach. For everyone else, let’s say you have UI components A and B that are bound to your application state (some view-model type stuff), and UI component A receives an update from the user. This propagates to the application state, and if component B is watching it will update as well. But what if component B also contains some state that component A is watching? Now A will update, and so on. In the real world more than two components are usually involved, and the potential for this behavior makes the program harder to reason about.

So then why do we need a special library? Can’t we just move the state out to a separate module and route every action through there (effectively making all bindings one-way)? The reason is that most libraries weren’t designed with that in mind, so throwing away the old state and fetching new state from an external source on every action means a lot of DOM operations, making for a slow application. This is where React’s virtual DOM comes in: you can go ahead and overwrite the state from your external source, and React will optimize what is sent to the real DOM.

More details will have to wait since I’m still learning the technology, but look into reactive programming for an extreme perspective on stateless UI, and check out the Windows message loop to learn about a precursor to the Flux architecture. Again, the technical reasons are important, but the easier argument to understand is that large teams at Facebook and elsewhere have had success with React (and Flux). Happy coding.

Architecture and Environment

August 10, 2014June 19, 2017 guscost2 Comments

Sometimes an architect has a wide-open space, a large budget, and unlimited freedom to design a structure. In these cases established best practices will lead to a technically impressive result. Looking at the Burj Dubai, the appearance is of a very tall structure which required engineering expertise and advanced materials to build. With this sort of tower superproject, genuinely new and unexpected patterns are comparatively rare. The technology and academic literature certainly progress as in other fields, but the result of one of these projects is pretty much what you would expect: a very tall structure on a flat foundation.

Now consider a heavily constrained project, like Frank Geary’s fallingwater house. In this case the client requested a house that fit on a unique piece of land, with forested hills and a waterfall going through the property. You might guess that limiting the possibilities available to the architect would produce limited results, but the reality is exactly the opposite. In those unique circumstances, Geary produced the famous cantilevered design and spectacular proportions which endure as iconic examples of the craft. I don’t want to suggest that one can design a skyscraper without any creativity, or build the fallingwater house without any engineering skill. In fact both projects require a great deal of both abilities. I will suggest that the skyscraper requires comparatively more of the engineering part, and the fallingwater house requires more of the creative part.

Here there is an analogy with other disciplines. Architectural concepts apply to any situation where complicated systems of interacting components have to be designed. Courses of education, business processes, and software systems are some examples.

Working programmers dream about having unlimited resources and freedom to build monumental skyscrapers of software design, but reality does not often accomodate these fantasies. Schedules and budgets are limited, often severely. Large systems usually include legacy infrastructure which was not intended to perform the task at hand. Instead of designing with a blank slate, working around these limitations makes up the bulk of a programmer’s day to day experience. I would argue that the reality makes professional programming a much more creative pursuit than it might seem after reading the textbooks and papers. This is good! It makes the job interesting and even fun at times.

The Ludum Dare competition is a great example of the same phenomenon. Each round the participants vote on a thematic constraint to impose on their game projects. Together with a deadline, this limitation acts as a catalyst for ideas. If I’m asked right now to come up with an arcade game, it probably won’t happen soon. Ask me for a game which has, say, a theme of “Escape” or “Tiny world” and the ideas come a lot faster.

The point is not just that constrained projects demand the creative side of architecture, but also that limiting the scope of a project can encourage creativity. The next time you find yourself at a loss for new ideas, try imposing some arbitrary constraints. It will probably help.

Space Frack!

April 29, 2014June 19, 2017 guscostLeave a comment

Here’s a game I put together over the weekend for Ludum Dare 29. The theme for this weekend was “Beneath the Surface” and the idea was to make an outer space fracking game.

You are controlling the latest in space fracking technology, a little rover with the task of retrieving space resources from a moon or asteroid or whatever. Use your shockwave (spacebar) to frack open the resources, and then drill (shift) to extract the resources for processing. But watch out for enemy rovers sent by rival corporations!

The enemy rovers are a less sophisticated model without cutting-edge fracking technology, but they have been extracting all the available resources for a while and will not hesitate to hit you with a shotgun blast and steal all your newly fracked resources. Avoid them carefully, or use your shockwave to blast them out of the way!

Watch out for the space fracking protesters, too. You don’t want to cause a PR disaster by accidentally running one of them over!

Play the current build here: http://guscost.github.io/spacefrack

Vote for Space Frack: http://www.ludumdare.com/compo/ludum-dare-29/?action=preview&uid=31734

Fractals

May 24, 2013January 15, 2018 guscost2 Comments

Last time we plotted a Mandelbrot set using a small Python script. This set is interesting because an unexpectedly large amount of detail emerges from the iteration of a relatively simple function. Even more interesting is the fact that this detail is not limited to any particular resolution. The Mandelbrot set exhibits fractal geometry, meaning that tiny areas of the set share features with the whole thing. If we zoom in, smaller details are visible no matter how far we choose to zoom.

But you don’t have to believe me – let’s modify our Python script and see what happens. Basically we have to map 600 pixels in each direction to arbitrary sections of the real and imaginary axes. We can have our script read in the desired ranges and calculate appropriate mappings. First, store the inputs in decimal variables:

def get_decimal(m):
    while True:
        try:
            x = float(input(m))
            return x
        except ValueError:
            print('Enter a decimal number.')

loX = get_decimal('Minimum X: ')
hiX = get_decimal('Maximum X: ')
loY = get_decimal('Minimum Y: ')
hiY = get_decimal('Maximum Y: ')

Divide 600 by the range in each direction to compute scale coefficients:

scaleX = 600.0/(hiX - loX)
scaleY = 600.0/(hiY - loY)

Now modify the drawing code to use our designated ranges:

for x in range(0,600):
    real = loX + x / scaleX
    for y in range(0, 600):
        imag = loY + y / scaleY
        c = complex(real, imag)
        p = mandel(c)
        w.create_line(x, 600-y, x+1, 601-y, fill=colors[p])
        w.pack()

With these changes we can zoom in and see some interesting features. Try using -0.15, -0.05, 0.9, and 1.0 as input values. More detail is visible, but it looks like some of the boundaries are smoothing out! Interestingly, that’s because our mandel() function only checks whether each candidate c escapes within 20 iterations. Points close to the boundary often take more than 20 iterations to escape, but they don’t actually belong in the set. Therefore, as the zoom level increases we have to test each candidate c for more iterations in order to maintain an accurate image. This is left as an exercise for the reader.

The Mandelbrot Set

April 17, 2013April 18, 2013 guscost1 Comment

Last year we considered complex numbers, quantities with two degrees of freedom. These numbers have many important applications in engineering, but can we immediately use them to do something interesting?

Well, we can draw the Mandelbrot set with a computer and a bit of ingenuity. This set includes every complex number for which the following recurrence equation never produces a result with an absolute value (distance from the complex origin) greater than two:

mandelbrotequation

For any complex number, we start with z = 0, square it, add our complex candidate c, then repeat the process with z equal to the new (complex) value. As long as z stays close to the origin after many iterations, our candidate is probably in the Mandelbrot set. Since complex numbers behave like points in a 2D plane, we can draw an image where each pixel is colored by testing a candidate c related to its horizontal and vertical position.

So let’s make a drawing! This program is based on Prez Jordan’s Python code, but we’ll add a gradient to show how many iterations each candidate takes to escape. First we set up a 600×600 canvas with Tkinter:

from Tkinter import Tk, Canvas, mainloop

tk = Tk() w = Canvas(tk, width=600, height=600)

Next we define our mandel() function which takes a complex number and tests whether it escapes in twenty iterations. If so the function returns the last iteration number, otherwise it returns 99:

def mandel(c): z = 0 i = 0 for h in range(0,20): z = z*z + c if abs(z) > 2: break else: i+=1 if abs(z) >= 2: return i else: return 99

In order to draw a gradient, let’s use a dictionary to map iterations to colors. We’ll need entries for keys 0-20 and 99:

colors = { 0: 'white', 1: 'white', 2: 'light yellow', 3: 'light yellow', 4: 'lemon chiffon', 5: 'lemon chiffon', 6: 'yellow', 7: 'yellow', 8: 'gold', 9: 'gold', 10: 'orange', 11: 'orange', 12: 'orange red', 13: 'orange red', 14: 'red', 15: 'red', 16: 'red', 17: 'dark red', 18: 'dark red', 19: 'dark red', 20: 'dark red', 99: 'black' }

Finally we loop over each pixel, convert its x and y coordinates to a complex number, test that number by passing it to the mandel() function, and use the returned key to look up the appropriate color in our dictionary:

print "Drawing..."

for x in range(0,600): real = x / 200.0 - 2.2 for y in range(0, 600): imag = y / 200.0 - 1.5 c = complex(real, imag) p = mandel(c) w.create_line(x, 600-y, x+1, 601-y, fill=colors[p]) w.pack()

print "Complete!" mainloop()

Run this code in your Python interpreter and see a picture of the Mandelbrot set!