React at Wagon
    October 01, 2015 | Mike Craig

    We’re building a hybrid web/native application that runs both in the browser and as a downloadable desktop app. Analysts use Wagon to query, analyze, visualize, and share data: the app is highly interactive and data-heavy. It has to be fast, furious, and stable even when used for hours.

    It ain’t all gravy: it’s difficult to maintain a UI with asynchronous updates, large scale data manipulation, and many other cross-cutting concerns. How can we build a sane frontend codebase, without losing our ability to iterate and ship quickly? The answer is to separate concerns. We break our UI into small self-contained components, and we isolate state and manage it separately from the UI. Facebook’s React and Flux libraries make this practical.

    Wagon loves React

    The big idea behind React is this: a UI component is just a function from its inputs to its content. All a component needs is a render() method that returns the elements we want the user to see. As an example, here’s a component that takes a size and color and renders a div displaying a filled-in square. Notice that users of this component don’t need to know about how it is implemented.

    React components are simple to reuse because they nest like HTML elements. It’s easy to wrap an existing component to add additional styles or behavior—React favors composition over inheritance. We can forget about carefully maintaining the DOM to avoid excessive redraws and flicker: we declare what our components should look like, and React makes it so.

    React is great for organizing view elements, but an application is more than static UI. Users generate events and we need to capture them, update state, and direct how the app should respond. Flux manages this flow by clearly seperating user action events from application responses.

    Actions encapsulate events. They’re the application logic that runs in response to users doing stuff. In our example, when a user clicks a colored square, we update the server and dispatch to let the rest of app know what happened:

    Stores encapsulate state. They listen to state changes dispatched from actions, and they update themselves to record the changes.

    UI components listen to stores and re-render when relevant state changes.

    Building a solid, maintainable frontend is still difficult despite these great libraries. Here are a few other strategies we’re using:

    • Build pure React components whenever possible. A pure component doesn’t make any external calls from render()—it’s a pure function of the component’s properties and state. Components like this are much easier to test, debug, and reuse. It’s such a good idea it’s included in React itself!

    • Separate React components that listen to Flux stores from those that render the UI. Wrapping UI components in container components is another win for reuse. Jason Bonta mentioned this in his great talk at React.js Conf 2015.

    • Take care when integrating non-React UI components. Mixing React’s declarative API with another library’s imperative API can be painful. Build wrapper components around external libraries, and use React’s lifecycle methods to handle setup and teardown. When possible, avoid exposing direct-update methods like drawChart() or setCursorPosition()—manage state through component properties or Flux stores.

    • Split Flux actions into modules by UX concern. We separate navigation and authentication from running queries and making charts. Carve out submodules for cross-cutting concerns, like AJAX requests or logging.

    • Split Flux stores by domain. It’s helpful to separate persisted server-side state from ephemeral page state, for example. We hide the state of the URL bar behind a store, too!

    We’re tackling fun engineering challenges at Wagon. If you want to learn more or work everyday on these technologies, check out our jobs page and get in touch!

    Weekly Roundup: React
    May 27, 2015 | Mike Craig

    Our hybrid native/web application’s frontend layer is built with Javascript, React, and Flux. In 2014, our friends at Facebook advised us to try React and it has since been a core part of our stack.

    React logo

    The React community is bustling and there are some great posts to help people start, organize, and scale frontend applications. These articles have helped us:

    1. Alex Lopatin details Sift Science’s migration from Backbone to React in Best practices for building large React applications.
    2. Alexander Early from Fluid maintains an in-depth set of Tips and Best Practices. The discussion in the comments is good too.
    3. Facebook’s Christopher Chedeau has many talks on React architectures but this review of how to scale CSS in a React stack guided our development of Wagon’s custom UI. Merci!
    4. Alex Schepanovski describes React from a jQuery perspective in Boiling React Down to a Few Lines in jQuery.
    5. Ryan Clark has a technical but still approachable Getting Started With React walkthrough. If you like that, read the sibling article Getting Started with Flux too.

    We’ll be doing a deeper dive into Wagon’s React architecture, tooling, and how we see it changing in our next engineering blog post. If you like to hack on React, Flux, Javascript, and Electron, we’re hiring a frontend engineer in San Francisco. Say hi!

    Engineering at Wagon
    February 13, 2015 | Mike Craig

    Over the last few months, we’ve been user-testing a preview version of Wagon—a modern SQL editor that powers a Google Docs-style experience for analysts, engineers, and their teams. We’ve iterated on both our design process and technology choices to build a tool that we and our users rely on every day.

    Wagon takes advantage of cloud-local computing, combining the strengths of remote servers like syncing, sharing, and storage with local client advantages like data locality and device-specific user interfaces. Our architecture has three components: a frontend app, a native client, and backend services.

    Wagon architecture Wagon architecture

    We make it easy to get started. You can download Wagon—a hybrid web application and native client, like Slack—and connect it to an existing local, private, or public database. The frontend is a slick interface for writing and running SQL, interacting with large results, and generating statistics and pivot tables. The native client locally handles connections, manages query runs, and processes results as they return from the database. The results, statistics, and pivots are cached locally and can be used to create sharable charts. When shared, Wagon securely transfers data from the local machine to the backend, making it available to permissioned teammates.

    Frontend App

    We use HTML, CSS, and JavaScript to maintain a consistent user experience across platforms and environments. The frontend is a single-page application built with React and Flux. We bundle it with the native client as a desktop application, and we have an in-browser experience for sharing and collaboration.

    Technology: Javascript, React, Flux, Ace

    • React gives us declarative, reusable UI elements. React components compose like HTML elements with rich state handling, so we manage fewer event listeners and nested callbacks.
    • Flux complements React and helps structure our frontend code. It separates logic, state, and UI into loosely coupled components, and has a straightforward data flow.
    • Ace is an extensible, embeddable code editor. We use its syntax highlighting and have extended its autocomplete.

    Native Client

    Our users want to query their databases without any new IT or admin setup, so users connect via their local machine with their existing credentials. Wagon handles connections through the native client, since most databases require socket-level network access that isn’t possible directly from browser.

    Exploration and analysis can be a time-consuming and tedious job with existing SQL tools. Each step is an additional database query subject to queuing, slow execution, and changing data. Our users want to quickly explore results and evaluate summary statistics without executing subsequent queries. To power that experience, the client also computes statistics and caches query results on the local file system.

    We built the native client in Haskell. It’s a high level language with great libraries, static compilation, a high performance ceiling, and an active community. We use MacGap to bundle the client and the frontend into a downloadable OS X application (Windows soon!).

    Technology: Haskell, MacGap

    • Prototyping and refactoring is safe with Haskell’s type-driven development. The compiler helps us avoid bugs and edge cases as we iterate.
    • Haskell’s opt-in control over compiler optimizations and strictness allows for incremental optimization. GHC includes a capable profiler, and the community maintains libraries for testing and performance measurement.
    • MacGap is a lightweight (~1MB) web-to-native wrapper. It provides a JS API to integrate with the file system, subprocesses, and OS-specific features like notifications and menus.

    *Note: Wagon is now powered by Electron rather than MacGap. Read our Electron blog post to learn more.

    Backend Services

    Our backend services persists state and shared data to allow users to securely collaborate on queries, results, and charts. The backend is also written in Haskell: using the same language for the client and the server helps us stay flexible and reduce complexity. We store application state in Postgres while larger immutable data, like shared query results, live in Amazon S3.

    Technology: Haskell, Postgres, Amazon S3

    • Haskell’s lightweight threads and the GHC runtime scale well on multi-core machines.
    • Postgres is powerful, flexible, and straightforward to host. We use features like TOAST and JSON support to test new features.

    There are exciting upcoming technology challenges including optimizing compute across local and remote environments, designing efficient streaming statistics algorithms, building features that learn and improve with usage.

    We’d love to hear from you! Follow @WagonHQ, check out our stack on Stackshare, and learn more about our engineering and design team. Stay tuned for future engineering deep dives!

    Thanks to Steve Pike and Doug Petkanics for reviewing this post. Image credit Adam Wilson from The Noun Project.