Bandwagon
May 23, 2015 | Jeff Weinstein

Delighting our users drives the Wagon product and motivates our team. We see startups as a community of people coming together around a goal: ours is to make data analysis easier for everyone. Building something great is the collective work of our team, along with our users, friends, family, open-source contributors, advisors, and investors. Sharing the journey is such a positive force for us and it also happens to be a really fun way to run a company.

I'm having one of those moments where @WagonHQ feels like collective work of a community, not just our team. Wild.

— Jeff Weinstein (@jeff_weinstein) March 19, 2015

We want people to adore Wagon. Iterating on feedback and shipping frequently is our tact to building the best product (see our design post). While people tend to inherently enjoy giving their opinion, we know detailing constructive criticism and bug reports can be a burden on our users especially when the product is changing quickly. We focus on a few ways to encourage our community to come together: be welcoming, actively listen, communicate directly, encourage others, and have fun.

Our main community space is a shared Slack channel: appropriately named #bandwagon. We have 50+ people along with our whole team. It’s a safe place to ask questions about data, engineering, analytics, SQL, and give feedback on Wagon. The tone is helpful, collaborative, and positive. We like to make a custom emoji for each new bandwagoner when they sign up!

The conversation is often led by others and many people contribute to answering questions— it’s really fabulous. One early user described it as: “#bandwagon is an awesome Slack channel for Wagon users that makes the app feel bespoke; some of my feedback has been implemented!” We’ve even hired someone from within #bandwagon (welcome @agrano!).

It’s hard work to be active and available for our growing userbase but it’s worth the energy. Here’s a look at our team’s Slack activity over time…

Wagon's Slack messages history.

We love to interact with people on Twitter and Instagram , host monthly happy hours at our office in San Francisco, and welcome anyone to hang out for a day (we have both coffee and internet). We have t-shirts for bandwagoners and enjoy sending handwritten cards. On the silly side, kids love our adorable stickers! And while we don’t encourage vandalism, it’s fun to see that someone tagged our favorite watering hole.

Wagon’s monthly happy hours!

Without our community, we’d be just a small team with a long list of TODOs. But together, we feel like a group marching toward a common goal.

Want to hop on? Send us an email or tweet at @WagonHQ.

Weekly Roundup: Cloud Data Posts
March 12, 2015 | Jeff Weinstein

Over burritos, our team enjoys discussing what’s new in cloud data infrastructure. Here are a few recent articles and announcements that caught our eye:

1. Three Myths about Hybrid Architectures Using the Cloud

Stephen Orban, Head of AWS Enterprise Strategy, unpacks a few commonly held beliefs about hybrid architectures. A must read for larger companies that are attempting to migrate from legacy systems.

2. Video of Mesosphere's DCOS

Check out this awesome demonstration and rad visualization of Mesosphere’s DCOS by Matt Trifiro.

3. Observations on the Importance of Cloud-based Analytics

Werner Vogels, CTO at Amazon, details how cloud based analytics can improve different sectors including healthcare, cities, internet of things as well as business intelligence.

4. Introducing Google Cloud Storage Nearline: (near)online data at an offline price

Google recently announced $0.01 per GB per month storage, available with ~3 second response time. Amazing!

5. Cloud-Client Computing

A few months ago, Andreessen Horowitz published 16 ideas they’re watching and the one that spoke to us the most is cloud-client computing. It’s how we’ve architected Wagon and is worth a repost!

Also, shoutout to our friend and Quizlet infrastructure lead Peter Bakkum on his open source tool for capturing and replaying HTTP traffic. We love seeing the old Hyperpublic crew doing big things.

How Wagon Rolls: Tooling
March 06, 2015 | Patrick Flor

In a recent engineering blog post, our CTO Mike walked through our system architecture. As a complement, I’d like to describe the tools and processes we use to build Wagon.

We’re always learning new things from our customers that impact the product and our technologies. Fast iteration based on feedback drives our development, so we have invested significant time in testing, integration, deployment, and monitoring frameworks. We want to minimize the friction needed to turn new ideas into features.

Here is an overview of our tools and engineering development flow:

Docker + CoreOS

Docker and CoreOS gives us a common interface to our backend systems across development, staging, and production environments. If it works on my machine, it works in staging; if it misbehaves in staging or production, I can debug it on my machine. CoreOS’ clustering and service discovery features also give us a path forward as we scale.

Github + CircleCI

CircleCI’s enthusiasm for experimentation and great customer service have helped us quickly evolve our technical architecture. Their highly configurable Linux and Mac build environments let us consistently test and deploy pieces of Wagon while we transitioned from a server/browser web app to a hybrid native/JS desktop tool. CircleCI’s inferred build processes and Github integration help us to get traction on quick spikes, and their configuration and documentation let us customize CI and get back to building our product.

Here are some details of how we use them to minimize manual testing and deployment:

• CircleCI builds and runs the full test suite on updates to every GitHub branch, so we know the state of any mainline or feature branch.
• We dial up the counts on our generative tests in CI to ferret out uncommon and transient bugs in the build farm.
• All changes are reviewed by at least one other team member before they are merged to a mainline branch. We use GitHub’s Pull Requests for code review and to accept changes.
• Staging and production deployments are a direct function of successful CI builds on mainline branches. Any pull request accepted into master goes to staging and anything that makes our production branch goes live to users, with no additional tagging or manual deploy steps.
• We use CircleCI’s API to kick off transitive builds for more complicated pipelines: this helps us keep the component projects small and single-purpose.
• Each deployment pushes a notification to Slack, so everyone knows there is a new Wagon to enjoy.

Structured logs + Wagon

We log all application events–native, web, and server-side–as structured JSON. We use this data to answer questions about our systems post-hoc. It also gives us a structured stream to pass along to monitoring and alerting tools.

After some light ETL, we use Postgres’ awesome JSON support to query our logs with Wagon itself. This combines dogfooding and product analytics into a workflow that’s so fun and beneficial we dedicate a Slack channel to it. We use Wagon to investigate our application performance and user engagement the same way our users analyze their own data: by writing queries, visualizing results, and sharing charts.

Here’s an example of how we use these tools together in practice.

We find a bug! If we’re unsure how to reproduce it, we use Wagon to query our logs to confirm the bug and find more test cases. Since our development setup works just like production, we can quickly spin up a local instance of the specific environment we’re debugging–this makes root cause isolation much easier. When the fix is reviewed (any emoji on the Github PR is our stamp of approval) and merged into master, it automatically kicks off a staging CI build, whose success transitively launches a new build of our hybrid native/JS Mac app. CircleCI gives us a native OS X build environment with scaled-up generative testing, so we have an additional level of verification that our fix performs as expected. Within 5 minutes, the new app build is available in staging and we are alerted in Slack to download and give it a spin. When we’ve queued up a cohesive set of features and fixes, a pull request from master to the production branch automatically cuts a new release and deploys it to users. We then explore and monitor the effects of those changes by querying Wagon’s logs with Wagon.

If this sounds like fun to you, say hi at our sunny office on Valencia St. in San Francisco and check out our open positions: we’d love to talk to you!

Weekly Roundup: Great SQL Posts
March 02, 2015 | Brant Gidwitz

We’re always on the lookout for great articles about SQL, analytics, and visualization. Here’s a roundup of the team’s favorite recent posts. Enjoy!

1. Embracing SQL In Postgres

Rob Conery dives into Postgres' rich features including regex, full text search, series generation, date math, and window functions.

2. Modern SQL in PostgreSQL [and other databases]

Markus Winand presented a talk about lateral joins, common table expressions, with recursive, filters, and fetch first. Do you know about fetch first?!

3. Tree structure query with PostgreSQL

Learn how to use WITH RECURSIVE to query hierarchical data from Trường TX.

4. Tinkering with Marcel (Pt. 1)

Awesome adaptation of Tom Tango's Marcel Projection system for baseball projections implemented in PostgreSQL. Hats off to @mattdennewitz!

These posts showcase SQL’s power and versatility. While the basics may be simple, it’s hard to master the language. We love seeing people carefully explain the complex topics. What are your favorite articles and blogs for learning SQL and exploring data?

Engineering at Wagon
February 13, 2015 | Mike Craig

Over the last few months, we’ve been user-testing a preview version of Wagon—a modern SQL editor that powers a Google Docs-style experience for analysts, engineers, and their teams. We’ve iterated on both our design process and technology choices to build a tool that we and our users rely on every day.

Wagon takes advantage of cloud-local computing, combining the strengths of remote servers like syncing, sharing, and storage with local client advantages like data locality and device-specific user interfaces. Our architecture has three components: a frontend app, a native client, and backend services.

Wagon architecture

We make it easy to get started. You can download Wagon—a hybrid web application and native client, like Slack—and connect it to an existing local, private, or public database. The frontend is a slick interface for writing and running SQL, interacting with large results, and generating statistics and pivot tables. The native client locally handles connections, manages query runs, and processes results as they return from the database. The results, statistics, and pivots are cached locally and can be used to create sharable charts. When shared, Wagon securely transfers data from the local machine to the backend, making it available to permissioned teammates.

Frontend App

We use HTML, CSS, and JavaScript to maintain a consistent user experience across platforms and environments. The frontend is a single-page application built with React and Flux. We bundle it with the native client as a desktop application, and we have an in-browser experience for sharing and collaboration.

Technology: Javascript, React, Flux, Ace

• React gives us declarative, reusable UI elements. React components compose like HTML elements with rich state handling, so we manage fewer event listeners and nested callbacks.
• Flux complements React and helps structure our frontend code. It separates logic, state, and UI into loosely coupled components, and has a straightforward data flow.
• Ace is an extensible, embeddable code editor. We use its syntax highlighting and have extended its autocomplete.

Native Client

Our users want to query their databases without any new IT or admin setup, so users connect via their local machine with their existing credentials. Wagon handles connections through the native client, since most databases require socket-level network access that isn’t possible directly from browser.

Exploration and analysis can be a time-consuming and tedious job with existing SQL tools. Each step is an additional database query subject to queuing, slow execution, and changing data. Our users want to quickly explore results and evaluate summary statistics without executing subsequent queries. To power that experience, the client also computes statistics and caches query results on the local file system.

We built the native client in Haskell. It’s a high level language with great libraries, static compilation, a high performance ceiling, and an active community. We use MacGap to bundle the client and the frontend into a downloadable OS X application (Windows soon!).

Technology: Haskell, MacGap

• Prototyping and refactoring is safe with Haskell’s type-driven development. The compiler helps us avoid bugs and edge cases as we iterate.
• Haskell’s opt-in control over compiler optimizations and strictness allows for incremental optimization. GHC includes a capable profiler, and the community maintains libraries for testing and performance measurement.
• MacGap is a lightweight (~1MB) web-to-native wrapper. It provides a JS API to integrate with the file system, subprocesses, and OS-specific features like notifications and menus.

*Note: Wagon is now powered by Electron rather than MacGap. Read our Electron blog post to learn more.

Backend Services

Our backend services persists state and shared data to allow users to securely collaborate on queries, results, and charts. The backend is also written in Haskell: using the same language for the client and the server helps us stay flexible and reduce complexity. We store application state in Postgres while larger immutable data, like shared query results, live in Amazon S3.

Technology: Haskell, Postgres, Amazon S3

• Haskell’s lightweight threads and the GHC runtime scale well on multi-core machines.
• Postgres is powerful, flexible, and straightforward to host. We use features like TOAST and JSON support to test new features.

There are exciting upcoming technology challenges including optimizing compute across local and remote environments, designing efficient streaming statistics algorithms, building features that learn and improve with usage.

We’d love to hear from you! Follow @WagonHQ, check out our stack on Stackshare, and learn more about our engineering and design team. Stay tuned for future engineering deep dives!

Thanks to Steve Pike and Doug Petkanics for reviewing this post. Image credit Adam Wilson from The Noun Project.

Design at Wagon
January 22, 2015 | Jeff Weinstein

We have a challenging design goal: build an easy-to-use product for analysts and their teams to collaborate on understanding data. Our product is (1) a database query editor for analysts and (2) a Google Docs-style experience for non-technical people to explore results together.

We’re looking for someone to be our first UX designer – come craft our user experience, our voice, and our pixels. You’ll grow a team of designers and engineers that love data analysis and collaboration software.

As a team of engineers and data scientists, we have more experience building systems than storyboarding. Over the past few months, we’ve developed a structured design workflow. It starts with stories, drawing, and getting feedback from customers. We then specify the application’s states and object models and finally code mockups. Here’s how it all works:

Mark and Mike sketch application states.

Story

Many friends and former colleagues have the analysis and collaboration problem that Wagon solves and they’ve been very helpful testing the product. We watch them work, document their pain points, and listen to their feature requests. We winnow our notes into specific user stories: “As an X person, I need to do Y, in order to get Z.” To avoid groupthink, we independently draft and prioritize these stories, then collectively compare and merge them.

Draw

We sketch on the whiteboard, use pen and paper, and drag and drop in Google Draw to visualize our ideas. We review screenshots and pictures in Github Issues and Slack and share paper prototypes using POP. One surprisingly successful mockup was built with ASCII in Sublime Text. As our friends at Wildcard detailed, there are plenty of other design tools that we’d be excited to learn.

Feedback

Early and frequent user testing is essential. Our beta testers’ eyes surface bugs and corner cases at a humbling rate. They contribute directly via our public Hipchat room and shared Google Docs. Fortunately, our users have been patient, thoughtful, and enjoy participating.

Specify

We define application states and user flows before writing any code. These detailed state diagrams are critical for building quickly and reducing the confusion and blockers that can arise during implementation.

Wireframe

During implementation, while APIs and data are developed and tested, we lay out the user experience with HTML/CSS. This helps us uncover usability problems and lets us tweak the design in conjunction with other tasks.

To stay focused as a team, our five core product values set expectations and help us measure progress.

Intuitive

Clean design and approachable voice

Aware

Communicate context and insight

Interoperable

Connect to the outside world

Responsive

Iterate at human speed, always in control

Collective

Shared knowledge improves individuals and unites teams

Design at Wagon has been a team effort. We’ve been lucky enough to be advised and supported by a set of design advisors. Curious about data and design at Wagon? Say hi.

Thank you to Hillary Kelly for reviewing this post.

Hello World
January 14, 2015 | Jeff Weinstein

Welcome to the Wagon blog. We’re excited to show you what we’ve been building, the technology behind it, and stories from the wonderful world of data.

At Wagon, our mission is to make data analysis easier for everyone.

Our first step on this path is a modern, easy to use, web-connected SQL editor. We’ve built a simple way to write queries, visualize statistics, pivot results, and securely share charts with your team. You can download Wagon and connect to a local, private, or public database without involving IT. Sign up for our private beta to get started.

We’re also hiring! If you love data and design, check out our jobs page. We’re looking for a user experience designer, a frontend engineer, a data visualization engineer, and a backend engineer.

We’d love to hear from you. Tweet us @wagonhq, email the team, or visit our San Francisco office.

Thank you to our early beta testers, investors, advisors, family, and friends. As we say at the office, gogogo!