Wagon Blog

Wagon Joins Box

Team Wagon — Wed, 31 Aug 2016 00:00:00 -0700

We’re excited to share that the Wagon team will be joining the Box family! We are thrilled to bring our data analytics and insights expertise to the Box enterprise platform.

INSERT INTO "Box" VALUES ('Wagon');

Wagon and Box believe that shared knowledge empowers individuals and unites teams. Box helps enterprises do more with their content and get their work done faster by acting as the central, modern content platform. Fortune 500 companies across every industry trust Box to sit at the center of their businesses. We’re excited to build data analytics products at Box to help people understand their data and work better, together.

Our team will be focused on building new products at Box, so we’ll be shutting down the current Wagon product on October 3rd, 2016. Please read our transition FAQ for details.

We started Wagon over two years ago to help teams collaborate on data analysis. Tens of thousands of SQL savvy people use Wagon to understand their data. We’re proud of what we’ve accomplished and are ready to continue building at Box.

We’re humbled by the support from our users, partners, investors, advisors, friends (#bandwagon), and family. Thank you for your feedback, bug reports, demands, and kind words.

Please read Box CEO Aaron Levie’s blog post.

We'll be rooting you all on. Gogogo!
— Team Wagon

Your customer data, together at last

Andy Granowitz — Wed, 06 Apr 2016 00:00:00 -0700

Customers interact with your company through many paths — they engage inside your product, open emails, view ads, answer surveys, etc. To understand your customers, you have to combine product engagement with their other interactions.

Unfortunately, it’s difficult to analyze customer data across these different experiences. Customer behavior in your product is usually measured through a 3rd party tracking service or by analyzing raw application logs. Other interactions, like as customer support chats or sales calls, are stored in SaaS apps or homegrown internal tools. You can only access this data through each of their limited reporting UIs, exports, or possibly an API. There’s no magic wand for joining these disparate streams.

There are two common approaches to join these multiple datasets: (1) send all data to one tracking service or (2) move the data to a common place. Sending all the data to a single tracking service, like Google Analytics or Mixpanel, is cumbersome, nearly impossible to implement, and inflexible to query. Dead end.

The better approach is to move all customer data into a common store. Amazon Redshift, Google BigQuery, and Azure SQL Warehouse are our favorite cloud data stores for large scale analytics. They’re fast, easy to manage, and decreasing in price. But moving data from 3rd party sources to these cloud databases is still hard. You have to work with multiple APIs, find join keys, schedule data pulls, handle backfill, etc… It’s a huge headache.

There are two strategies: build or buy. Building your own is complicated - we have a post about it!

💖

Today, thanks to Segment Sources, bringing all of your customer touch points together got easier. They’ve made it easy to pull customer data from many sources, such as Salesforce, Zendesk, and behavioral data from your app or website, into a single warehouse. We worked with Segment to write some starter queries - give them a go. If you’re already using Wagon and connect a Segment Warehouse, you can open these queries right in the app.

Segment will be expanding its Source catalog in the coming months, and you can always check out the current catalog to see the latest additions. We’re excited to partner with Segment on our shared mission to make answering questions with data easy.

We also recommend other tools to help unify your various customer data sets: Fivetran, RJ Metrics Pipeline, and Snowplow – so that you’ll always have access to the data you need in Wagon, no matter how you choose to set up your pipeline.

Modern ETL + modern SQL tools = win!

Deploying Electron

Matt DeLand — Tue, 23 Feb 2016 00:00:00 -0800

Electron makes it easy to write cross-platform desktop apps using web technology— but how do you build and deploy these hybrid apps in production?

At January’s Bay Area Electron Meetup, we presented how Wagon builds and delivers new updates to our users. This talk covers both standard and custom Electron usage for background updating, uninterrupted usage, helper processes, cross-platform builds, OS notifications, and the occasional hotfix.

Thanks to Ben Gotow at Nylas for organizing!

Haskell for commercial software development

Mike Craig — Tue, 02 Feb 2016 00:00:00 -0800

Inquiring minds on Quora want to know, is Haskell suitable for commercial software development?

I’m looking for an alternative to Python mainly for reasons of Python having so much trouble with workable concurrency, and Go seemed great at first look (it’s still not bad, but indeed error handling and lack of generics are a few features cut away too far).

This pretty much leaves Haskell as workable alternative. But is it really a practical alternative?

My biggest fear that while Haskell seems very powerful, it is also difficult and so it may be too difficult for person A to read code written by person B (what seems to have killed Lisp in my view is that problem and person-specific DSLs written in Lisp were simply unreadable for too many other people).

PS, in other words: I do not ask about strong sides of Haskell (or language X), those are well known, but rather about a lack of show-stoppers.

Absolutely, yes! At Wagon, we’ve built server- and client-side systems in Haskell, running in AWS and distributed to our users in a cross-platform desktop app.

Possible show-stoppers to using Haskell commercially:

Community support
A good open-source community is a big deal. It’s the difference between fixing other people’s code and finding an open GitHub issue with a workaround and an in-progress fix. With Haskell, we find the latter much more frequently than the former. The community is positive, active, and generally helpful when problems come up. Lively discussion takes place on Reddit, IRC, a public Slack channel, and several mailing lists.

Hiring
There are many factors involved in hiring: team size, location, remove vs on-site, other expertise required, etc. At Wagon we’re an on-site team of ~10 in San Francisco, with lots of web- and database-related problems to work on. Finding qualified Haskellers has not been an issue.

Haskell is different from the languages lots of developers are used to. ML-style syntax, strong types, and lazy evaluation make for a steep initial learning curve. Anecdotally, we’ve found Haskell’s secondary learning curve smooth and productive. Intermediate Haskellers quickly pick up advanced concepts on the job: parsers, monad transformers, higher-order types, profiling and optimization, etc.

Available industrial-strength libraries
Haskell lends itself to lightweight-but-powerful libraries that do one thing very well. We get a lot done by composing these small libraries rather than relying on big feature-complete frameworks. Some libraries have become de-facto standards that are stable and performant, and Stackage gives us a reliable way to find additional packages that usually “just work”.

We do sometimes find gaps in Haskell’s available libraries. For example, Haskell can’t yet match Python’s tools for numerical computing. And like every open-source ecosystem, there is some bad code out there. We rely on community recommendations and a sense of dependency hygiene to stay productive.

Performance
A quick Google search turns up benchmarks comparing Haskell to anything else you can imagine. In practice, we’re interested in tools with two properties:

Fast enough under most circumstances. “Fast enough” means performance is never an issue for a given piece of code.
Easy to optimize when needed. When we do need more speed, we’d like to get it incrementally rather than immediately rewrite that code in C.

Haskell has both of these properties. We almost never have to address performance in our Haskell code. When we do, we turn to GHC’s profiler and its many performance knobs: optimization flags, inlining, strictness and memory layout controls, etc. Haskell also has good interopability with C for when that’s appropriate.

Memory leaks
There’s a lot of talk about memory leaks caused by Haskell’s lazy evaluation model. We run into this very infrequently in practice, and in those rare situations GHC’s profiler has led us to solutions quickly. The real problem is not lazy evaluation but lazy IO, which we avoid with tools like conduit and pipes.

Debuggability
Haskell doesn’t have a debugger in the same sense as Python or Java. This is mostly a non-issue, because exceptions are rare and GHCi gives us a flexible way to run code interactively. Nonetheless, hunting down problems in a big application can be difficult. This has improved in recent versions of GHC with support for proper stack traces.

Haskellers reading other Haskellers’ code
This hasn’t been a problem for us. Haskell is a flexible language both in syntax and semantics, but this leads to better solutions more than it leads to opaque or unreadable code. Heavy indirection—via embedded DSLs or deep typeclass hierarchies—is unusual in practice. We reach for those tools as a last resort, knowing that they come with a serious maintenance cost. Agreeing on a basic style guide helps smooth out minor syntax differences.

Read our other Haskell engineering blog posts, come to our community events, or better yet, join our team!

Python UDFs in Amazon Redshift

Elise Breda (Yhat) — Mon, 01 Feb 2016 00:00:00 -0800

Elise Breda is the Growth Strategist at Yhat, a tech company that builds software for deploying predictive analytics to mobile and web apps. When she isn’t at their awesome new office in DUMBO, she can be found exploring the bike paths and Thai restaurants of NYC.

In the Beginning…

In the early 2000’s, the behemoth collection of cloud computing services we now know as Amazon Web Services (AWS) was little more than sparks firing in Chris Pinkham and Benjamin Black’s neurons. In 2003, the two presented a paper (blog post here) outlining a radical vision for a retail computing infrastructure that would be standardized, automated and built upon web services. The next year, Simple Queue Service, the first AWS service for public usage, was launched.

Fast forward almost a decade, and AWS is now the most commonly used cloud platform among enterprise software developers. AWS products span the gamut of web services, from computation (eg EC2) to networking (eg VPC) and content delivery (eg S3). In this post we’ll explore a small fraction of a fraction of the AWS ecosystem–a database that’s generating all kinds of groundswell right now: Amazon Redshift.

The rest of this post will talk about Redshift at a high level and then dive into a mini overview of User Defined Functions (UDFs), how they work, why they’re great, and how to start using them.

Amazon Redshift

Amazon Redshift is a hosted data warehouse that’s accessible / easy to set up, and built for speed and suitable for a variety of combining, storing, and compute-heavy analytics tasks.

Two things make Redshift particularly attractive. First, Redshift can handle insane amounts of data–it is a petabyte-scale warehouse. A petabyte is a lot (10¹⁵ bytes) of data. As a point of reference, the entire master catalog of Netflix video in 2013 amounted to about 3.14 petabytes of storage space (interesting read on Quora). Second, unlike Amazon’s other hosted database product, Amazon RDS, Redshift stores data according to column-based structure. Column orientation is good for tables containing columns with lots of repeated values (i.e. Credit Card Names, County/State, Product Type, etc, like CRM data. The benefit of column data is that because it’s uniform, there are opportunities for storage size optimization via compression. You can read more about how to maximize compression here.

Redshift handles large scale column-oriented datasets using massive parallel processing, performing coordinated computations across a large number of processors in parallel, making it a fast and powerful data warehouse option.

Data Warehouse Setup in the Data Age

Even just a few years ago, getting a data warehouse and proper ETL processes in place was a long, painful and probably very expensive ordeal. But we’ve arrived in the data age where easy-to-use, affordable data solutions are bountiful.

At Yhat, we use a Redshift to warehouse everything–CRM data (we use SFDC), product data, site metrics from Google Analytics, and data from a bunch of other data. It took us about 20 mins to set up the database on AWS, and it took us…wait for it…another 20 mins or so to set up all of our ETL using Fivetran which we couldn’t be more impressed with.

SQL IDE Done Right

Most SQL IDEs of yesteryear leave something to be desired in terms of UX. The majority are clunky and have super old school frankenstein UIs. Why they all focus on making exploring the DB schema rather than on making it easy to write queries, view results and think critically about your data has always been a mystery.

Well those days are also over. Wagon is the query-focused SQL app I’ve been looking for for years. Wagon boasts a clean UX designed analysts. Features are carefully chosen with a keen eye for usability for people writing tens or hundreds of queries per day. Wagon gets it in spades.

Overview of Python UDFs in Redshift

UDF stands for user-defined function, meaning that you can add functions to an environment (in this case, Redshift) in addition to those that come built in. Python UDFs allow you combine the power of Redshift with what you know and love about the Python programming language without switching between IDEs or systems.

The great thing about UDFs in Redshift is that Redshift will automatically execute it using its MPP architecture. One caveat to keep in mind is that your Python code still won’t execute as quickly as native SQL functions (AVG, MIN, MAX, etc.) that are baked into the database.

How to Use UDFs

You can certainly work with text in pure SQL, but some tasks are just easier to do in a scripting language like Python instead. Here’s a toy example to illustrate how to use Python functionality within Redshift using a UDF.

Suppose a column in one of our tables contains huge chunks of text or html, and we’re interested to find any email addresses within any one record. Let’s write a function that will take in raw text and return a pipe | separated string containing any email addresses found within the input text document. Define the function like so:

Once defined, you can use it like this:

This is a scalar function, so it’ll return one record for each input row (i.e. not an aggregate function). One thing to remember is that your UDFs are per-database, meaning that if you have multiple in your Redshift cluster, you’ll need to define your functions in each database.

Example

Redshift Python UDFs are based on Python 2.7 and come preloaded with a lot of our favorite libraries, including NumPy, SciPy and Pandas. You can also import custom modules from S3 and the web.

Here’s the template published on the AWS blog that you can use to start creating your own scalar functions:

The scalar UDFs that you create will return a single result value for each input value. Once you’ve defined a UDF, you can use it in any SQL statement. One thing to remember is that your UDFs are per-database, meaning that if you have multiple in your Redshift cluster, you’ll need to define your functions in each database.

Helpful Resources

To learn more about Python UDFs in Redshift, check out Amazon’s documentation, which is super helpful and covers everything from constraints to security and python support. You can also check out the initial release blogpost and a more extensive post that uses UDFs to analyze the CMS Open Payments data set.

Yhat

Yhat’s flagship product, ScienceOps, empowers teams of data scientists deploy their models into web and mobile applications. These models are embedded into production applications via REST APIs without any recoding from their native statistical language.

FUNctional Programming with Haskell - Twitter Tech Talk

Jeff Weinstein — Mon, 21 Dec 2015 00:00:00 -0800

Haskell is our primary backend language at Wagon— it helps us (safely) iterate faster, build for multiple environments, and attracts great engineering talent. It is fun to meet engineers from startups and large companies at Haskell meetups, BayHac, and in the functional programming Slack channel. Many engineers are curious how we use Haskell in production and have invited us to speak at their companies.

Wagon’s CTO Mike Craig spoke last month at Twitter (and earlier this year at Square) on the pros and cons of using Haskell in production. Thanks to Twitter’s Peter Seibel for inviting us and to their team for posting the video.

Here’s the talk:

If you’re using Haskell at your company, let us know.

Calculating Active Users in SQL

Andy Granowitz — Fri, 13 Nov 2015 00:00:00 -0800

How engaged are your users? How frequently do they visit your website or app? Analytics services like Google Analytics and MixPanel calculate basic counts of daily, weekly, and monthly active users, but it’s difficult to customize or join these results with other data. Writing this query in SQL gives you more control. Let’s do it!

Here’s a table of user logins by day. How many users were active in the last week and month?

date	user_id	num_logins
10/1/15	1	3
10/1/15	2	null
10/1/15	3	1
10/2/15	1	null
10/2/15	2	1
10/2/15	3	3

Like calculating running totals, there are two approaches: self join or window function.

In either approach, it’s helpful to have a table of logins per user for each day, even if the user didn’t login (null in this example). If your data isn’t already organized like this, you can generate a table with a row per day, per user, with the following query (this is Postgres syntax, for other databases, modify the generate_series function to generate a table of dates).

select d.date, u.user_id, c.num_logins from ( select * from -- fill in the minimum date in your dataset generate_series('01-05-15'::timestamp, current_date::timestamp, '24 hours') as date ) d full outer join (select distinct(user_id) as user_id from userActivityTable) u on 1 = 1 full outer join userActivityTable c on u.user_id = c.user_id and c.date = d.date;

To use this data, you can create a temporary table, use a common table expression, or include it as a subselect.

Approach 1: Self Join

A self join is when you join a table with itself. How meta is that? For each row, we ask how many logins that user had in the last week. The join condition requires emails to match and for the date to be within the last 7 days. In line 5, the query sums num_logins for those dates. The case statement identifies the user as active on that day if she had any logins in the prior week.

select o.user_id, o.date, case when sum(i.num_logins) >= 1 then TRUE else FALSE end as active from userActivityTable as o join userActivityTable as i on i.date <= o.date AND i.date >= (o.date :: date) - integer '7' AND i.user_id = o.user_id group by o.user_id, o.date;

This query generates a table that tells us which users are seven-day-active over time. This result can be aggregated further, filtered for specific dates, used to find inactive users, and joined with other data. In Wagon, we can create a graph of the number of 7 day active users over time.

Approach 2: Window Functions

The self join works great, but modern databases have a more efficient way to get the same results. With window functions, we can explicitly aggregate only over rows that we care about with just a single pass through the data. If you have millions or billions of rows (lucky you), the self join will take a long time to compute. In line 5, the query sums num_logins for the user’s previous 14 days. It first partitions the table by email, then evaluates over a set of rows - in this case we’re looking at a specific date range. The case statement classifies the user as active or not just as before.

select user_id, date, case when sum(num_logins) over (partition by user_id order by date rows between 14 preceding and current row) >= 1 then TRUE else FALSE end as active from userActivityTable;

This query makes it easier to add additional metrics for 7 and 30 day active users. As expected, the wider your definition of active user, the more you’ll have. Use these new powers carefully!

Want to learn more SQL? Join us on Monday, November 16 at the Wagon office in San Francisco for a free SQL workshop. Please RSVP!

Electron meetup at Microsoft

Jeff Weinstein — Fri, 06 Nov 2015 00:00:00 -0800

Electron has momentum. The open source project for building cross platform desktop apps with web technologies now has 6000+ commits from 200+ contributors, a 3500+ person Slack room, and now it’s 4th meetup with 90+ RSVPs (the 1st was a few of us at a bar).

On Monday, the Bay Area Electron group met at Microsoft Reactor, an event space in San Francisco. A few members of Microsoft’s open source team were in town to hear how people are using Electron to build Windows apps. The team is dedicated to helping projects run well with Microsoft platforms. It’s also exciting to see that Visual Studio Code is using Electron. Two of the five talks were Windows related:

Kevin Sawicki from GitHub spoke about how to test Electron apps using ChromeDriver and his project Spectron. Try this in your next project.

Felix Rieseberg from Microsoft’s open source team gave us walkthrough of WinJS. It seems like a great way to make Electron apps look native on Windows 10. Here are the slides.

John Haley from Axosoft’s GitKraken team talked about how they handle task scheduling in their Electron apps. This strategy seems like a new standard way to handle both UI and processing intensive applications.

Wagon’s Matt DeLand presented a few tips and tricks for developing, building, and deploying on Windows along with a demo of our latest version. Sign up for early access to try Wagon on Windows.

Edgar Aroutiounian gave a quick demo of how he used OCaml to build an Electron app. Checkout the example project here.

Special thanks to Microsoft for hosting. We’re excited the Electron community is growing and that large companies are adopting and supporting this platform. See you at the next event!

Haskell in Production - Square Tech Talk

Mike Craig — Wed, 28 Oct 2015 00:00:00 -0700

The Square engineering team invited Wagon to give a tech talk on how we use Haskell in production. Their teams are interested in functional programming and we were honored to walkthrough our experience building a modern analytics tool using Haskell, React, and Electron. Thanks Square for welcoming us!

If you’re using Haskell at your company, let us know. We’d love to trade notes.

Calculating Running Totals using SQL

Andy Granowitz — Tue, 20 Oct 2015 00:00:00 -0700

How many users joined in the last 5 months? What were total sales in Q2? How much revenue came from the March sign up cohort?

Although these questions can be answered with a single number, it can be useful to see a running total over time: how many unique users joined, or how much cumulative revenue was received by day over some period.

Usually, data is stored incrementally. For example, here’s a table of sales per day:

Date	Sales
10/1/2015	5
10/2/2015	3
10/3/2015	7
10/4/2015	8
10/5/2015	2
10/6/2015	3
10/7/2015	6

How do we generate the following table of cumulative sales over time? In SQL, there are two typical approaches: a self join or a window function.

Date	Running Total of Sales
10/1/2015	5
10/2/2015	8
10/3/2015	15
10/4/2015	23
10/5/2015	25
10/6/2015	28
10/7/2015	34

A self join is a query that compares a table to itself. In this case, we’re comparing each date to any date less than or equal to it in order to calculate the running total. Concretely, we take the sum of sales in the second table over every row that has a date less than or equal to the date coming from the first table. This is Postgres/Redshift syntax, but other SQL dialects are very similar.

This is not a bad approach; it is a nice showcase of how extensible SQL can be using only select, from, join, and group by statements.

But it is a lot of code for a simple task. Let’s try a window function. They are designed to calculate a metric over a set of rows. In our case, we want to sum every row where the date is less than or equal to the date in the current row.

The window function can filter and arrange the set of rows to run the function over. Here the order by date rows unbounded preceding limits the sum function to only sales before the date of the current row. Window functions are incredibly useful for time-based analytical queries; to learn more, the Postgres docs are a great place to start.

The final step of creating a chart and sharing it triumphantly with your teammates is easily accomplished using Wagon. Window functions for the win!

Wagon is a modern SQL editor for analysts and engineers: write queries, visualize data, and share charts with your team. Signup for free:

React at Wagon

Mike Craig — Thu, 01 Oct 2015 00:00:00 -0700

We’re building a hybrid web/native application that runs both in the browser and as a downloadable desktop app. Analysts use Wagon to query, analyze, visualize, and share data: the app is highly interactive and data-heavy. It has to be fast, furious, and stable even when used for hours.

It ain’t all gravy: it’s difficult to maintain a UI with asynchronous updates, large scale data manipulation, and many other cross-cutting concerns. How can we build a sane frontend codebase, without losing our ability to iterate and ship quickly? The answer is to separate concerns. We break our UI into small self-contained components, and we isolate state and manage it separately from the UI. Facebook’s React and Flux libraries make this practical.

The big idea behind React is this: a UI component is just a function from its inputs to its content. All a component needs is a render() method that returns the elements we want the user to see. As an example, here’s a component that takes a size and color and renders a div displaying a filled-in square. Notice that users of this component don’t need to know about how it is implemented.

React components are simple to reuse because they nest like HTML elements. It’s easy to wrap an existing component to add additional styles or behavior—React favors composition over inheritance. We can forget about carefully maintaining the DOM to avoid excessive redraws and flicker: we declare what our components should look like, and React makes it so.

React is great for organizing view elements, but an application is more than static UI. Users generate events and we need to capture them, update state, and direct how the app should respond. Flux manages this flow by clearly seperating user action events from application responses.

Actions encapsulate events. They’re the application logic that runs in response to users doing stuff. In our example, when a user clicks a colored square, we update the server and dispatch to let the rest of app know what happened:

Stores encapsulate state. They listen to state changes dispatched from actions, and they update themselves to record the changes.

class UserStore extends EventEmitter { // administrivia omitted! onDispatch(action) { switch (action.type) { case USER_INFO_SAVED: // save the updated state in an instance property this._users[action.payload.id].favoriteColor = action.payload.favoriteColor; // tell the rest of the application to come and get it this.emitChange(); break; // etc } } }

UI components listen to stores and re-render when relevant state changes.

Building a solid, maintainable frontend is still difficult despite these great libraries. Here are a few other strategies we’re using:

Build pure React components whenever possible. A pure component doesn’t make any external calls from render()—it’s a pure function of the component’s properties and state. Components like this are much easier to test, debug, and reuse. It’s such a good idea it’s included in React itself!
Separate React components that listen to Flux stores from those that render the UI. Wrapping UI components in container components is another win for reuse. Jason Bonta mentioned this in his great talk at React.js Conf 2015.
Take care when integrating non-React UI components. Mixing React’s declarative API with another library’s imperative API can be painful. Build wrapper components around external libraries, and use React’s lifecycle methods to handle setup and teardown. When possible, avoid exposing direct-update methods like drawChart() or setCursorPosition()—manage state through component properties or Flux stores.
Split Flux actions into modules by UX concern. We separate navigation and authentication from running queries and making charts. Carve out submodules for cross-cutting concerns, like AJAX requests or logging.
Split Flux stores by domain. It’s helpful to separate persisted server-side state from ephemeral page state, for example. We hide the state of the URL bar behind a store, too!

We’re tackling fun engineering challenges at Wagon. If you want to learn more or work everyday on these technologies, check out our jobs page and get in touch!

Migrating from SQL Workbench/J to Wagon

Andy Granowitz — Wed, 30 Sep 2015 00:00:00 -0700

Many Wagon users previously used SQL Workbench/J to query Amazon Redshift. Older SQL tools are focused on DBA tasks like managing tables, updating schemas, and provisioning users. Analysts just want a simple way to query data, analyze it, visualize it, and collaborate with others. It’s no surprise that we’re frequently asked how to move from legacy tools like SQL Workbench/J to Wagon. It’s super easy.

If you are currently using SQL Workbench/J and want to try Wagon, here are the quick steps to connect to Redshift in Wagon:

In SQL Workbench/J, open the connection window
Grab the hostname, port, and database from the URL, the username, and the password (in the Redshift interface, the URL is called the JDBC URL)
Paste into Wagon (no need to install any drivers!)

Happy querying!