Transactions in on-line transaction processing (OLTP) workloads typically have the following characteristics: (1) they are short-lived, (2) they work on a small subset of the data, (3) they are repetitive. Traditional disk-based database management systems (DBMS) incur too much overhead for OLTP datasets that could simply be memory resident. This is because of the presence of heavyweight concurrency control and... Read More

How can we visualize billion-scale graphs? How to spot outliers in such graphs quickly? Visualizing graphs is the most direct way of understand- ing them; however, billion-scale graphs are very difficult to visualize since the amount of information overflows the resolution of a typical screen. In this paper we propose NET-RAY, an open-source package for visualization- based mining on billion-scale... Read More

Trained systems that apply machine learning to very large datasets, such as web search and IBM's Watson question-answering system, are among the most important and sophisticated software systems being constructed today. Such trained systems are frequently based on supervised learning tasks that require features, signals extracted from the data that distill complicated raw data objects into a small number of... Read More

How can we correlate the neural activity in the human brain as it responds to typed words, with properties of these terms (like ’edible’, ’fits in hand’)? In short, we want to find latent variables, that jointly explain both the brain activity, as well as the behavioral responses. This is one of many settings of the Coupled Matrix- Tensor Factorization... Read More

Proponents of functional programming tout its many benefits, most of which are available only within a particular process, or afforded by a particular programming language feature. Anything outside of that is considered I/O, dangerous and difficult to reason about. But real systems almost always cross process and language boundaries, and most require, crucially, a very gnarly bit of shared state... Read More

Summingbird, an open-source project recently released by Twitter, allows engineers to easily build data processing pipelines that work both in a streaming context provided by Twitter Storm, and in offline batch context through Apache Hadoop. This talk will cover the practical motivation for building such a thing, and explain the core Summingbird architecture and components. Read More

Cassandra has fulfilled its original design as a Bigtable/Dynamo hybrid and is moving beyond it with the release of 2.0 last month. (I have annotated the original Cassandra LADIS paper with comparisons to modern Apache Cassandra at [1].) I will talk about integrating Paxos with an eventually consistent system, using cardinality estimation to improve compaction efficiency, and improving request isolation... Read More

SQL databases have been the dominant form of data management since the late 1970's. However, computing and technology have evolved, leading to an influx of data and new systems to manage it. For this new generation, "Document Databases" quickly emerged as a popular solution and became widely used among companies of all sizes. Using MongoDB as the example, we'll look... Read More