I was recently approached by an entrepreneur who had an interesting way to correlate short term performance of a stock with news reports about the stock. Needless to say, there are many places from which one can get the news, and what results one gets from this sort of analysis does depend on the input news sources. Surprisingly, within two minutes the conversation had drifted from characteristics of news sources to the challenges of running SVM on Hadoop. The reason for this is not that Hadoop is the right infrastructure for this problem. But rather that the problem can legitimately be considered a Big Data problem. In consequence, in the minds of many, it must be addressed by running analytics in the cloud. I have nothing against cloud services. In fact, I think they are an important part of the computational eco-system, permitting organizations to out-source selected aspects of their computational needs, and to provision peak capacity for load bursts. The map-reduce paradigm is a fantastic abstraction with which to handle tasks that are “embarrassingly parallelizable.” In short, there are many circumstances in which cloud services are called for. However, they are not always the solution, and are rarely the complete solution. For the stock price data analysis problem, based solely on the brief outline I’ve given you, one cannot say whether they are appropriate. I have nothing against Support Vector Machines, or other machine learning techniques. They can be immensely useful, and I have used them myself in many situations. Scaling up these techniques for large data sets can be an issue, and certainly is a Big Data challenge. But for the problem at hand, I would be much more concerned about how it was modeled than how the model was scaled. What should the features be? Do we worry about duplicates in news appearances? Into how many categories should we classify news mentions? These are by far the more important questions to answer, because how we answer them can change what results we get: scaling better will only change how fast we get them. It is hard to avoid mention of Big Data anywhere we turn today. There is broad recognition of the value of data, and products obtained through analyzing it. Industry is abuzz with the promise of big data. Government agencies have recently announced significant programs towards addressing challenges of big data. Yet, many have a very narrow interpretation of what that means, and we lose track of the fact that there are multiple steps to the data analysis pipeline, whether the data are big or small. At each step, there is work to be done, and there are challenges with Big Data. The first step is data acquisition. Some data sources, such as sensor networks, can produce staggering amounts of raw data. Much of this data is of no interest, and it can be filtered and compressed by orders of magnitude. One challenge is to define these filters in such a way that they do not discard useful information. For example, in considering news reports, is it enough to retain only those that mention the name of a company of interest? Do we need the full report, or just a snippet around the mentioned name? The second big challenge is to automatically generate the right metadata to describe what data is recorded and how it is recorded and measured. This metadata is likely to be crucial to downstream analysis. For example, we may need to know the source for each report if we wish to examine duplicates. Frequently, the information collected will not be in a format ready for analysis. The second step is an information extraction process that pulls out the required information from the underlying sources and expresses it in a structured form suitable for analysis. A news report will get reduced to a concrete structure, such as a set of tuples, or even a single class label, to facilitate analysis. Furthermore, we are used to thinking of Big Data as always telling us the truth, but this is actually far from reality. We have to deal with erroneous data: some news reports are inaccurate. Data analysis is considerably more challenging than simply locating, identifying, understanding, and citing data. For effective large-scale analysis all of this has to happen in a completely automated manner. This requires differences in data structure and semantics to be expressed in forms that are computer understandable, and then “robotically” resolvable. Even for simpler analyses that depend on only one data set, there remains an important question of suitable database design. Usually, there will be many alternative ways in which to store the same information. Certain designs will have advantages over others for certain purposes, and possibly drawbacks for other purposes. Mining requires integrated, cleaned, trustworthy, and efficiently accessible data, declarative query and mining interfaces, scalable mining algorithms, and big-data computing environments. A problem with current Big Data analysis is the lack of coordination between database systems, which host the data and provide SQL querying, with analytics packages that perform various forms of non-SQL processing, such as data mining and statistical analyses. Today’s analysts are impeded by a tedious process of exporting data from the database, performing a non-SQL process and bringing the data back. Having the ability to analyze Big Data is of limited value if users cannot understand the analysis. Ultimately, a decision-maker, provided with the result of analysis, has to interpret these results. Usually, this involves examining all the assumptions made and retracing the analysis. Furthermore, as we saw above, there are many possible sources of error: computer systems can have bugs, models almost always have assumptions, and results can be based on erroneous data. For all of these reasons, users will try to understand, and verify, the results produced by the computer. The computer system must make it easy for her to do so by providing supplementary information that explains how each result was derived, and based upon precisely what inputs. In short, there is a multi-step pipeline required to extract value from data. Heterogeneity, incompleteness, scale, timeliness, privacy and process complexity give rise to challenges at all phases of the pipeline. Furthermore, this pipeline isn’t a simple linear flow – rather there are frequent loops back as downstream steps suggest changes to upstream steps. There is more than enough here that we in the database research community can work on. To highlight this fact, several of us got together electronically last winter, and wrote a white paper, available at http://cra.org/ccc/docs/init/bigdatawhitepaper.pdf . Please read it, and say what you think. The database community came very late to much of the web. We should make sure not to miss the boat on Big Data. My post is loosely based on an extract from this white paper, which was created through a distributed conversation among many prominent researchers listed below. Divyakant Agrawal, UC Santa Barbara |
| Blogger’s Profile: H. V. Jagadish is Bernard A Galler Collegiate Professor of Electrical Engineering and Computer Science and Director of the Software Systems Research Laboratory at the University of Michigan , Ann Arbor. He is well-known for his broad-ranging research on information management, and particularly its use in biology, medicine, telecommunications, finance, engineering, and the web. He is an ACM Fellow and founding Editor in Chief of PVLDB. He serves on the board of the Computing Research Association. |
Big Data: It’s Not Just the Analytics
MADlib: An Open-Source Library for Scalable Analytics
 tl;dr: MADlib is an open-source library of scalable in-database algorithms for machine learning, statistics and other analytic tasks. MADlib is supported with people-power from Greenplum; researchers at Berkeley, Florida and Wisconsin are also contributing. The project recently released a MADlib TR, and is now welcoming additional community contributions. Warehousing → ScienceBack in 2008, I had the good fortune to fall in with a group of data professionals documenting new usage patterns in scalable analytics. It was an interesting team: a computational advertising analyst at a large social networking firm, a seasoned DBMS consultant formerly employed at a major Internet retailer, a pair of DBMS engine developers and an academic. The usage patterns we were seeing represented a shift from accountancy to analytics—from the cautious record-keeping of “Data Warehousing” to the open-ended, predictive task of “Data Science”. This shift was turning many Data Warehousing tenets on their heads. Rather than “architecting” an integrated permanent record that repelled data until it was well-conditioned, the groups we observed were interested in fostering a data-centric computational “watering hole”, where analysts could bring any kind of relevant data into a shared infrastructure, and experiment with ad-hoc integration and rich algorithmic analysis at very large scales. In response to the dry TLAs of Data Warehousing, we dubbed this usage model MAD, to reflect
We wrote the MAD Skills paper in VLDB 2009 to capture these practices in broad terms. The paper describes the usage patterns mentioned above in more detail. It also includes a fairly technical section with a number of non-trivial analytics techniques adapted from the field, implemented via simple SQL excerpts. MADlib (MAD Skills, the SQL)When we released the MAD Skills paper, many people were interested not only in its design aspects, but also in the promise of sophisticated statistical methods in SQL. This interest came from multiple directions: DBMS customers were requesting it of consultants and vendors, and academics were increasingly publishing papers on in-database analytics. What was missing was a software framework to harness the energy of the community, and connect the various interested constituencies. To this end, a group formed to build MADlib, a free, open-source library of SQL-based algorithms for machine learning, statistics, and related analytic tasks. The methods in MADlib are designed both for in- and out-of-core execution, and for the shared-nothing, “scale-out” parallelism offered by modern parallel database engines, ensuring that computation is done close to the data. The core functionality is written in declarative SQL statements, which orchestrate data movement to and from disk, and across networked machines. Single-node inner loops take advantage of SQL extensibility to call out to high-performance math libraries (currently, Eigen) in user-defined scalar and aggregate functions. At the highest level, tasks that require iteration and/or structure definition are coded in Python driver routines, which are used only to kick off the data-rich computations that happen within the database engine. The primary goal of the MADlib open-source project is to accelerate innovation and technology transfer in the Data Science community via a shared library of scalable in-database analytics, much as the CRAN library serves the R community. Unlike CRAN, which is customized to the R analytics tool, we hope that MADlib’s grounding in standard SQL can result in community ports to a variety of parallel database engines. Open-Source Algorithms in Parallel DBMSs?
The state of scalable analytics today depends very much on who you talk to. The motivation for considering parallel databases comes from both the database market and technology issues. There is a large and growing installed base of massively parallel commercial DBMSs in industry, fueled in part by a recent wave of startup acquisitions. Meanwhile, it is no surprise to database researchers that a massively parallel DBMS is a powerful platform for dataflow programming of sophisticated analytic algorithms. Research on sophisticated in-database analytics has been growing in recent years, in part as an offshoot of work on Probabilistic Databases. Education is hopefully shifting as well. For example, in my own CS186 database course this spring, the students not only wrote traditional SQL queries, they also had to implement a non-trivial social network analysis algorithm in SQL (betweenness centrality). The open-source nature of MADlib represents a serious commitment by the entire team, and differs from the proprietary approaches traditionally associated with DBMS vendors. The decision to go open-source was motivated by a number of goals, including:
MADlib StatusMADlib is still young, at Version 0.3. The initial versions focused on establishing infrastructure and a baseline of textbook and some advanced methods; this initial suite actually covers a fair bit of ground (Table 1). Most methods were chosen because they were frequently requested from customers we met through contacts at Greenplum. More recently, we made a point of validating MADlib as a research vehicle, by fostering a small number of university groups who were working in the area to experiment with the platform and get their code disseminated. Profs. Chris Ré at Wisconsin and Daisy Wang at Florida have written up their work in a MADLib tech report that expands upon this post.
MADlib is currently ported to PostgreSQL (single-node, open-source) and Greenplum (shared-nothing parallel, commercial). Greenplum inherits the PostgreSQL extensibility interfaces almost completely, so these two ports were easy to pursue simultaneously in the early days of the project. Another attraction of Greenplum is that it offers a free download of a massively parallel DBMS for researchers, so there is no limitation on scaling experiments. (This is surprisingly unusual: most DBMS vendors still only advertise free trial downloads of “crippleware” that artificially limits database size or the number of nodes. I would imagine that market forces will change this story relatively soon.) MADlib is hosted publicly at github, and readers are encouraged to browse the code and documentation via the MADlib website. The initial MADlib codebase reflects contributions from both industry (a team at Greenplum) and academia (Berkeley, Wisconsin, Florida). Project oversight and Quality Assurance efforts have been contributed by Greenplum. Our MADlib TR expands on the architecture and status, and also includes extensive discussion of related work. Pitch in!At this time, MADlib is ready to consider contributions from additional parties, including both new methods and ports to new platforms. Like any serious open-source project, contributions will have to be managed carefully to maintain code quality. I hope that more researchers will find it worthwhile to contribute serious code to the MADlib effort. It’s a bit more work than getting an algorithm ready to run experiments in a paper, but it’s really satisfying to develop and refine production-quality open-source code, and get it delivered to end-users. If you are doing research on scalable analytic methods, consider going the extra mile and contributing your code to the MADlib effort. For more information on MADlib, please see the website at http://madlib.net. Thanks to Chris Ré, Florian Schoppmann and Daisy Wang for their help writing up the recent MADlib TR that this post excerpts, and to Azza Abouzied, Peter Bailis, and Neil Conway for feedback on this version. |
|
Blogger’s Profile: Joseph M. Hellerstein is a Chancellor’s Professor of Computer Science at the University of California, Berkeley, whose work focuses on data-centric systems and the way they drive computing. He is an ACM Fellow, an Alfred P. Sloan Research Fellow and the recipient of two ACM-SIGMOD “Test of Time” awards for his research. In 2010, Fortune Magazine included him in their list of 50 smartest people in technology , and MIT’s Technology Review magazine included his Bloom language for cloud computing on their TR10 list of the 10 technologies “most likely to change our world”. A past research lab director for Intel, Hellerstein maintains an active role in the high tech industry, currently serving on the technical advisory boards of a number of computing and Internet companies including EMC, SurveyMonkey, Platfora and Captricity. |
From 100 Students to 100,000
 Typically I teach around 100 students per year in my introductory database course. This past fall my enrollment was a whopping 60,000. Admittedly, only 25,000 of them chose to submit assignments, and a mere 6500 achieved a strong final score. But even with 6500 students, I more than quadrupled the total number of students I’ve taught in my entire 18-year academic career.
The story begins a couple of years earlier, when Stanford computer science faculty started thinking about shaking up the way we teach. We were tired of delivering the same lectures year after year, often to a half-empty classroom because our classes were being videotaped. (The primary purpose of the videotaping is for Stanford’s Center for Professional Development, but the biggest effect is that many Stanford students skip lectures and watch them later online.) Why not “purpose-build” better videos: shorter, topic-specific segments, punctuated with in-video quizzes to let watchers check their understanding? Then class time could be made more enticing for students and instructor alike, with interactive activities, advanced or exotic topics, and guest speakers. This “flipped classroom” idea was evangelized in the Stanford C.S. department by Daphne Koller; I was one of the early adopters, creating my videos during the first few months of 2011. Recording was a low-tech affair, involving a computer, Cintiq tablet, cheap webcam and microphone, Camtasia software, and a teaching assistant to help with editing.
I put my videos online for the public, and soon realized that with a little extra work, I could make available what amounted to an entire course. With further help from the teaching assistant, I added slides (annotated as lecture notes, and unannotated for teaching use by others), demo scripts, pointers to textbook readings and other course materials, a comprehensive suite of written and programming exercises, and quick-guides for relevant software. The site got a reasonable amount of traffic, but the turning point came when my colleague Sebastian Thrun decided to open up his fall 2011 introductory artificial intelligence course to the world. After one email announcement promising a free online version of the Stanford AI course, including automatically-graded weekly assignments and a “statement of accomplishment” upon completion, Sebastian’s public course garnered tens of thousands of sign-ups within a week. Having already prepared lots of materials, I jumped on the free-to-the-world bandwagon, as did my colleague Andrew Ng with his machine learning course. What transpired over the next ten weeks was one of the most rewarding things I’ve done in my life. The sign-ups poured in, and soon the “Q&A Forum” was buzzing with activity. The fact that I had a lot of materials ready before the course started turned out to be a bit deceptive—for ten weeks I worked nearly full-time on the course (never mind my other job as department chair, much less my research program), in part because there was a lot to do, but mostly because there was a lot I could do to make it even better, and I was having a grand time.
In addition to the video lectures, in-video quizzes, course materials, and self-guided exercises, I added two very popular components: quizzes that generate different combinations of correct and incorrect answers each time they’re launched (using technology pioneered a decade ago by my colleague Jeff Ullman in his Gradiance system), and interactive workbenches for topics ranging from XML DTD validation to view-update triggers. I offered midterm and final exams—multiple-choice, and crafted carefully so the problems weren’t solvable by running queries or checking Wikipedia. (Creating these exams, at just the right level, turned out to be one of the most challenging tasks of the entire endeavor.) To add a personal touch, and to amplify the strong sense of community that quickly welled up through the Q&A Forum, each week I posted a “screenside chat” video—modeled after Franklin D. Roosevelt’s fireside chats—covering topics ranging from logistical issues, to technical clarifications, to full-on cheerleading for those who were struggling. Meanwhile back on the campus front, the Stanford students worked through exactly the same materials as the public students (except for the multiple-choice exams), but they did get something more for their money: hand-graded written problems with more depth than the automated exercises, a significant programming project, traditional written exams, and classroom activities ranging from interactive problem-solving to presentations by data architects at Facebook and Twitter. There’s no question that the Stanford students were satisfied: I’ve taught the course enough times to know that the uptick in my teaching ratings was statistically significant.
One interesting and surprisingly large effect of having 60,000 students is the need for absolute perfection: not one tiny flaw or ambiguity goes unnoticed. And when there’s a downright mistake, especially in, say, an exam question … well, I shudder to remember. The task of correcting small (and larger) errors and ambiguities in videos, quizzes, exercises, and other materials, was a continuing chore, but certainly instructive. What kept me most engaged throughout the course was the attitude of the public students, conveyed primarily through emails and posts on the Q&A Forum. They were unabashedly, genuinely, deeply appreciative. Many said the course was a gift they could scarcely believe had come their way. As the course came to a close, several students admitted to shedding tears. One posted a heartfelt poem. A particularly noteworthy student named Amy became an absolute folk hero: Over the duration of the course Amy answered almost 900 posted questions. Regardless of whether the questions were silly or naive, complex or deep, her answers were patient, correct, of just the right length, included examples as appropriate, and were crafted in perfect English. Amy never revealed anything about herself (although she agreed to visit me after the course was over), despite hundreds of adoring public thank-you’s from her classmates, and one marriage proposal!
So who were these thousands and thousands of students? I ran a survey that revealed some interesting statistics. For example, although ages and occupations spanned the gamut, the largest contingent of students were software professionals wanting to sharpen their job skills. Many students commented that they’d been programming with databases for years without really knowing what they were doing. Males outnumbered females four to one, which is actually a little better than the ratio among U.S. college computer science majors. Students hailed from 130 countries; the U.S. had the highest number by a wide margin, followed by India and Russia. (China unfortunately blocked some of the content, although a few enterprising students helped each other out with workarounds.) On db-class.org you can find the full survey results via the FAQ page, as well as some participation and performance statistics. Were there any negatives to the experience? Naturally there were a few complainers. For example, in my screenside chats I often referred to the “eager beavers” who were working well ahead of the schedule, and the “procrastinators” who were barely meeting deadlines. Most students enjoyed self-identifying into the categories (some eager-beavers even planned to make T-shirts), but a few procrastinators objected to the term, pointing out that they were squeezing the course between a full-time job or two and significant family obligations. A number of students were disappointed by the low-tech, non-Stanford-endorsed “statement of accomplishment” they received at the end; despite ample warnings from the start, apparently some students were still expecting official certification. I can’t help but wonder if some of those students were the same ones who cheated; I did appear to have quite a number of secondary accounts created expressly for achieving a perfect score. I made it clear from the start that I was assuming students were in it to learn, and cheating was not something I planned to prevent or even think about. Of course in the long run of online education, the interrelated topics of certification and cheating will need to be addressed. So what happens next? Stanford is launching quite a few more courses in the same style, and I’ll offer mine again next fall. MIT has jumped on the bandwagon; other universities can’t be far behind. Independent enterprises such as the pioneering Khan Academy, and the recently-announced Udacity, are sure to play into the scene. There’s no doubt we’re at a major inflection point in higher education, both on campus and through internet distribution to the world. I’m thrilled to have been an early part of it. Meanwhile here are a few more numbers: A few months after the initial launch we now have over 100,000 accounts, and we’ve accumulated millions of video views. Even with the course in a self-serve dormant state, each day there are a couple of thousand video views and around 100 assignments submitted for automated grading. All to learn about databases! Wow. Check it out at db-class.org. |
| Blogger’s Profile: Jennifer Widom is the Fletcher Jones Professor and (currently) Chair of the Computer Science Department at Stanford University. She was a Research Staff Member at the IBM Almaden Research Center before joining the Stanford faculty in 1993. Her research interests span many aspects of nontraditional data management. She is an ACM Fellow and a member of the National Academy of Engineering and the American Academy of Arts & Sciences; she received the ACM SIGMOD Edgar F. Codd Innovations Award in 2007. |
The ACM SIGMOD Blog
| In the era of blogs and social networks, ACM SIGMOD gets social!
The ACM SIGMOD Blog is the official blog site for ACM SIGMOD. This blog aims at catching the heartbeat of our community on exciting and controversial topics that are of interest to the community, and facilitate intelligent discussions among researchers on such topics. Its purpose is to be both interesting and fun. The Blog will periodically host one featured blogger to share his/her view on a matter of interest. People can participate by leaving comments and opinions. In this way, the ensuing discussion can take a form that can hopefully lead to an interesting conclusion. Who can be a featured blogger: Anyone in our community! If you are passionate on a topic and would like to write a few paragraphs staking out your position, please contact Georgia Koutrika (by sending an email to sigmodblog [at] acm.org). We also plan to invite people to blog on selected topics. Participating in a discussion: You are welcome to participate in the discussion following the current blog post. When you do so, we encourage you to identify yourself by providing your name, and optionally your institution. In this way, everyone can get acknowledged, and discussions can be open, interesting and constructive. Living in the era of social networking sites, we also offer the option for you to leave comments by signing up using your Facebook account. In this way, people will be able to see your photo and your name next to your comments! Comments will be moderated to ensure a healthy and fun discussion environment. Enjoy!
|
